GSM 03.38

Last updated

In mobile telephony GSM 03.38 or 3GPP 23.038 is a character encoding used in GSM networks for SMS (Short Message Service), CB (Cell Broadcast) and USSD (Unstructured Supplementary Service Data). The 3GPP TS 23.038 standard (originally GSM recommendation 03.38) defines GSM 7-bit default alphabet which is mandatory for GSM handsets and network elements, [1] but the character set is suitable only for English and a number of Western-European languages. Languages such as Chinese, Korean or Japanese must be transferred using the 16-bit UCS-2 character encoding. A limited number of languages, like Portuguese, Spanish, Turkish and a number of languages used in India written with a Brahmic scripts may use 7-bit encoding with national language shift table defined in 3GPP 23.038. For binary messages, 8-bit encoding is used.

Contents

GSM 7-bit default alphabet and extension table of 3GPP TS 23.038 / GSM 03.38

The standard encoding for GSM messages is the 7-bit default alphabet as defined in the 23.038 recommendation.

Seven-bit characters must be encoded into octets following one of three packing modes:

Basic Character Set [2]
 0x000x100x200x300x400x500x600x70
0x00 @ Δ SP 0 ¡ P ¿ p
0x01 £ _ ! 1 A Q a q
0x02 $ Φ " 2 B R b r
0x03 ¥ Γ # 3 C S c s
0x04 è Λ ¤ 4 D T d t
0x05 é Ω % 5 E U e u
0x06 ù Π & 6 F V f v
0x07 ì Ψ ' 7 G W g w
0x08 ò Σ ( 8 H X h x
0x09 Ç Θ ) 9 I Y i y
0x0ALF Ξ * : J Z j z
0x0B Ø ESC + ; K Ä k ä
0x0C ø Æ , < L Ö l ö
0x0DCR æ - = M Ñ m ñ
0x0E Å ß . > N Ü n ü
0x0F å É / ? O § o à
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape to extension table (maps to NBSP).
  • SP is a Space character.
Basic Character Set Extension [2]
 0x000x100x200x300x400x500x600x70
0x00     |    
0x01        
0x02        
0x03        
0x04  ^       
0x05        
0x06        
0x07        
0x08   {      
0x09   }      
0x0AFF       
0x0B SS2      
0x0C    [     
0x0DCR2   ~     
0x0E    ]     
0x0F   \      
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

It is important (especially when a message is to be segmented using concatenated SMS mechanism) that characters from the Basic Character Set table take one septet, characters from the Basic Character Set Extension table take two septets.

Note that the second part of the table is only accessible if the GSM device supports the 7-bit extension mechanism, using the ESC character prefix. Otherwise, the ESC code itself is interpreted as a space, and the following character will be treated as if there was no leading ESC code.

Most of the high part of the table is not used in the default character set, but the GSM standard defines some language code indicators that allows the system to identify national variants of this part, to support more characters than those displayed in the above table.

In a standard GSM text message, all characters are encoded using 7-bit code units, packed together to fill all bits of octets. So, for example, the 140-octet envelope of an SMS, [3] with no other language indicator but only the standard class prefix, can transport up to (140*8)/7=160, that is 160 GSM 7-bit characters (but note that the ESC code counts for one of them, if characters in the high part of the table are used).

Longer messages may be sent, but will require a continuation prefix and a sequence number on subsequent SMS messages (these prefix bytes and sequence number are counted within the maximum length of the 140-octet payload of the envelope format).

When there are 1 to 6 spare bits in the last octet of a message, these bits are set to zero (these bits do not count as a character but only as a filler). When there are 7 spare bits in the last octet of a message, these bits are set to the 7-bit code of the CR control (also used as a padding filler) instead of being set to zero (where they would be confused with the 7-bit code of an '@' character).

This 7-bit encoding allows the transport of texts consisting of printable characters from Basic Latin (Unicode block) (with the exception of the grave accent/backtick), as well as some characters of the ISO Latin 1 character set. It also allows the encoding of texts written in the Greek script, but only capitals; for such use in Greek, the Latin capital letters that look like the Greek letters are reused with the same code, so that the above character set is complete only for modern monotonic Greek restricted to capital letters. A complete support for the Greek alphabet (including small letters) requires a national version of the shifted 7-bit table (using the ESC code for each national character encoded in this shifted table), or an unspecified proprietary 8-bit encoding, or the use of the UCS-2 encoding (see below).

Note that the special code marked SS2 in the table above has also been assigned (and encoded as 0x1B,0x1B) to allow using another alternate 7-bit shift table. But this mechanism has never been used and the UCS-2 encoding has been preferred.

Note that the character 0x09 (Ç, capital C with cedilla) should instead be replaced by ç (small c with cedilla) in modern implementation, as recommended by Unicode, [4] since the uppercase version is of little use.

GSM 8-bit data encoding

8-bit data encoding mode treats the information as raw data. According to the standard, the alphabet for this encoding is user-specific.

UCS-2 Encoding

This encoding allows use of a greater range of characters and languages. UCS-2 can represent the most commonly used Latin and eastern characters at the cost of a greater space expense. Strictly speaking, UCS-2 is limited to characters in the Basic Multilingual Plane. However, since modern programming environments do not provide encoders or decoders for UCS-2, some cell phones (e.g. iPhones) use UTF-16 instead of UCS-2. [5] This works, because for characters in the Basic Multilingual Plane (including full alphabets of most modern human languages) UCS-2 and UTF-16 encodings are identical. To encode characters outside of the BMP (unreachable in plain UCS-2), such as Emoji, UTF-16 uses surrogate pairs, which when decoded with UCS-2 would appear as two valid but unmapped code points.

A single SMS GSM message using this encoding can have at most 70 characters (140 octets).

Note that on many GSM cell phones, there's no specific preselection of the UCS-2 encoding. The default is to use the 7-bit encoding described above, until one enters a character that is not present in the GSM 7-bit table (for example the lowercase 'a' with acute: 'á'). In that case, the whole message gets reencoded using the UCS-2 encoding, and the maximum length of the message sent in a single SMS is immediately reduced to 70 characters, instead of 160. Others vary based on the choice and configuration of SMS application, and the length of the message[ citation needed ].

To avoid unexpected costs for senders that have a subscription for a limited pack of sent SMS, applications should[ according to whom? ] display the number of character used and the maximum number of characters in the composed SMS. When a message exceeds this maximum, the message will[ clarification needed ] be sent as multiple successive SMS containing parts of the message (each one containing a sequence number, which also uses a few leading characters in each part); these parts are intended to be[ citation needed ] reassembled later by the recipient.

Some applications alert the user when a message will require splitting, or even send a longer message as a multimedia message (MMS).

National language shift tables

Since release 8 of the 3GPP 23.038 standard of March 2008, additional characters sets can be accessed through the use of a National Language Shift Tables.

These tables allow using of different character sets according to the language the text is going to be written. The choice of table for a given message is selected in the User Data Header section of an SMS message and can be specified for the whole text (a Locking shift table replacing standard GSM 7-bit default alphabet table) or a single character (Single shift table replacing the GSM 7-bit default alphabet extension table). Locking and Single shift tables together in the same message are possible, if both standard default alphabet table and default alphabet extension table are to be replaced.

Using a shift table, a message can still use 7-bit encoding for the characters, but a different set can be chosen to correctly show accented and language specific characters. This allows up to 155 characters, encoded in 136 octets (140 octets, minus the 4-octets of User Data Header required to indicate the use of a shift table and the language code). With both Locking and Single shift tables, up to 152 characters are allowed, encoded in 133 octets (140 octets, minus 7-octets User Data Header).

Characters from any locking shift table take one septet, characters from the single shift table (or Basic Character Set Extension table) take two septets.

Initially, shift tables only for Turkish were specified; Spanish and Portuguese were added in later revisions of release 8. Release 9 introduced 10 languages used in India written with a Brahmic scripts (Bengali, Gujarati, Hindi, Kannada, Malayalam, Oriya, Punjabi, Tamil, Telugu) and Urdu.

There is still no defined national language shift table for French, Greek, Russian, Bulgarian, Arabic, Hebrew and most Central European languages that need a better coverage than the default 7-bit standard character set and its default 7-bit extension character set: if ever any character is composed that cannot be represented in those default GSM 7-bit sets, the message will be automatically reencoded using UCS-2, with the effect of dividing by more than two the maximum length in characters of messages that can be sent at the price of a single SMS (when a message is split in multiple parts, a few other octets are needed in the User Data Header to indicate the sequence number of each part).

Although a revision of GSM 03.38 (as early as in version 4.0.1 of September 1994) has defined Data Coding Scheme values for Cell Broadcast System (CBS) for German, English, Italian, French, Spanish, Dutch, Swedish, Danish, Finnish, Norwegian, Greek and Turkish; with Hungarian, Polish, Czech, Hebrew, Arabic, Russian and Icelandic added in later revisions, no coding tables were defined for these languages. The purpose of this field was purely to identify the language of the message.

There's also no language shift table for Japanese written in basic kanas, or for Korean written in Hangul jamos, or for Chinese written in the Han script. This is often not a problem in Japan, because it uses other standards than GSM and WAP for messaging. The two other languages also have too many distinct characters to fit into a 7-bit shift table.

Spanish language (Latin script)

There's no specific Locking Shift Character Set for the Spanish language. Uses the default Basic Character Set.

Basic Character Set
by default
(No Locking Shift Table Defined for Spanish) [2]
 0x000x100x200x300x400x500x600x70
0x00 @ Δ SP 0 ¡ P ¿ p
0x01 £ _ ! 1 A Q a q
0x02 $ Φ " 2 B R b r
0x03 ¥ Γ # 3 C S c s
0x04 è Λ ¤ 4 D T d t
0x05 é Ω % 5 E U e u
0x06 ù Π & 6 F V f v
0x07 ì Ψ ' 7 G W g w
0x08 ò Σ ( 8 H X h x
0x09 Ç Θ ) 9 I Y i y
0x0ALF Ξ * : J Z j z
0x0B Ø ESC + ; K Ä k ä
0x0C ø Æ , < L Ö l ö
0x0DCR æ - = M Ñ m ñ
0x0E Å ß . > N Ü n ü
0x0F å É / ? O § o à
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Spanish language
UDH contains 0x24 0x01 0x02 [2]
 0x000x100x200x300x400x500x600x70
0x00     |    
0x01     Á   á  
0x02        
0x03        
0x04  ^       
0x05      Ú ú
0x06        
0x07        
0x08   {      
0x09 ç   }   Í   í  
0x0AFF       
0x0B SS2      
0x0C    [     
0x0DCR2   ~     
0x0E    ]     
0x0F   \   Ó   ó  
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Portuguese language (Latin script)

Locking Shift Character Set
for Portuguese language
UDH contains 0x25 0x01 0x03 [2]
 0x000x100x200x300x400x500x600x70
0x00 @ Δ SP 0 Í P ~ p
0x01 £ _ ! 1 A Q a q
0x02 $ ª " 2 B R b r
0x03 ¥ Ç # 3 C S c s
0x04 ê À º 4 D T d t
0x05 é % 5 E U e u
0x06 ú ^ & 6 F V f v
0x07 í \ ' 7 G W g w
0x08 ó ( 8 H X h x
0x09 ç Ó ) 9 I Y i y
0x0ALF | * : J Z j z
0x0B Ô ESC + ; K Ã k ã
0x0C ô Â , < L Õ l õ
0x0DCR â - = M Ú m `
0x0E Á Ê . > N Ü n ü
0x0F á É / ? O § o à
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Portuguese language
UDH contains 0x24 0x01 0x03 [2]
 0x000x100x200x300x400x500x600x70
0x00     |    
0x01     À   Â  
0x02  Φ       
0x03  Γ       
0x04  ^       
0x05 ê Ω     Ú ú
0x06  Π       
0x07  Ψ       
0x08  Σ {      
0x09 ç Θ }   Í   í  
0x0AFF       
0x0B Ô SS2    Ã   ã
0x0C ô    [   Õ   õ
0x0DCR2   ~     
0x0E Á    ]     
0x0F á Ê \   Ó   ó â
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Turkish language (Latin script)

Locking Shift Character Set
for Turkish language
UDH contains 0x25 0x01 0x01 [2]
 0x000x100x200x300x400x500x600x70
0x00 @ Δ SP 0 İ P ç p
0x01 £ _ ! 1 A Q a q
0x02 $ Φ " 2 B R b r
0x03 ¥ Γ # 3 C S c s
0x04 Λ ¤ 4 D T d t
0x05 é Ω % 5 E U e u
0x06 ù Π & 6 F V f v
0x07 ı Ψ ' 7 G W g w
0x08 ò Σ ( 8 H X h x
0x09 Ç Θ ) 9 I Y i y
0x0ALF Ξ * : J Z j z
0x0B Ğ ESC + ; K Ä k ä
0x0C ğ Ş , < L Ö l ö
0x0DCR ş - = M Ñ m ñ
0x0E Å ß . > N Ü n ü
0x0F å É / ? O § o à
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Turkish language
UDH contains 0x24 0x01 0x01 [2]
 0x000x100x200x300x400x500x600x70
0x00     |    
0x01        
0x02        
0x03      Ş ç ş
0x04  ^       
0x05        
0x06        
0x07     Ğ   ğ  
0x08   {      
0x09   }   İ   ı  
0x0AFF       
0x0B SS2      
0x0C    [     
0x0DCR2   ~     
0x0E    ]     
0x0F   \      
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Urdu language (Arabic and basic Latin scripts)

It may also be used for the Sindhi language also written in the Arabic script.

Sometimes it may be used for Arabic language as well, but the Eastern digits (encoded here in their Persian-Hindu variant) won't be used in that case because standard Arabic prefer its traditional Eastern Arabic digits, and will frequently be replaced by Western Arabic digits (encoded in the locking shift character set in column 0x30) which are also used now frequently in Urdu as well. However, in India, phones recognizing the Arabic language indication may substitute the Persian-Hindu variants of the Eastern Arabic digits by the traditional Eastern Arabic digits.

Locking Shift Character Set
for Urdu language
UDH contains 0x25 0x01 0x0D [2]
 0x000x100x200x300x400x500x600x70
0x00 ا ث SP 0 ص ں ٔ p
0x01 آ ج ! 1 ض ڻ a q
0x02 ب ځ ڏ 2 ط ڼ b r
0x03 ٻ ڄ ڍ 3 ظ و c s
0x04 ڀ ڃ ذ 4 ع ۄ d t
0x05 پ څ ر 5 ف ە e u
0x06 ڦ چ ڑ 6 ق ہ f v
0x07 ت ڇ ړ 7 ک ھ g w
0x08 ۂ ح ) 8 ڪ ء h x
0x09 ٿ خ ( 9 ګ ی i y
0x0ALF د ڙ : گ ې j z
0x0B ٹ ESC ز ; ڳ ے k ٕ
0x0C ٽ ڌ , ښ ڱ ٍ l ّ
0x0DCR ڈ ږ س ل ِ m ٓ
0x0E ٺ ډ . ش م ُ n ٖ
0x0F ټ ڊ ژ ? ن ٗ o ٰ
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Urdu language
UDH contains 0x24 0x01 0x0D [2]
 0x000x100x200x300x400x500x600x70
0x00 @ < ۴ ؓ | P   
0x01 £ = ۵ ؔ A Q   
0x02 $ > ۶ ؛ B R   
0x03 ¥ ¡ ۷ ؟ C S   
0x04 ¿ ^ ۸ ـ D T   
0x05 " ¡ ۹ ْ E U  
0x06 ¤ _ ، ٘ F V   
0x07 % # ؍ ٫ G W   
0x08 & * { ٬ H X   
0x09 ' ؀ } ٲ I Y   
0x0AFF ؁ ؎ ٳ J Z   
0x0B * SS2 ؏ ۍ K    
0x0C + ۰ ؐ [ L    
0x0DCR2 ۱ ؑ ~ M    
0x0E - ۲ ؒ ] N    
0x0F / ۳ \ ۔ O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Hindi language (Devanagari and basic Latin scripts)

Locking Shift Character Set
for Hindi language
UDH contains 0x25 0x01 0x06 [2]
 0x000x100x200x300x400x500x600x70
0x00 ◌ँ SP 0 ◌ा p
0x01 ◌ं ! 1 ◌ि a q
0x02 ◌ः 2 ◌ी b r
0x03 3 ◌ु c s
0x04 4 ◌ू d t
0x05 5 ◌ृ e u
0x06 6 ◌ॄ f v
0x07 7 ◌ॅ g w
0x08 ) 8 ◌ॆ h x
0x09 ( 9 ◌े i y
0x0ALF : ◌ै j z
0x0B ESC ; ◌ॉ k
0x0C , ◌ॊ l
0x0DCR ◌ो m
0x0E . ◌ौ n
0x0F ? o ॿ
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Hindi language
UDH contains 0x24 0x01 0x06 [2]
 0x000x100x200x300x400x500x600x70
0x00 @ < ज़ | P   
0x01 £ = ड़ A Q   
0x02 $ > ढ़ B R   
0x03 ¥ ¡ फ़ C S   
0x04 ¿ ^ य़ D T   
0x05 " ¡ E U  
0x06 ¤ _ ◌॑ F V   
0x07 % # ◌॒ ◌ॢ G W   
0x08 & * { ◌ॣ H X   
0x09 ' } I Y   
0x0AFF ◌॓ J Z   
0x0B * SS2 ◌॔   K    
0x0C + क़ [ L    
0x0DCR2 ख़ ~ M    
0x0E - ग़ ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Bengali and Assamese languages (Bengali and basic Latin scripts)

Locking Shift Character Set
for Bengali and Assamese languages
UDH contains 0x25 0x01 0x04 [2]
 0x000x100x200x300x400x500x600x70
0x00 ◌ঁ SP 0 ◌ব p
0x01 ◌ং   ! 1 ◌ি a q
0x02 ◌ঃ   2 ◌ী b r
0x03 3 ◌ু c s
0x04 4 ◌ূ d t
0x05 5   ◌ৃ e u
0x06 6 ◌ৄ f v
0x07 7    g w
0x08 ) 8    h x
0x09 ( 9   ◌ে i y
0x0ALF : ◌ৈ j z
0x0B ESC ;   k ◌ৗ
0x0C  ,     l ড়
0x0DCR ◌ো m ঢ়
0x0E  . ◌ৌ n
0x0F ? o
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Bengali and Assamese languages
UDH contains 0x24 0x01 0x04 [2]
 0x000x100x200x300x400x500x600x70
0x00 @ < | P   
0x01 £ = A Q   
0x02 $ > B R   
0x03 ¥ ¡ C S   
0x04 ¿ ^ য় D T   
0x05 " ¡   E U  
0x06 ¤ _   F V   
0x07 % # ◌ৢ   G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF ◌ৣ   J Z   
0x0B * SS2   K    
0x0C + [ L    
0x0DCR2 ~ M    
0x0E - ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Punjabi language (Gurmukhī and basic Latin scripts)

Locking Shift Character Set
for Punjabi language
UDH contains 0x25 0x01 0x0A [2]
 0x000x100x200x300x400x500x600x70
0x00 ◌ਁ SP 0 ◌ਾ ◌ੑ p
0x01 ◌ਂ   ! 1 ◌ਿ a q
0x02 ◌ਃ   2 ◌ੀ b r
0x03 3 ◌ੁ c s
0x04 4 ◌ੂ d t
0x05 5    e u
0x06 6   f v
0x07 7 ਲ਼   g w
0x08 ) 8    h x
0x09  ( 9 ◌ੇ i y
0x0ALF : ਸ਼ ◌ੈ j z
0x0B ESC ;    k ◌ੰ
0x0C  ,     l ◌ੱ
0x0DCR ◌ੋ m
0x0E  . ◌ੌ n
0x0F ?   o
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Punjabi language
UDH contains 0x24 0x01 0x0A [2]
 0x000x100x200x300x400x500x600x70
0x00 @ <   | P   
0x01 £ =   A Q   
0x02 $ >   B R   
0x03 ¥ ¡   C S   
0x04 ¿ ^   D T   
0x05 " ¡   E U  
0x06 ¤ _ ਖ਼   F V   
0x07 % # ਗ਼   G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF ਜ਼   J Z   
0x0B * SS2   K    
0x0C + ਫ਼ [ L    
0x0DCR2 ◌ੵ ~ M    
0x0E -   ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Gujarati language (Gujarati and basic Latin scripts)

Locking Shift Character Set
for Gujarati language
UDH contains 0x25 0x01 0x05 [2]
 0x000x100x200x300x400x500x600x70
0x00 ◌ઁ SP 0 ◌ા p
0x01 ◌ં ! 1 ◌િ a q
0x02 ◌ઃ   2 ◌ી b r
0x03 3 ◌ુ c s
0x04 4 ◌ૂ d t
0x05 5   ◌ૃ e u
0x06 6 ◌ૄ f v
0x07 7 ◌ૅ g w
0x08 ) 8    h x
0x09 ( 9 ◌ે i y
0x0ALF : ◌ૈ j z
0x0B ESC ; ◌ૉ k
0x0C ,     l
0x0DCR ◌ો m ◌ૢ
0x0E  . ◌ૌ n ◌ૣ
0x0F ? o
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Gujarati language
UDH contains 0x24 0x01 0x05 [2]
 0x000x100x200x300x400x500x600x70
0x00 @ <   | P   
0x01 £ =   A Q   
0x02 $ >   B R   
0x03 ¥ ¡   C S   
0x04 ¿ ^   D T   
0x05 " ¡   E U  
0x06 ¤ _    F V   
0x07 % #    G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF    J Z   
0x0B * SS2   K    
0x0C +   [ L    
0x0DCR2   ~ M    
0x0E -   ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Oriya language (Oriya and basic Latin scripts)

Locking Shift Character Set
for Oriya language
UDH contains 0x25 0x01 0x09 [2]
 0x000x100x200x300x400x500x600x70
0x00 ◌ଁ SP 0 ◌ା ◌ୖ p
0x01 ◌ଂ   ! 1 ◌ି a q
0x02 ◌ଃ   2 ◌ୀ b r
0x03 3 ◌ୁ c s
0x04 4 ◌ୂ d t
0x05 5   ◌ୃ e u
0x06 6 f v
0x07 7   g w
0x08 ) 8    h x
0x09 ( 9 ◌େ i y
0x0ALF : ◌ୈ j z
0x0B ESC ;   k ◌ୗ
0x0C  ,     l
0x0DCR ◌ୋ m
0x0E  . ◌ୌ n ◌ୢ
0x0F ? o ◌ୣ
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Oriya language
UDH contains 0x24 0x01 0x09 [2]
 0x000x100x200x300x400x500x600x70
0x00 @ <   | P   
0x01 £ =   A Q   
0x02 $ >   B R   
0x03 ¥ ¡   C S   
0x04 ¿ ^   D T   
0x05 " ¡   E U  
0x06 ¤ _ ଡ଼   F V   
0x07 % # ଢ଼   G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF   J Z   
0x0B * SS2   K    
0x0C + [ L    
0x0DCR2   ~ M    
0x0E -   ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Tamil language (Tamil and basic Latin scripts)

Locking Shift Character Set
for Tamil language
UDH contains 0x25 0x01 0x0B [2]
 0x000x100x200x300x400x500x600x70
0x00  SP 0   ◌ா p
0x01 ◌ஂ   ! 1   ◌ி a q
0x02 ◌ஃ 2 ◌ீ b r
0x03   3 ◌ு c s
0x04   4 ◌ூ d t
0x05   5   e u
0x06   6   f v
0x07   7   g w
0x08   ) 8 ◌ெ h x
0x09  ( 9 ◌ே i y
0x0ALF   : ◌ை j z
0x0B ESC  ;   k ◌ௗ
0x0C   , ◌ொ l
0x0DCR   ◌ோ m
0x0E   .    ◌ௌ n
0x0F ?   o
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Tamil language
UDH contains 0x24 0x01 0x0B [2]
 0x000x100x200x300x400x500x600x70
0x00 @ <   | P   
0x01 £ =   A Q   
0x02 $ >   B R   
0x03 ¥ ¡   C S   
0x04 ¿ ^   D T   
0x05 " ¡   E U  
0x06 ¤ _   F V   
0x07 % #   G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF   J Z   
0x0B * SS2   K    
0x0C + [ L    
0x0DCR2 ~ M    
0x0E - ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Telugu language (Telugu and basic Latin scripts)

Locking Shift Character Set
for Telugu language
UDH contains 0x25 0x01 0x0C [2]
 0x000x100x200x300x400x500x600x70
0x00 ◌ఁ SP 0 ◌ా p
0x01 ◌ం   ! 1 ◌ి a q
0x02 ◌ః 2 ◌ీ b r
0x03 3 ◌ు c s
0x04 4 ◌ూ d t
0x05 5 ◌ృ e u
0x06 6 ◌ౄ f v
0x07 7   g w
0x08 ) 8   ◌ె h x
0x09 ( 9 ◌ే i y
0x0ALF : ◌ై j z
0x0B ESC ;   k
0x0C  ,   ◌ొ l
0x0DCR ◌ో m
0x0E .   ◌ౌ n ◌ౢ
0x0F ? o ◌ౣ
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Telugu language
UDH contains 0x24 0x01 0x0C [2]
 0x000x100x200x300x400x500x600x70
0x00 @ < | P   
0x01 £ = A Q   
0x02 $ > ౿ B R   
0x03 ¥ ¡   C S   
0x04 ¿ ^   D T   
0x05 " ¡   E U   
0x06 ¤ _   F V   
0x07 % #   G W   
0x08 & * {   H X   
0x09 '   }   I Y   
0x0AFF    J Z   
0x0B * SS2   K    
0x0C + [ L    
0x0DCR2 ~ M    
0x0E - ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Kannada language (Kannada and basic Latin scripts)

Locking Shift Character Set
for Kannada language
UDH contains 0x25 0x01 0x07 [2]
 0x000x100x200x300x400x500x600x70 !  
0x00  SP 0 p
0x01   ! 1 ಿ a q
0x02 2 b r
0x03 3 c s
0x04 4 d t
0x05 5 e u
0x06 6 f v
0x07 7   g w
0x08 ) 8   h x
0x09 ( 9 i y
0x0ALF : j z
0x0B ESC ;   k
0x0C  ,   l
0x0DCR m
0x0E . n
0x0F ? o
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Kannada language
UDH contains 0x24 0x01 0x07 [2]
 0x000x100x200x300x400x500x600x70 !  
0x00 @ <   | P   
0x01 £ =   A Q   
0x02 $ >   B R   
0x03 ¥ ¡   C S   
0x04 ¿ ^   D T   
0x05 " ¡   E U  
0x06 ¤ _   F V   
0x07 % #   G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF   J Z   
0x0B * SS2   K    
0x0C +   ] L    
0x0DCR2   ~ M    
0x0E -   ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

Malayalam language (Malayalam and basic Latin scripts)

Locking Shift Character Set
for Malayalam language
UDH contains 0x25 0x01 0x08 [2]
 0x000x100x200x300x400x500x600x70 !  
0x00  SP 0 p
0x01   ! 1 ി a q
0x02 2 b r
0x03 3 c s
0x04 4 d t
0x05 5 e u
0x06 6 f v
0x07 7   g w
0x08 ) 8 h x
0x09 ( 9 i y
0x0ALF : j z
0x0B ESC ;   k
0x0C  ,   l
0x0DCR m
0x0E .   n
0x0F ? o
  • LF is a Line Feed control.
  • CR is a Carriage Return control, or filler.
  • ESC is an Escape control.
  • SP is a Space character.
Single Shift Character Set
for Malayalam language
UDH contains 0x25 0x01 0x08 [2]
 0x000x100x200x300x400x500x600x70 !  
0x00 @ < - P   
0x01 £ = A Q   
0x02 $ > B R   
0x03 ¥ ¡ C S   
0x04 ¿ ^ ൿ D T   
0x05 " ¡   E U  
0x06 ¤ _   F V   
0x07 % #   G W   
0x08 & * {   H X   
0x09 ' }   I Y   
0x0AFF   J Z   
0x0B * SS2   K    
0x0C + [ L    
0x0DCR2 ~ M    
0x0E - ] N    
0x0F / \   O    
  • FF is a Page Break control. If not recognized, it shall be treated like LF.
  • CR2 is a control character. No language specific character shall be encoded at this position.
  • SS2 is a second Single Shift Escape control reserved for future extensions.

See also

Related Research Articles

<span class="mw-page-title-main">ASCII</span> American character encoding standard

ASCII, an acronym for American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. Because of technical limitations of computer systems at the time it was invented, ASCII has just 128 code points, of which only 95 are printable characters, which severely limited its scope. Modern computer systems have evolved to use Unicode, which has millions of code points, but the first 128 of these are the same as the ASCII set.

<span class="mw-page-title-main">Character encoding</span> Using numbers to represent text characters

Character encoding is the process of assigning numbers to graphical characters, especially the written characters of human language, allowing them to be stored, transmitted, and transformed using digital computers. The numerical values that make up a character encoding are known as "code points" and collectively comprise a "code space", a "code page", or a "character map".

Extended Binary Coded Decimal Interchange Code is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. It descended from the code used with punched cards and the corresponding six-bit binary-coded decimal code used with most of IBM's computer peripherals of the late 1950s and early 1960s. It is supported by various non-IBM platforms, such as Fujitsu-Siemens' BS2000/OSD, OS-IV, MSP, and MSP-EX, the SDS Sigma series, Unisys VS/9, Unisys MCP and ICL VME.

ISO/IEC 8859 is a joint ISO and IEC series of standards for 8-bit character encodings. The series of standards consists of numbered parts, such as ISO/IEC 8859-1, ISO/IEC 8859-2, etc. There are 15 parts, excluding the abandoned ISO/IEC 8859-12. The ISO working group maintaining this series of standards has been disbanded.

<span class="mw-page-title-main">SMS</span> Text messaging service component

Short Message/Messaging Service, commonly abbreviated as SMS, is a text messaging service component of most telephone, Internet and mobile device systems. It uses standardized communication protocols that let mobile devices exchange short text messages. An intermediary service can facilitate a text-to-voice conversion to be sent to landlines.

Short Message Peer-to-Peer (SMPP) in the telecommunications industry is an open, industry standard protocol designed to provide a flexible data communication interface for the transfer of short message data between External Short Messaging Entities (ESMEs), Routing Entities (REs) and SMSC.

<span class="mw-page-title-main">Unicode</span> Character encoding standard

Unicode, formally The Unicode Standard, is a text encoding standard maintained by the Unicode Consortium designed to support the use of text written in all of the world's major writing systems. Version 15.1 of the standard defines 149813 characters and 161 scripts used in various ordinary, literary, academic, and technical contexts.

Web pages authored using HyperText Markup Language (HTML) may contain multilingual text represented with the Unicode universal character set. Key to the relationship between Unicode and HTML is the relationship between the "document character set", which defines the set of characters that may be present in an HTML document and assigns numbers to them, and the "external character encoding", or "charset", used to encode a given document as a sequence of bytes.

UTF-8 is a variable-length character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation Format – 8-bit.

<span class="mw-page-title-main">UTF-16</span> Variable-width encoding of Unicode, using one or two 16-bit code units

UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as code points are encoded with one or two 16-bit code units. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as UCS-2 (for 2-byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names.

ISO/IEC 2022Information technology—Character code structure and extension techniques, is an ISO/IEC standard in the field of character encoding. It is equivalent to the ECMA standard ECMA-35, the ANSI standard ANSI X3.41 and the Japanese Industrial Standard JIS X 0202. Originating in 1971, it was most recently revised in 1994.

External Machine Interface (EMI), an extension to Universal Computer Protocol (UCP), is a protocol primarily used to connect to short message service centres (SMSCs) for mobile telephones. The protocol was developed by CMG Wireless Data Solutions, now part of Mavenir.

The currency sign¤ is a character used to denote an unspecified currency. It can be described as a circle the size of a lowercase character with four short radiating arms at 45° (NE), 135° (SE), 225° (SW) and 315° (NW). It is raised slightly above the baseline. The character is sometimes called scarab.

<span class="mw-page-title-main">Cell Broadcast</span> Method of sending messages to multiple mobile phone users

Cell Broadcast (CB) is a method of sending messages to multiple mobile telephone users in a defined area at the same time. It is defined by the ETSI's GSM committee and 3GPP and is part of the 2G, 3G, 4G LTE (telecommunication) and 5G standards. It is also known as Short Message Service-Cell Broadcast (SMS-CB) or CB SMS.

<span class="mw-page-title-main">Universal Character Set characters</span> Complete list of the characters available on most computers

The Unicode Consortium and the ISO/IEC JTC 1/SC 2/WG 2 jointly collaborate on the list of the characters in the Universal Coded Character Set. The Universal Coded Character Set, most commonly called the Universal Character Set, is an international standard to map characters, discrete symbols used in natural language, mathematics, music, and other domains, to unique machine-readable data values. By creating this mapping, the UCS enables computer software vendors to interoperate, and transmit—interchange—UCS-encoded text strings from one to another. Because it is a universal map, it can be used to represent multiple languages at the same time. This avoids the confusion of using multiple legacy character encodings, which can result in the same sequence of codes having multiple interpretations depending on the character encoding in use, resulting in mojibake if the wrong one is chosen.

In the cellular phone industry, mobile phones and their networks sometimes support concatenated short message service to overcome the limitation on the number of characters that can be sent in a single SMS text message transmission. Using this method, long messages are split into smaller messages by the sending device and recombined at the receiving end. Each message is then billed separately. When the feature works properly, it is nearly transparent to the user, appearing as a single long text message. Previously, due to incompatibilities between providers and lack of support in some phone models, there was not widespread use of this feature.

The Universal Coded Character Set is a standard set of characters defined by the international standard ISO/IEC 10646, Information technology — Universal Coded Character Set (UCS), which is the basis of many character encodings, improving as characters from previously unrepresented typing systems are added.

User Data Header (UDH) is a binary structure which may be present at the start of a short message in the Short Message Service in GSM. It does not contain any text, but it specifies how the message should be formatted and processed.

GSM 03.40 or 3GPP TS 23.040 is a mobile telephony standard describing the format of the Transfer Protocol Data Units (TPDU) of the Short Message Transfer Protocol (SM-TP) used in the GSM networks to carry Short Messages. This format is used throughout the whole transfer of the message in the GSM mobile network. In contrast, application servers use different protocols, like Short Message Peer-to-Peer or Universal Computer Protocol, to exchange messages between them and the Short Message Service Center (SMSC).

Data Coding Scheme is a one-octet field in Short Messages (SM) and Cell Broadcast Messages (CB) which carries a basic information how the recipient handset should process the received message. The information includes:

References

  1. 3GPP TS 23.038, Alphabets and language-specific information.
  2. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Alphabets and language-specific information (3G TS 23.038 version 12.0.0) (zipped .doc file), ETSI, September 2014.
  3. "The text messages [...] contain up to 140 octets." in 3GPP TS 23.040 Technical realization of the Short Message Service (SMS)
  4. GSM 03.38 to Unicode GSM 03.38 to Unicode
  5. Chad Selph (2012-11-08). "Adventures in Unicode SMS". Twilio. Archived from the original on 2015-09-08. Retrieved 2015-08-28.