In mobile telephony GSM 03.38 or 3GPP 23.038 is a character encoding used in GSM networks for SMS (Short Message Service), CB (Cell Broadcast) and USSD (Unstructured Supplementary Service Data). The 3GPP TS 23.038 standard (originally GSM recommendation 03.38) defines GSM 7-bit default alphabet which is mandatory for GSM handsets and network elements, [1] but the character set is suitable only for English and a number of Western-European languages. Languages such as Chinese, Korean or Japanese must be transferred using the 16-bit UCS-2 character encoding. A limited number of languages, like Portuguese, Spanish, Turkish and a number of languages used in India written with a Brahmic scripts may use 7-bit encoding with national language shift table defined in 3GPP 23.038. For binary messages, 8-bit encoding is used.
The standard encoding for GSM messages is the 7-bit default alphabet as defined in the 23.038 recommendation.
Seven-bit characters must be encoded into octets following one of three packing modes:
|
|
It is important (especially when a message is to be segmented using concatenated SMS mechanism) that characters from the Basic Character Set table take one septet, characters from the Basic Character Set Extension table take two septets.
Note that the second part of the table is only accessible if the GSM device supports the 7-bit extension mechanism, using the ESC character prefix. Otherwise, the ESC code itself is interpreted as a space, and the following character will be treated as if there was no leading ESC code.
Most of the high part of the table is not used in the default character set, but the GSM standard defines some language code indicators that allows the system to identify national variants of this part, to support more characters than those displayed in the above table.
In a standard GSM text message, all characters are encoded using 7-bit code units, packed together to fill all bits of octets. So, for example, the 140-octet envelope of an SMS, [3] with no other language indicator but only the standard class prefix, can transport up to (140*8)/7=160, that is 160 GSM 7-bit characters (but note that the ESC code counts for one of them, if characters in the high part of the table are used).
Longer messages may be sent, but will require a continuation prefix and a sequence number on subsequent SMS messages (these prefix bytes and sequence number are counted within the maximum length of the 140-octet payload of the envelope format).
When there are 1 to 6 spare bits in the last octet of a message, these bits are set to zero (these bits do not count as a character but only as a filler). When there are 7 spare bits in the last octet of a message, these bits are set to the 7-bit code of the CR control (also used as a padding filler) instead of being set to zero (where they would be confused with the 7-bit code of an '@' character).
This 7-bit encoding allows the transport of texts consisting of printable characters from Basic Latin (Unicode block) (with the exception of the grave accent/backtick), as well as some characters of the ISO Latin 1 character set. It also allows the encoding of texts written in the Greek script, but only capitals; for such use in Greek, the Latin capital letters that look like the Greek letters are reused with the same code, so that the above character set is complete only for modern monotonic Greek restricted to capital letters. A complete support for the Greek alphabet (including small letters) requires a national version of the shifted 7-bit table (using the ESC code for each national character encoded in this shifted table), or an unspecified proprietary 8-bit encoding, or the use of the UCS-2 encoding (see below).
Note that the special code marked SS2 in the table above has also been assigned (and encoded as 0x1B,0x1B) to allow using another alternate 7-bit shift table. But this mechanism has never been used and the UCS-2 encoding has been preferred.
Note that the character 0x09 (Ç, capital C with cedilla) should instead be replaced by ç (small c with cedilla) in modern implementation, as recommended by Unicode, [4] since the uppercase version is of little use.
8-bit data encoding mode treats the information as raw data. According to the standard, the alphabet for this encoding is user-specific.
This encoding allows use of a greater range of characters and languages. UCS-2 can represent the most commonly used Latin and eastern characters at the cost of a greater space expense. Strictly speaking, UCS-2 is limited to characters in the Basic Multilingual Plane. However, since modern programming environments do not provide encoders or decoders for UCS-2, some cell phones (e.g. iPhones) use UTF-16 instead of UCS-2. [5] This works, because for characters in the Basic Multilingual Plane (including full alphabets of most modern human languages) UCS-2 and UTF-16 encodings are identical. To encode characters outside of the BMP (unreachable in plain UCS-2), such as Emoji, UTF-16 uses surrogate pairs, which when decoded with UCS-2 would appear as two valid but unmapped code points.
A single SMS GSM message using this encoding can have at most 70 characters (140 octets).
Note that on many GSM cell phones, there's no specific preselection of the UCS-2 encoding. The default is to use the 7-bit encoding described above, until one enters a character that is not present in the GSM 7-bit table (for example the lowercase 'a' with acute: 'á'). In that case, the whole message gets reencoded using the UCS-2 encoding, and the maximum length of the message sent in a single SMS is immediately reduced to 70 characters, instead of 160. Others vary based on the choice and configuration of SMS application, and the length of the message[ citation needed ].
To avoid unexpected costs for senders that have a subscription for a limited pack of sent SMS, applications should[ according to whom? ] display the number of character used and the maximum number of characters in the composed SMS. When a message exceeds this maximum, the message will[ clarification needed ] be sent as multiple successive SMS containing parts of the message (each one containing a sequence number, which also uses a few leading characters in each part); these parts are intended to be[ citation needed ] reassembled later by the recipient.
Some applications alert the user when a message will require splitting, or even send a longer message as a multimedia message (MMS).
Since release 8 of the 3GPP 23.038 standard of March 2008, additional characters sets can be accessed through the use of a National Language Shift Tables.
These tables allow using of different character sets according to the language the text is going to be written. The choice of table for a given message is selected in the User Data Header section of an SMS message and can be specified for the whole text (a Locking shift table replacing standard GSM 7-bit default alphabet table) or a single character (Single shift table replacing the GSM 7-bit default alphabet extension table). Locking and Single shift tables together in the same message are possible, if both standard default alphabet table and default alphabet extension table are to be replaced.
Using a shift table, a message can still use 7-bit encoding for the characters, but a different set can be chosen to correctly show accented and language specific characters. This allows up to 155 characters, encoded in 136 octets (140 octets, minus the 4-octets of User Data Header required to indicate the use of a shift table and the language code). With both Locking and Single shift tables, up to 152 characters are allowed, encoded in 133 octets (140 octets, minus 7-octets User Data Header).
Characters from any locking shift table take one septet, characters from the single shift table (or Basic Character Set Extension table) take two septets.
Initially, shift tables only for Turkish were specified; Spanish and Portuguese were added in later revisions of release 8. Release 9 introduced 10 languages used in India written with a Brahmic scripts (Bengali, Gujarati, Hindi, Kannada, Malayalam, Oriya, Punjabi, Tamil, Telugu) and Urdu.
There is still no defined national language shift table for French, Greek, Russian, Bulgarian, Arabic, Hebrew and most Central European languages that need a better coverage than the default 7-bit standard character set and its default 7-bit extension character set: if ever any character is composed that cannot be represented in those default GSM 7-bit sets, the message will be automatically reencoded using UCS-2, with the effect of dividing by more than two the maximum length in characters of messages that can be sent at the price of a single SMS (when a message is split in multiple parts, a few other octets are needed in the User Data Header to indicate the sequence number of each part).
Although a revision of GSM 03.38 (as early as in version 4.0.1 of September 1994) has defined Data Coding Scheme values for Cell Broadcast System (CBS) for German, English, Italian, French, Spanish, Dutch, Swedish, Danish, Finnish, Norwegian, Greek and Turkish; with Hungarian, Polish, Czech, Hebrew, Arabic, Russian and Icelandic added in later revisions, no coding tables were defined for these languages. The purpose of this field was purely to identify the language of the message.
There's also no language shift table for Japanese written in basic kanas, or for Korean written in Hangul jamos, or for Chinese written in the Han script. This is often not a problem in Japan, because it uses other standards than GSM and WAP for messaging. The two other languages also have too many distinct characters to fit into a 7-bit shift table.
There's no specific Locking Shift Character Set for the Spanish language. Uses the default Basic Character Set.
|
|
|
|
|
|
It may also be used for the Sindhi language also written in the Arabic script.
Sometimes it may be used for Arabic language as well, but the Eastern digits (encoded here in their Persian-Hindu variant) won't be used in that case because standard Arabic prefer its traditional Eastern Arabic digits, and will frequently be replaced by Western Arabic digits (encoded in the locking shift character set in column 0x30) which are also used now frequently in Urdu as well. However, in India, phones recognizing the Arabic language indication may substitute the Persian-Hindu variants of the Eastern Arabic digits by the traditional Eastern Arabic digits.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ASCII, an acronym for American Standard Code for Information Interchange, is a character encoding standard for electronic communication. ASCII codes represent text in computers, telecommunications equipment, and other devices. ASCII has just 128 code points, of which only 95 are printable characters, which severely limit its scope. The set of available punctuation had significant impact on the syntax of computer languages and text markup. ASCII hugely influenced the design of character sets used by modern computers, including Unicode which has over a million code points, but the first 128 of these are the same as ASCII.
Character encoding is the process of assigning numbers to graphical characters, especially the written characters of human language, allowing them to be stored, transmitted, and transformed using digital computers. The numerical values that make up a character encoding are known as "code points" and collectively comprise a "code space", a "code page", or a "character map".
Extended Binary Coded Decimal Interchange Code is an eight-bit character encoding used mainly on IBM mainframe and IBM midrange computer operating systems. It descended from the code used with punched cards and the corresponding six-bit binary-coded decimal code used with most of IBM's computer peripherals of the late 1950s and early 1960s. It is supported by various non-IBM platforms, such as Fujitsu-Siemens' BS2000/OSD, OS-IV, MSP, and MSP-EX, the SDS Sigma series, Unisys VS/9, Unisys MCP and ICL VME.
ISO/IEC 8859 is a joint ISO and IEC series of standards for 8-bit character encodings. The series of standards consists of numbered parts, such as ISO/IEC 8859-1, ISO/IEC 8859-2, etc. There are 15 parts, excluding the abandoned ISO/IEC 8859-12. The ISO working group maintaining this series of standards has been disbanded.
Short Message Service, commonly abbreviated as SMS, is a text messaging service component of most telephone, Internet and mobile device systems. It uses standardized communication protocols that let mobile phones exchange short text messages, typically transmitted over cellular networks.
Short Message Peer-to-Peer (SMPP) in the telecommunications industry is an open, industry standard protocol designed to provide a flexible data communication interface for the transfer of short message data between External Short Messaging Entities (ESMEs), Routing Entities (REs) and SMSC.
Unicode, formally The Unicode Standard, is a text encoding standard maintained by the Unicode Consortium designed to support the use of text in all of the world's writing systems that can be digitized. Version 16.0 of the standard defines 154998 characters and 168 scripts used in various ordinary, literary, academic, and technical contexts.
Web pages authored using HyperText Markup Language (HTML) may contain multilingual text represented with the Unicode universal character set. Key to the relationship between Unicode and HTML is the relationship between the "document character set", which defines the set of characters that may be present in an HTML document and assigns numbers to them, and the "external character encoding", or "charset", used to encode a given document as a sequence of bytes.
UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation Format – 8-bit. Almost every web page is stored in UTF-8.
UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as code points are encoded with one or two 16-bit code units. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as "UCS-2" (for 2-byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names.
External Machine Interface (EMI), an extension to Universal Computer Protocol (UCP), is a protocol primarily used to connect to short message service centres (SMSCs) for mobile telephones. The protocol was developed by CMG Wireless Data Solutions, now part of Mavenir.
The currency sign¤ is a character used to denote an unspecified currency. It can be described as a circle the size of a lowercase character with four short radiating arms at 45° (NE), 135° (SE), 225° (SW) and 315° (NW). It is raised slightly above the baseline. The character is sometimes called scarab.
In Unicode, a Private Use Area (PUA) is a range of code points that, by definition, will not be assigned characters by the standard. Three private use areas are defined: one in the Basic Multilingual Plane, and one each in, and nearly covering, planes 15 and 16. They are intentionally left undefined so that third parties may assign their own characters without conflicting with Unicode Consortium assignments. Under the Unicode Stability Policy, the Private Use Areas will remain allocated for that purpose in all future Unicode versions.
Cell Broadcast (CB) is a method of sending short messages to multiple mobile telephone users in a defined area at the same time. It is defined by the ETSI's GSM committee and 3GPP and is part of the 2G, 3G, 4G LTE (telecommunication) and 5G standards. It is also known as Short Message Service-Cell Broadcast . Unlike Short Message Service-Point to Point (SMS-PP), Cell Broadcast is a one-to-many geo-targeted and geo-fenced messaging service.
The Unicode Consortium and the ISO/IEC JTC 1/SC 2/WG 2 jointly collaborate on the list of the characters in the Universal Coded Character Set. The Universal Coded Character Set, most commonly called the Universal Character Set, is an international standard to map characters, discrete symbols used in natural language, mathematics, music, and other domains, to unique machine-readable data values. By creating this mapping, the UCS enables computer software vendors to interoperate, and transmit—interchange—UCS-encoded text strings from one to another. Because it is a universal map, it can be used to represent multiple languages at the same time. This avoids the confusion of using multiple legacy character encodings, which can result in the same sequence of codes having multiple interpretations depending on the character encoding in use, resulting in mojibake if the wrong one is chosen.
In the cellular phone industry, mobile phones and their networks sometimes support concatenated short message service to overcome the limitation on the number of characters that can be sent in a single SMS text message transmission. Using this method, long messages are split into smaller messages by the sending device and recombined at the receiving end. Each message is then billed separately. When the feature works properly, it is nearly transparent to the user, appearing as a single long text message. Previously, due to incompatibilities between providers and lack of support in some phone models, there was not widespread use of this feature.
The Universal Coded Character Set is a standard set of characters defined by the international standard ISO/IEC 10646, Information technology — Universal Coded Character Set (UCS), which is the basis of many character encodings, improving as characters from previously unrepresented typing systems are added.
User Data Header (UDH) is a binary structure which may be present at the start of a short message in the Short Message Service in GSM. It does not contain any text, but it specifies how the message should be formatted and processed.
GSM 03.40 or 3GPP TS 23.040 is a mobile telephony standard describing the format of the Transfer Protocol Data Units (TPDU) part of the Short Message Transfer Protocol (SM-TP) used in the GSM networks to carry Short Messages. This format is used throughout the whole transfer of the message in the GSM mobile network. In contrast, application servers use different protocols, like Short Message Peer-to-Peer or Universal Computer Protocol, to exchange messages between them and the Short Message service center (SMSC).
Data Coding Scheme is a one-octet field in Short Messages (SM) and Cell Broadcast Messages (CB) which carries a basic information how the recipient handset should process the received message. The information includes: