Japanese language in EBCDIC

Last updated

Several mutually incompatible versions of the Extended Binary Coded Decimal Interchange Code (EBCDIC) have been used to represent the Japanese language on computers, including variants defined by Hitachi, Fujitsu, IBM and others. Some are variable-width encodings, employing locking shift codes to switch between single-byte and double-byte modes. [1] Unlike other EBCDIC locales, the lowercase basic Latin letters are often not preserved in their usual locations. [2]

Contents

The characters which are found in the double-byte Japanese code used with EBCDIC by IBM, but not found in the first edition of JIS X 0208, also influenced the vendor extensions found in some non-EBCDIC encodings such as IBM code page 932 ("DBCS-PC") and Windows code page 932. [3]

Single-byte codes

Similarly to JIS X 0201 (itself incorporated into Shift JIS), Japanese EBCDIC encodings often include a set of single-byte katakana. Several different variants of the single-byte EBCDIC code are used in the Japanese locale, by different vendors; a given vendor may also define two different single-byte codes, one favoured for half-width katakana and one favoured for Latin script. Variants of EBCDIC favoured by a given vendor for use for katakana are sometimes referred to as EBCDIK, standing for Extended Binary-Coded Decimal Interchange Kana code. [1] [4]

Code pages incorporating half-width kana are an exception to IBM's EBCDIC invariant character set, which specifies a set of characters which are usually encoded the same across all EBCDIC code pages. Most notably, they sometimes include katakana characters at code points which are used for lowercase letters of the Basic Latin alphabet in the invariant set. [2] Encoding of lowercase letters when katakana characters are included at those locations, and encoding of katakana characters when lowercase letters are retained in their usual locations, can vary between vendors, as shown below.

Microsoft Windows implements two Japanese single-byte EBCDIC variants, with code page numbers 20000 higher than IBM's code page numbers for its variants, as code pages 20290 (documented as IBM290, "IBM EBCDIC Japanese Katakana Extended") [5] and 21027 ("Extended/Ext Alpha Lowercase"). Code page 21027 as implemented in Windows is an incomplete implementation, lacking two-way mappings for several letters and kana, [6] and is currently deprecated. [5]

IBM's code pages were later updated to include the Euro sign at 0xE1, retaining their original CPGID numbers, but being assigned new CCSID numbers. Hence, the CCSID 290 refers to the original version of code page 290, while the version of code page 290 with the Euro sign is also known as CCSID 8482. [7] Similarly, CCSID 1027 refers to the original version of code page 1027, while the version of CPGID 1027 with the Euro sign is given the CCSID 5123. [8]

Alongside versions of IBM's double-byte Japanese DBCS-Host code page (CPGID 300, CCSID 300 or 16684) as a double-byte component, IBM code page 290 is used as the single-byte component of the multi-byte code page IBM-930 [9] [10] and (as the Euro-updated CCSID 8482) the updated version IBM-1390. [11] [12] IBM code page 1027 is used as the single-byte component of the multi-byte code page IBM-939 [13] [14] and (as the Euro-updated CCSID 5123) the updated version IBM-1399. [15] [16]

In the following table, conformance to the invariant set is marked with green; collision with the invariant set is marked with red.

Single-byte codes in Japanese EBCDIC variants (excluding control codes)
Hexadecimal Lowercase in invariant locations, kana absent or displacedLowercase absent or displaced by kana
Fujitsu EBCDIC [17] Hitachi EBCDIC [18] HP EBCDIK [19] IBM 298 [20] IBM 1027
[13] [15] [21] [22]
IBM 1031 [23] Fujitsu EBCDIK [17] Hitachi EBCDIK [18]
IBM 1136 [24]
NEC EBCDIC [25] [26] IBM 290
[9] [11] [27] [28]
IBM 887 [29] IBM 1030 [30]
0x40 SP SP SP SP SP SP SP SP SP SP SP SP
0x41
0x42
0x43
0x44
0x45
0x46
0x47
0x48
0x49
0x4A£[¢¢¢£[[£££
0x4B............
0x4C<<)<<<<<<<<<
0x4D(([(((((((((
0x4E++<+++++++++
0x4F|!||||!!|||
0x50&&+&&&&&&&&&
0x51
0x52
0x53
0x54
0x55
0x56
0x57a
0x58
0x59ab
0x5A!]!!!!]]!!!
0x5B\¥$$$$\¥\¥¥¥
0x5C************
0x5D))])))))))))
0x5E;;;;;;;;;;;;
0x5F¬^¬¬¬¬^^¬¬¬
0x60------------
0x61////////////
0x62bcaa
0x63cdbb
0x64decc
0x65efdd
0x66fgee
0x67ghff
0x68higg
0x69ijhh
0x6A¦||¦||
0x6B,,,,,,,,,,,,
0x6C%%(%%%%%%%%%
0x6D____________
0x6E>>¥>>>>>>>>>
0x6F???????????
0x70jk[[
0x71klii
0x72lmjj
0x73mnkk
0x74noll
0x75ソソソソソopmm
0x76pqnn
0x77qroo
0x78rspp
0x79```````````
0x7A:::::::::::
0x7B##=#########
0x7C@@'@@@@@@@@@
0x7D'':'''''''''
0x7E==>=========
0x7F""""""""""""
0x80st]]
0x81aaaaaa
0x82bbbbbb
0x83cccccc
0x84dddddd
0x85eeeeee
0x86ffffff
0x87gggggg
0x88hhhhhh
0x89iiiiii
0x8A
0x8Btuqq
0x8C
0x8D
0x8E
0x8F
0x90ソソソソソソ
0x91jjjjjj
0x92kkkkkk
0x93llllll
0x94mmmmmm
0x95nnnnnn
0x96oooooo
0x97pppppp
0x98qqqqqq
0x99rrrrrr
0x9A
0x9Buvrr
0x9Cvw
0x9D
0x9E
0x9F
0xA0¯¯wx~~
0xA1~~¯~~~~~~¯¯¯
0xA2ssssss
0xA3tttttt
0xA4uuuuuu
0xA5vvvvvv
0xA6wwwwww
0xA7xxxxxx
0xA8yyyyyy
0xA9zzzzzz
0xAA
0xABxyss
0xAC
0xAD[[
0xAE
0xAF
0xB0^^yz^^
0xB1££z¢¢
0xB2¥¥\\
0xB3tt
0xB4uu
0xB5vv
0xB6ww
0xB7xx
0xB8yy
0xB9zz
0xBA
0xBB
0xBC
0xBD]]
0xBE
0xBF
0xC0{{?{{{{{{{{
0xC1AAAAAAAAAAAA
0xC2BBBBBBBBBBBB
0xC3CCCCCCCCCCCC
0xC4DDDDDDDDDDDD
0xC5EEEEEEEEEEEE
0xC6FFFFFFFFFFFF
0xC7GGGGGGGGGGGG
0xC8HHHHHHHHHHHH
0xC9IIIIIIIIIIII
0xCA
0xCB
0xCC
0xCD
0xCE
0xCF
0xD0}}!}}}}}}}}
0xD1JJJJJJJJJJJJ
0xD2KKKKKKKKKKKK
0xD3LLLLLLLLLLLL
0xD4MMMMMMMMMMMM
0xD5NNNNNNNNNNNN
0xD6OOOOOOOOOOOO
0xD7PPPPPPPPPPPP
0xD8QQQQQQQQQQQQ
0xD9RRRRRRRRRRRR
0xDA
0xDB°
0xDC±
0xDD
0xDE
0xDF
0xE0$$¥\\$$$$$$
0xE1
0xE2SSSSSSSSSSSS
0xE3TTTTTTTTTTTT
0xE4UUUUUUUUUUUU
0xE5VVVVVVVVVVVV
0xE6WWWWWWWWWWWW
0xE7XXXXXXXXXXXX
0xE8YYYYYYYYYYYY
0xE9ZZZZZZZZZZZZ
0xEA
0xEB
0xEC
0xED
0xEE
0xEFΩ
0xF0000000000000
0xF1111111111111
0xF2222222222222
0xF3333333333333
0xF4444444444444
0xF5555555555555
0xF6666666666666
0xF7777777777777
0xF8888888888888
0xF9999999999999
0xFA
0xFB
0xFC
0xFDµ
0xFE

Double-byte codes

There are three double-byte character codes used for Japanese with EBCDIC: [31] [32] [3] IBM code page 300 (also called IBM Kanji or IBM Japanese DBCS-Host) from IBM, [33] KEIS from Hitachi, [31] and JEF from Fujitsu. [17] These are DBCS-Host encodings, using different shift codes to switch between single-byte EBCDIC and double-byte modes. [1] Codes 0x41 through 0xFE (those used for graphic characters in EBCDIC) are used in pairs to represent characters from a 190×190 grid; code 0x40 (space in EBCDIC) is used doubled as an ideographic space, but not as part of any other double-byte code. [1] [33]

In the IBM version of the DBCS-Host code, the code 0x0F switches to single-byte mode and the code 0x0E switches to double-byte mode, [1] [9] [13] [11] [15] in common with IBM double-byte EBCDIC codes for other CJK languages, [1] such as the EBCDIC version of Johab for Korean. [34] In contrast to KEIS and JEF, the layout of IBM code page 300 is unrelated to JIS X 0208, and conversion between the two must be done via a table; [31] however, its character repertoire has been kept up-to-date with successive revisions of JIS X 0208 so as to remain a superset of JIS X 0208's repertoire. [3] Lead bytes 0x41 through 0x44 are used for non-Kanji characters, lead bytes 0x45 through 0x68 are used for Kanji characters, and lead bytes 0x69 through 0x89 are used for UDC (user-defined characters). [33]

The existence of IBM's Japanese DBCS-Host code had impact beyond EBCDIC systems, since IBM also defined variants of Shift JIS ("DBCS-PC", defined in the pure double-byte Code page 301 and used in the variable width Code page 932 and Code page 942) and of EUC-JP which encode the entire repertoire of IBM code page 300, including 28 non-Kanji and 360 Kanji additional to those originally included in JIS X 0208 (although the non-Kanji because sign ∵ and not sign ¬ were later added to JIS X 0208 in 1983). These are referred to as "IBM-selected" characters and are included as extensions in, for example, Windows code page 932. [3]

Some newer revisions of the IBM-300 code page add additional Kanji with lead bytes 0xB8 through 0xD5 and additional non-Kanji with lead bytes 0xD6 through 0xE9. [33] This revision updated the set for JIS X 0213, including the Euro sign and, while retaining the CPGID 300, was assigned the new CCSID 16684. [35] The code pages IBM-930 (with code page 290 as the single-byte set), [9] IBM-931 (with code page 37 as the single-byte set), and IBM-939 (with code page 1027 as the single-byte set) [13] exclude these additions, while IBM-1390 (with the Euro sign version of code page 290 / CCSID 8482) [11] and IBM-1399 (with the Euro sign version of code page 1027 / CCSID 5123) [15] include them.

In Hitachi KEIS [31] (Kanji-processing Extended Information System), the sequence 0x0A 0x41 switches to single-byte mode and the sequence 0x0A 0x42 switches to double-byte mode. [lower-alpha 1] JIS X 0208 characters are encoded using the same byte sequences used to encode them in EUC-JP, i.e. with both bytes being between 0xA1 and 0xFE inclusive. This results in duplicate encodings for the ideographic space—0x4040 per the DBCS-Host code structure, and 0xA1A1 as in EUC-JP. However, the lead byte range is extended back to 0x59, out of which the lead bytes 0x81–A0 are designated for user-defined characters, [1] and the remainder are used for corporate-defined characters, including both kanji and non-kanji. [3]

In Fujitsu JEF [17] (Japanese-processing Extended Feature), 0x29 switches to single-byte mode and 0x28 switches to double-byte mode. Similarly to KEIS, JIS X 0208 codes are represented the same as in EUC-JP. [1] Differing from KEIS, the JIS X 0208 edition used in this JEF zone is the original JIS C 6226:1978. [31] [17] The lead byte range is extended back to 0x41, with 0x80–A0 designated for user definition; lead bytes 0x41–7F are assigned row numbers 101 through 163 for kuten purposes, although row 162 (lead byte 0x7E) is unused. [1] [3] Rows 101 through 148 are used for extended kanji, while rows 149 through 163 are used for extended non-kanji. [3]

Footnotes

  1. These hexadecimal forms of these shift sequences match the decimal forms (10 65 and 10 66) listed by Lunde. [1] Lunde lists the hexadecimal forms for both shifts as 0xA0 0x42, seemingly in error.

Related Research Articles

Big-5 or Big5 is a Chinese character encoding method used in Taiwan, Hong Kong, and Macau for traditional Chinese characters.

In computing, a code page is a character encoding and as such it is a specific association of a set of printable characters and control characters with unique numbers. Typically each number represents the binary value in a single byte.

Shift JIS is a character encoding for the Japanese language, originally developed by a Japanese company called ASCII Corporation in conjunction with Microsoft and standardized as JIS X 0208 Appendix 1.

Extended Unix Code (EUC) is a multibyte character encoding system used primarily for Japanese, Korean, and simplified Chinese (characters).

A CCSID is a 16-bit number that represents a particular encoding of a specific code page. For example, Unicode is a code page that has several character encoding schemes —including UTF-8, UTF-16 and UTF-32—but which may or may not actually be accompanied by a CCSID number to indicate that this encoding is being used.

<span class="mw-page-title-main">Code page 950</span> Windows code page for Traditional Chinese, based on Big5

Code page 950 is the code page used on Microsoft Windows for Traditional Chinese. It is Microsoft's implementation of the de facto standard Big5 character encoding. The code page is not registered with IANA, and hence, it is not a standard to communicate information over the internet, although it is usually labelled simply as big5, including by Microsoft library functions.

IBM code page 932 is one of IBM's extensions of Shift JIS. The coded character sets are JIS X 0201:1976, JIS X 0208:1983, IBM extensions and IBM extensions for IBM 1880 UDC. It is the combination of the single-byte Code page 897 and the double-byte Code page 301. Code page 301 is designed to encode the same repertoire as IBM Japanese DBCS-Host.

<span class="mw-page-title-main">JIS X 0201</span> Japanese single byte character encoding

JIS X 0201, a Japanese Industrial Standard developed in 1969, was the first Japanese electronic character set to become widely used. The character set was initially known as JIS C 6220 before the JIS category reform. Its two forms were a 7-bit encoding or an 8-bit encoding, although the 8-bit form was dominant until Unicode replaced it. The full name of this standard is 7-bit and 8-bit coded character sets for information interchange (7ビット及び8ビットの情報交換用符号化文字集合).

Half-width kana are katakana characters displayed compressed at half their normal width, instead of the usual square (1:1) aspect ratio. For example, the usual (full-width) form of the katakana ka is カ while the half-width form is カ. Half-width hiragana is not included in Unicode, although it is usable on Web or in e-books via CSS's font-feature-settings: "hwid" 1 with Adobe-Japan1-6 based OpenType fonts. Half-width kanji is not usable on modern computers, but is used in some receipt printers, electric bulletin board and old computers.

<span class="mw-page-title-main">Halfwidth and fullwidth forms</span> Alternative width characters in East Asian typography

In CJK computing, graphic characters are traditionally classed into fullwidth and halfwidth characters. Unlike monospaced fonts, a halfwidth character occupies half the width of a fullwidth character, hence the name.

JIS X 0212 is a Japanese Industrial Standard defining a coded character set for encoding supplementary characters for use in Japanese. This standard is intended to supplement JIS X 0208. It is numbered 953 or 5049 as an IBM code page.

JIS X 0208 is a 2-byte character set specified as a Japanese Industrial Standard, containing 6879 graphic characters suitable for writing text, place names, personal names, and so forth in the Japanese language. The official title of the current standard is 7-bit and 8-bit double byte coded KANJI sets for information interchange. It was originally established as JIS C 6226 in 1978, and has been revised in 1983, 1990, and 1997. It is also called Code page 952 by IBM. The 1978 version is also called Code page 955 by IBM.

KS X 1001, "Code for Information Interchange ", formerly called KS C 5601, is a South Korean coded character set standard to represent hangul and hanja characters on a computer.

Code page 895 is a 7-bit character set and is Japan's national ISO 646 variant. It is the Roman set of the JIS X 0201 Japanese Standard and is variously called Japan 7-Bit Latin, JISCII, JIS Roman, JIS C6220-1969-ro, ISO646-JP or Japanese-Roman. Its ISO-IR registration number is 14.

Microsoft Windows code page 932, also called Windows-31J amongst other names, is the Microsoft Windows code page for the Japanese language, which is an extended variant of the Shift JIS Japanese character encoding. It contains standard 7-bit ASCII codes, and Japanese characters are indicated by the high bit of the first byte being set to 1. Some code points in this page require a second byte, so characters use either 8 or 16 bits for encoding.

Code page 942 is one of IBM's extensions of Shift JIS. The coded character sets are JIS X 0201, JIS X 0208, IBM extensions for IBM 1880 UDC and IBM extensions. It is the combination of the single-byte Code page 1041 and the double-byte Code page 301.

<span class="mw-page-title-main">Code page 949 (IBM)</span>

IBM code page 949 (IBM-949) is a character encoding which has been used by IBM to represent Korean language text on computers. It is a variable-width encoding which represents the characters from the Wansung code defined by the South Korean standard KS X 1001 in a format compatible with EUC-KR, but adds IBM extensions for additional hanja, additional precomposed Hangul syllables, and user-defined characters.

IBM code page 936 is a character encoding for Simplified Chinese including 1880 user-defined characters (UDC), which was superseded in 1993. It is a combination of the single-byte Code page 903 and the double-byte Code page 928. Code page 946 uses the same double-byte component, but an extended single-byte component.

Code page 897 is IBM's implementation of the 8-bit form of JIS X 0201. It includes several additional graphical characters in the C0 control characters area, and the code points in question may be used as control characters or graphical characters depending on the context, similarly in concept to OEM-US, but with different graphical characters. The C0 rows are shown below.

Code page 896, called Japan 7-Bit Katakana Extended, is IBM's code page for code-set G2 of EUC-JP, a 7-bit code page representing the Kana set of JIS X 0201 and accompanying Code page 895 which corresponds to the lower half of that standard. It encodes half-width katakana.

References

  1. 1 2 3 4 5 6 7 8 9 10 Lunde, Ken (2009). "Appendix F: Vendor Encoding Methods" (PDF). CJKV Information Processing: Chinese, Japanese, Korean & Vietnamese Computing (2nd ed.). Sebastopol, CA: O'Reilly. ISBN   978-0-596-51447-1.
  2. 1 2 "Invariant character set". IBM i 7.1 Documentation. IBM. 14 August 2018.
  3. 1 2 3 4 5 6 7 Lunde, Ken (2009). "Appendix E: Vendor Character Set Standards" (PDF). CJKV Information Processing: Chinese, Japanese, Korean & Vietnamese Computing (2nd ed.). Sebastopol, CA: O'Reilly. ISBN   978-0-596-51447-1.
  4. "EBCDIK: Extended Binary Coded Decimal Interchange Kana Code". 通信用語の基礎知識.
  5. 1 2 "Code Page Identifiers". Windows Dev Center. Microsoft.
  6. Steele, Shawn. "Code Page 21027 "Extended/Ext Alpha Lowercase"". Microsoft.
  7. "CCSID 8482". Coded character set identifiers. IBM. Archived from the original on 2014-11-29.
  8. "CCSID 5123". Coded character set identifiers. IBM. Archived from the original on 2014-11-29.
  9. 1 2 3 4 "ibm-930_P120-1999". International Components for Unicode . Unicode Consortium.
  10. "CCSID 930". Coded character set identifiers. IBM. Archived from the original on 2014-12-01.
  11. 1 2 3 4 "ibm-1390_P110-2003". International Components for Unicode . Unicode Consortium.
  12. "CCSID 1390". Coded character set identifiers. IBM. Archived from the original on 2014-11-29.
  13. 1 2 3 4 "ibm-939_P120-1999". International Components for Unicode . Unicode Consortium.
  14. "CCSID 939". Coded character set identifiers. IBM. Archived from the original on 2014-12-01.
  15. 1 2 3 4 "ibm-1399_P110-2003". International Components for Unicode . Unicode Consortium.
  16. "CCSID 1399". Coded character set identifiers. IBM. Archived from the original on 2014-11-29.
  17. 1 2 3 4 5 Izuno, Hidekatsu (20 October 2021). "jef4j: JEF charset support for Java". GitHub .
  18. 1 2 "EBCDIC/EBCDIKのコード表". Hitachi. Archived from the original on 2021-06-03. Retrieved 2021-05-29.
  19. "JIS/EBCDIK Conversion Table". FCOPY Reference Manual. Hewlett-Packard. Note: table in source maps all input bytes, sometimes to codes not actually defined in JIS X 0201, but is asymmetric and does not always round-trip. Characters shown here are those where the tables in the source round-trip map characters to codes actually defined in JIS X 0201.
  20. "Code Page 00298" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  21. "Code Page 01027" (PDF). REGISTRY: Graphic Character Sets and Code Pages. IBM. 1999.
  22. "Code Page 01027" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  23. "Code Page 01031" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  24. "Code Page 01136" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  25. "EBCDICカナ文字からの変換". コード変換 マニュアル - はじめに (HULFT8) (in Japanese). Saison Information Systems.
  26. Koizumi, Moriyoshi. "EBCDIC-kana".
  27. "Code Page 00290" (PDF). REGISTRY: Graphic Character Sets and Code Pages. IBM. 1999.
  28. "Code Page 00290" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  29. "Code Page 00887" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  30. "Code Page 01030" (PDF). IBM. Archived from the original (PDF) on 2015-07-08.
  31. 1 2 3 4 5 "付録K.3 文字コード変換". Hitachi.
  32. "2 Codesets and Codeset Conversion". DIGITAL UNIX Technical Reference for Using Japanese Features. Compaq.
  33. 1 2 3 4 "IBM Japanese Graphic Character Set, Kanji" (PDF). IBM. 1999. IBM Corporate Specification C-H 3-3220-024.
  34. "ibm-1364_P110-2007". International Components for Unicode . Unicode Consortium.
  35. "CCSID 16684". Coded character set identifiers. IBM. Archived from the original on 2014-11-29.