Unicode in Microsoft Windows

Last updated

Microsoft was one of the first companies to implement Unicode in their products. Windows NT was the first operating system that used "wide characters" in system calls. Using the (now obsolete) UCS-2 encoding scheme at first, it was upgraded to the variable-width encoding UTF-16 starting with Windows 2000, allowing a representation of additional planes with surrogate pairs. However Microsoft did not support UTF-8 in its API until May 2019.

Contents

Before 2019, Microsoft emphasized UTF-16 (i.e. -W API), but has since recommended to use UTF-8 (at least in some cases), [1] on Windows and Xbox (and in other of its products), even states "UTF-8 is the universal code page for internationalization [and] UTF-16 [..] a unique burden that Windows places on code that targets multiple platforms. [..] Windows [is] moving forward to support UTF-8 to remove this unique burden [resulting] in fewer internationalization issues in apps and games". [2]

A large amount of Microsoft documentation uses the word "Unicode" to refer explicitly to the UTF-16 encoding. Anything else, including UTF-8, is not "Unicode" in Microsoft's outdated language (while UTF-8 and UTF-16 are both Unicode according to the Unicode Standard, or encodings/"transformation formats" thereof).

In various Windows families

Windows NT based systems

Current Windows versions and all back to Windows XP and prior Windows NT (3.x, 4.0) are shipped with system libraries that support string encoding of two types: 16-bit "Unicode" (UTF-16 since Windows 2000) and a (sometimes multibyte) encoding called the "code page" (or incorrectly referred to as ANSI code page). 16-bit functions have names suffixed with 'W' (from "wide") such as SetWindowTextW. Code page oriented functions use the suffix 'A' for "ANSI" such as SetWindowTextA (some other conventions were used for APIs that were copied from other systems, such as _wfopen/fopen or wcslen/strlen). This split was necessary because many languages, including C, did not provide a clean way to pass both 8-bit and 16-bit strings to the same function.

Microsoft attempted to support Unicode "portably" by providing a "UNICODE" switch to the compiler, that switches unsuffixed "generic" calls from the 'A' to the 'W' interface and converts all string constants to "wide" UTF-16 versions. [3] [4] This does not actually work because it does not translate UTF-8 outside of string constants, resulting in code that attempts to open files just not compiling.[ citation needed ]

Earlier, and independent of the "UNICODE" switch, Windows also provided the Multibyte Character Sets (MBCS) API switch. [5] This changes some functions that don't work in MBCS such as strrev to an MBCS-aware one such as _mbsrev. [6] [7]

Windows CE

In (the now discontinued) Windows CE, UTF-16 was used almost exclusively, with the 'A' API mostly missing. [8] A limited set of ANSI API is available in Windows CE 5.0, for use on a reduced set of locales that may be selectively built onto the runtime image. [9]

Windows 9x

In 2001, Microsoft released a special supplement to Microsoft's old Windows 9x systems. It includes a dynamic link library, 'unicows.dll', (only 240 KB) containing the 16-bit flavor (the ones with the letter W on the end) of all the basic functions of Windows API. It is merely a translation layer: SetWindowTextW will simply convert its input using the current codepage and call SetWindowTextA.

UTF-8

Microsoft Windows (Windows XP and later) has a code page designated for UTF-8, code page 65001 [10] or CP_UTF8. For a long time, it was impossible to set the locale code page to 65001, leaving this code page only available for (a) explicit conversion functions such as MultiByteToWideChar and/or (b) the Win32 console command chcp 65001 to translate stdin/out between UTF-8 and UTF-16. This meant that "narrow" functions, in particular fopen (which opens files), couldn't be called with UTF-8 strings, and in fact there was no way to open all possible files using fopen no matter what the locale was set to and/or what bytes were put in the string, as none of the available locales could produce all possible UTF-16 characters. This problem also applied to all other APIs that take or return 8-bit strings, including Windows ones such as SetWindowText.

Programs that wanted to use UTF-8, in particular code intended to be portable to other operating systems, needed a workaround for this deficiency. The usual work-around was to add new functions to open files that convert UTF-8 to UTF-16 using MultiByteToWideChar and call the "wide" function instead of fopen. [11] Dozens of multi-platform libraries added wrapper functions to do this conversion on Windows (and pass UTF-8 through unchanged on others), an example is a proposed addition to Boost, Boost.Nowide. [12] Another popular work-around was to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library. None of these workarounds are considered good, as they require changes to the code that works on non-Windows.

In April 2018 (or possibly November 2017 [13] ), with insider build 17035 (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8. [lower-alpha 1] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings. However this is a system-wide setting and a program cannot assume it is set.

In May 2019, Microsoft added the ability for a program to set the code page to UTF-8 itself, [1] [14] allowing programs written to use UTF-8 to be run by non-expert users.

As of 2019, Microsoft recommends programmers use UTF-8 (e.g. instead of any other 8-bit encoding), [1] on Windows and Xbox, and may be recommending its use instead of UTF-16, even stating "UTF-8 is the universal code page for internationalization [and] UTF-16 [..] is a unique burden that Windows places on code that targets multiple platforms." [2] Microsoft does appear to be transitioning to UTF-8, stating it previously emphasized its alternative, and in Windows 11 some system files are required to use UTF-8 and do not require a Byte Order Mark. [15] Notepad can now recognize UTF-8 without the Byte Order Mark, and can be told to write UTF-8 without a Byte Order Mark.[ citation needed ] Some other Microsoft products are using UTF-8 internally, including Visual Studio[ citation needed ] and their SQL Server 2019, with Microsoft claiming 35% speed increase from use of UTF-8, and "nearly 50% reduction in storage requirements." [16]

Programming platforms

Microsoft's compilers often fail at producing UTF-8 string constants from UTF-8 source files. The most reliable method is to turn off UNICODE, not mark the input file as being UTF-8 (i.e. do not use a BOM), and arrange the string constants to have the UTF-8 bytes. If a BOM was added, a Microsoft compiler will interpret the strings as UTF-8, convert them to UTF-16, then convert them back into the current locale, thus destroying the UTF-8. [17] Without a BOM and using a single-byte locale, Microsoft compilers will leave the bytes in a quoted string unchanged. On modern systems setting the code page to UTF-8 helps considerably, but invalid byte sequences are still not preserved (using \x can work around this).

See also

Notes

  1. ↑ Found under control panel, "Region" entry, "Administrative" tab, "Change system locale" button.

Related Research Articles

<span class="mw-page-title-main">Character encoding</span> Using numbers to represent text characters

Character encoding is the process of assigning numbers to graphical characters, especially the written characters of human language, allowing them to be stored, transmitted, and transformed using digital computers. The numerical values that make up a character encoding are known as "code points" and collectively comprise a "code space", a "code page", or a "character map".

UTF-8 is a variable-length character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from Unicode Transformation Format – 8-bit.

<span class="mw-page-title-main">UTF-16</span> Variable-width encoding of Unicode, using one or two 16-bit code units

UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode (in fact this number of code points is dictated by the design of UTF-16). The encoding is variable-length, as code points are encoded with one or two 16-bit code units. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as UCS-2 (for 2-byte Universal Character Set), once it became clear that more than 216 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names.

<span class="mw-page-title-main">Character (computing)</span> Primitive data type

In computer and machine-based telecommunications terminology, a character is a unit of information that roughly corresponds to a grapheme, grapheme-like unit, or symbol, such as in an alphabet or syllabary in the written form of a natural language.

The byte order mark (BOM) is a particular usage of the special Unicode character, U+FEFFZERO WIDTH NO-BREAK SPACE, whose appearance as a magic number at the start of a text stream can signal several things to a program reading the text:

UTF-32 (32-bit Unicode Transformation Format) is a fixed-length encoding used to encode Unicode code points that uses exactly 32 bits (four bytes) per code point (but a number of leading bits must be zero as there are far fewer than 232 Unicode code points, needing actually only 21 bits). UTF-32 is a fixed-length encoding, in contrast to all other Unicode transformation formats, which are variable-length encodings. Each 32-bit value in UTF-32 represents one Unicode code point and is exactly equal to that code point's numerical value.

A text file is a kind of computer file that is structured as a sequence of lines of electronic text. A text file exists stored as data within a computer file system. In operating systems such as CP/M and DOS, where the operating system does not keep track of the file size in bytes, the end of a text file is denoted by placing one or more special characters, known as an end-of-file (EOF) marker, as padding after the last line in a text file. On modern operating systems such as Microsoft Windows and Unix-like systems, text files do not contain any special EOF character, because file systems on those operating systems keep track of the file size in bytes. Most text files need to have end-of-line delimiters, which are done in a few different ways depending on operating system. Some operating systems with record-orientated file systems may not use new line delimiters and will primarily store text files with lines separated as fixed or variable length records.

<span class="mw-page-title-main">Windows-1252</span> Windows character set for Latin alphabet

Windows-1252 or CP-1252 is a single-byte character encoding of the Latin alphabet that was used by default in Microsoft Windows for English and many Romance and Germanic languages including Spanish, Portuguese, French, and German. This character-encoding scheme is used throughout the Americas, Western Europe, Oceania, and much of Africa.

In computing, a locale is a set of parameters that defines the user's language, region and any special variant preferences that the user wants to see in their user interface. Usually a locale identifier consists of at least a language code and a country/region code. Locale is an important aspect of i18n.

A double-byte character set (DBCS) is a character encoding in which either all characters are encoded in two bytes, or merely every graphic character not representable by an accompanying single-byte character set (SBCS) is encoded in two bytes. A DBCS supports national languages that contain many unique characters or symbols. Examples of such languages include Japanese and Chinese. Korean Hangul does not contain as many characters, but KS X 1001 supports both Hangul and Hanja, and uses two bytes per character.

<span class="mw-page-title-main">GB 18030</span> Unicode character encoding mostly used for Simplified Chinese

GB 18030 is a Chinese government standard, described as Information Technology — Chinese coded character set and defines the required language and character support necessary for software in China. GB18030 is the registered Internet name for the official character set of the People's Republic of China (PRC) superseding GB2312. As a Unicode Transformation Format, GB18030 supports both simplified and traditional Chinese characters. It is also compatible with legacy encodings including GB2312, CP936, and GBK 1.0.

<span class="mw-page-title-main">Windows Console</span> Infrastructure for console applications in Microsoft Windows

Windows Console is the infrastructure for console applications in Microsoft Windows. An instance of a Windows Console has a screen buffer and an input buffer. It allows console apps to run inside a window or in hardware text mode. The user can switch between the two using the Alt+↵ Enter key combination. The text mode is unavailable in Windows Vista and later. Starting with Windows 10, however, a native full-screen mode is available.

A wide character is a computer character datatype that generally has a size greater than the traditional 8-bit character. The increased datatype size allows for the use of larger coded character sets.

A variable-width encoding is a type of character encoding scheme in which codes of differing lengths are used to encode a character set for representation, usually in a computer. Most common variable-width encodings are multibyte encodings, which use varying numbers of bytes (octets) to encode different characters. (Some authors, notably in Microsoft documentation, use the term multibyte character set, which is a misnomer, because representation size is an attribute of the encoding, not of the character set.)

This article compares Unicode encodings. Two situations are considered: 8-bit-clean environments, and environments that forbid use of byte values that have the high bit set. Originally such prohibitions were to allow for links that used only seven data bits, but they remain in some standards and so some standard-conforming software must generate messages that comply with the restrictions. Standard Compression Scheme for Unicode and Binary Ordered Compression for Unicode are excluded from the comparison tables because it is difficult to simply quantify their size.

Windows code pages are sets of characters or code pages used in Microsoft Windows from the 1980s and 1990s. Windows code pages were gradually superseded when Unicode was implemented in Windows, although they are still supported both within Windows and other platforms, and still apply when Alt code shortcuts are used.

The C++ programming language has support for string handling, mostly implemented in its standard library. The language standard specifies several string types, some inherited from C, some designed to make use of the language's features, such as classes and RAII. The most-used of these is std::string.

<span class="mw-page-title-main">Bush hid the facts</span> Bug in Microsoft Windows

Bush hid the facts is a common name for a bug present in some versions of Microsoft Windows, which causes text encoded in ASCII to be interpreted as if it were UTF-16LE, resulting in garbled text. When the string "Bush hid the facts", without quotes, was put in a new Notepad document and saved, closed, and reopened, the nonsensical sequence of the Chinese characters "畂桳栠摩琠敨映捡獴" would appear instead.

The C programming language has a set of functions implementing operations on strings in its standard library. Various operations, such as copying, concatenation, tokenization and searching are supported. For character strings, the standard library uses the convention that strings are null-terminated: a string of n characters is represented as an array of n + 1 elements, the last of which is a "NUL character" with numeric value 0.

Microsoft Windows code page 932, also called Windows-31J amongst other names, is the Microsoft Windows code page for the Japanese language, which is an extended variant of the Shift JIS Japanese character encoding. It contains standard 7-bit ASCII codes, and Japanese characters are indicated by the high bit of the first byte being set to 1. Some code points in this page require a second byte, so characters use either 8 or 16 bits for encoding.

References

  1. 1 2 3 "Use UTF-8 code pages in Windows apps". learn.microsoft.com. Retrieved 2020-06-06. As of Windows version 1903 (May 2019 update), you can use the ActiveCodePage property in the appxmanifest for packaged apps, or the fusion manifest for unpackaged apps, to force a process to use UTF-8 as the process code page. [...] CP_ACP equates to CP_UTF8 only if running on Windows version 1903 (May 2019 update) or above and the ActiveCodePage property described above is set to UTF-8. Otherwise, it honors the legacy system code page. We recommend using CP_UTF8 explicitly.
  2. 1 2 "UTF-8 support in the Microsoft Game Development Kit (GDK) - Microsoft Game Development Kit". learn.microsoft.com. 19 August 2022. Retrieved 2023-03-05. By operating in UTF-8, you can ensure maximum compatibility [..] Windows operates natively in UTF-16 (or WCHAR), which requires code page conversions by using MultiByteToWideChar and WideCharToMultiByte. This is a unique burden that Windows places on code that targets multiple platforms. [..] The Microsoft Game Development Kit (GDK) and Windows in general are moving forward to support UTF-8 to remove this unique burden of Windows on code targeting or interchanging with multiple platforms and the web. Also, this results in fewer internationalization issues in apps and games and reduces the test matrix that's required to get it right.
  3. ↑ "Unicode in the Windows API" . Retrieved 7 May 2018.
  4. ↑ "Conventions for Function Prototypes (Windows)". MSDN. Retrieved 7 May 2018.
  5. ↑ "Support for Multibyte Character Sets (MBCSs)" . Retrieved 2020-06-15.
  6. ↑ "Double-byte Character Sets". MSDN. 2018-05-31. Retrieved 2020-06-15. our applications use DBCS Windows code pages with the "A" versions of Windows functions.
  7. ↑ _strrev, _wcsrev, _mbsrev, _mbsrev_l Microsoft Docs
  8. ↑ "Differences Between the Windows CE and Windows NT Implementations of TAPI". MSDN. 28 August 2006. Retrieved 7 May 2018. Windows CE is Unicode-based. You might have to recompile source code that was written for a Windows NT-based application.
  9. ↑ "Code Pages (Windows CE 5.0)". Microsoft Docs. 14 September 2012. Retrieved 7 May 2018.
  10. ↑ "Code Page Identifiers (Windows)". msdn.microsoft.com. 7 January 2021.
  11. ↑ "UTF-8 in Windows". Stack Overflow . Retrieved July 1, 2011.
  12. ↑ "Boost.Nowide". GitHub .
  13. ↑ "Windows10 Insider Preview Build 17035 Supports UTF-8 as ANSI". Hacker News. Retrieved 7 May 2018.
  14. ↑ "Windows 10 1903 and later versions finally support UTF-8 with the A forms of the Win32 functions".
  15. ↑ "Customize the Windows 11 Start menu". docs.microsoft.com. Retrieved 2021-06-29. Make sure your LayoutModification.json uses UTF-8 encoding.
  16. ↑ "Introducing UTF-8 support for SQL Server". techcommunity.microsoft.com. 2019-07-02. Retrieved 2021-08-24. For example, changing an existing column data type from NCHAR(10) to CHAR(10) using an UTF-8 enabled collation, translates into nearly 50% reduction in storage requirements. [..] In the ASCII range, when doing intensive read/write I/O on UTF-8, we measured an average 35% performance improvement over UTF-16 using clustered tables with a non-clustered index on the string column, and an average 11% performance improvement over UTF-16 using a heap.
  17. ↑ UTF-8 Everywhere FAQ: How do I write UTF-8 string literal in my C++ code?
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.