Bidirectional text

Last updated

A bidirectional text contains two text directionalities, right-to-left (RTL) and left-to-right (LTR). It generally involves text containing different types of alphabets, but may also refer to boustrophedon, which is changing text direction in each row.

Contents

An example is the RTL Hebrew name Sarah: שרה, spelled sin (ש) on the right, resh (ר) in the middle, and heh (ה) on the left. Many computer program failed to display this correctly, because they were designed to display text in one direction only.

Some so-called right-to-left scripts such as the Persian script and Arabic are mostly, but not exclusively, right-to-left—mathematical expressions, numeric dates and numbers bearing units are embedded from left to right. That also happens if text from a left-to-right language such as English is embedded in them; or vice versa, if Arabic is embedded in a left-to-right script such as English.

Bidirectional script support

Bidirectional script support is the capability of a computer system to correctly display bidirectional text. The term is often shortened to "BiDi" or "bidi".

Early computer installations were designed only to support a single writing system, typically for left-to-right scripts based on the Latin alphabet only. Adding new character sets and character encodings enabled a number of other left-to-right scripts to be supported, but did not easily support right-to-left scripts such as Arabic or Hebrew, and mixing the two was not practical. Right-to-left scripts were introduced through encodings like ISO/IEC 8859-6 and ISO/IEC 8859-8, storing the letters (usually) in writing and reading order. It is possible to simply flip the left-to-right display order to a right-to-left display order, but doing this sacrifices the ability to correctly display left-to-right scripts. With bidirectional script support, it is possible to mix characters from different scripts on the same page, regardless of writing direction.

In particular, the Unicode standard provides foundations for complete BiDi support, with detailed rules as to how mixtures of left-to-right and right-to-left scripts are to be encoded and displayed.

Unicode bidi support

The Unicode standard calls for characters to be ordered 'logically', i.e. in the sequence they are intended to be interpreted, as opposed to 'visually', the sequence they appear. This distinction is relevant for bidi support because at any bidi transition, the visual presentation ceases to be the 'logical' one. Thus, in order to offer bidi support, Unicode prescribes an algorithm for how to convert the logical sequence of characters into the correct visual presentation. For this purpose, the Unicode encoding standard divides all its characters into one of four types: 'strong', 'weak', 'neutral', and 'explicit formatting'. [1]

Strong characters

Strong characters are those with a definite direction. Examples of this type of character include most alphabetic characters, syllabic characters, Han ideographs, non-European or non-Arabic digits, and punctuation characters that are specific to only those scripts.

Weak characters

Weak characters are those with vague direction. Examples of this type of character include European digits, Eastern Arabic-Indic digits, arithmetic symbols, and currency symbols.

Neutral characters

Neutral characters have direction indeterminable without context. Examples include paragraph separators, tabs, and most other whitespace characters. Punctuation symbols that are common to many scripts, such as the colon, comma, full-stop, and the no-break-space also fall within this category.

Explicit formatting

Explicit formatting characters, also referred to as "directional formatting characters", are special Unicode sequences that direct the algorithm to modify its default behavior. These characters are subdivided into "marks", "embeddings", "isolates", and "overrides". Their effects continue until the occurrence of either a paragraph separator, or a "pop" character.

Marks

If a "weak" character is followed by another "weak" character, the algorithm will look at the first neighbouring "strong" character. Sometimes this leads to unintentional display errors. These errors are corrected or prevented with "pseudo-strong" characters. Such Unicode control characters are called marks. The mark (U+200ELEFT-TO-RIGHT MARK (LRM) or U+200FRIGHT-TO-LEFT MARK (RLM)) is to be inserted into a location to make an enclosed weak character inherit its writing direction.

For example, to correctly display the U+2122TRADE MARK SIGN for an English name brand (LTR) in an Arabic (RTL) passage, an LRM mark is inserted after the trademark symbol if the symbol is not followed by LTR text (e.g. "قرأ Wikipedia™ طوال اليوم."). If the LRM mark is not added, the weak character ™ will be neighbored by a strong LTR character and a strong RTL character. Hence, in an RTL context, it will be considered to be RTL, and displayed in an incorrect order (e.g. "قرأ Wikipedia™ طوال اليوم.").

Embeddings

The "embedding" directional formatting characters are the classical Unicode method of explicit formatting, and as of Unicode 6.3, are being discouraged in favor of "isolates". An "embedding" signals that a piece of text is to be treated as directionally distinct. The text within the scope of the embedding formatting characters is not independent of the surrounding text. Also, characters within an embedding can affect the ordering of characters outside. Unicode 6.3 recognized that directional embeddings usually have too strong an effect on their surroundings and are thus unnecessarily difficult to use.

Isolates

The "isolate" directional formatting characters signal that a piece of text is to be treated as directionally isolated from its surroundings. As of Unicode 6.3, these are the formatting characters that are being encouraged in new documents – once target platforms are known to support them. These formatting characters were introduced after it became apparent that directional embeddings usually have too strong an effect on their surroundings and are thus unnecessarily difficult to use. Unlike the legacy 'embedding' directional formatting characters, 'isolate' characters have no effect on the ordering of the text outside their scope. Isolates can be nested, and may be placed within embeddings and overrides.

Overrides

The "override" directional formatting characters allow for special cases, such as for part numbers (e.g. to force a part number made of mixed English, digits and Hebrew letters to be written from right to left), and are recommended to be avoided wherever possible. As is true of the other directional formatting characters, "overrides" can be nested one inside another, and in embeddings and isolates.

Pops

The "pop" directional formatting characters terminate the scope of the most recent "embedding", "override", or "isolate".

Runs

In the algorithm, each sequence of concatenated strong characters is called a "run". A "weak" character that is located between two "strong" characters with the same orientation will inherit their orientation. A "weak" character that is located between two "strong" characters with a different writing direction will inherit the main context's writing direction (in an LTR document the character will become LTR, in an RTL document, it will become RTL).

Table of possible BiDi character types

Bidirectional character type(Bidi_Class Unicode character property) [1]
Type [2] DescriptionStrengthDirectionalityGeneral scopeBidi_Control character [3]
LLeft-to-RightStrongL-to-RMost alphabetic and syllabic characters, Chinese characters, non-European or non-Arabic digits, LRM character, ...U+200E LEFT-TO-RIGHT MARK (LRM)
RRight-to-LeftStrongR-to-LAdlam, Hebrew, Mandaic, Mende Kikakui, N'Ko, Samaritan, ancient scripts like Kharoshthi and Nabataean, RLM character, ...U+200F RIGHT-TO-LEFT MARK (RLM)
ALArabic LetterStrongR-to-LArabic, Hanifi Rohingya, Sogdian, Syriac, and Thaana alphabets, and most punctuation specific to those scripts, ALM character, ...U+061C ARABIC LETTER MARK (ALM)
ENEuropean NumberWeakEuropean digits, Eastern Arabic-Indic digits, Coptic epact numbers, ...
ESEuropean SeparatorWeakplus sign, minus sign, ...
ETEuropean Number TerminatorWeakdegree sign, currency symbols, ...
ANArabic NumberWeakArabic-Indic digits, Arabic decimal and thousands separators, Rumi digits, Hanifi Rohingya digits, ...
CSCommon Number SeparatorWeakcolon, comma, full stop, no-break space, ...
NSMNonspacing MarkWeakCharacters in General Categories Mark, nonspacing, and Mark, enclosing (Mn, Me)
BNBoundary NeutralWeakDefault ignorables, non-characters, control characters other than those explicitly given other types
BParagraph SeparatorNeutralparagraph separator, appropriate Newline Functions, higher-level protocol paragraph determination
SSegment SeparatorNeutralTabs
WSWhitespaceNeutralspace, figure space, line separator, form feed, General Punctuation block spaces (smaller set than the Unicode whitespace list)
ONOther NeutralsNeutralAll other characters, including object replacement character
LRELeft-to-Right EmbeddingExplicitL-to-RLRE character onlyU+202A LEFT-TO-RIGHT EMBEDDING (LRE)
LROLeft-to-Right OverrideExplicitL-to-RLRO character onlyU+202D LEFT-TO-RIGHT OVERRIDE (LRO)
RLERight-to-Left EmbeddingExplicitR-to-LRLE character onlyU+202B RIGHT-TO-LEFT EMBEDDING (RLE)
RLORight-to-Left OverrideExplicitR-to-LRLO character onlyU+202E RIGHT-TO-LEFT OVERRIDE (RLO)
PDFPop Directional FormatExplicitPDF character onlyU+202C POP DIRECTIONAL FORMATTING (PDF)
LRILeft-to-Right IsolateExplicitL-to-RLRI character onlyU+2066 LEFT-TO-RIGHT ISOLATE (LRI)
RLIRight-to-Left IsolateExplicitR-to-LRLI character onlyU+2067 RIGHT-TO-LEFT ISOLATE (RLI)
FSIFirst Strong IsolateExplicitFSI character onlyU+2068 FIRST STRONG ISOLATE (FSI)
PDIPop Directional IsolateExplicitPDI character onlyU+2069 POP DIRECTIONAL ISOLATE (PDI)
Notes
1. ^ Unicode Bidirectional Algorithm (UAX#9), As of Unicode version 12.0
2. ^ Possible Bidirectional character types for character property: Bidi_Class or 'type'
3. ^ Bidi_Control characters: Twelve Bidi_Control formatting characters are defined. They are invisible, and have no effect apart from directionality. Nine of them have a unique, overruling BiDi-type that is used by the algorithm. Their type is also their acronym (e.g. character 'LRE' has BiDi type 'LRE').

Security

Unicode bidirectional characters are used in the Trojan Source vulnerability. [2]

Visual Studio Code highlights BiDi control characters since version 1.62 released in October 2021. [3]

Visual Studio highlights BiDi control characters since version 17.0.3 released on December 14, 2021. [4]

Scripts using bidirectional text

Egyptian hieroglyphs

Egyptian hieroglyphs were written bidirectionally, where the signs that had a distinct "head" or "tail" faced the beginning of the line.

Chinese characters and other CJK scripts

Chinese characters can be written in either direction as well as vertically (top to bottom then right to left), especially in signs (such as plaques), but the orientation of the individual characters does not change. This can often be seen on tour buses in China, where the company name customarily runs from the front of the vehicle to its rear — that is, from right to left on the right side of the bus, and from left to right on the left side of the bus. English texts on the right side of the vehicle are also quite commonly written in reverse order. (See pictures of tour bus and post vehicle below.)

Likewise, other CJK scripts made up of the same square characters, such as the Japanese writing system and Korean writing system, can also be written in any direction, although horizontally left-to-right, top-to-bottom and vertically top-to-bottom right-to-left are the two most common forms.

Boustrophedon

Boustrophedon is a writing style found in ancient Greek inscriptions, in Old Sabaic (an Old South Arabian language) and in Hungarian runes. This method of writing alternates direction, and usually reverses the individual characters, on each successive line.

Moon type

Moon type is an embossed adaptation of the Latin alphabet invented as a tactile alphabet for the blind. Initially the text changed direction (but not character orientation) at the end of the lines. Special embossed lines connected the end of a line and the beginning of the next. [5] Around 1990, it changed to a left-to-right orientation.

See also

Related Research Articles

<span class="mw-page-title-main">Plain text</span> Term for computer data consisting only of unformatted characters of readable material

In computing, plain text is a loose term for data that represent only characters of readable material but not its graphical representation nor other objects. It may also include a limited number of "whitespace" characters that affect simple arrangement of text, such as spaces, line breaks, or tabulation characters. Plain text is different from formatted text, where style information is included; from structured text, where structural parts of the document such as paragraphs, sections, and the like are identified; and from binary files in which some portions must be interpreted as binary objects.

ISO/IEC 8859-8, Information technology — 8-bit single-byte coded graphic character sets — Part 8: Latin/Hebrew alphabet, is part of the ISO/IEC 8859 series of ASCII-based standard character encodings. ISO/IEC 8859-8:1999 from 1999 represents its second and current revision, preceded by the first edition ISO/IEC 8859-8:1988 in 1988. It is informally referred to as Latin/Hebrew. ISO/IEC 8859-8 covers all the Hebrew letters, but no Hebrew vowel signs. IBM assigned code page 916 to it. This character set was also adopted by Israeli Standard SI1311:2002, with some extensions.

ISO/IEC 8859-6:1999, Information technology — 8-bit single-byte coded graphic character sets — Part 6: Latin/Arabic alphabet, is part of the ISO/IEC 8859 series of ASCII-based standard character encodings, first edition published in 1987. It is informally referred to as Latin/Arabic. It was designed to cover Arabic. Only nominal letters are encoded, no preshaped forms of the letters, so shaping processing is required for display. It does not include the extra letters needed to write most Arabic-script languages other than Arabic itself.

<span class="mw-page-title-main">Complex text layout</span> Neighbour-dependent grapheme positioning

Complex text layout (CTL) or complex text rendering is the typesetting of writing systems in which the shape or positioning of a grapheme depends on its relation to other graphemes. The term is used in the field of software internationalization, where each grapheme is a character.

This article provides basic comparisons for notable text editors. More feature details for text editors are available from the Category of text editor features and from the individual products' articles. This article may not be up-to-date or necessarily all-inclusive.

<span class="mw-page-title-main">Horizontal and vertical writing in East Asian scripts</span> Writing conventions of eastern Asian countries

Many East Asian scripts can be written horizontally or vertically. Chinese characters, Korean hangul, and Japanese kana may be oriented along either axis, as they consist mainly of disconnected logographic or syllabic units, each occupying a square block of space, thus allowing for flexibility for which direction texts can be written, be it horizontally from left-to-right, horizontally from right-to-left, vertically from top-to-bottom, and even vertically from bottom-to-top.

A whitespace character is a character data element that represents white space when text is rendered for display by a computer.

<span class="mw-page-title-main">Right-to-left script</span> Type of writing system

In a right-to-left, top-to-bottom script, writing starts from the right of the page and continues to the left, proceeding from top to bottom for new lines. Arabic, Hebrew, and Persian are the most widespread RTL writing systems in modern times.

<span class="mw-page-title-main">Hebrew keyboard</span> Keyboard layout

A Hebrew keyboard comes in two different keyboard layouts. Most Hebrew keyboards are bilingual, with Latin characters, usually in a US Qwerty layout. Trilingual keyboard options also exist, with the third script being Arabic or Russian, due to the sizable Arabic- and Russian-speaking populations in Israel.

The left-to-right mark (LRM) is a control character used in computerized typesetting of text containing a mix of left-to-right scripts and right-to-left scripts. It is used to set the way adjacent characters are grouped with respect to text direction.

‏The right-to-left mark (RLM) is a non-printing character used in the computerized typesetting of bi-directional text containing a mix of left-to-right scripts and right-to-left scripts.

<span class="mw-page-title-main">Universal Character Set characters</span> Complete list of the characters available on most computers

The Unicode Consortium and the ISO/IEC JTC 1/SC 2/WG 2 jointly collaborate on the list of the characters in the Universal Coded Character Set. The Universal Coded Character Set, most commonly called the Universal Character Set, is an international standard to map characters, discrete symbols used in natural language, mathematics, music, and other domains, to unique machine-readable data values. By creating this mapping, the UCS enables computer software vendors to interoperate, and transmit—interchange—UCS-encoded text strings from one to another. Because it is a universal map, it can be used to represent multiple languages at the same time. This avoids the confusion of using multiple legacy character encodings, which can result in the same sequence of codes having multiple interpretations depending on the character encoding in use, resulting in mojibake if the wrong one is chosen.

Many Unicode characters are used to control the interpretation or display of text, but these characters themselves have no visual or spatial representation. For example, the null character is used in C-programming application environments to indicate the end of a string of characters. In this way, these programs only require a single starting memory address for a string, since the string ends once the program reads the null character.

<span class="mw-page-title-main">Panorama (typesetting software)</span>

Panorama is a line layout and text composition engine to render text in various worldwide languages made by Bitstream Inc. Panorama uses Font Fusion as the base to support rendering of the text. The engine allows the user to manage different text formatting aspects like spacing, alignment, style effects.

The Universal Coded Character Set is a standard set of characters defined by the international standard ISO/IEC 10646, Information technology — Universal Coded Character Set (UCS), which is the basis of many character encodings, improving as characters from previously unrepresented typing systems are added.

<span class="mw-page-title-main">Writing system</span> Convention of symbols representing language

A writing system comprises a particular set of symbols, called a script, as well as the rules by which the script represents a particular language. Writing systems can generally be classified according to how symbols function according to these rules, with the most common types being alphabets, syllabaries, and logographies. Alphabets use symbols called letters that correspond to spoken phonemes. Abjads generally only have letters for consonants, while pure alphabets have letters for both consonants and vowels. Abugidas use characters that correspond to consonant–vowel pairs. Syllabaries use symbols called syllabograms to represent syllables or moras. Logographies use characters that represent semantic units, such as words or morphemes.

The Unicode Standard assigns various properties to each Unicode character and code point.

General Punctuation is a Unicode block containing punctuation, spacing, and formatting characters for use with all scripts and writing systems. Included are the defined-width spaces, joining formats, directional formats, smart quotes, archaic and novel punctuation such as the interrobang, and invisible mathematical operators.

The Arabic letter mark (ALM) is a non-printing character used in the computerized typesetting of bi-directional text containing mixed left-to-right scripts and right-to-left scripts.

Trojan Source is the name of a software vulnerability that abuses Unicode's bidirectional characters to display source code differently than the actual execution of the source code. The exploit utilizes how writing scripts of different reading directions are displayed and encoded on computers. It was discovered by Nicholas Boucher and Ross Anderson at Cambridge University in late 2021.

References

  1. "UAX #9: Unicode Bi-directional Algorithm". Unicode.org. 2018-05-09. Retrieved 2018-06-26.
  2. "Trojan Source Attacks". trojansource.codes. Retrieved 17 January 2022.
  3. "Visual Studio Code October 2021". code.visualstudio.com. Retrieved 11 November 2021.
  4. "Visual Studio 2022 version 17.0 Release Notes". docs.microsoft.com. Retrieved 17 January 2022.
  5. Moon Type for the Blind , Ramseyer Bible Collection, Kathryn A. Martin Library, University of Minnesota Duluth.