Apertium

Last updated
Apertium
Stable release
3.9.4 [1]   OOjs UI icon edit-ltr-progressive.svg / 28 December 2023;7 months ago (28 December 2023)
Repository github.com/apertium
Written in C++
Operating system POSIX compatible and Windows NT (limited support)
Available in35 languages, see below
Type Rule-based machine translation
License GNU General Public License
Website www.apertium.org

Apertium is a free/open-source rule-based machine translation platform. It is free software and released under the terms of the GNU General Public License.

Contents

Overview

Apertium is a transfer-based machine translation system, which uses finite state transducers for all of its lexical transformations, and Constraint Grammar taggers as well as hidden Markov models or Perceptrons for part-of-speech tagging / word category disambiguation. [2] A structural transfer component is responsible for word movement and agreement; most Apertium language pairs up until now have used "chunking" or shallow transfer rules, though newer pairs use (possibly recursive) rules defined in a Context-free grammar. [3]

Many existing machine translation systems available at present are commercial or use proprietary technologies, which makes them very hard to adapt to new usages. Apertium code and data is free software and uses a language-independent specification, to allow for the ease of contributing to Apertium, more efficient development, and enhancing the project's overall growth.

At present (December 2020), Apertium has released 51 stable language pairs, [4] delivering fast translation with reasonably intelligible results (errors are easily corrected). Being an open-source project, Apertium provides tools for potential developers to build their own language pair and contribute to the project.

History

Apertium originated as one of the machine translation engines in the project OpenTrad, which was funded by the Spanish government, and developed by the Transducens research group at the Universitat d'Alacant. It was originally designed to translate between closely related languages, although it has recently been expanded to treat more divergent language pairs. To create a new machine translation system, one just has to develop linguistic data (dictionaries, rules) in well-specified XML formats.

Language data developed for it (in collaboration with the Universidade de Vigo, the Universitat Politècnica de Catalunya and the Universitat Pompeu Fabra) currently support (in stable version) the Arabic, Aragonese, Asturian, Basque, Belarusian, Breton, Bulgarian, Catalan, Crimean Tatar, Danish, English, Esperanto, French, Galician, Hindi, Icelandic, Indonesian, Italian, Kazakh, Macedonian, Malaysian, Maltese, Northern Sami, Norwegian (Bokmål and Nynorsk), Occitan, Polish, Portuguese, Romanian, Russian, Sardinian, Serbo-Croatian, Silesian, Slovene, Spanish, Swedish, Tatar, Ukrainian, Urdu, and Welsh languages. A full list is available below. Several companies are also involved in the development of Apertium, including Prompsit Language Engineering, Imaxin Software and Eleka Ingeniaritza Linguistikoa.

The project has taken part in the 2009, [5] 2010, [6] 2011, [7] 2012, [8] 2013 [9] and 2014 [10] editions of Google Summer of Code and the 2010, [11] 2011, [12] 2012, [13] 2013, [14] 2014, [15] 2015, [16] 2016 [17] and 2017 [18] editions of Google Code-In.

Translation methodology

Pipeline of Apertium machine translation system Apertium-pipeline.png
Pipeline of Apertium machine translation system

This is an overall, step-by-step view how Apertium works.

The diagram displays the steps that Apertium takes to translate a source-language text (the text we want to translate) into a target-language text (the translated text).

  1. Source language text is passed into Apertium for translation.
  2. The deformatter removes formatting markup (HTML, RTF, etc.) that should be kept in place but not translated.
  3. The morphological analyser segments the text (expanding elisions, marking set phrases, etc.), and looks up segments in the language dictionaries, returning dictionary forms and tags for all matches. In pairs that involve agglutinative morphology, including a number of Turkic languages, a Helsinki Finite State Transducer (HFST) is used. Otherwise, an Apertium-specific finite state transducer system called lttoolbox, [19] is used.
  4. The morphological disambiguator (the morphological analyser and the morphological disambiguator together form the part of speech tagger ) resolves ambiguous segments (i.e., when there is more than one match) by choosing one match. Apertium uses Constraint Grammar rules (with the vislcg3 parser [20] ) for most of its language pairs.
  5. Retokenisation uses a finite state transducer to match sequences of lexical units and may reorder or translate tags (often used for translating idiomatic expressions into something that more approaches the target language grammar)
  6. Lexical transfer looks up disambiguated source-language basewords to find their target-language equivalents (i.e., mapping source language to target language). For lexical transfer, Apertium uses an XML-based dictionary format called bidix. [21]
  7. Lexical selection chooses between alternative translations when the source text word has alternative meanings. Apertium uses a specific XML-based technology, apertium-lex-tools, [22] to perform lexical selection.
  8. Structural transfer (i.e., it is an XML format that allows writing complex structural transfer rules) can consist of one-step chunking transfer, three-step chunking transfer or a CFG-based transfer module. The chunking modules flag grammatical differences between the source language and target language (e.g. gender or number agreement) by creating a sequence of chunks containing markers for this. They then reorder or modify chunks in order to produce a grammatical translation in the target-language. The newer CFG-based module matches input sequences into possible parse trees, selecting the best-ranking one and applying transformation rules on the tree.
  9. The morphological generator uses the tags to deliver the correct target language surface form. The morphological generator is a morphological transducer, [23] just like the morphological analyser. A morphological transducer both analyses and generates forms.
  10. The post-generator makes any necessary orthographic changes due to the contact of words (e.g. elisions).
  11. The reformatter replaces formatting markup (HTML, RTF, etc.) that was removed by the deformatter in the first step.
  12. Apertium delivers the target-language translation.

Supported languages

As of August2024, the following 108 pairs and 50 languages and languages varieties are supported by Apertium.

  1. Afrikaans to Dutch
  2. Arabic to Maltese
  3. Aragonese to Catalan
  4. Aragonese to Spanish
  5. Arpitan (Franco-Provençal) to French
  6. Basque to English
  7. Basque to Spanish
  8. Belarusian to Russian
  9. Breton to French
  10. Bulgarian to Macedonian
  11. Catalan to Aragonese
  12. Catalan to English
  13. Catalan to Esperanto
  14. Catalan to French
  15. Catalan to Italian
  16. Catalan to Occitan
  17. Catalan to Aranese
  18. Catalan to Portuguese
  19. Catalan to Brazilian Portuguese
  20. Catalan to European Portuguese (traditional spelling)
  21. Catalan to Romanian
  22. Catalan to Sardinian
  23. Catalan to Spanish
  24. Crimean Tatar to Turkish
  25. Danish to Norwegian (Bokmål)
  26. Danish to Norwegian (Nynorsk)
  27. Danish to Swedish
  28. Dutch to Afrikaans
  29. English to Catalan
  30. English to Valencian
  31. English to Esperanto
  32. English to Galician
  33. English to Serbo-Croatian
  34. English to Spanish
  35. Esperanto to English
  36. French to Arpitan (Franco-Provençal)
  37. French to Catalan
  38. French to Esperanto
  39. French to Occitan
  40. French to Gascon
  41. French to Spanish
  42. Galician to English
  43. Galician to Portuguese
  44. Galician to Spanish
  45. Hindi to Urdu
  46. Icelandic to English
  47. Icelandic to Swedish
  48. Indonesian to Malay
  49. Italian to Catalan
  50. Italian to Sardinian
  51. Italian to Spanish
  52. Kazakh to Tatar
  53. Macedonian to Bulgarian
  54. Macedonian to English
  55. Malay to Indonesian
  56. Maltese to Arabic
  57. Northern Sámi to Norwegian (Bokmål)
  58. Norwegian (Bokmål) to Danish
  59. Norwegian (Bokmål) to Norwegian (Nynorsk)
  60. Norwegian (Bokmål) to East Norwegian, vi→vi
  61. Norwegian (Bokmål) to Swedish
  62. Norwegian (Nynorsk) to Danish
  63. Norwegian (Nynorsk) to Norwegian (Bokmål)
  64. Norwegian (Nynorsk) to East Norwegian, vi→vi
  65. Norwegian (Nynorsk) to Swedish
  66. East Norwegian, vi→vi to Norwegian (Nynorsk)
  67. Occitan to Catalan
  68. Occitan to French
  69. Occitan to Spanish
  70. Aranese to Catalan
  71. Aranese to Spanish
  72. Gascon to French
  73. Polish to Silesian
  74. Portuguese to Catalan
  75. Portuguese to Galician
  76. Portuguese to Spanish
  77. Romanian to Catalan
  78. Romanian to Spanish
  79. Russian to Belarusian
  80. Russian to Ukrainian
  81. Sardinian to Italian
  82. Serbo-Croatian to English
  83. Serbo-Croatian to Macedonian
  84. Serbo-Croatian to Slovenian
  85. Silesian to Polish
  86. Slovenian to Serbo-Croatian
  87. Spanish to Aragonese
  88. Spanish to Asturian
  89. Spanish to Catalan
  90. Spanish to Valencian
  91. Spanish to English
  92. Spanish to Esperanto
  93. Spanish to French
  94. Spanish to Galician
  95. Spanish to Italian
  96. Spanish to Occitan
  97. Spanish to Aranese
  98. Spanish to Portuguese
  99. Spanish to Brazilian Portuguese
  100. Swedish to Danish
  101. Swedish to Icelandic
  102. Swedish to Norwegian (Bokmål)
  103. Swedish to Norwegian (Nynorsk)
  104. Tatar to Kazakh
  105. Turkish to Crimean Tatar
  106. Ukrainian to Russian
  107. Urdu to Hindi
  108. Welsh to English

See also

Notes

  1. . 28 December 2023 https://github.com/apertium/apertium/releases/tag/v3.9.4.{{cite web}}: Missing or empty |title= (help)
  2. Francis M. Tyers (2010) "Rule-based Breton to French machine translation Archived 2016-11-17 at the Wayback Machine ". 'Proceedings of the 14th Annual Conference of the European Association of Machine Translation, EAMT10', pp. 174--181
  3. Khanna, Tanmai; Washington, Jonathan N.; Tyers, Francis M.; Bayatlı, Sevilay; Swanson, Daniel G.; Pirinen, Tommi A.; Tang, Irene; Alòs i Font, Hèctor (1 December 2021). "Recent advances in Apertium, a free/open-source rule-based machine translation platform for low-resource languages". Machine Translation. 35 (4): 475–502. doi: 10.1007/s10590-021-09260-6 . hdl: 10037/22990 .
  4. "Apertium".
  5. "Accepted organizations for Google Summer of Code 2009".
  6. "Accepted organizations for Google Summer of Code 2010".
  7. "Accepted organizations for Google Summer of Code 2011".
  8. "Accepted organizations for Google Summer of Code 2012".
  9. "Accepted organizations for Google Summer of Code 2013".
  10. "Accepted organizations for Google Summer of Code 2014".
  11. "Accepted organizations for Google Code-in 2010".
  12. "Accepted organizations for Google Code-in 2011".
  13. "Accepted organizations for Google Code In 2012".
  14. "Accepted organizations for Google Code-in 2013".
  15. "Accepted organizations for Google Code-in 2014".
  16. "Accepted organizations for Google Code-in 2015".
  17. "Accepted organizations for Google Code-in 2016".
  18. "Accepted organizations for Google Code-in 2017".
  19. "Lttoolbox - Apertium". wiki.apertium.org. Retrieved 2016-01-19.
  20. "VISL". beta.visl.sdu.dk. Retrieved 2016-01-19.
  21. "Bilingual dictionary - Apertium". wiki.apertium.org. Retrieved 2016-01-19.
  22. "Constraint-based lexical selection module - Apertium". wiki.apertium.org. Retrieved 2016-01-19.
  23. "Morphological dictionary - Apertium". wiki.apertium.org. Retrieved 2016-01-19.

Related Research Articles

<span class="mw-page-title-main">Catalan language</span> Western Romance language

Catalan, known in the Valencian Community and Carche as Valencian, is a Western Romance language. It is the official language of Andorra, and an official language of three autonomous communities in eastern Spain: Catalonia, the Balearic Islands and the Valencian Community, where it is called Valencian. It has semi-official status in the Italian comune of Alghero, and it is spoken in the Pyrénées-Orientales department of France and in two further areas in eastern Spain: the eastern strip of Aragon and the Carche area in the Region of Murcia. The Catalan-speaking territories are often called the Països Catalans or "Catalan Countries".

<span class="mw-page-title-main">Cognate</span> Words inherited by different languages

In historical linguistics, cognates or lexical cognates are sets of words that have been inherited in direct descent from an etymological ancestor in a common parent language.

<span class="mw-page-title-main">Norwegian language</span> North Germanic language spoken in Norway

Norwegian is a North Germanic language from the Indo-European language family spoken mainly in Norway, where it is an official language. Along with Swedish and Danish, Norwegian forms a dialect continuum of more or less mutually intelligible local and regional varieties; some Norwegian and Swedish dialects, in particular, are very close. These Scandinavian languages, together with Faroese and Icelandic as well as some extinct languages, constitute the North Germanic languages. Faroese and Icelandic are not mutually intelligible with Norwegian in their spoken form because continental Scandinavian has diverged from them. While the two Germanic languages with the greatest numbers of speakers, English and German, have close similarities with Norwegian, neither is mutually intelligible with it. Norwegian is a descendant of Old Norse, the common language of the Germanic peoples living in Scandinavia during the Viking Age.

<span class="mw-page-title-main">Occitan language</span> Romance language of Western Europe

Occitan, also known as lenga d'òc by its native speakers, sometimes also referred to as Provençal, is a Romance language spoken in Southern France, Monaco, Italy's Occitan Valleys, as well as Spain's Val d'Aran in Catalonia; collectively, these regions are sometimes referred to as Occitania. It is also spoken in Calabria in a linguistic enclave of Cosenza area. Some include Catalan in Occitan, as the distance between this language and some Occitan dialects is similar to the distance between different Occitan dialects. Catalan was considered a dialect of Occitan until the end of the 19th century and still today remains its closest relative.

<span class="mw-page-title-main">Gascon dialect</span> Occitan dialect spoken in France and Spain

Gascon is the vernacular Romance variety spoken mainly in the region of Gascony, France. It is often considered a variety of Occitan, although some authors consider it a different language.

<span class="mw-page-title-main">Ó</span> Latin letter O with acute accent

Ó, ó (o-acute) is a letter in the Czech, Emilian-Romagnol, Faroese, Hungarian, Icelandic, Kashubian, Polish, Slovak, Karakalpak, and Sorbian languages. This letter also appears in the Afrikaans, Catalan, Dutch, Irish, Nynorsk, Bokmål, Occitan, Portuguese, Spanish, Italian and Galician languages as a variant of letter "o". In some cases, the letter "ó" is used in some languages as in a high rising tone. It is sometimes also used in English for loanwords.

<span class="mw-page-title-main">Aranese dialect</span> Occitan dialect near the French-Spanish border

Aranese is a standardized form of the Pyrenean Gascon variety of the Occitan language spoken in the Val d'Aran, in northwestern Catalonia close to the Spanish border with France, where it is one of the three official languages beside Catalan and Spanish. In 2010, it was declared the third official language in Catalonia by the Parliament of Catalonia.

<i>lernu!</i> Esperanto learning tool

lernu! is a multilingual, web-based free project for promoting and teaching Esperanto. The name Lernu comes from the imperative form of the Esperanto verb lerni, meaning "to learn". The site is run by E@I, an international youth organization, which started as a working group of the World Esperanto Youth Organization.

<span class="mw-page-title-main">Norwegian Wikipedia</span> Two of the Norwegian-language editions of Wikipedia

There are two Norwegian language editions of Wikipedia: one for articles written in Bokmål or Riksmål, and one for articles written in Nynorsk or Høgnorsk. There are currently 634,233 articles on the Norwegian Wikipedia edition in Bokmål/Riksmål, and 170,247 articles on the Nynorsk edition.

<span class="mw-page-title-main">Softcatalà</span>

Softcatalà is a non-profit association that promotes the use of the Catalan language on computing, Internet and new technologies. This association consists of computer specialists, philologists, translators, students and all kind of volunteers that work in the field of translating software into Catalan, in order to preserve this language in the English-controlled software environment. They also offer several linguistic tools to help users improve their language knowledge.

<span class="mw-page-title-main">Google Translate</span> Multilingual neural machine translation service

Google Translate is a multilingual neural machine translation service developed by Google to translate text, documents and websites from one language into another. It offers a website interface, a mobile app for Android and iOS, as well as an API that helps developers build browser extensions and software applications. As of August 2024, Google Translate supports 243 languages at various levels. It served over 200 million people daily in May 2013,, and over 500 million total users as of April 2016, with more than 100 billion words translated daily.

<span class="mw-page-title-main">Gollum browser</span> Web application for accessing Wikipedia

Gollum browser is a web application for accessing the encyclopedia, Wikipedia. Since 2017, Gollum is no longer accessible online.

<span class="mw-page-title-main">Languages of Spain</span>

The majority of languages of Spain belong to the Romance language family, of which Spanish is the only one with official status in the whole country. Others, including Catalan/Valencian and Galician, enjoy official status in their respective autonomous regions, similar to Basque in the northeast of the country. A number of other languages and dialects belonging to the Romance continuum exist in Spain, such as Aragonese, Asturian, Fala and Aranese Occitan.

Constraint grammar (CG) is a methodological paradigm for natural language processing (NLP). Linguist-written, context-dependent rules are compiled into a grammar that assigns grammatical tags ("readings") to words or other tokens in running text. Typical tags address lemmatisation, inflexion, derivation, syntactic function, dependency, valency, case roles, semantic type etc. Each rule either adds, removes, selects or replaces a tag or a set of grammatical tags in a given sentence context. Context conditions can be linked to any tag or tag set of any word anywhere in the sentence, either locally or globally. Context conditions in the same rule may be linked, i.e. conditioned upon each other, negated, or blocked by interfering words or tags. Typical CGs consist of thousands of rules, that are applied set-wise in progressive steps, covering ever more advanced levels of analysis. Within each level, safe rules are used before heuristic rules, and no rule is allowed to remove the last reading of a given kind, thus providing a high degree of robustness.

This article compares several selected client-based genealogy programs. Web-based genealogy software is not included.

<span class="mw-page-title-main">Transfer-based machine translation</span>

Transfer-based machine translation is a type of machine translation (MT). It is currently one of the most widely used methods of machine translation. In contrast to the simpler direct model of MT, transfer MT breaks translation into three steps: analysis of the source language text to determine its grammatical structure, transfer of the resulting structure to a structure suitable for generating text in the target language, and finally generation of this text. Transfer-based MT systems are thus capable of using knowledge of the source and target languages.

<span class="mw-page-title-main">Languages of Catalonia</span>

There are four languages with official status in Catalonia : Catalan; Spanish, which is official throughout Spain; Aranese, a dialect of Occitan spoken in the Aran Valley; and Catalan Sign Language. Many other languages are spoken in Catalonia as a result of recent immigration from all over the world.

<span class="mw-page-title-main">Western Romance languages</span> Subdivision of the Romance languages

Western Romance languages are one of the two subdivisions of a proposed subdivision of the Romance languages based on the La Spezia–Rimini Line. They include the Gallo-Romance, Occitano-Romance and Iberian Romance branches. Gallo-Italic may also be included. The subdivision is based mainly on the use of the "s" for pluralization, the weakening of some consonants and the pronunciation of "Soft C" as /t͡s/ rather than /t͡ʃ/ as in Italian and Romanian.

<span class="mw-page-title-main">Yandex Translate</span> Translation web service by Yandex

Yandex Translate is a web service provided by Yandex, intended for the translation of web pages into another language.

Google Neural Machine Translation (GNMT) was a neural machine translation (NMT) system developed by Google and introduced in November 2016 that used an artificial neural network to increase fluency and accuracy in Google Translate. The neural network consisted of two main blocks, an encoder and a decoder, both of LSTM architecture with 8 1024-wide layers each and a simple 1-layer 1024-wide feedforward attention mechanism connecting them. The total number of parameters has been variously described as over 160 million, approximately 210 million, 278 million or 380 million. It used WordPiece tokenizer, and beam search decoding strategy. It ran on Tensor Processing Units.

References

End-user services and software

(All services are based on the Apertium engine)

Online translation websites

Offline applications