Machine translation software usability

Last updated

The sections below give objective criteria for evaluating the usability of machine translation software output.

Contents

Stationarity or canonical form

Do repeated translations converge on a single expression in both languages? I.e. does the translation method show stationarity or produce a canonical form? Does the translation become stationary without losing the original meaning? This metric has been criticized as not being well correlated with BLEU (BiLingual Evaluation Understudy) scores. [1]

Adaptive to colloquialism, argot or slang

Is the system adaptive to colloquialism, argot or slang? The French language has many rules for creating words in the speech and writing of popular culture. Two such rules are: (a) The reverse spelling of words such as femme to meuf. (This is called verlan.) (b) The attachment of the suffix -ard to a noun or verb to form a proper noun. For example, the noun faluche means "student hat". The word faluchard formed from faluche colloquially can mean, depending on context, "a group of students", "a gathering of students" and "behavior typical of a student". The Google translator as of 28 December 2006 doesn't derive the constructed words as for example from rule (b), as shown here:

Il y a une chorale falucharde mercredi, venez nombreux, les faluchards chantent des paillardes! ==> There is a choral society falucharde Wednesday, come many, the faluchards sing loose-living women!

French argot has three levels of usage: [2]

  1. familier or friendly, acceptable among friends, family and peers but not at work
  2. grossier or swear words, acceptable among friends and peers but not at work or in family
  3. verlan or ghetto slang, acceptable among lower classes but not among middle or upper classes

The United States National Institute of Standards and Technology conducts annual evaluations of machine translation systems based on the BLEU-4 criterion . A combined method called IQmt which incorporates BLEU and additional metrics NIST, GTM, ROUGE and METEOR has been implemented by Gimenez and Amigo .

Well-formed output

Is the output grammatical or well-formed in the target language? Using an interlingua should be helpful in this regard, because with a fixed interlingua one should be able to write a grammatical mapping to the target language from the interlingua. Consider the following Arabic language input and English language translation result from the Google translator as of 27 December 2006 . This Google translator output doesn't parse using a reasonable English grammar:

وعن حوادث التدافع عند شعيرة رمي الجمرات -التي كثيرا ما يسقط فيها العديد من الضحايا- أشار الأمير نايف إلى إدخال "تحسينات كثيرة في جسر الجمرات ستمنع بإذن الله حدوث أي تزاحم". ==> And incidents at the push Carbuncles-throwing ritual, which often fall where many of the victims - Prince Nayef pointed to the introduction of "many improvements in bridge Carbuncles God would stop the occurrence of any competing."

Semantics preservation

Do repeated re-translations preserve the semantics of the original sentence? For example, consider the following English input passed multiple times into and out of French using the Google translator as of 27 December 2006:

Better a day earlier than a day late. ==>

Améliorer un jour plus tôt qu'un jour tard. ==>

To improve one day earlier than a day late. ==>

Pour améliorer un jour plus tôt qu'un jour tard. ==>

To improve one day earlier than a day late.

As noted above and in, [1] this kind of round-trip translation is a very unreliable method of evaluation.

Trustworthiness and security

An interesting peculiarity of Google Translate as of 24 January 2008 (corrected as of 25 January 2008) is the following result when translating from English to Spanish, which shows an embedded joke in the English-Spanish dictionary which has some added poignancy given recent events:

Heath Ledger is dead ==>

Tom Cruise está muerto

This raises the issue of trustworthiness when relying on a machine translation system embedded in a Life-critical system in which the translation system has input to a Safety Critical Decision Making process. Conjointly it raises the issue of whether in a given use the software of the machine translation system is safe from hackers.

It is not known whether this feature of Google Translate was the result of a joke/hack or perhaps an unintended consequence of the use of a method such as statistical machine translation. Reporters from CNET Networks asked Google for an explanation on January 24, 2008; Google said only that it was an "internal issue with Google Translate". [3] The mistranslation was the subject of much hilarity and speculation on the Internet. [4] [5]

If it is an unintended consequence of the use of a method such as statistical machine translation, and not a joke/hack, then this event is a demonstration of a potential source of critical unreliability in the statistical machine translation method.

In human translations, in particular on the part of interpreters, selectivity on the part of the translator in performing a translation is often commented on when one of the two parties being served by the interpreter knows both languages.

This leads to the issue of whether a particular translation could be considered verifiable. In this case, a converging round-trip translation would be a kind of verification.

See also

Notes

  1. 1 2 Somers, Harold (2005). "Round-trip translation: What is it good for?". Proceedings of Australasian Language Technology Workshop ALTW 2005. Sydney: 127–133.
  2. ↑ "The Agony of Argot", Chitlins & Camembert, October 28, 2005
  3. ↑ "Google Translate bug mixes up Heath Ledger, Tom Cruise", by Caroline McCarthy, CNET Networks, January 24, 2008
  4. ↑ '"Tom Cruise" is Spanish for "Heath Ledger"', gawker.com, January 24, 2008 Archived January 28, 2008, at the Wayback Machine
  5. ↑ "Tom Cruise está muerto", Ray Leon Blog Project, January 24, 2008 Archived October 29, 2008, at the Wayback Machine

Related Research Articles

<span class="mw-page-title-main">Interlingua</span> Constructed language

Interlingua is an international auxiliary language (IAL) developed between 1937 and 1951 by the American International Auxiliary Language Association (IALA). It is a constructed language of the "naturalistic" variety, whose vocabulary, grammar, and other characteristics are derived from natural languages. Interlingua literature maintains that (written) Interlingua is comprehensible to the hundreds of millions of people who speak Romance languages, though it is actively spoken by only a few hundred.

<span class="mw-page-title-main">Leet</span> Online slang and alternative orthography

Leet, also known as eleet or leetspeak, is a system of modified spellings used primarily on the Internet. It often uses character replacements in ways that play on the similarity of their glyphs via reflection or other resemblance. Additionally, it modifies certain words on the basis of a system of suffixes and alternative meanings. There are many dialects or linguistic varieties in different online communities.

<span class="mw-page-title-main">Machine translation</span> Use of software for language translation

Machine translation is use of either rule-based or probabilistic machine learning approaches to translation of text or speech from one language to another, including the contextual, idiomatic and pragmatic nuances of both languages.

In linguistics, a neologism is any newly formed word, term, or phrase that nevertheless has achieved popular or institutional recognition and is becoming accepted into mainstream language. Most definitively, a word can be considered a neologism once it is published in a dictionary.

<span class="mw-page-title-main">Latino sine flexione</span> Latin-based international auxiliary language

Latino sine flexione, Interlingua de Academia pro Interlingua or Peano's Interlingua is an international auxiliary language compiled by the Academia pro Interlingua under the chairmanship of the Italian mathematician Giuseppe Peano (1858–1932) from 1887 until 1914. It is a simplified version of Latin, and retains its vocabulary. Interlingua-IL was published in the journal Revue de Mathématiques in an article of 1903 entitled De Latino Sine Flexione, Lingua Auxiliare Internationale, which explained the reason for its creation. The article argued that other auxiliary languages were unnecessary, since Latin was already established as the world's international language. The article was written in classical Latin, but it gradually dropped its inflections until there were none.

A pivot language, sometimes also called a bridge language, is an artificial or natural language used as an intermediary language for translation between many different languages – to translate between any pair of languages A and B, one translates A to the pivot language P, then from P to B. Using a pivot language avoids the combinatorial explosion of having translators across every combination of the supported languages, as the number of combinations of language is linear, rather than quadratic – one need only know the language A and the pivot language P, rather than needing a different translator for every possible combination of A and B.

<span class="mw-page-title-main">Interlingual machine translation</span> Type of machine translation

Interlingual machine translation is one of the classic approaches to machine translation. In this approach, the source language, i.e. the text to be translated is transformed into an interlingua, i.e., an abstract language-independent representation. The target language is then generated from the interlingua. Within the rule-based machine translation paradigm, the interlingual approach is an alternative to the direct approach and the transfer approach.

Round-trip translation (RTT), also known as back-and-forth translation, recursive translation and bi-directional translation, is the process of translating a word, phrase or text into another language, then translating the result back into the original language, using machine translation (MT) software. It is often used by laypeople to evaluate a machine translation system, or to test whether a text is suitable for MT when they are unfamiliar with the target language. Because the resulting text can often differ substantially from the original, RTT can also be a source of entertainment.

Linguistic categories include

Various methods for the evaluation for machine translation have been employed. This article focuses on the evaluation of the output of machine translation, rather than on performance or usability evaluation.

<span class="mw-page-title-main">Transfer-based machine translation</span>

Transfer-based machine translation is a type of machine translation (MT). It is currently one of the most widely used methods of machine translation. In contrast to the simpler direct model of MT, transfer MT breaks translation into three steps: analysis of the source language text to determine its grammatical structure, transfer of the resulting structure to a structure suitable for generating text in the target language, and finally generation of this text. Transfer-based MT systems are thus capable of using knowledge of the source and target languages.

Mobile translation is any electronic device or software application that provides audio translation. The concept includes any handheld electronic device that is specifically designed for audio translation. It also includes any machine translation service or software application for hand-held devices, including mobile telephones, Pocket PCs, and PDAs. Mobile translation provides hand-held device users with the advantage of instantaneous and non-mediated translation from one human language to another, usually against a service fee that is, nevertheless, significantly smaller than a human translator charges.

A language game is a system of manipulating spoken words to render them incomprehensible to an untrained listener. Language games are used primarily by groups attempting to conceal their conversations from others. Some common examples are Pig Latin; the Gibberish family, prevalent in the United States and Sweden; and Verlan, spoken in France.

<span class="mw-page-title-main">Microsoft Translator</span> Machine translation cloud service by Microsoft

Microsoft Translator is a multilingual machine translation cloud service provided by Microsoft. Microsoft Translator is a part of Microsoft Cognitive Services and integrated across multiple consumer, developer, and enterprise products, including Bing, Microsoft Office, SharePoint, Microsoft Edge, Microsoft Lync, Yammer, Skype Translator, Visual Studio, and Microsoft Translator apps for Windows, Windows Phone, iPhone and Apple Watch, and Android phone and Android Wear.

Rule-based machine translation is machine translation systems based on linguistic information about source and target languages basically retrieved from dictionaries and grammars covering the main semantic, morphological, and syntactic regularities of each language respectively. Having input sentences, an RBMT system generates them to output sentences on the basis of morphological, syntactic, and semantic analysis of both the source and the target languages involved in a concrete translation task.

MedSLT is a medium-ranged open source spoken language translator developed by the University of Geneva. It is funded by the Swiss National Science Foundation. The system has been designed for the medical domain. It currently covers the doctor-patient diagnosis dialogues for the domains of headache, chest and abdominal pain in English, French, Japanese, Spanish, Catalan and Arabic. The vocabulary used ranges from 350 to 1000 words depending on the domain and language pair.

Kurt Gingold (1929–1997) was an Austrian-American scientific translator, and a charter member and second president of the American Translators Association.

The Europarl Corpus is a corpus that consists of the proceedings of the European Parliament from 1996 to 2012. In its first release in 2001, it covered eleven official languages of the European Union. With the political expansion of the EU the official languages of the ten new member states have been added to the corpus data. The latest release (2012) comprised up to 60 million words per language with the newly added languages being slightly underrepresented as data for them is only available from 2007 onwards. This latest version includes 21 European languages: Romanic, Germanic, Slavic, Finno-Ugric, Baltic, and Greek.

LEPOR is an automatic language independent machine translation evaluation metric with tunable parameters and reinforced factors.

Google Neural Machine Translation (GNMT) was a neural machine translation (NMT) system developed by Google and introduced in November 2016 that used an artificial neural network to increase fluency and accuracy in Google Translate. The neural network consisted of two main blocks, an encoder and a decoder, both of LSTM architecture with 8 1024-wide layers each and a simple 1-layer 1024-wide feedforward attention mechanism connecting them. The total number of parameters has been variously described as over 160 million, approximately 210 million, 278 million or 380 million. By 2020, the system had been replaced by another deep learning system based on transformers.

References

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.