Google Neural Machine Translation

Last updated

Google Neural Machine Translation (GNMT) was a neural machine translation (NMT) system developed by Google and introduced in November 2016 that used an artificial neural network to increase fluency and accuracy in Google Translate. [1] [2] [3] [4] The neural network consisted of two main blocks, an encoder and a decoder, both of LSTM architecture with 8 1024-wide layers each and a simple 1-layer 1024-wide feedforward attention mechanism connecting them. [4] [5] The total number of parameters has been variously described as over 160 million, [6] approximately 210 million, [7] 278 million [8] or 380 million. [9] It used WordPiece tokenizer, and beam search decoding strategy. It ran on Tensor Processing Units.

Contents

By 2020, the system had been replaced by another deep learning system based on a Transformer encoder and an RNN decoder. [10]

GNMT improved on the quality of translation by applying an example-based (EBMT) machine translation method in which the system learns from millions of examples of language translation. [2] GNMT's proposed architecture of system learning was first tested on over a hundred languages supported by Google Translate. [2] With the large end-to-end framework, the system learns over time to create better, more natural translations. [1] GNMT attempts to translate whole sentences at a time, rather than just piece by piece. [1] The GNMT network can undertake interlingual machine translation by encoding the semantics of the sentence, rather than by memorizing phrase-to-phrase translations. [2] [11]

History

The Google Brain project was established in 2011 in the "secretive Google X research lab" [12] by Google Fellow Jeff Dean, Google Researcher Greg Corrado, and Stanford University Computer Science professor Andrew Ng. [13] [14] [15] Ng's work has led to some of the biggest breakthroughs at Google and Stanford. [12]

In November 2016, Google Neural Machine Translation system (GNMT) was introduced. Since then, Google Translate began using neural machine translation (NMT) in preference to its previous statistical methods (SMT) [1] [16] [17] [18] which had been used since October 2007, with its proprietary, in-house SMT technology. [19] [20]

Training GNMT was a big effort at the time and took, by a 2021 OpenAI estimate, on the order of 100 PFLOP/s*day (up to 1022 FLOPs) of compute which was 1.5 orders of magnitude larger than Seq2seq model of 2014 [21] (but about 2x smaller than GPT-J-6B in 2021 [22] ).

Google Translate's NMT system uses a large artificial neural network capable of deep learning. [1] [2] [3] By using millions of examples, GNMT improves the quality of translation, [2] using broader context to deduce the most relevant translation. The result is then rearranged and adapted to approach grammatically based human language. [1] GNMT's proposed architecture of system learning was first tested on over a hundred languages supported by Google Translate. [2] GNMT did not create its own universal interlingua but rather aimed at finding the commonality between many languages using insights from psychology and linguistics. [23] The new translation engine was first enabled for eight languages: to and from English and French, German, Spanish, Portuguese, Chinese, Japanese, Korean and Turkish in November 2016. [24] In March 2017, three additional languages were enabled: Russian, Hindi and Vietnamese along with Thai for which support was added later. [25] [26] Support for Hebrew and Arabic was also added with help from the Google Translate Community in the same month. [27] In mid April 2017 Google Netherlands announced support for Dutch and other European languages related to English. [28] Further support was added for nine Indian languages: Hindi, Bengali, Marathi, Gujarati, Punjabi, Tamil, Telugu, Malayalam and Kannada at the end of April 2017. [29]

By 2020, Google had changed methodology to use a different neural network system based on transformers, and had phased out NMT. [30]

Evaluation

The GNMT system was said to represent an improvement over the former Google Translate in that it will be able to handle "zero-shot translation", that is it directly translates one language into another (for example, Japanese to Korean). [2] Google Translate previously first translated the source language into English and then translated the English into the target language rather than translating directly from one language to another. [11]

A July 2019 study in Annals of Internal Medicine found that "Google Translate is a viable, accurate tool for translating non–English-language trials". Only one disagreement between reviewers reading machine-translated trials was due to a translation error. Since many medical studies are excluded from systematic reviews because the reviewers do not understand the language, GNMT has the potential to reduce bias and improve accuracy in such reviews. [31]

Languages supported by GNMT

As of December 2021, all of the languages of Google Translate support GNMT, with Latin being the most recent addition.

  1. Afrikaans
  2. Albanian
  3. Amharic
  4. Arabic
  5. Armenian
  6. Azerbaijani
  7. Basque
  8. Belarusian
  9. Bengali
  10. Bosnian
  11. Bulgarian
  12. Burmese
  13. Catalan
  14. Cebuano
  15. Chewa
  16. Chinese (Simplified)
  17. Chinese (Traditional)
  18. Corsican
  19. Croatian
  20. Czech
  21. Danish
  22. Dutch
  23. English
  24. Esperanto
  25. Estonian
  26. Filipino (Tagalog)
  27. Finnish
  28. French
  29. Galician
  30. Georgian
  31. German
  32. Greek
  33. Gujarati
  34. Haitian Creole
  35. Hausa
  36. Hawaiian
  37. Hebrew
  38. Hindi
  39. Hmong
  40. Hungarian
  41. Icelandic
  42. Igbo
  43. Indonesian
  44. Irish
  45. Italian
  46. Japanese
  47. Javanese
  48. Kannada
  49. Kazakh
  50. Khmer
  51. Kinyarwanda
  52. Korean
  53. Kurdish (Kurmanji)
  54. Kyrgyz
  55. Lao
  56. Latin
  57. Latvian
  58. Lithuanian
  59. Luxembourgish
  60. Macedonian
  61. Malagasy
  62. Malay
  63. Malayalam
  64. Maltese
  65. Maori
  66. Marathi
  67. Mongolian
  68. Nepali
  69. Norwegian (Bokmål)
  70. Odia
  71. Pashto
  72. Persian
  73. Polish
  74. Portuguese
  75. Punjabi (Gurmukhi)
  76. Romanian
  77. Russian
  78. Samoan
  79. Scottish Gaelic
  80. Serbian
  81. Shona
  82. Sindhi
  83. Sinhala
  84. Slovak
  85. Slovenian
  86. Somali
  87. Sotho
  88. Spanish
  89. Sundanese
  90. Swahili
  91. Swedish
  92. Tajik
  93. Tamil
  94. Tatar
  95. Telugu
  96. Thai
  97. Turkish
  98. Turkmen
  99. Ukrainian
  100. Urdu
  101. Uyghur
  102. Uzbek
  103. Vietnamese
  104. Welsh
  105. West Frisian
  106. Xhosa
  107. Yiddish
  108. Yoruba
  109. Zulu

See also

Related Research Articles

Artificial intelligence (AI), in its broadest sense, is intelligence exhibited by machines, particularly computer systems. It is a field of research in computer science that develops and studies methods and software that enable machines to perceive their environment and use learning and intelligence to take actions that maximize their chances of achieving defined goals. Such machines may be called AIs.

<span class="mw-page-title-main">Jürgen Schmidhuber</span> German computer scientist

Jürgen Schmidhuber is a German computer scientist noted for his work in the field of artificial intelligence, specifically artificial neural networks. He is a scientific director of the Dalle Molle Institute for Artificial Intelligence Research in Switzerland. He is also director of the Artificial Intelligence Initiative and professor of the Computer Science program in the Computer, Electrical, and Mathematical Sciences and Engineering (CEMSE) division at the King Abdullah University of Science and Technology (KAUST) in Saudi Arabia.

<span class="mw-page-title-main">Geoffrey Hinton</span> British-Canadian computer scientist and psychologist (born 1947)

Geoffrey Everest Hinton is a British-Canadian computer scientist and cognitive psychologist, most noted for his work on artificial neural networks. From 2013 to 2023, he divided his time working for Google and the University of Toronto, before publicly announcing his departure from Google in May 2023, citing concerns about the risks of artificial intelligence (AI) technology. In 2017, he co-founded and became the chief scientific advisor of the Vector Institute in Toronto.

<span class="mw-page-title-main">Google Translate</span> Multilingual neural machine translation service

Google Translate is a multilingual neural machine translation service developed by Google to translate text, documents and websites from one language into another. It offers a website interface, a mobile app for Android and iOS, as well as an API that helps developers build browser extensions and software applications. As of August 2024, Google Translate supports 243 languages at various levels. It served over 200 million people daily in May 2013, and over 500 million total users as of April 2016, with more than 100 billion words translated daily.

In the history of artificial intelligence, an AI winter is a period of reduced funding and interest in artificial intelligence research. The field has experienced several hype cycles, followed by disappointment and criticism, followed by funding cuts, followed by renewed interest years or even decades later.

Machine translation is a sub-field of computational linguistics that investigates the use of software to translate text or speech from one natural language to another.

The following outline is provided as an overview of and topical guide to artificial intelligence:

<span class="mw-page-title-main">Long short-term memory</span> Artificial recurrent neural network architecture used in deep learning

Long short-term memory (LSTM) is a type of recurrent neural network (RNN) aimed at dealing with the vanishing gradient problem present in traditional RNNs. Its relative insensitivity to gap length is its advantage over other RNNs, hidden Markov models and other sequence learning methods. It aims to provide a short-term memory for RNN that can last thousands of timesteps, thus "long short-term memory". The name is made in analogy with long-term memory and short-term memory and their relationship, studied by cognitive psychologists since early 20th century.

Artificial intelligence (AI) has been used in applications throughout industry and academia. In a manner analogous to electricity or computers, AI serves as a general-purpose technology that has numerous applications, including language translation, image recognition, decision-making, credit scoring and e-commerce. AI includes the development of machines which can perceive, understand, act and learn a scientific discipline.

<span class="mw-page-title-main">Otávio Good</span> American computer programmer

Otávio Good is a Brazilian and American computer programmer and inventor. He is the original author of Word Lens, the first augmented reality translation application that replaces printed text into the desired language in video without connection to the Internet.

<span class="mw-page-title-main">Deep learning</span> Branch of machine learning

Deep learning is the subset of machine learning methods based on neural networks with representation learning. The adjective "deep" refers to the use of multiple layers in the network. Methods used can be either supervised, semi-supervised or unsupervised.

<span class="mw-page-title-main">Jeff Dean</span> American computer scientist and software engineer

Jeffrey Adgate "Jeff" Dean is an American computer scientist and software engineer. Since 2018, he has been the lead of Google AI. He was appointed Alphabet's chief scientist in 2023 after a reorganization of Alphabet's AI focused groups.

Google Brain was a deep learning artificial intelligence research team under the umbrella of Google AI, a research division at Google dedicated to artificial intelligence. Formed in 2011, it combined open-ended machine learning research with information systems and large-scale computing resources. It created tools such as TensorFlow, which allow neural networks to be used by the public, and multiple internal AI research projects, and aimed to create research opportunities in machine learning and natural language processing. It was merged into former Google sister company DeepMind to form Google DeepMind in April 2023.

Google DeepMind Technologies Limited is a British-American artificial intelligence research laboratory which serves as a subsidiary of Google. Founded in the UK in 2010, it was acquired by Google in 2014 and merged with Google AI's Google Brain division to become Google DeepMind in April 2023. The company is based in London, with research centres in Canada, France, Germany, and the United States.

Neural machine translation (NMT) is an approach to machine translation that uses an artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model.

This glossary of artificial intelligence is a list of definitions of terms and concepts relevant to the study of artificial intelligence, its sub-disciplines, and related fields. Related glossaries include Glossary of computer science, Glossary of robotics, and Glossary of machine vision.

An AI accelerator, deep learning processor or neural processing unit (NPU) is a class of specialized hardware accelerator or computer system designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and computer vision. Typical applications include algorithms for robotics, Internet of Things, and other data-intensive or sensor-driven tasks. They are often manycore designs and generally focus on low-precision arithmetic, novel dataflow architectures or in-memory computing capability. As of 2024, a typical AI integrated circuit chip contains tens of billions of MOSFETs.

<span class="mw-page-title-main">Ian Goodfellow</span> American computer scientist

Ian J. Goodfellow is an American computer scientist, engineer, and executive, most noted for his work on artificial neural networks and deep learning. He was previously employed as a research scientist at Google Brain and director of machine learning at Apple and has made several important contributions to the field of deep learning including the invention of the generative adversarial network (GAN). Goodfellow co-wrote, as the first author, the textbook Deep Learning (2016) and wrote the chapter on deep learning in the authoritative textbook of the field of artificial intelligence, Artificial Intelligence: A Modern Approach.

<span class="mw-page-title-main">Meta AI</span> Artificial intelligence division of Meta Platforms

Meta AI is an American company owned by Meta that develops artificial intelligence and augmented and artificial reality technologies. Meta AI deems itself an academic research laboratory, focused on generating knowledge for the AI community, and should not be confused with Meta's Applied Machine Learning (AML) team, which focuses on the practical applications of its products.

References

  1. 1 2 3 4 5 6 Barak Turovsky (November 15, 2016), "Found in translation: More accurate, fluent sentences in Google Translate", Google Blog, retrieved January 11, 2017
  2. 1 2 3 4 5 6 7 8 Mike Schuster; Melvin Johnson; Nikhil Thorat (November 22, 2016), "Zero-Shot Translation with Google's Multilingual Neural Machine Translation System", Google Research Blog, retrieved January 11, 2017
  3. 1 2 Gil Fewster (January 5, 2017), "The mind-blowing AI announcement from Google that you probably missed", freeCodeCamp, retrieved January 11, 2017
  4. 1 2 Wu, Yonghui; Schuster, Mike; Chen, Zhifeng; Le, Quoc V.; Norouzi, Mohammad (2016). "Google's neural machine translation system: Bridging the gap between human and machine translation". arXiv: 1609.08144 . Bibcode:2016arXiv160908144W.{{cite journal}}: Cite journal requires |journal= (help)
  5. ↑ "Peeking into the neural network architecture used for Google's Neural Machine Translation".
  6. ↑ Qin, Minghai; Zhang, Tianyun; Sun, Fei; Chen, Yen-Kuang; Fardad, Makan; Wang, Yanzhi; Xie, Yuan (2021). "Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting". arXiv: 2112.10930 [cs.NE].
  7. ↑ "Compression of Google Neural Machine Translation Model – NLP Architect by Intel® AI Lab 0.5.5 documentation".
  8. ↑ Langroudi, Hamed F.; Karia, Vedant; Pandit, Tej; Kudithipudi, Dhireesha (2021). "TENT: Efficient Quantization of Neural Networks on the tiny Edge with Tapered FixEd PoiNT". arXiv: 2104.02233 [cs.LG].
  9. ↑ "Data Augmentation | How to use Deep Learning when you have Limited Data". May 19, 2021.
  10. ↑ "Recent Advances in Google Translate". research.google. Retrieved May 8, 2024.
  11. 1 2 Boitet, Christian; Blanchon, Hervé; Seligman, Mark; Bellynck, Valérie (2010). "MT on and for the Web" (PDF). Archived from the original (PDF) on March 29, 2017. Retrieved December 1, 2016.
  12. 1 2 Robert D. Hof (August 14, 2014). "A Chinese Internet Giant Starts to Dream: Baidu is a fixture of online life in China, but it wants to become a global power. Can one of the world's leading artificial intelligence researchers help it challenge Silicon Valley's biggest companies?". Technology Review. Retrieved January 11, 2017.
  13. ↑ Jeff Dean and Andrew Ng (June 26, 2012). "Using large-scale brain simulations for machine learning and A.I." Official Google Blog. Retrieved January 26, 2015.
  14. ↑ "Google's Large Scale Deep Neural Networks Project". YouTube . Retrieved October 25, 2015.
  15. ↑ Markoff, John (June 25, 2012). "How Many Computers to Identify a Cat? 16,000". New York Times . Retrieved February 11, 2014.
  16. ↑ Katyanna Quach (November 17, 2016), "Google's neural network learns to translate languages it hasn't been trained on: First time machine translation has used true transfer learning", The Register, retrieved January 11, 2017
  17. ↑ Lewis-Kraus, Gideon (December 14, 2016). "The Great A.I. Awakening". The New York Times . Retrieved January 11, 2017.
  18. ↑ Le, Quoc; Schuster, Mike (September 27, 2016). "A Neural Network for Machine Translation, at Production Scale". Google Research Blog. Retrieved December 1, 2016.
  19. ↑ Google Switches to its Own Translation System, October 22, 2007
  20. ↑ Barry Schwartz (October 23, 2007). "Google Translate Drops SYSTRAN for Home-Brewed Translation". Search Engine Land.
  21. ↑ "AI and compute".
  22. ↑ "Table of contents". GitHub .
  23. ↑ Chris McDonald (January 7, 2017), Commenting on Gil Fewster's January 5th article in the Atlantic , retrieved January 11, 2017
  24. ↑ Turovsky, Barak (November 15, 2016). "Found in translation: More accurate, fluent sentences in Google Translate". The Keyword Google Blog. Retrieved December 1, 2016.
  25. ↑ Perez, Sarah (March 6, 2017). "Google's smarter, A.I.-powered translation system expands to more languages". TechCrunch. Oath Inc.
  26. ↑ Turovsky, Barak (March 6, 2017). "Higher quality neural translations for a bunch more languages". The Keyword Google Blog. Retrieved March 6, 2017.
  27. ↑ Novet, Jordan (March 30, 2017). "Google now provides AI-powered translations for Arabic and Hebrew". VentureBeat .
  28. ↑ Finge, Rachid (April 19, 2017). "Grote verbetering voor het Nederlands in Google Translate" [Big improvement for Dutch in Google Translate]. Google Netherlands Blog (in Dutch).
  29. ↑ Turovsky, Barak (April 25, 2017). "Making the internet more inclusive in India". The Keyword.
  30. ↑ "Recent Advances in Google Translate". research.google. Retrieved May 8, 2024.
  31. ↑ Jackson, Jeffrey L; Kuriyama, Akira; Anton, Andreea; Choi, April; Fournier, Jean-Pascal; Geier, Anne-Kathrin; Jacquerioz, Frederique; Kogan, Dmitry; Scholcoff, Cecilia; Sun, Rao (July 30, 2019). "The Accuracy of Google Translate for Abstracting Data From Non–English-Language Trials for Systematic Reviews". Annals of Internal Medicine. 171 (9): 678. doi:10.7326/M19-0891. ISSN   0570-183X. PMID   31357212. S2CID   198980789.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.