Neural machine translation

Last updated

Neural machine translation (NMT) is an approach to machine translation that uses an artificial neural network to predict the likelihood of a sequence of words, typically modeling entire sentences in a single integrated model.

Contents

It is the dominant approach today [1] : 293  [2] : 1  and can produce translations that rival human translations when translating between high-resource languages under specific conditions. [3] However, there still remain challenges, especially with languages where less high-quality data is available, [4] [5] [1] : 293  and with domain shift between the data a system was trained on and the texts it is supposed to translate. [1] : 293  NMT systems also tend to produce fairly literal translations. [5]

Overview

In the translation task, a sentence (consisting of tokens ) in the source language is to be translated into a sentence (consisting of tokens ) in the target language. The source and target tokens (which in the simple event are used for each other in order for a particular game ] vectors, so they can be processed mathematically.

NMT models assign a probability [2] : 5  [6] : 1  to potential translations y and then search a subset of potential translations for the one with the highest probability. Most NMT models are auto-regressive: They model the probability of each target token as a function of the source sentence and the previously predicted target tokens. The probability of the whole translation then is the product of the probabilities of the individual predicted tokens: [2] : 5  [6] : 2 

NMT models differ in how exactly they model this function , but most use some variation of the encoder-decoder architecture: [6] : 2  [7] : 469  They first use an encoder network to process and encode it into a vector or matrix representation of the source sentence. Then they use a decoder network that usually produces one target word at a time, taking into account the source representation and the tokens it previously produced. As soon as the decoder produces a special end of sentence token, the decoding process is finished. Since the decoder refers to its own previous outputs during, this way of decoding is called auto-regressive.

History

Early approaches

In 1987, Robert B. Allen demonstrated the use of feed-forward neural networks for translating auto-generated English sentences with a limited vocabulary of 31 words into Spanish. In this experiment, the size of the network's input and output layers was chosen to be just large enough for the longest sentences in the source and target language, respectively, because the network did not have any mechanism to encode sequences of arbitrary length into a fixed-size representation. In his summary, Allen also already hinted at the possibility of using auto-associative models, one for encoding the source and one for decoding the target. [8]

Lonnie Chrisman built upon Allen's work in 1991 by training separate recursive auto-associative memory (RAAM) networks (developed by Jordan B. Pollack [9] ) for the source and the target language. Each of the RAAM networks is trained to encode an arbitrary-length sentence into a fixed-size hidden representation and to decode the original sentence again from that representation. Additionally, the two networks are also trained to share their hidden representation; this way, the source encoder can produce a representation that the target decoder can decode. [10] Forcada and Ñeco simplified this procedure in 1997 to directly train a source encoder and a target decoder in what they called a recursive hetero-associative memory. [11]

Also in 1997, Castaño and Casacuberta employed an Elman's recurrent neural network in another machine translation task with very limited vocabulary and complexity. [12] [13]

Even though these early approaches were already similar to modern NMT, the computing resources of the time were not sufficient to process datasets large enough for the computational complexity of the machine translation problem on real-world texts. [1] : 39  [14] : 2  Instead, other methods like statistical machine translation rose to become the state of the art of the 1990s and 2000s.

Hybrid approaches

During the time when statistical machine translation was prevalent, some works used neural methods to replace various parts in the statistical machine translation while still using the log-linear approach to tie them together. [1] : 39  [2] : 1  For example, in various works together with other researchers, Holger Schwenk replaced the usual n-gram language model with a neural one [15] [16] and estimated phrase translation probabilities using a feed-forward network. [17]

seq2seq

In 2013 and 2014, end-to-end neural machine translation had their breakthrough with Kalchbrenner & Blunsom using a convolutional neural network (CNN) for encoding the source [18] and both Cho et al. and Sutskever et al. using a recurrent neural network (RNN) instead. [19] [20] All three used an RNN conditioned on a fixed encoding of the source as their decoder to produce the translation. However, these models performed poorly on longer sentences. [21] : 107  [1] : 39  [2] : 7  This problem was addressed when Bahdanau et al. introduced attention to their encoder-decoder architecture: At each decoding step, the state of the decoder is used to calculate a source representation that focuses on different parts of the source and uses that representation in the calculation of the probabilities for the next token. [22] Based on these RNN-based architectures, Baidu launched the "first large-scale NMT system" [23] : 144  in 2015, followed by Google Neural Machine Translation in 2016. [23] : 144  [24] From that year on, neural models also became the prevailing choice in the main machine translation conference Workshop on Statistical Machine Translation. [25]

Gehring et al. combined a CNN encoder with an attention mechanism in 2017, which handled long-range dependencies in the source better than previous approaches and also increased translation speed because a CNN encoder is parallelizable, whereas an RNN encoder has to encode one token at a time due to its recurrent nature. [26] : 230  In the same year, “Microsoft Translator released AI-powered online neural machine translation (NMT). [27] DeepL Translator, which was at the time based on a CNN encoder, was also released in the same year and was judged by several news outlets to outperform its competitors. [28] [29] [30] It has also been seen that OpenAI's GPT-3 released in 2020 can function as a neural machine translation system. Some other machine translation systems, such as Microsoft translator and SYSTRAN can be also seen to have integrated neural networks into their operations.

Transformer

Another network architecture that lends itself to parallelization is the transformer, which was introduced by Vaswani et al. also in 2017. [31] Like previous models, the transformer still uses the attention mechanism for weighting encoder output for the decoding steps. However, the transformer's encoder and decoder networks themselves are also based on attention instead of recurrence or convolution: Each layer weights and transforms the previous layer's output in a process called self-attention. Since the attention mechanism does not have any notion of token order, but the order of words in a sentence is obviously relevant, the token embeddings are combined with an explicit encoding of their position in the sentence. [2] : 15  [6] : 7  Since both the transformer's encoder and decoder are free from recurrent elements, they can both be parallelized during training. However, the original transformer's decoder is still auto-regressive, which means that decoding still has to be done one token at a time during inference.

The transformer model quickly became the dominant choice for machine translation systems [2] : 44  and was still by far the most-used architecture in the Workshop on Statistical Machine Translation in 2022 and 2023. [32] : 35–40  [33] : 28–31 

Usually, NMT models’ weights are initialized randomly and then learned by training on parallel datasets. However, since using large language models (LLMs) such as BERT pre-trained on large amounts of monolingual data as a starting point for learning other tasks has proven very successful in wider NLP, this paradigm is also becoming more prevalent in NMT. This is especially useful for low-resource languages, where large parallel datasets do not exist. [4] : 689–690  An example of this is the mBART model, which first trains one transformer on a multilingual dataset to recover masked tokens in sentences, and then fine-tunes the resulting autoencoder on the translation task. [34]

Generative LLMs

Instead of fine-tuning a pre-trained language model on the translation task, sufficiently large generative models can also be directly prompted to translate a sentence into the desired language. This approach was first comprehensively tested and evaluated for GPT 3.5 in 2023 by Hendy et al. They found that "GPT systems can produce highly fluent and competitive translation outputs even in the zero-shot setting especially for the high-resource language translations". [35] : 22  The WMT23 evaluated the same approach (but using GPT-4) and found that it was on par with the state of the art when translating into English, but not quite when translating into lower-resource languages. [33] : 16–17  This is plausible considering that GPT models are trained mainly on English text. [36]

Comparison with statistical machine translation

NMT has overcome several challenges that were present in statistical machine translation (SMT):

Training procedure

Cross-entropy loss

NMT models are usually trained to maximize the likelihood of observing the training data. I.e., for a dataset of source sentences and corresponding target sentences , the goal is finding the model parameters that maximize the sum of the likelihood of each target sentence in the training data given the corresponding source sentence:

Expanding to token level yields:

Since we are only interested in the maximum, we can just as well search for the maximum of the logarithm instead (which has the advantage that it avoids floating point underflow that could happen with the product of low probabilities).

Using the fact that the logarithm of a product is the sum of the factors’ logarithms and flipping the sign yields the classic cross-entropy loss:

In practice, this minimization is done iteratively on small subsets (mini-batches) of the training set using stochastic gradient descent.

Teacher forcing

During inference, auto-regressive decoders use the token generated in the previous step as the input token. However, the vocabulary of target tokens is usually very large. So, at the beginning of the training phase, untrained models will pick the wrong token almost always; and subsequent steps would then have to work with wrong input tokens, which would slow down training considerably. Instead, teacher forcing is used during the training phase: The model (the “student” in the teacher forcing metaphor) is always fed the previous ground-truth tokens as input for the next token, regardless of what it predicted in the previous step.

Translation by prompt engineering LLMs

As outlined in the history section above, instead of using an NMT system that is trained on parallel text, one can also prompt a generative LLM to translate a text. These models differ from an encoder-decoder NMT system in a number of ways: [35] : 1 

A generative LLM can be prompted in a zero-shot fashion by just asking it to translate a text into another language without giving any further examples in the prompt. Or one can include one or several example translations in the prompt before asking to translate the text in question. This is then called one-shot or few-shot learning, respectively. For example, the following prompts were used by Hendy et al. (2023) for zero-shot and one-shot translation: [35] 

### Translate this sentence from [source language] to [target language], Source: [source sentence] ### Target: 
Translate this into 1. [target language]: [shot 1 source] 1. [shot 1 reference] Translate this into 1. [target language]: [input] 1.

Literature

See also

Related Research Articles

Pattern recognition is the task of assigning a class to an observation based on patterns extracted from data. While similar, pattern recognition (PR) is not to be confused with pattern machines (PM) which may possess (PR) capabilities but their primary function is to distinguish and create emergent patterns. PR has applications in statistical data analysis, signal processing, image analysis, information retrieval, bioinformatics, data compression, computer graphics and machine learning. Pattern recognition has its origins in statistics and engineering; some modern approaches to pattern recognition include the use of machine learning, due to the increased availability of big data and a new abundance of processing power.

In probability theory and statistics, a Gaussian process is a stochastic process, such that every finite collection of those random variables has a multivariate normal distribution. The distribution of a Gaussian process is the joint distribution of all those random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.

Random forests or random decision forests is an ensemble learning method for classification, regression and other tasks that works by creating a multitude of decision trees during training. For classification tasks, the output of the random forest is the class selected by most trees. For regression tasks, the output is the average of the predictions of the trees. Random forests correct for decision trees' habit of overfitting to their training set.

Recurrent neural networks (RNNs) are a class of artificial neural network commonly used for sequential data processing. Unlike feedforward neural networks, which process data in a single pass, RNNs process data across multiple time steps, making them well-adapted for modelling and processing text, speech, and time series.

<span class="mw-page-title-main">Autoencoder</span> Neural network that learns efficient data encoding in an unsupervised manner

An autoencoder is a type of artificial neural network used to learn efficient codings of unlabeled data. An autoencoder learns two functions: an encoding function that transforms the input data, and a decoding function that recreates the input data from the encoded representation. The autoencoder learns an efficient representation (encoding) for a set of data, typically for dimensionality reduction, to generate lower-dimensional embeddings for subsequent use by other machine learning algorithms.

Multimodal learning is a type of deep learning that integrates and processes multiple types of data, referred to as modalities, such as text, audio, images, or video. This integration allows for a more holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval, text-to-image generation, aesthetic ranking, and image captioning.

Google Neural Machine Translation (GNMT) was a neural machine translation (NMT) system developed by Google and introduced in November 2016 that used an artificial neural network to increase fluency and accuracy in Google Translate. The neural network consisted of two main blocks, an encoder and a decoder, both of LSTM architecture with 8 1024-wide layers each and a simple 1-layer 1024-wide feedforward attention mechanism connecting them. The total number of parameters has been variously described as over 160 million, approximately 210 million, 278 million or 380 million. It used WordPiece tokenizer, and beam search decoding strategy. It ran on Tensor Processing Units.

Mixture of experts (MoE) is a machine learning technique where multiple expert networks (learners) are used to divide a problem space into homogeneous regions. MoE represents a form of ensemble learning.

Paraphrase or paraphrasing in computational linguistics is the natural language processing task of detecting and generating paraphrases. Applications of paraphrasing are varied including information retrieval, question answering, text summarization, and plagiarism detection. Paraphrasing is also useful in the evaluation of machine translation, as well as semantic parsing and generation of new samples to expand existing corpora.

<span class="mw-page-title-main">Transformer (deep learning architecture)</span> Deep learning architecture for modelling sequential data

A transformer is a deep learning architecture developed by researchers at Google and based on the multi-head attention mechanism, proposed in the 2017 paper "Attention Is All You Need". Text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished.

Bidirectional encoder representations from transformers (BERT) is a language model introduced in October 2018 by researchers at Google. It learns to represent text as a sequence of vectors using self-supervised learning. It uses the encoder-only transformer architecture. It is notable for its dramatic improvement over previous state-of-the-art models, and as an early example of a large language model. As of 2020, BERT is a ubiquitous baseline in natural language processing (NLP) experiments.

<span class="mw-page-title-main">Variational autoencoder</span> Deep learning generative model to encode data representation

In machine learning, a variational autoencoder (VAE) is an artificial neural network architecture introduced by Diederik P. Kingma and Max Welling. It is part of the families of probabilistic graphical models and variational Bayesian methods.

In machine learning, knowledge distillation or model distillation is the process of transferring knowledge from a large model to a smaller one. While large models have more knowledge capacity than small models, this capacity might not be fully utilized. It can be just as computationally expensive to evaluate a model even if it utilizes little of its knowledge capacity. Knowledge distillation transfers knowledge from a large model to a smaller one without loss of validity. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware.

In the study of artificial neural networks (ANNs), the neural tangent kernel (NTK) is a kernel that describes the evolution of deep artificial neural networks during their training by gradient descent. It allows ANNs to be studied using theoretical tools from kernel methods.

<span class="mw-page-title-main">Attention (machine learning)</span> Machine learning technique

Attention is a machine learning method that determines the relative importance of each component in a sequence relative to the other components in that sequence. In natural language processing, importance is represented by "soft" weights assigned to each word in a sentence. More generally, attention encodes vectors called token embeddings across a fixed-width sequence that can range from tens to millions of tokens in size.

<span class="mw-page-title-main">Vision transformer</span> Machine learning model for vision processing

A vision transformer (ViT) is a transformer designed for computer vision. A ViT decomposes an input image into a series of patches, serializes each patch into a vector, and maps it to a smaller dimension with a single matrix multiplication. These vector embeddings are then processed by a transformer encoder as if they were token embeddings.

Prompt engineering is the process of structuring an instruction that can be interpreted and understood by a generative artificial intelligence (AI) model. A prompt is natural language text describing the task that an AI should perform. A prompt for a text-to-text language model can be a query such as "what is Fermat's little theorem?", a command such as "write a poem in the style of Edgar Allan Poe about leaves falling", or a longer statement including context, instructions, and conversation history.

In machine learning, diffusion models, also known as diffusion probabilistic models or score-based generative models, are a class of latent variable generative models. A diffusion model consists of three major components: the forward process, the reverse process, and the sampling procedure. The goal of diffusion models is to learn a diffusion process for a given dataset, such that the process can generate new elements that are distributed similarly as the original dataset. A diffusion model models data as generated by a diffusion process, whereby a new datum performs a random walk with drift through the space of all possible data. A trained diffusion model can be sampled in many ways, with different efficiency and quality.

<span class="mw-page-title-main">Neural scaling law</span> Law in machine learning

In machine learning, a neural scaling law is an empirical scaling law that describes how neural network performance changes as key factors are scaled up or down. These factors typically include the number of parameters, training dataset size, and training cost.

<span class="mw-page-title-main">Attention Is All You Need</span> 2017 research paper by Google

"Attention Is All You Need" is a 2017 landmark research paper in machine learning authored by eight scientists working at Google. The paper introduced a new deep learning architecture known as the transformer, based on the attention mechanism proposed in 2014 by Bahdanau et al. It is considered a foundational paper in modern artificial intelligence, as the transformer approach has become the main architecture of large language models like those based on GPT. At the time, the focus of the research was on improving Seq2seq techniques for machine translation, but the authors go further in the paper, foreseeing the technique's potential for other tasks like question answering and what is now known as multimodal Generative AI.

References

  1. 1 2 3 4 5 6 Koehn, Philipp (2020). Neural Machine Translation. Cambridge University Press.
  2. 1 2 3 4 5 6 7 Stahlberg, Felix (2020-09-29). "Neural Machine Translation: A Review and Survey". arXiv: 1912.02047v2 [cs.CL].
  3. ↑ Popel, Martin; Tomkova, Marketa; Tomek, Jakub; Kaiser, Łukasz; Uszkoreit, Jakob; Bojar, Ondřej; Žabokrtský, Zdeněk (2020-09-01). "Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals". Nature Communications. 11 (1): 4381. Bibcode:2020NatCo..11.4381P. doi:10.1038/s41467-020-18073-9. hdl: 11346/BIBLIO@id=368112263610994118 . ISSN   2041-1723. PMC   7463233 . PMID   32873773.
  4. 1 2 Haddow, Barry; Bawden, Rachel; Miceli Barone, Antonio Valerio; Helcl, Jindřich; Birch, Alexandra (2022). "Survey of Low-Resource Machine Translation". Computational Linguistics. 48 (3): 673–732. arXiv: 2109.00486 . doi:10.1162/coli_a_00446.
  5. 1 2 Poibeau, Thierry (2022). Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente (eds.). "On "Human Parity" and "Super Human Performance" in Machine Translation Evaluation". Proceedings of the Thirteenth Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association: 6018–6023.
  6. 1 2 3 4 Tan, Zhixing; Wang, Shuo; Yang, Zonghan; Chen, Gang; Huang, Xuancheng; Sun, Maosong; Liu, Yang (2020-12-31). "Neural Machine Translation: A Review of Methods, Resources, and Tools". arXiv: 2012.15515 [cs.CL].
  7. ↑ Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). "12.4.5 Neural Machine Translation". Deep Learning. MIT Press. pp. 468–471. Retrieved 2022-12-29.
  8. ↑ Allen, Robert B. (1987). Several Studies on Natural Language and Back-Propagation. IEEE First International Conference on Neural Networks. Vol. 2. San Diego. pp. 335–341. Retrieved 2022-12-30.
  9. ↑ Chrisman, Lonnie (1991). "Learning Recursive Distributed Representations for Holistic Computation". Connection Science. 3 (4): 345–366. doi:10.1080/09540099108946592. ISSN   0954-0091.
  10. ↑ Pollack, Jordan B. (1990). "Recursive distributed representations". Artificial Intelligence. 46 (1): 77–105. doi:10.1016/0004-3702(90)90005-K.
  11. ↑ Forcada, Mikel L.; Ñeco, Ramón P. (1997). "Recursive hetero-associative memories for translation". Biological and Artificial Computation: From Neuroscience to Technology. Lecture Notes in Computer Science. Vol. 1240. pp. 453–462. doi:10.1007/BFb0032504. ISBN   978-3-540-63047-0.
  12. ↑ Castaño, Asunción; Casacuberta, Francisco (1997). A connectionist approach to machine translation. 5th European Conference on Speech Communication and Technology (Eurospeech 1997). Rhodes, Greece. pp. 91–94. doi:10.21437/Eurospeech.1997-50.
  13. ↑ Castaño, Asunción; Casacuberta, Francisco; Vidal, Enrique (1997-07-23). Machine translation using neural networks and finite-state models. Proceedings of the 7th Conference on Theoretical and Methodological Issues in Machine Translation of Natural Languages. St John's College, Santa Fe.
  14. ↑ Yang, Shuoheng; Wang, Yuxin; Chu, Xiaowen (2020-02-18). "A Survey of Deep Learning Techniques for Neural Machine Translation". arXiv: 2002.07526 [cs.CL].
  15. ↑ Schwenk, Holger; Dechelotte, Daniel; Gauvain, Jean-Luc (2006). Continuous Space Language Models for Statistical Machine Translation. Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions. Sydney, Australia. pp. 723–730.
  16. ↑ Schwenk, Holger (2007). "Contiuous space language models". Computer Speech and Language. 3 (21): 492–518. doi:10.1016/j.csl.2006.09.003.
  17. ↑ Schwenk, Holger (2012). Continuous Space Translation Models for Phrase-Based Statistical Machine Translation. Proceedings of COLING 2012: Posters. Mumbai, India. pp. 1071–1080.
  18. 1 2 Kalchbrenner, Nal; Blunsom, Philip (2013). "Recurrent Continuous Translation Models". Proceedings of the Association for Computational Linguistics: 1700–1709.
  19. ↑ Cho, Kyunghyun; van Merriënboer, Bart; Gulcehre, Caglar; Bahdanau, Dzmitry; Bougares, Fethi; Schwenk, Holger; Bengio, Yoshua (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics. pp. 1724–1734. arXiv: 1406.1078 . doi:10.3115/v1/D14-1179.
  20. ↑ Sutskever, Ilya; Vinyals, Oriol; Le, Quoc V. (2014). "Sequence to Sequence Learning with Neural Networks". Advances in Neural Information Processing Systems. 27. Curran Associates, Inc. arXiv: 1409.3215 .
  21. ↑ Cho, Kyunghyun; van Merriënboer, Bart; Bahdanau, Dzmitry; Bengio, Yoshua (2014). On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Doha, Qatar: Association for Computational Linguistics. pp. 103–111. arXiv: 1409.1259 . doi:10.3115/v1/W14-4012.
  22. ↑ Bahdanau, Dzmitry; Cho, Kyunghyun; Bengio, Yoshua (2014). "Neural Machine Translation by Jointly Learning to Align and Translate". arXiv: 1409.0473 [cs.CL].
  23. 1 2 Wang, Haifeng; Wu, Hua; He, Zhongjun; Huang, Liang; Church, Kenneth Ward (2022-11-01). "Progress in Machine Translation". Engineering. 18: 143–153. Bibcode:2022Engin..18..143W. doi:10.1016/j.eng.2021.03.023.
  24. ↑ Wu, Yonghui; Schuster, Mike; Chen, Zhifeng; Le, Quoc V.; Norouzi, Mohammad; Macherey, Wolfgang; Krikun, Maxim; Cao, Yuan; Gao, Qin; Macherey, Klaus; Klingner, Jeff; Shah, Apurva; Johnson, Melvin; Liu, Xiaobing; Kaiser, Łukasz (2016). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation". arXiv: 1609.08144 [cs.CL].
  25. ↑ Bojar, Ondrej; Chatterjee, Rajen; Federmann, Christian; Graham, Yvette; Haddow, Barry; Huck, Matthias; Yepes, Antonio Jimeno; Koehn, Philipp; Logacheva, Varvara; Monz, Christof; Negri, Matteo; Névéol, Aurélie; Neves, Mariana; Popel, Martin; Post, Matt; Rubino, Raphael; Scarton, Carolina; Specia, Lucia; Turchi, Marco; Verspoor, Karin; Zampieri, Marcos (2016). "Findings of the 2016 Conference on Machine Translation" (PDF). ACL 2016 First Conference on Machine Translation (WMT16). The Association for Computational Linguistics: 131–198. Archived from the original (PDF) on 2018-01-27. Retrieved 2018-01-27.
  26. ↑ Gehring, Jonas; Auli, Michael; Grangier, David; Dauphin, Yann (2017). A Convolutional Encoder Model for Neural Machine Translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics. pp. 123–135. arXiv: 1611.02344 . doi:10.18653/v1/P17-1012.
  27. ↑ Translator, Microsoft (2018-04-18). "Microsoft brings AI-powered translation to end users and developers, whether you're online or offline". Microsoft Translator Blog. Retrieved 2024-04-19.{{cite web}}: |last= has generic name (help)
  28. ↑ Coldewey, Devin (2017-08-29). "DeepL schools other online translators with clever machine learning". TechCrunch. Retrieved 2023-12-26.
  29. ↑ Leloup, Damien; Larousserie, David (2022-08-29). "Quel est le meilleur service de traduction en ligne?". Le Monde. Retrieved 2023-01-10.
  30. ↑ Pakalski, Ingo (2017-08-29). "DeepL im Hands On: Neues Tool übersetzt viel besser als Google und Microsoft". Golem. Retrieved 2023-01-10.
  31. 1 2 Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Gomez, Aidan N.; Kaiser, Łukasz; Polosukhin, Illia (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017). pp. 5998–6008.
  32. ↑ Kocmi, Tom; Bawden, Rachel; Bojar, Ondřej; Dvorkovich, Anton; Federmann, Christian; Fishel, Mark; Gowda, Thamme; Graham, Yvette; Grundkiewicz, Roman; Haddow, Barry; Knowles, Rebecca; Koehn, Philipp; Monz, Christof; Morishita, Makoto; Nagata, Masaaki (2022). Koehn, Philipp; Barrault, Loïc; Bojar, Ondřej; Bougares, Fethi; Chatterjee, Rajen; Costa-jussà, Marta R.; Federmann, Christian; Fishel, Mark; Fraser, Alexander (eds.). Findings of the 2022 Conference on Machine Translation (WMT22). Proceedings of the Seventh Conference on Machine Translation (WMT). Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics. pp. 1–45.
  33. 1 2 Kocmi, Tom; Avramidis, Eleftherios; Bawden, Rachel; Bojar, Ondřej; Dvorkovich, Anton; Federmann, Christian; Fishel, Mark; Freitag, Markus; Gowda, Thamme; Grundkiewicz, Roman; Haddow, Barry; Koehn, Philipp; Marie, Benjamin; Monz, Christof; Morishita, Makoto (2023). Koehn, Philipp; Haddow, Barry; Kocmi, Tom; Monz, Christof (eds.). Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet. Proceedings of the Eighth Conference on Machine Translation. Singapore: Association for Computational Linguistics. pp. 1–42. doi: 10.18653/v1/2023.wmt-1.1 .
  34. 1 2 Liu, Yinhan; Gu, Jiatao; Goyal, Naman; Li, Xian; Edunov, Sergey; Ghazvininejad, Marjan; Lewis, Mike; Zettlemoyer, Luke (2020). "Multilingual Denoising Pre-training for Neural Machine Translation". Transactions of the Association for Computational Linguistics. 8: 726–742. arXiv: 2001.08210 . doi:10.1162/tacl_a_00343.
  35. 1 2 3 Hendy, Amr; Abdelrehim, Mohamed; Sharaf, Amr; Raunak, Vikas; Gabr, Mohamed; Matsushita, Hitokazu; Kim, Young Jin; Afify, Mohamed; Awadalla, Hany (2023-02-18). "How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation". arXiv: 2302.09210 [cs.CL].
  36. 1 2 "GPT 3 dataset statistics: languages by character count". OpenAI. 2020-06-01. Retrieved 2023-12-23.
  37. 1 2 Russell, Stuart; Norvig, Peter. Artificial Intelligence: A Modern Approach (4th, global ed.). Pearson.
  38. ↑ Federico, Marcello; Cettolo, Mauro (2007). Callison-Burch, Chris; Koehn, Philipp; Fordyce, Cameron Shaw; Monz, Christof (eds.). "Efficient Handling of N-gram Language Models for Statistical Machine Translation". Proceedings of the Second Workshop on Statistical Machine Translation. Prague, Czech Republic: Association for Computational Linguistics: 88–95. doi:10.3115/1626355.1626367.
  39. ↑ Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya (2018). Improving Language Understanding by Generative Pre-Training (PDF) (Technical report). OpenAI. Retrieved 2023-12-26.
  40. ↑ Brown, Tom; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, Jared D; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini; Herbert-Voss, Ariel; Krueger, Gretchen; Henighan, Tom; Child, Rewon (2020). "Language Models are Few-Shot Learners". Advances in Neural Information Processing Systems. 33. Curran Associates, Inc.: 1877–1901.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.