Llama (language model)

Last updated
Llama
Developer(s) Meta AI
Initial releaseFebruary 24, 2023;20 months ago (2023-02-24)
Stable release
Llama 3.2 / September 25, 2024;51 days ago (2024-09-25)
Repository github.com/meta-llama/llama-models
Written in Python
Type
License Source available (Meta Llama 3.2 Community License) [1]
Website llama.com

Llama (Large Language Model Meta AI, formerly stylized as LLaMA) is a family of autoregressive large language models (LLMs) released by Meta AI starting in February 2023. [2] [3] The latest version is Llama 3.2, released in September 2024. [4]

Contents

Model weights for the first version of Llama were made available to the research community under a non-commercial license, and access was granted on a case-by-case basis. [5] [3] Unauthorized copies of the model were shared via BitTorrent. In response, Meta AI issued DMCA takedown requests against repositories sharing the link on GitHub. [6] [7] Subsequent versions of Llama were made accessible outside academia and released under licenses that permitted some commercial use. [8] [9] Llama models are trained at different parameter sizes, ranging between 1B and 405B. [10] Originally, Llama was only available as a foundation model. [11] Starting with Llama 2, Meta AI started releasing instruction fine-tuned versions alongside foundation models. [9]

Alongside the release of Llama 3, Meta added virtual assistant features to Facebook and WhatsApp in select regions, and a standalone website. Both services use a Llama 3 model. [12]

Background

After the release of large language models such as GPT-3, a focus of research was up-scaling models which in some instances showed major increases in emergent capabilities. [13] The release of ChatGPT and its surprise success caused an increase in attention to large language models. [14]

Compared with other responses to ChatGPT, Meta's Chief AI scientist Yann LeCun stated that large language models are best for aiding with writing. [15] [16] [17]

An empirical investigation of the Llama series was the scaling laws. It was observed that the Llama 3 models showed that when a model is trained on data that is more than the "Chinchilla-optimal" amount, the performance continues to scale log-linearly. For example, the Chinchilla-optimal dataset for Llama 3 8B is 200 billion tokens, but performance continued to scale log-linearly to the 75-times larger dataset of 15 trillion tokens. [18]

Initial release

LLaMA was announced on February 24, 2023, via a blog post and a paper describing the model's training, architecture, and performance. [2] [3] The inference code used to run the model was publicly released under the open-source GPLv3 license. [19] Access to the model's weights was managed by an application process, with access to be granted "on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world". [3]

Llama was trained on only publicly available information, and was trained at various model sizes, with the intention to make it more accessible to different hardware.

Meta AI reported the 13B parameter model performance on most NLP benchmarks exceeded that of the much larger GPT-3 (with 175B parameters), and the largest 65B model was competitive with state of the art models such as PaLM and Chinchilla. [2]

Leak

On March 3, 2023, a torrent containing LLaMA's weights was uploaded, with a link to the torrent shared on the 4chan imageboard and subsequently spread through online AI communities. [6] That same day, a pull request on the main LLaMA repository was opened, requesting to add the magnet link to the official documentation. [20] [21] On March 4, a pull request was opened to add links to HuggingFace repositories containing the model. [22] [20] On March 6, Meta filed takedown requests to remove the HuggingFace repositories linked in the pull request, characterizing it as "unauthorized distribution" of the model. HuggingFace complied with the requests. [23] On March 20, Meta filed a DMCA takedown request for copyright infringement against a repository containing a script that downloaded LLaMA from a mirror, and GitHub complied the next day. [7]

Reactions to the leak varied. Some speculated that the model would be used for malicious purposes, such as more sophisticated spam. Some have celebrated the model's accessibility, as well as the fact that smaller versions of the model can be run relatively cheaply, suggesting that this will promote the flourishing of additional research developments. [6] Multiple commentators, such as Simon Willison, compared LLaMA to Stable Diffusion, a text-to-image model which, unlike comparably sophisticated models which preceded it, was openly distributed, leading to a rapid proliferation of associated tools, techniques, and software. [6] [24]

Llama 2

On July 18, 2023, in partnership with Microsoft, Meta announced Llama 2, the next generation of Llama. Meta trained and released Llama 2 in three model sizes: 7, 13, and 70 billion parameters. [9] The model architecture remains largely unchanged from that of LLaMA-1 models, but 40% more data was used to train the foundational models. [25] The accompanying preprint [25] also mentions a model with 34B parameters that might be released in the future upon satisfying safety targets.

Llama 2 includes foundation models and models fine-tuned for chat. In a further departure from LLaMA, all models are released with weights and are free for many commercial use cases. However, due to some remaining restrictions, Meta's description of LLaMA as open source has been disputed by the Open Source Initiative (known for maintaining the Open Source Definition). [26]

Code Llama is a fine-tune of Llama 2 with code specific datasets. 7B, 13B, and 34B versions were released on August 24, 2023, with the 70B releasing on the January 29, 2024. [27] Starting with the foundation models from Llama 2, Meta AI would train an additional 500B tokens of code datasets, before an additional 20B token of long-context data, creating the Code Llama foundation models. This foundation model was further trained on 5B instruction following token to create the instruct fine-tune. Another foundation model was created for Python code, which trained on 100B tokens of Python-only code, before the long-context data. [28]

Llama 3

On April 18, 2024, Meta released Llama-3 with two sizes: 8B and 70B parameters. [18] The models have been pre-trained on approximately 15 trillion tokens of text gathered from “publicly available sources” with the instruct models fine-tuned on “publicly available instruction datasets, as well as over 10M human-annotated examples". Meta AI's testing showed in April 2024 that Llama 3 70B was beating Gemini pro 1.5 and Claude 3 Sonnet on most benchmarks. Meta also announced plans to make Llama 3 multilingual and multimodal, better at coding and reasoning, and to increase its context window. [29] [30]

During an interview with Dwarkesh Patel, Mark Zuckerberg said that the 8B version of Llama 3 was nearly as powerful as the largest Llama 2. Compared to previous models, Zuckerberg stated the team was surprised that the 70B model was still learning even at the end of the 15T tokens training. The decision was made to end training to focus GPU power elsewhere. [31]

Llama-3.1 was released on July 23, 2024, with three sizes: 8B, 70B, and 405B parameters. [10] [32]

Comparison of models

For the training cost column, only the largest model's cost is written. So for example, "21,000" is the training cost of Llama 2 69B in units of petaFLOP-day. Also, 1 petaFLOP-day = 1 petaFLOP/sec × 1 day = 8.64E19 FLOP. "T" means "trillion" and "B" means "billion".

NameRelease dateParametersTraining cost (petaFLOP-day)Context length (tokens)Corpus size (tokens)Commercial viability?
LLaMAFebruary 24, 2023
  • 6.7B
  • 13B
  • 32.5B
  • 65.2B
6,300 [33] 20481–1.4TNo
Llama 2July 18, 2023
  • 6.7B
  • 13B
  • 69B
21,000 [34] 40962TYes
Code LlamaAugust 24, 2023
  • 6.7B
  • 13B
  • 33.7B
  • 69B
Llama 3April 18, 2024
  • 8B
  • 70.6B
100,000 [35] [36] 819215T
Llama 3.1July 23, 2024
  • 8B
  • 70.6B
  • 405B
440,000 [32] [37] 128,000
Llama 3.2September 25, 2024128,000 [40]

Architecture and training

"recommendation letter for the Magic Unicorn Corporation"

Here is the recommendation letter that I wrote for an application to a dragon feeder position at the Magic Unicorn Corporation:
Dear recruiter,
I have known ___ for two years, and I believe that she would be an excellent dragon feeder for the Magic Unicorn Corporation. ___ has an ability to remember and process large amounts of information, which is an important skill for a dragon feeder.
___, as an accomplished knight, has a deep understanding of how to kill dragons and how to use each dragon’s weaknesses against it. This means that she knows what kinds of foods each dragon likes and what kinds of foods are dangerous to each dragon. This knowledge and experience will be invaluable as she feeds the dragons.
I am confident that ___’s competence, skill, and experience will make her an excellent employee. Please contact me at (___) ___-___ if you have any questions. I look forward to hearing from you.
Best regards,
Honorable Knight
Sir George

– Output of 65 billion parameter LLaMA model before instruction tuning, given the prompt (in bold) [2]

Architecture

Like GPT-3, the Llama series of models are decoder-only Transformers, but there are some minor differences:

key hyperparameters of Llama 3.1
8B70B405B
Layers3280126
Model Dimension4,096819216,384
FFN Dimension14,33628,67253,248
Attention Heads3264128
Key/Value Heads888
Peak Learning Rate3 × 10−41.5 × 10−40.8 × 10−4
Activation FunctionSwiGLU
Vocabulary Size128,000
Positional Embeddings

Training datasets

LLaMA's developers focused their effort on scaling the model's performance by increasing the volume of training data, rather than the number of parameters, reasoning that the dominating cost for LLMs is from doing inference on the trained model rather than the computational cost of the training process.

LLaMA 1 foundational models were trained on a data set with 1.4 trillion tokens, drawn from publicly available data sources, including: [2]

On April 17, 2023, TogetherAI launched a project named RedPajama to reproduce and distribute an open source version of the LLaMA dataset. [45] The dataset has approximately 1.2 trillion tokens and is publicly available for download. [46]

Llama 2 foundational models were trained on a data set with 2 trillion tokens. This data set was curated to remove Web sites that often disclose personal data of people. It also upsamples sources considered trustworthy. [25] Llama 2 - Chat was additionally fine-tuned on 27,540 prompt-response pairs created for this project, which performed better than larger but lower-quality third-party datasets. For AI alignment, reinforcement learning with human feedback (RLHF) was used with a combination of 1,418,091 Meta examples and seven smaller datasets. The average dialog depth was 3.9 in the Meta examples, 3.0 for Anthropic Helpful and Anthropic Harmless sets, and 1.0 for five other sets, including OpenAI Summarize, StackExchange, etc.

Llama 3 consists of mainly English data, with over 5% in over 30 other languages. Its dataset was filtered by a text-quality classifier, and the classifier was trained by text synthesized by Llama 2. [18]

Fine-tuning

Llama 1 models are only available as foundational models with self-supervised learning and without fine-tuning. Llama 2 – Chat models were derived from foundational Llama 2 models. Unlike GPT-4 which increased context length during fine-tuning, Llama 2 and Code Llama - Chat have the same context length of 4K tokens. Supervised fine-tuning used an autoregressive loss function with token loss on user prompts zeroed out. The batch size was 64.

For AI alignment, human annotators wrote prompts and then compared two model outputs (a binary protocol), giving confidence levels and separate safety labels with veto power. Two separate reward models were trained from these preferences for safety and helpfulness using Reinforcement learning from human feedback (RLHF). A major technical contribution is the departure from the exclusive use of Proximal Policy Optimization (PPO) for RLHF – a new technique based on Rejection sampling was used, followed by PPO.

Multi-turn consistency in dialogs was targeted for improvement, to make sure that "system messages" (initial instructions, such as "speak in French" and "act like Napoleon") are respected during the dialog. This was accomplished using the new "Ghost attention" technique during training, which concatenates relevant instructions to each new user message but zeros out the loss function for tokens in the prompt (earlier parts of the dialog).

Applications

The Stanford University Institute for Human-Centered Artificial Intelligence (HAI) Center for Research on Foundation Models (CRFM) released Alpaca, a training recipe based on the LLaMA 7B model that uses the "Self-Instruct" method of instruction tuning to acquire capabilities comparable to the OpenAI GPT-3 series text-davinci-003 model at a modest cost. [47] [48] [49] The model files were officially removed on March 21, 2023, over hosting costs and safety concerns, though the code and paper remain online for reference. [50] [51] [52]

Meditron is a family of Llama-based finetuned on a corpus of clinical guidelines, PubMed papers, and articles. It was created by researchers at École Polytechnique Fédérale de Lausanne School of Computer and Communication Sciences, and the Yale School of Medicine. It shows increased performance on medical-related benchmarks such as MedQA and MedMCQA. [53] [54] [55]

Zoom used Meta Llama 2 to create an AI Companion that can summarize meetings, provide helpful presentation tips, and assist with message responses. This AI Companion is powered by multiple models, including Meta Llama 2. [56]

Reuters reported in 2024 that many Chinese foundation models relied on Llama models for their training. [57]

llama.cpp

Software developer Georgi Gerganov released llama.cpp as open-source on March 10, 2023. It's a re-implementation of LLaMA in C++, allowing systems without a powerful GPU to run the model locally. [58] The llama.cpp project introduced the GGUF file format, a binary format that stores both tensors and metadata. [59] The format focuses on supporting different quantization types, which can reduce memory usage, and increase speed at the expense of lower model precision. [60]

llamafile created by Justine Tunney is an open-source tool that bundles llama.cpp with the model into a single executable file. Tunney et al. introduced new optimized matrix multiplication kernels for x86 and ARM CPUs, improving prompt evaluation performance for FP16 and 8-bit quantized data types. [61]

Military

In 2024, researchers from the People's Liberation Army Academy of Military Sciences (top military academy of China) were reported to have developed a military tool using Llama, which Meta Platforms stated was unauthorized due to its model use prohibition for military purposes. [62] [63]

Reception

Wired describes the 8B parameter version of Llama 3 as being "surprisingly capable" given its size. [64]

The response to Meta's integration of Llama into Facebook was mixed, with some users confused after Meta AI told a parental group that it had a child. [65]

According to the Q4 2023 Earnings transcript, Meta adopted the strategy of open weights to improve on model safety, iteration speed, increase adoption among developers and researchers, and to become the industry standard. Llama 5, 6, and 7 are planned for the future. [66]

The release of Llama models has sparked significant debates on the benefits and misuse risks of open weight models. Such models can be fine-tuned to remove safeguards, notably by cyber criminals, until they comply with harmful requests. Some experts contend that future models may facilitate causing damage more than defending against it, for example by making it relatively easy to engineer advanced bioweapons without specialized knowledge. Conversely, open-weight models can be useful for a wide variety of purposes, including for safety research. [67] Open Source Initiative head Stefano Maffulli criticized Meta for describing Llama as open source, saying that it was causing confusion among users and "polluting" the term. [68]

See also

Related Research Articles

Multimodal learning is a type of deep learning that integrates and processes multiple types of data, referred to as modalities, such as text, audio, images, or video. This integration allows for a more holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval, text-to-image generation, aesthetic ranking, and image captioning.

Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.

GitHub Copilot is a code completion and automatic programming tool developed by GitHub and OpenAI that assists users of Visual Studio Code, Visual Studio, Neovim, and JetBrains integrated development environments (IDEs) by autocompleting code. Currently available by subscription to individual developers and to businesses, the generative artificial intelligence software was first announced by GitHub on 29 June 2021, and works best for users coding in Python, JavaScript, TypeScript, Ruby, and Go. In March 2023 GitHub announced plans for "Copilot X", which will incorporate a chatbot based on GPT-4, as well as support for voice commands, into Copilot.

Prompt engineering is the process of structuring an instruction that can be interpreted and understood by a generative artificial intelligence (AI) model. A prompt is natural language text describing the task that an AI should perform. A prompt for a text-to-text language model can be a query such as "what is Fermat's little theorem?", a command such as "write a poem in the style of Edgar Allan Poe about leaves falling", or a longer statement including context, instructions, and conversation history.

A foundation model, also known as large AI model, is a machine learning or deep learning model that is trained on vast datasets so it can be applied across a wide range of use cases. Generative AI applications like Large Language Models are often examples of foundation models.

Hugging Face, Inc. is an American company incorporated under the Delaware General Corporation Law and based in New York City that develops computation tools for building applications using machine learning. It is most notable for its transformers library built for natural language processing applications and its platform that allows users to share machine learning models and datasets and showcase their work.

Generative Pre-trained Transformer 4 (GPT-4) is a multimodal large language model created by OpenAI, and the fourth in its series of GPT foundation models. It was launched on March 14, 2023, and made publicly available via the paid chatbot product ChatGPT Plus, via OpenAI's API, and via the free chatbot Microsoft Copilot. As a transformer-based model, GPT-4 uses a paradigm where pre-training using both public data and "data licensed from third-party providers" is used to predict the next token. After this step, the model was then fine-tuned with reinforcement learning feedback from humans and AI for human alignment and policy compliance.

<span class="mw-page-title-main">Generative pre-trained transformer</span> Type of large language model

A generative pre-trained transformer (GPT) is a type of large language model (LLM) and a prominent framework for generative artificial intelligence. It is an artificial neural network that is used in natural language processing by machines. It is based on the transformer deep learning architecture, pre-trained on large data sets of unlabeled text, and able to generate novel human-like content. As of 2023, most LLMs had these characteristics and are sometimes referred to broadly as GPTs.

<span class="mw-page-title-main">GPT-J</span> Open source artificial intelligence text generating language model developed by EleutherAI

GPT-J or GPT-J-6B is an open-source large language model (LLM) developed by EleutherAI in 2021. As the name suggests, it is a generative pre-trained transformer model designed to produce human-like text that continues from a prompt. The optional "6B" in the name refers to the fact that it has 6 billion parameters.

A large language model (LLM) is a type of computational model designed for natural language processing tasks such as language generation. As language models, LLMs acquire these abilities by learning statistical relationships from vast amounts of text during a self-supervised and semi-supervised training process.

In deep learning, fine-tuning is an approach to transfer learning in which the parameters of a pre-trained neural network model are trained on new data. Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen". A model may also be augmented with "adapters" that consist of far fewer parameters than the original model, and fine-tuned in a parameter-efficient way by tuning the weights of the adapters and leaving the rest of the model's weights frozen.

The Pile is an 886.03 GB diverse, open-source dataset of English text created as a training dataset for large language models (LLMs). It was constructed by EleutherAI in 2020 and publicly released on December 31 of that year. It is composed of 22 smaller datasets, including 14 new ones.

<span class="mw-page-title-main">PaLM</span> Large language model developed by Google

PaLM is a 540 billion-parameter transformer-based large language model (LLM) developed by Google AI. Researchers also trained smaller versions of PaLM to test the effects of model scale.

Open-source artificial intelligence is an AI system that is freely available to use, study, modify, and share. These attributes extend to each of the system’s components, including datasets, code, and model parameters, promoting a collaborative and transparent approach to AI development.

<span class="mw-page-title-main">Gemini (language model)</span> Large language model developed by Google

Gemini is a family of multimodal large language models developed by Google DeepMind, serving as the successor to LaMDA and PaLM 2. Comprising Gemini Ultra, Gemini Pro, Gemini Flash, and Gemini Nano, it was announced on December 6, 2023, positioned as a competitor to OpenAI's GPT-4. It powers the chatbot of the same name.

<span class="mw-page-title-main">Mistral AI</span> French artificial intelligence company

Mistral AI is a French company specializing in artificial intelligence (AI) products. Founded in April 2023 by former employees of Meta Platforms and Google DeepMind, the company has quickly risen to prominence in the AI sector.

<span class="mw-page-title-main">IBM Granite</span> 2023 text-generating language model

IBM Granite is a series of decoder-only AI foundation models created by IBM. It was announced on September 7, 2023, and an initial paper was published 4 days later. Initially intended for use in the IBM's cloud-based data and generative AI platform Watsonx along with other models, IBM opened the source code of some code models. Granite models are trained on datasets curated from Internet, academic publishings, code datasets, legal and finance documents.

DBRX is an open-sourced large language model (LLM) developed by Mosaic ML team at Databricks, released on March 27, 2024. It is a mixture-of-experts Transformer model, with 132 billion parameters in total. 36 billion parameters are active for each token. The released model comes in either a base foundation model version or an instruct-tuned variant.

llama.cpp Software library for LLM inference

llama.cpp is an open source software library written mostly in C++ that performs inference on various large language models such as Llama. It is co-developed alongside the GGML project, a general-purpose tensor library.

<span class="mw-page-title-main">01.AI</span> Artificial intelligence company

01.AI is an artificial intelligence (AI) company based in Beijing, China. It focuses on developing open source products.

References

  1. ↑ "llama-models/models/llama3_2/LICENSE at main · meta-llama/llama-models · GitHub". GitHub. Archived from the original on 2024-09-29. Retrieved 2024-10-20.
  2. 1 2 3 4 5 Touvron, Hugo; Lavril, Thibaut; Izacard, Gautier; Martinet, Xavier; Lachaux, Marie-Anne; Lacroix, Timothée; Rozière, Baptiste; Goyal, Naman; Hambro, Eric; Azhar, Faisal; Rodriguez, Aurelien; Joulin, Armand; Grave, Edouard; Lample, Guillaume (2023). "LLaMA: Open and Efficient Foundation Language Models". arXiv: 2302.13971 [cs.CL].
  3. 1 2 3 4 "Introducing LLaMA: A foundational, 65-billion-parameter large language model". Meta AI. 24 February 2023. Archived from the original on 3 March 2023. Retrieved 16 March 2023.
  4. ↑ Knight, Will. "Meta Releases Llama 3.2—and Gives Its AI a Voice". Wired. ISSN   1059-1028 . Retrieved 2024-09-25.
  5. ↑ Malik, Yuvraj; Paul, Katie (25 February 2023). "Meta heats up Big Tech's AI arms race with new language model". Reuters.
  6. 1 2 3 4 Vincent, James (8 March 2023). "Meta's powerful AI language model has leaked online — what happens now?". The Verge. Archived from the original on 3 November 2023. Retrieved 16 March 2023.
  7. 1 2 OpSec Online LLC (21 March 2023). "github/dmca - Notice of Claimed Infringement via Email". GitHub. Archived from the original on 10 April 2023. Retrieved 25 March 2023.
  8. ↑ David, Emilia (30 October 2023). "Meta's AI research head wants open source licensing to change". The Verge. Archived from the original on 14 September 2024. Retrieved 20 October 2024.
  9. 1 2 3 "Meta and Microsoft Introduce the Next Generation of LLaMA". Meta. 18 July 2023. Archived from the original on 14 September 2023. Retrieved 21 July 2023.
  10. 1 2 "Introducing Llama 3.1: Our most capable models to date". ai.meta.com. July 23, 2024. Archived from the original on 2024-07-23. Retrieved 2024-07-23.
  11. ↑ Peters, Jay; Vincent, James (24 February 2023). "Meta has a new machine learning language model to remind you it does AI too". The Verge.
  12. ↑ "Meet Your New Assistant: Meta AI, Built With Llama 3". Meta. 18 April 2024. Archived from the original on 7 October 2024. Retrieved 20 October 2024.
  13. ↑ "Examining Emergent Abilities in Large Language Models". hai.stanford.edu. 13 September 2022.
  14. ↑ "The inside story of how ChatGPT was built from the people who made it". MIT Technology Review. Archived from the original on 2023-03-03. Retrieved 2024-10-20.
  15. ↑ "ChatGPT is 'not particularly innovative,' and 'nothing revolutionary', says Meta's chief AI scientist". ZDNET. Archived from the original on 2023-02-17. Retrieved 2024-10-20.
  16. ↑ Badminton, Nik (13 February 2023). "Meta's Yann LeCun on auto-regressive Large Language Models (LLMs)". Futurist.com. Archived from the original on 22 July 2024. Retrieved 20 October 2024.
  17. ↑ "Yann LeCun on LinkedIn: My unwavering opinion on current (auto-regressive) LLMs". www.linkedin.com. Archived from the original on 2024-09-17. Retrieved 2024-10-20.
  18. 1 2 3 "Introducing Meta Llama 3: The most capable openly available LLM to date". ai.meta.com. April 18, 2024. Archived from the original on 2024-05-15. Retrieved 2024-04-21.
  19. ↑ "llama". GitHub. Archived from the original on 15 March 2023. Retrieved 16 March 2023.
  20. 1 2 VK, Anirudh (6 March 2023). "Meta's LLaMA Leaked to the Public, Thanks To 4chan". Analytics India Magazine. Archived from the original on 26 March 2023. Retrieved 17 March 2023.
  21. ↑ "Save bandwidth by using a torrent to distribute more efficiently by ChristopherKing42 · Pull Request #73 · facebookresearch/llama". GitHub. Archived from the original on 10 April 2023. Retrieved 25 March 2023.
  22. ↑ "Download weights from hugging face to help us save bandwidth by Jainam213 · Pull Request #109 · facebookresearch/llama". GitHub. Archived from the original on 21 March 2023. Retrieved 17 March 2023.
  23. ↑ Cox, Joseph (7 March 2023). "Facebook's Powerful Large Language Model Leaks Online". Vice. Archived from the original on 6 April 2023. Retrieved 17 March 2023.
  24. ↑ Willison, Simon (11 March 2023). "Large language models are having their Stable Diffusion moment". Simon Willison's Weblog. Archived from the original on 16 March 2023. Retrieved 16 March 2023.
  25. 1 2 3 Touvron, Hugo; Martin, Louis; et al. (18 Jul 2023). "LLaMA-2: Open Foundation and Fine-Tuned Chat Models". arXiv: 2307.09288 [cs.CL].
  26. ↑ Edwards, Benj (2023-07-18). "Meta launches LLaMA-2, a source-available AI model that allows commercial applications [Updated]". Ars Technica. Archived from the original on 2023-11-07. Retrieved 2023-08-08.
  27. ↑ "Introducing Code Llama, a state-of-the-art large language model for coding". ai.meta.com. Archived from the original on 2024-09-27. Retrieved 2024-10-20.
  28. ↑ Rozière, Baptiste; Gehring, Jonas; Gloeckle, Fabian; Sootla, Sten; Gat, Itai; Tan, Xiaoqing Ellen; Adi, Yossi; Liu, Jingyu; Sauvestre, Romain (2024-01-31). "Code Llama: Open Foundation Models for Code". arXiv: 2308.12950 [cs.CL].
  29. ↑ Wiggers, Kyle (18 April 2024). "Meta releases Llama 3, claims it's among the best open models available". TechCrunch. Archived from the original on 18 September 2024. Retrieved 20 October 2024.
  30. ↑ Mann, Tobias (April 19, 2024). "Meta debuts third-generation Llama large language model". The Register. Archived from the original on August 25, 2024. Retrieved October 20, 2024.
  31. ↑ Patel, Dwarkesh (2024-07-24). "Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus". www.dwarkeshpatel.com. Archived from the original on 2024-07-16. Retrieved 2024-08-01. the 8 billion is nearly as powerful as the biggest version of Llama 2 that we released [...] even by the end, it was... still learning right it's like we probably could have fed it more tokens and it would have gotten somewhat better but i mean at some point you know you're running a company you need to do these meta reasoning questions of [...] how do I want to spend our GPUs
  32. 1 2 Dubey, Abhimanyu; Jauhri, Abhinav; Pandey, Abhinav; Kadian, Abhishek; Al-Dahle, Ahmad; Letman, Aiesha; Mathur, Akhil; Schelten, Alan; Yang, Amy (2024-07-31), The Llama 3 Herd of Models, arXiv: 2407.21783
  33. ↑ "The Falcon has landed in the Hugging Face ecosystem". huggingface.co. Archived from the original on 2023-06-20. Retrieved 2023-06-20.
  34. ↑ "llama/MODEL_CARD.md at main · meta-llama/llama". GitHub. Archived from the original on 2024-05-28. Retrieved 2024-05-28.
  35. ↑ "Andrej Karpathy (Apr 18, 2024), The model card has some more interesting info too". Archived from the original on August 17, 2024. Retrieved October 20, 2024.
  36. ↑ "llama3/MODEL_CARD.md at main · meta-llama/llama3". GitHub. Archived from the original on 2024-05-21. Retrieved 2024-05-28.
  37. ↑ "llama-models/models/llama3_1/MODEL_CARD.md at main · meta-llama/llama-models". GitHub. Archived from the original on 2024-07-23. Retrieved 2024-07-23.
  38. ↑ Robison, Kylie (2024-09-25). "Meta releases its first open AI model that can process images". The Verge. Retrieved 2024-09-25.
  39. ↑ Wiggers, Kyle (2024-09-25). "Meta's Llama AI models get multimodal". TechCrunch. Archived from the original on 2024-09-25. Retrieved 2024-09-25.
  40. ↑ "Archived copy". ai.meta.com. Archived from the original on 2024-09-25. Retrieved 2024-09-26.{{cite web}}: CS1 maint: archived copy as title (link)
  41. ↑ Shazeer, Noam (2020-02-01). "GLU Variants Improve Transformer". arXiv: 2002.05202 [cs.CL].
  42. ↑ Su, Jianlin; Lu, Yu; Pan, Shengfeng; Murtadha, Ahmed; Wen, Bo; Liu, Yunfeng (2021-04-01). "RoFormer: Enhanced Transformer with Rotary Position Embedding". arXiv: 2104.09864 [cs.CL].
  43. ↑ Zhang, Biao; Sennrich, Rico (2019-10-01). "Root Mean Square Layer Normalization". arXiv: 1910.07467 [cs.LG].
  44. ↑ Lei Ba, Jimmy; Kiros, Jamie Ryan; Hinton, Geoffrey E. (2016-07-01). "Layer Normalization". arXiv: 1607.06450 [stat.ML].
  45. ↑ "RedPajama-Data: An Open Source Recipe to Reproduce LLaMA training dataset". GitHub. Together. Archived from the original on 7 November 2023. Retrieved 4 May 2023.
  46. ↑ "RedPajama-Data-1T". Hugging Face. Together. Archived from the original on 3 November 2023. Retrieved 4 May 2023.
  47. ↑ Taori, Rohan; Gulrajani, Ishaan; Zhang, Tianyi; Dubois, Yann; Li, Xuechen; Guestrin, Carlos; Liang, Percy; Hashimoto, Tatsunori B. (13 March 2023). "Alpaca: A Strong, Replicable Instruction-Following Model". Stanford Center for Research on Foundation Models. Archived from the original on 6 April 2023.
  48. ↑ Wang, Yizhong; Kordi, Yeganeh; Mishra, Swaroop; Liu, Alisa; Smith, Noah A.; Khashabi, Daniel; Hajishirzi, Hannaneh (2022). "Self-Instruct: Aligning Language Models with Self-Generated Instructions". arXiv: 2212.10560 [cs.CL].
  49. ↑ "Stanford CRFM". crfm.stanford.edu. Archived from the original on 2023-04-06. Retrieved 2023-03-20.
  50. ↑ Quach, Katyanna. "Stanford takes costly, risky Alpaca AI model offline". www.theregister.com.
  51. ↑ "Stanford Researchers Take Down Alpaca AI Over Cost and Hallucinations". Gizmodo. 21 March 2023. Archived from the original on 12 May 2024. Retrieved 20 October 2024.
  52. ↑ "alpaca-lora". GitHub. Archived from the original on 4 April 2023. Retrieved 5 April 2023.
  53. ↑ "Meditron: An LLM suite for low-resource medical settings leveraging Meta Llama". ai.meta.com.
  54. ↑ Petersen, Tanya (28 November 2023). "EPFL's new Large Language Model for Medical Knowledge". Archived from the original on 17 September 2024. Retrieved 20 October 2024.
  55. ↑ "epfLLM/meditron". epfLLM. 11 May 2024. Archived from the original on 27 September 2024. Retrieved 20 October 2024.
  56. ↑ "How Companies Are Using Meta Llama". Meta. 7 May 2024. Archived from the original on 27 September 2024. Retrieved 20 October 2024.
  57. ↑ "How dependent is China on US artificial intelligence technology?". Reuters. May 9, 2024.
  58. ↑ Edwards, Benj (2023-03-13). "You can now run a GPT-3-level AI model on your laptop, phone, and Raspberry Pi". Ars Technica. Archived from the original on 2024-01-09. Retrieved 2024-01-04.
  59. ↑ "GGUF". huggingface.co. Retrieved 9 May 2024.
  60. ↑ Labonne, Maxime (29 November 2023). "Quantize Llama models with GGUF and llama.cpp". Medium. Towards Data Science. Archived from the original on 9 May 2024. Retrieved 9 May 2024.
  61. ↑ Connatser, Matthew. "Llamafile LLM driver project boosts performance on CPU cores". www.theregister.com. Archived from the original on 10 May 2024. Retrieved 10 May 2024.
  62. ↑ Cheung, Sunny (October 31, 2024). "PRC Adapts Meta's Llama for Military and Security AI Applications". Jamestown Foundation . Retrieved 2024-11-03.
  63. ↑ Pomfret, James; Pang, Jessie (November 1, 2024). "Chinese researchers develop AI model for military use on back of Meta's Llama". Reuters . Retrieved November 1, 2024.
  64. ↑ Knight, Will. "Meta's Open Source Llama 3 Is Already Nipping at OpenAI's Heels". Wired. Archived from the original on 2024-09-27. Retrieved 2024-10-20.
  65. ↑ "Meta's amped-up AI agents confusing Facebook users". ABC News. 19 April 2024. Archived from the original on 2024-09-17. Retrieved 2024-10-20.
  66. ↑ "Archived copy" (PDF). Archived (PDF) from the original on 2024-09-17. Retrieved 2024-10-20.{{cite web}}: CS1 maint: archived copy as title (link)
  67. ↑ Knight, Will. "Meta's New Llama 3.1 AI Model Is Free, Powerful, and Risky". Wired. ISSN   1059-1028. Archived from the original on 2024-08-03. Retrieved 2024-08-04.
  68. ↑ Waters, Richard (October 17, 2024). "Meta under fire for 'polluting' open-source". Financial Times .

Further reading

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.