Mistral AI

Last updated

Mistral AI
Company typePrivate
Industry Artificial intelligence
Founded28 April 2023
Founders
  • Arthur Mensch
    (Co-Founder & CEO)
  • Guillaume Lample
    (Co-Founder & Chief Scientist)
  • Timothée Lacroix
    (Co-Founder & CTO)
Headquarters
Paris
,
France
Products
  • Mistral 7B
  • Mixtral 8x7B
  • Mistral Medium
  • Mistral Large
  • Mistral Large 2 (123B)
  • Mixtral 8x22B
  • Codestral 22B
  • Codestral Mamba (7B)
  • Mathstral (7B)
  • Mistral NeMo 12B
  • Mistral Embed
Website mistral.ai

Mistral AI, headquartered in Paris, France specializes in artificial intelligence (AI) products and focuses on open-weight large language models, [1] [2] (LLMs). Founded in April 2023 by former engineers from Google DeepMind [3] and Meta Platforms, the company has gained prominence as an alternative to proprietary AI systems. Named after the mistral  –a powerful, cold wind in southern France [4]  –the company emphasized openness and innovation in the AI field. Mistral AI positions itself as an alternative to proprietary models. [5]

Contents

In October 2023, Mistral AI raised €385 million. [6] By December 2023, it was valued at over $2 billion. [7] [8] [9]

In June 2024, Mistral AI secured a €600 million ($645 million) founding round, elevating its valuation to €5.8 billion ($6.2 billion). [10] Led by venture capital firm General Catalyst, [11] this round resulted in additional contributions from existing investors. The funds aim to support the company's expansion.

Mistral AI has published three open-source models available as weights. [12] Additionally, three more models –Small, Medium, and Large –are available via API only. [13] [14]

Based on valuation, the company is in fourth place in the global AI race and in first place outside the San Francisco Bay Area, ahead of several of its peers, such as Cohere, Hugging Face, Inflection, Perplexity and Together. [15] Mistral AI aims to "democratize" AI by focusing on open-source innovation. [16]

History

Mistral AI was established in April 2023 by three French AI researchers: Arthur Mensch, Guillaume Lample and Timothée Lacroix. [17] Mensch, a former researcher at Google DeepMind, brought expertise in advanced AI systems, while Lample and Lacroix contributed their experience from Meta Platforms, [18] where they specialized in developing large-scale AI models. The trio initially met during their studies at École Polytechnique, [4] a public university in France.

In June 2023, the start-up carried out a first fundraising of €105 million ($117 million) with investors including the American fund Lightspeed Venture Partners, Eric Schmidt, Xavier Niel and JCDecaux. The valuation is then estimated by the Financial Times at €240 million ($267 million).

On 27 September 2023, the company made its language processing model “Mistral 7B” available under the free Apache 2.0 license. This model has 7 billion parameters, a small size compared to its competitors.

On 10 December 2023, Mistral AI announced that it had raised €385 million ($428 million) as part of its second fundraising. This round of financing involves the Californian fund Andreessen Horowitz, BNP Paribas and the software publisher Salesforce. [19]

On 11 December 2023, the company released the Mixtral 8x7B model with 46.7 billion parameters but using only 12.9 billion per token with mixture of experts architecture. The model masters 5 languages (French, Spanish, Italian, English and German) and outperforms, according to its developers' tests, the "LLama 2 70B" model from Meta. A version trained to follow instructions and called “Mixtral 8x7B Instruct” is also offered. [20]

Example of an image generated with Le Chat, the prompt is Generate an image you feel represents yourself, Mistral AI Mistral AI representation (FLUX Pro) (cropped).webp
Example of an image generated with Le Chat, the prompt is Generate an image you feel represents yourself, Mistral AI
Screenshot of Le Chat, Mistral AI chatbot, describing Wikipedia in a thoughtful way Le Chat chatbot example screenshot.webp
Screenshot of Le Chat, Mistral AI chatbot, describing Wikipedia in a thoughtful way

On 26 February 2024, Microsoft announced a new partnership with the company to expand its presence in the artificial intelligence industry. Under the agreement, Mistral's language models will be available on Microsoft's Azure cloud, while the multilingual conversational assistant Le Chat will be launched in the style of ChatGPT. [21]

On 10 April 2024, the company released the mixture of expert models, Mixtral 8x22B, offering high performance on various benchmarks compared to other open models. [22]

On 16 April 2024, reporting revealed that Mistral was in talks to raise €500 million, a deal that would more than double its current valuation to at least €5 billion. [23]

On November 19, 2024, the company announced updates for Le Chat. It added the ability to create images, in partnership with Black Forest Labs, utilizing the Flux Pro model. Additionally, it introduced the capability to search for information on the internet to provide reliable and up-to-date information. Furthermore, it launched the Canvas system, a collaborative interface where the AI generates code and the user can modify it. The company also introduced a new model, Pixtral Large, which is an improvement over Pixtral 12B, integrating a 1-billion-parameter visual encoder coupled with Mistral Large 2. This model has also been enhanced, particularly for long contexts and function calls. [24]

The company had over 100 employees by late fall 2024.

Models

Open-weight models

Mistral 7B

Mistral 7B is a 7.3B parameter language model using the transformers architecture. Officially released on September 27, 2023, via a BitTorrent magnet link, [25] and Hugging Face. [26] The model was released under the Apache 2.0 license. The release blog post claimed the model outperforms LLaMA 2 13B on all benchmarks tested, and is on par with LLaMA 34B on many benchmarks tested. [27]

Mistral 7B employs grouped-query attention (GQA), which is a variant of the standard attention mechanism. This architecture optimizes performance by calculating attention within specific groups of hidden states rather than across all hidden states, improving efficiency and scalability. [28]

Both a base model and "instruct" model were released with the latter receiving additional tuning to follow chat-style prompts. The fine-tuned model is only intended for demonstration purposes, and does not have guardrails or moderation built-in. [27]

Mixtral 8x7B

Much like Mistral's first model, Mixtral 8x7B was released via a BitTorrent link posted on Twitter on December 9, 2023, [1] and later Hugging Face and a blog post were released two days later. [20]

Unlike the previous Mistral model, Mixtral 8x7B uses a sparse mixture of experts architecture. The model has 8 distinct groups of "experts", giving the model a total of 46.7B usable parameters. [29] [30] Each single token can only use 12.9B parameters, therefore giving the speed and cost that a 12.9B parameter model would incur. [20]

Mistral AI's testing shows the model beats both LLaMA 70B, and GPT-3.5 in most benchmarks. [31]

In March 2024, research conducted by Patronus AI comparing performance of LLMs on a 100-question test with prompts to generate text from books protected under U.S. copyright law found that Open AI's GPT-4, Mixtral, Meta AI's LLaMA-2, and Anthropic's Claude 2 generated copyrighted text verbatim in 44%, 22%, 10%, and 8% of responses respectively. [32] [33]

Mixtral 8x22B

Similar to Mistral's previous open models, Mixtral 8x22B was released via a BitTorrent link on Twitter on April 10, 2024, [34] with a release on Hugging Face soon after. [35] The model uses an architecture similar to that of Mistral 8x7B, but with each expert having 22 billion parameters instead of 7. In total, the model contains 141 billion parameters, as some parameters are shared among the experts. [35]

Mistral Large 2

Mistral Large 2 was announced on July 24, 2024, and released on Hugging Face. Unlike the previous Mistral Large, this version was released with open weights. It is available for free with a Mistral Research Licence, and with a commercial licence for commercial purposes. Mistral AI claims that it is fluent in dozens of languages, including many programming languages. The model has 123 billion parameters and a context length of 128,000 tokens. Its performance in benchmarks is competitive with Llama 3.1 405B, particularly in programming-related tasks. [36] [37]

Codestral 22B

Codestral is Mistral's first code focused open weight model. Codestral was launched on 29 May 2024. It is a lightweight model specifically built for code generation tasks. As of its release date, this model surpasses Meta's Llama3 70B and DeepSeek Coder 33B (78.2% - 91.6%), another code-focused model on the HumanEval FIM benchmark. [38] Mistral claims Codestral is fluent in more than 80 programming languages [39] Codestral has its own license which forbids the usage of Codestral for commercial purposes. [40]

Mathstral 7B

Mathstral 7B is a model with 7 billion parameters released by Mistral AI on July 16, 2024. It focuses on STEM subjects, achieving a score of 56.6% on the MATH benchmark and 63.47% on the MMLU benchmark. [41] The model was produced in collaboration with Project Numina, [42] and was released under the Apache 2.0 License. It has a context length of 32k tokens. [41]

Codestral Mamba 7B

Codestral Mamba is based on the Mamba 2 architecture, which allows it to generate responses even with longer input. [42] Unlike Codestral, it was released under the Apache 2.0 license. While previous releases often included both the base model and the instruct version, only the instruct version of Codestral Mamba was released. [43]

API-only models

Unlike Mistral 7B, Mixtral 8x7B and Mixtral 8x22B, the following models are closed-source and only available through the Mistral API. [44]

Mistral Large

Mistral Large was launched on February 26, 2024, and Mistral claims it is second in the world only to OpenAI's GPT-4.

It is fluent in English, French, Spanish, German, and Italian, with Mistral claiming understanding of both grammar and cultural context, and provides coding capabilities. As of early 2024, it is Mistral's flagship AI. [45] It is also available on Microsoft Azure.

In July 2024, Mistral Large 2 was released, replacing the original Mistral Large. [46] Unlike the original model, it was released with open weights. [37]

Mistral Medium

Mistral Medium is trained in various languages including English, French, Italian, German, Spanish and code with a score of 8.6 on MT-Bench. [47] It is ranked in performance above Claude and below GPT-4 on the LMSys ELO Arena benchmark. [48]

The number of parameters, and architecture of Mistral Medium is not known as Mistral has not published public information about it.

Mistral Small

Like the Large model, Mistral Small was launched on February 26, 2024.

Related Research Articles

<span class="mw-page-title-main">Databricks</span> American software company

Databricks, Inc. is a global data, analytics, and artificial intelligence (AI) company, founded in 2013 by the original creators of Apache Spark. The company provides a cloud-based platform to help enterprises build, scale, and govern data and AI, including generative AI and other machine learning models.

Generative Pre-trained Transformer 3 (GPT-3) is a large language model released by OpenAI in 2020.

<span class="mw-page-title-main">GPT-2</span> 2019 text-generating language model

Generative Pre-trained Transformer 2 (GPT-2) is a large language model by OpenAI and the second in their foundational series of GPT models. GPT-2 was pre-trained on a dataset of 8 million web pages. It was partially released in February 2019, followed by full release of the 1.5-billion-parameter model on November 5, 2019.

A foundation model, also known as large X model (LxM), is a machine learning or deep learning model that is trained on vast datasets so it can be applied across a wide range of use cases. Generative AI applications like Large Language Models are often examples of foundation models.

Hugging Face, Inc. is an American company incorporated under the Delaware General Corporation Law and based in New York City that develops computation tools for building applications using machine learning. It is most notable for its transformers library built for natural language processing applications and its platform that allows users to share machine learning models and datasets and showcase their work.

<span class="mw-page-title-main">Stable Diffusion</span> Image-generating machine learning model

Stable Diffusion is a deep learning, text-to-image model released in 2022 based on diffusion techniques. The generative artificial intelligence technology is the premier product of Stability AI and is considered to be a part of the ongoing artificial intelligence boom.

<span class="mw-page-title-main">GPT-J</span> Open source artificial intelligence text generating language model developed by EleutherAI

GPT-J or GPT-J-6B is an open-source large language model (LLM) developed by EleutherAI in 2021. As the name suggests, it is a generative pre-trained transformer model designed to produce human-like text that continues from a prompt. The optional "6B" in the name refers to the fact that it has 6 billion parameters.

<span class="mw-page-title-main">EleutherAI</span> Artificial intelligence research collective

EleutherAI is a grass-roots non-profit artificial intelligence (AI) research group. The group, considered an open-source version of OpenAI, was formed in a Discord server in July 2020 by Connor Leahy, Sid Black, and Leo Gao to organize a replication of GPT-3. In early 2023, it formally incorporated as the EleutherAI Institute, a non-profit research institute.

A large language model (LLM) is a type of machine learning model designed for natural language processing tasks such as language generation. As language models, LLMs acquire these abilities by learning statistical relationships from vast amounts of text during a self-supervised and semi-supervised training process.

In deep learning, fine-tuning is an approach to transfer learning in which the parameters of a pre-trained neural network model are trained on new data. Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen". A model may also be augmented with "adapters" that consist of far fewer parameters than the original model, and fine-tuned in a parameter-efficient way by tuning the weights of the adapters and leaving the rest of the model's weights frozen.

The Pile is an 886.03 GB diverse, open-source dataset of English text created as a training dataset for large language models (LLMs). It was constructed by EleutherAI in 2020 and publicly released on December 31 of that year. It is composed of 22 smaller datasets, including 14 new ones.

<span class="mw-page-title-main">Llama (language model)</span> Large language model by Meta AI

Llama is a family of autoregressive large language models (LLMs) released by Meta AI starting in February 2023. The latest version is Llama 3.3, released in December 2024.

<span class="mw-page-title-main">Gemini (language model)</span> Large language model developed by Google

Gemini is a family of multimodal large language models developed by Google DeepMind, serving as the successor to LaMDA and PaLM 2. Comprising Gemini Ultra, Gemini Pro, Gemini Flash, and Gemini Nano, it was announced on December 6, 2023, positioned as a competitor to OpenAI's GPT-4. It powers the chatbot of the same name.

Vicuna LLM is an omnibus Large Language Model used in AI research. Its methodology is to enable the public at large to contrast and compare the accuracy of LLMs "in the wild" and to vote on their output; a question-and-answer chat format is used. At the beginning of each round two LLM chatbots from a diverse pool of nine are presented randomly and anonymously, their identities only being revealed upon voting on their answers. The user has the option of either replaying ("regenerating") a round, or beginning an entirely fresh one with new LLMs. Based on Llama 2, it is an open source project, and it itself has become the subject of academic research in the burgeoning field. A non-commercial, public demo of the Vicuna-13b model is available to access using LMSYS.

Mamba is a deep learning architecture focused on sequence modeling. It was developed by researchers from Carnegie Mellon University and Princeton University to address some limitations of transformer models, especially in processing long sequences. It is based on the Structured State Space sequence (S4) model.

<span class="mw-page-title-main">IBM Granite</span> 2023 text-generating language model

IBM Granite is a series of decoder-only AI foundation models created by IBM. It was announced on September 7, 2023, and an initial paper was published 4 days later. Initially intended for use in the IBM's cloud-based data and generative AI platform Watsonx along with other models, IBM opened the source code of some code models. Granite models are trained on datasets curated from Internet, academic publishings, code datasets, legal and finance documents.

<span class="mw-page-title-main">DBRX</span> Open-sourced large language model

DBRX is an open-sourced large language model (LLM) developed by Mosaic ML team at Databricks, released on March 27, 2024. It is a mixture-of-experts transformer model, with 132 billion parameters in total. 36 billion parameters are active for each token. The released model comes in either a base foundation model version or an instruction-tuned variant.

llama.cpp Software library for LLM inference

llama.cpp is an open source software library that performs inference on various large language models such as Llama. It is co-developed alongside the GGML project, a general-purpose tensor library.

<span class="mw-page-title-main">01.AI</span> Artificial intelligence company

01.AI is an artificial intelligence (AI) company based in Beijing, China. It focuses on developing open source products.

Qwen is a family of Large Language Models developed by Alibaba. In July of 2024 it ranked as the top Chinese language model in some benchmarks and third globally behind Anthropic and OpenAI.

References

  1. 1 2 "Buzzy Startup Just Dumps AI Model That Beats GPT-3.5 Into a Torrent Link". Gizmodo. 12 December 2023. Retrieved 16 December 2023.
  2. ↑ "What is Mistral AI?". IBM . October 2024.
  3. ↑ "France's unicorn start-up Mistral AI embodies its artificial intelligence hopes". Le Monde.fr. 12 December 2023. Retrieved 16 December 2023.
  4. 1 2 Journal, Sam Schechner | Photographs by Edouard Jacquinet for The Wall Street. "The 9-Month-Old AI Startup Challenging Silicon Valley's Giants". WSJ. Retrieved 31 March 2024.
  5. ↑ "Bringing open AI models to the frontier". Mistral AI. 27 September 2023. Retrieved 4 January 2024.
  6. ↑ Metz, Cade (10 December 2023). "Mistral, French A.I. Start-Up, Is Valued at $2 Billion in Funding Round". The New York Times.
  7. ↑ Fink, Charlie. "This Week In XR: Epic Triumphs Over Google, Mistral AI Raises $415 Million, $56.5 Million For Essential AI". Forbes. Retrieved 16 December 2023.
  8. ↑ "A French AI start-up may have commenced an AI revolution, silently". Hindustan Times. 12 December 2023.
  9. ↑ Abboud, Leila; Levingston, Ivan; Hammond, George (8 December 2023). "French AI start-up Mistral secures €2bn valuation". Financial Times. ft.com Financial Times.
  10. ↑ Kharpal, Arjun (24 May 2024). "CEOs of AI startups backed by Microsoft and Amazon are the new tech rockstars". CNBC. Retrieved 13 June 2024.
  11. ↑ "Tripling Down on Mistral AI | General Catalyst". www.generalcatalyst.com. Retrieved 13 June 2024.
  12. ↑ "Open-weight models and Mistral AI Large Language Models". docs.mistral.ai. Retrieved 4 January 2024.
  13. ↑ "Endpoints and Mistral AI Large Language Models". docs.mistral.ai.
  14. ↑ "Endpoints and benchmarks | Mistral AI Large Language Models". docs.mistral.ai. Retrieved 6 March 2024.
  15. ↑ Bratton, Laura (12 June 2024). "OpenAI's French rival Mistral AI is now worth $6 billion. That's still a fraction of its top competitors". Quartz (publication) . Retrieved 13 June 2024.
  16. ↑ Webb, Maria (2 January 2024). "Mistral AI: Exploring Europe's Latest Tech Unicorn". techopedia.com. Retrieved 13 June 2024.
  17. ↑ "Spotlight Interview: Mistral AI CEO Arthur Mensch". The French Tech Journal. 24 June 2024.
  18. ↑ "France's unicorn start-up Mistral AI embodies its artificial intelligence hopes". Le Monde.fr. 12 December 2023.
  19. ↑ "Mistral lève 385 M€ et devient une licorne française - le Monde Informatique". 11 December 2023.
  20. 1 2 3 "Mixtral of experts". mistral.ai. 11 December 2023. Retrieved 4 January 2024.
  21. ↑ Bableshwar (26 February 2024). "Mistral Large, Mistral AI's flagship LLM, debuts on Azure AI Models-as-a-Service". techcommunity.microsoft.com. Retrieved 26 February 2024.
  22. ↑ "Mistral Releases Latest Open Source Model, Mixtral 8x22B". Pure AI. 17 April 2024.
  23. ↑ Abboud, Leila; Levingston, Ivan; Hammond, George (19 April 2024). "Mistral in talks to raise €500mn at €5bn valuation". Financial Times. Retrieved 19 April 2024.
  24. ↑ "Mistral has entered the chat". Mistral AI. 18 November 2024. Retrieved 11 December 2024.
  25. ↑ Goldman, Sharon (8 December 2023). "Mistral AI bucks release trend by dropping torrent link to new open source LLM". VentureBeat. Retrieved 4 January 2024.
  26. ↑ Coldewey, Devin (27 September 2023). "Mistral AI makes its first large language model free for everyone". TechCrunch. Retrieved 4 January 2024.
  27. 1 2 "Mistral 7B". mistral.ai. Mistral AI. 27 September 2023. Retrieved 4 January 2024.
  28. ↑ Jiang, Albert Q.; Sablayrolles, Alexandre; Mensch, Arthur; Bamford, Chris; Chaplot, Devendra Singh; Casas, Diego de las; Bressand, Florian; Lengyel, Gianna; Lample, Guillaume (10 October 2023). "Mistral 7B". arXiv: 2310.06825v1 [cs.CL].
  29. ↑ "Mixture of Experts Explained". huggingface.co. Retrieved 4 January 2024.
  30. ↑ Marie, Benjamin (15 December 2023). "Mixtral-8x7B: Understanding and Running the Sparse Mixture of Experts". Medium. Retrieved 4 January 2024.
  31. ↑ Franzen, Carl (11 December 2023). "Mistral shocks AI community as latest open source model eclipses GPT-3.5 performance". VentureBeat. Retrieved 4 January 2024.
  32. ↑ Field, Hayden (6 March 2024). "Researchers tested leading AI models for copyright infringement using popular books, and GPT-4 performed worst". CNBC. Retrieved 6 March 2024.
  33. ↑ "Introducing CopyrightCatcher, the first Copyright Detection API for LLMs". Patronus AI. 6 March 2024. Retrieved 6 March 2024.
  34. ↑ @MistralAI (10 April 2024). "Torrent" (Tweet) – via Twitter.
  35. 1 2 "mistralai/Mixtral-8x22B-v0.1 · Hugging Face". huggingface.co. Retrieved 5 May 2024.
  36. ↑ AI, Mistral (24 July 2024). "Large Enough". mistral.ai. Retrieved 24 July 2024.
  37. 1 2 "mistralai/Mistral-Large-Instruct-2407 · Hugging Face". huggingface.co. Retrieved 24 August 2024.
  38. ↑ AI, Mistral (29 May 2024). "Codestral: Hello, World!". mistral.ai. Retrieved 30 May 2024.
  39. ↑ Sharma, Shubham (29 May 2024). "Mistral announces Codestral, its first programming focused AI model". VentureBeat. Retrieved 30 May 2024.
  40. ↑ Wiggers, Kyle (29 May 2024). "Mistral releases Codestral, its first generative AI model for code". TechCrunch. Retrieved 30 May 2024.
  41. 1 2 AI, Mistral (16 July 2024). "MathΣtral". mistral.ai. Retrieved 16 July 2024.
  42. 1 2 David, Emilia (16 July 2024). "Mistral releases Codestral Mamba for faster, longer code generation". VentureBeat. Retrieved 17 July 2024.
  43. ↑ AI, Mistral (16 July 2024). "Codestral Mamba". mistral.ai. Retrieved 16 July 2024.
  44. ↑ "Pricing and rate limits | Mistral AI Large Language Models". docs.mistral.ai. Retrieved 22 January 2024.
  45. ↑ AI, Mistral (26 February 2024). "Au Large". mistral.ai. Retrieved 6 March 2024.
  46. ↑ "Models | Mistral AI Large Language Models". docs.mistral.ai. Retrieved 24 August 2024.
  47. ↑ AI, Mistral (11 December 2023). "La plateforme". mistral.ai. Retrieved 22 January 2024.
  48. ↑ "LMSys Chatbot Arena Leaderboard - a Hugging Face Space by lmsys". huggingface.co. Retrieved 22 January 2024.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.