 | |
| Company type | Private company |
|---|---|
| Industry | Artificial intelligence |
| Founded | 2022 |
| Founders |
|
| Headquarters | New York City, United States |
| Website | elevenlabs |
| Part of a series on |
| Artificial intelligence |
|---|
 |
ElevenLabs is a software company that specializes in developing natural-sounding speech synthesis and text-to-speech software, using artificial intelligence and deep learning.
It has been recognized as one of the major companies behind the ongoing AI Spring. [1]
ElevenLabs was co-founded in 2022 by Piotr Dąbkowski, an ex-Google machine learning engineer and Mateusz Staniszewski, an ex-Palantir deployment strategist. [2] Both were raised in Poland, and their inspiration for founding ElevenLabs reportedly came from watching inadequately dubbed American films. [3] [4]
Dąbkowski and Staniszewski initially considered different funding options, including the possibility of collaborating with a startup accelerator. In January 2023 they revealed having secured a $2 million pre-seed round. The startup's specialization in AI voice intelligence, a still-emerging field in Europe, played a significant role in attracting investors. The pre-seed funding was primarily led by Credo Ventures, and joined by Concept Ventures. [5]
In January 2023, ElevenLabs publicly released its beta platform. [6]
In June 2023, ElevenLabs raised a $19 million Series A funding round at a valuation of about $100 million, [7] [8] despite the company having no office and only 15 employees. [4] [8] The funding round was co-led by the venture capital firm Andreessen Horowitz, ex-GitHub CEO Nat Friedman, and entrepreneur Daniel Gross. It also saw participation from prominent individuals such as SV Angel, Mike Krieger (co-founder of Instagram), Brendan Iribe (co-founder of Oculus), Mustafa Suleyman (co-founder of Deepmind), and Tim O'Reilly (founder of O'Reilly Media). It was also announced that Andreessen Horowitz would be joining ElevenLabs' board. [3]
On January 22, 2024, ElevenLabs raised an additional $80 million in Series B funding raising the total valuation of the company to $1.1 billion. The funding round was led by Andreessen Horowitz, Friedman, Gross, and Sequoia Capital. Additionally, the company announced a series of new products, such as their Voice Marketplace, AI Dubbing Studio, and mobile app. [9]
ElevenLabs is primarily known for its browser-based, AI-assisted text-to-speech software, Speech Synthesis, which can produce lifelike speech by synthesizing vocal emotion and intonation. [10] The company states its software is built to adjust the intonation and pacing of delivery based on the context of language input used. [11] It uses advanced algorithms to analyze the contextual aspects of text, aiming to detect emotions like anger, sadness, happiness, or alarm, which enables the system to understand the user's sentiment, [12] resulting in achieving a more realistic and human-like inflection. The startup is in the process of patenting this technology. [5] Through its beta site, users can submit text and generate audio files from a selection of default voices. Paying users are given the ability to upload custom voice samples to create new vocal styles using the company's voice cloning tool. [13]
Voice Library is the company's feature for sharing unique voice profiles created using their Voice Design technology. These pre-designed voice profiles allow users to select a voice that best suits their needs, rather than creating one from scratch. [14] Another tool called VoiceLab allows users to clone voices from just a few short snippets of audio and can create entirely new synthetic voices. [3]
On 20 June 2023, ElevenLabs released an AI recognition tool called the AI Speech Classifier, which it claims is the first of its kind. [3] The tool is accessible through an API and designed to determine if an uploaded audio sample originates from ElevenLabs' proprietary AI technology. [4] The company has expressed its intention to collaborate with other AI developers in creating a universal detection system that could be adopted industry-wide. [15]
In July 2023, ElevenLabs announced "Projects", a tool for creating long-form spoken content such as audiobooks and dialogue segments with contextually-aware synthetic or custom voices. [4] [16] The tool was released in September. In August, ElevenLabs expanded its voice generation capabilities to 28 languages. Using an in-house AI model, it automatically detects languages like Korean, Dutch, and Vietnamese, allowing for "emotionally rich" multilingual speech generation. The company also announced that its technology had officially exited its beta phase. [17] [18]
In October 2023, ElevenLabs presented "AI Dubbing," a tool that is able to translate speech into more than 20 languages. The feature is capable of preserving the speaker's original voice, emotions, and intonation, by employing proprietary methods to handle tasks like noise removal, speaker differentiation, transcription, and synchronization of translated speech with the original audio. [19]
ElevenLabs' use cases span a range of sectors.
Content creators have used ElevenLabs for podcasts, narration, and comedy shows. [20] [21] [22] In March 2023, comedian Drew Carey used ElevenLabs' voice cloning tool to recreate his voice for an episode of his radio show, Friday Night Freakout. [11] In April 2023, Polish TV and radio presenter Jaroslaw Kuzniar has also used a synthesized version of his voice to deliver a series of podcasts on the War in Ukraine. [23] Seth Godin has also used ElevenLabs to narrate his AI-focused podcast. [3]
In March 2023, Super-Hi-Fi, a streaming automation service, partnered with ElevenLabs to launch a fully automated radio service called "AI Radio", using ElevenLabs' software to voice its virtual DJ from prompts generated with ChatGPT. [24] ElevenLabs has also been employed for narrating games and voicing game characters in partnerships with Swedish game developer Paradox Interactive and the United Kingdom-based Magicave. [3] [25]
Publishers and authors have used ElevenLabs to narrate audiobooks and newsletters. [5] [26] On 13 June 2023, Storytel announced an exclusive partnership with the company. In this collaboration, ElevenLabs will create voices tailored specifically to Storytel's core markets and to produce AI-narrated audiobooks. A voice-changing feature called VoiceSwitcher was implemented to enhance personalization for users, providing unique listening experiences customized for each individual. [27] [28]
ElevenLabs has been used to generate audio for dubbing videos in different languages, including by content creators. [5] [8] The platform has the capability to accurately replicate almost any accent in any language. [29] Celebrity fans have used ElevenLabs to create inspirational messages using the voices of their favorite celebrities. [30]
In February, VICE reporter Joseph Cox published findings that he had recorded five minutes of himself talking and then used ElevenLabs to create voice deepfakes that defeated a bank's voice-authentication system. [31]
ElevenLabs sets explicit guidelines regarding the use of its technology, forbidding the cloning of voices for abusive purposes such as fraud, discrimination, hate speech, or online abuse, although it does support the use of its platform for “caricature, parody and satire” and “artistic and political speech contributing to public debates." The company asserts its authority to suspend the accounts and content of users found in violation of these guidelines, and it also highlights its commitment to cooperate with authorities and report any illegal activities in accordance with applicable laws. [3] In January, the company admitted that its platform has been used for “voice cloning misuse cases” [32] and toughened its safeguards against vexatious use of its technology. [33]
Following its launch in January 2023, ElevenLabs gained rapid momentum and was commended for its voice output quality, fast generation times, and a "generous free tier". It has also been praised for its ability to accurately pronounce names with unique or uncommon pronunciations, addressing a common shortcoming in similar tools that often cater primarily to Western names. [34] The company reached over one million registered users between its launch and June 2023. [3] [4] [35]
ElevenLabs was criticized after users were able to abuse its software to generate controversial statements in the vocal style of celebrities, public officials, and other famous individuals, [36] [37] [38] [39] [33] particularly attracting attention after users on 4chan used the tool to share hateful messages. [40] [15] The software's ability to closely copy real voices has raised ethical concerns, with critics considering it a form of deepfaking. [41] In response, the company said it would work on mitigating potential abuse through safeguards and identity verification. [6] The company has subsequently limited access to its voice cloning feature to paid subscribers, [42] citing the requirement to provide payment information as means for improving accountability, [43] and has implemented bans on users who repeatedly violate the terms of service.
In the leadup to the January 2024 New Hampshire democratic primary, AI-generated robocalls seemingly from Joe Biden encouraging voters to skip voting on the day of the primary were sent out to thousands of residents. The New Hampshire attorney general's office launched an investigation into the incident and linked it to a company based in Texas, with audio experts concluding the call was made using ElevenLabs. In response to the incident, CEO Mati Staniszewski stated that the company was “dedicated to preventing the misuse of audio AI tools” but provided no comment on specific incidents. [44]
Additional concerns have been raised over the ethics of the source of ElevenLabs' training data, with multiple voice actors claiming ElevenLabs used samples of their voices without their consent. [45] ElevenLabs, along with other companies in its category, has thus been seen as a potential challenge to the voice acting sector. [18]
Speech synthesis is the artificial production of human speech. A computer system used for this purpose is called a speech synthesizer, and can be implemented in software or hardware products. A text-to-speech (TTS) system converts normal language text into speech; other systems render symbolic linguistic representations like phonetic transcriptions into speech. The reverse process is speech recognition.
A digital audio workstation is an electronic device or application software used for recording, editing and producing audio files. DAWs come in a wide variety of configurations from a single software program on a laptop, to an integrated stand-alone unit, all the way to a highly complex configuration of numerous components controlled by a central computer. Regardless of configuration, modern DAWs have a central interface that allows the user to alter and mix multiple recordings and tracks into a final produced piece.
An impersonator is someone who imitates or copies the behavior or actions of another. There are many reasons for impersonating someone:
Audible is an American online audiobook and podcast service that allows users to purchase and stream audiobooks and other forms of spoken word content. This content can be purchased individually or under a subscription model where the user receives "credits" that can be redeemed for content monthly and receive access to a curated on-demand library of content. Audible is the United States' largest audiobook producer and retailer. The service is owned by Audible, a wholly-owned subsidiary of Amazon.com, Inc., headquartered in Newark, New Jersey.
Human image synthesis is technology that can be applied to make believable and even photorealistic renditions of human-likenesses, moving or still. It has effectively existed since the early 2000s. Many films using computer generated imagery have featured synthetic images of human-like characters digitally composited onto the real or other simulated film material. Towards the end of the 2010s deep learning artificial intelligence has been applied to synthesize images and video that look like humans, without need for human assistance, once the training phase has been completed, whereas the old school 7D-route required massive amounts of human work .
Music and artificial intelligence is the development of music software programs which use AI to generate music. As with applications in other fields, AI in music also simulates mental tasks. A prominent feature is the capability of an AI algorithm to learn based on past data, such as in computer accompaniment technology, wherein the AI is capable of listening to a human performer and performing accompaniment. Artificial intelligence also drives interactive composition technology, wherein a computer composes music in response to a live performance. There are other AI applications in music that cover not only music composition, production, and performance but also how music is marketed and consumed. Several music player programs have also been developed to use voice recognition and natural language processing technology for music voice control. Current research includes the application of AI in music composition, performance, theory and digital sound processing.
Uniphore is a conversational automation technology company. Uniphore sells software for conversational analytics, conversational assistants and conversational security. The company is headquartered in Palo Alto, California, with offices in the United States, Singapore, India, Japan, Spain, and Israel. Its products are used by up to 75,000 customer service agents during approximately 160 million interactions per month.
Deepfakes are synthetic media that have been digitally manipulated to replace one person's likeness convincingly with that of another. It can also refer to computer-generated images of human subjects that do not exist in real life. While the act of creating fake content is not new, deepfakes leverage tools and techniques from machine learning and artificial intelligence, including facial recognition algorithms and artificial neural networks such as variational autoencoders (VAEs) and generative adversarial networks (GANs). In turn the field of image forensics develops techniques to detect manipulated images.
Digital cloning is an emerging technology, that involves deep-learning algorithms, which allows one to manipulate currently existing audio, photos, and videos that are hyper-realistic. One of the impacts of such technology is that hyper-realistic videos and photos makes it difficult for the human eye to distinguish what is real and what is fake. Furthermore, with various companies making such technologies available to the public, they can bring various benefits as well as potential legal and ethical concerns.
Otter.ai, Inc. is a Mountain View, California-based technology company that develops speech to text transcription applications using artificial intelligence and machine learning. Its software, called Otter, shows captions for live speakers, and generates written transcriptions of speech.
Synthetic media is a catch-all term for the artificial production, manipulation, and modification of data and media by automated means, especially through the use of artificial intelligence algorithms, such as for the purpose of misleading people or changing an original meaning. Synthetic media as a field has grown rapidly since the creation of generative adversarial networks, primarily through the rise of deepfakes as well as music synthesis, text generation, human image synthesis, speech synthesis, and more. Though experts use the term "synthetic media," individual methods such as deepfakes and text synthesis are sometimes not referred to as such by the media but instead by their respective terminology Significant attention arose towards the field of synthetic media starting in 2017 when Motherboard reported on the emergence of AI altered pornographic videos to insert the faces of famous actresses. Potential hazards of synthetic media include the spread of misinformation, further loss of trust in institutions such as media and government, the mass automation of creative and journalistic jobs and a retreat into AI-generated fantasy worlds. Synthetic media is an applied form of artificial imagination.
Deepfake pornography, or simply fake pornography, is a type of synthetic porn that is created via altering already-existing pornographic material by applying deepfake technology to the faces of the actors. The use of deepfake porn has sparked controversy because it involves the making and sharing of realistic videos featuring non-consenting individuals, typically female celebrities, and is sometimes used for revenge porn. Efforts are being made to combat these ethical concerns through legislation and technology-based solutions.
15.ai is a non-commercial freeware artificial intelligence web application that generates natural emotive high-fidelity text-to-speech voices from an assortment of fictional characters from a variety of media sources. Developed by a pseudonymous MIT researcher under the name 15, the project uses a combination of audio synthesis algorithms, speech synthesis deep neural networks, and sentiment analysis models to generate and serve emotive character voices faster than real-time, particularly those with a very small amount of trainable data.
An audio deepfake is a product of artificial intelligence used to create convincing speech sentences that sound like specific people saying things they did not say. This technology was initially developed for various applications to improve human life. For example, it can be used to produce audiobooks, and also to help people who have lost their voices to get them back. Commercially, it has opened the door to several opportunities. This technology can also create more personalized digital assistants and natural-sounding text-to-speech as well as speech translation services.
Krisp is an Armenian AI-based audio processing software company that offers real-time noise and voice suppression technology. The company was founded in 2017 in Yerevan, Armenia, by Davit Baghdasaryan and Artavazd Minasyan, and is based in Berkeley, California.
Synthesia is a synthetic media generation company that develops software used to create AI generated video content. It is based in London, England.
Generative artificial intelligence is artificial intelligence capable of generating text, images, videos, or other data using generative models, often in response to prompts. Generative AI models learn the patterns and structure of their input training data and then generate new data that has similar characteristics.
The AI boom, or AI spring, is the ongoing period of rapid progress in the field of artificial intelligence (AI). Prominent examples include protein folding prediction led by Google DeepMind and generative AI led by OpenAI.
Respeecher is a Ukrainian software company developing speech synthesis software enabling one person to speak in the voice of another particular person using artificial intelligence.
In late January 2024, sexually explicit AI-generated deepfake images of American musician Taylor Swift were proliferated on social media platforms 4chan and X. The images led Microsoft to enhance Microsoft Designer's text-to-image model to prevent future abuse. Several artificial images of Swift of a sexual or violent nature were quickly spread, with one post reported to have been seen over 47 million times before its eventual removal. These images prompted responses from anti sexual assault advocacy groups, US politicians, Swift's fans, Microsoft CEO Satya Nadella, among others, and it has been suggested that Swift's influence could result in new legislation regarding the creation of deepfake pornography.
{{cite web}}: CS1 maint: multiple names: authors list (link)