ElevenLabs Bets Voice AI Can Be Fast and Smart at the Same Time

For the most part, voice assistants still keep you waiting. Put in a question and there is a moment of silence before an answer comes back, one that is either hasty and superficial or careful to a fault. ElevenLabs, the $11 billion voice AI firm, put out two new models on Tuesday that it says end that tradeoff.

Eleven Realtime starts speaking in roughly half a second and, by an independent measure of spoken reasoning, gets the right answer to nearly every question. It also speaks the same language as OpenAI's developer tools, so a changeover requires little more than a new address and a model name.

What ElevenLabs Released

Eleven Realtime is a set of two speech to speech models, each a finished voice agent. The v1 Mini is the standard offering, designed for the kind of natural conversation an AI assistant, tutor or interactive character would have, without much expense; the company says it should cover most needs. For tougher jobs like customer support, lead qualification or multi step workflows, where an agent must work through a process and make calls to outside tools, there is the v1 Max.

With these models, developers no longer have to put the parts together themselves. They take care of the speech recognition, determining when the caller is done, the reasoning and the answering voice. Over 90 languages are supported, as well as any of the more than 17,000 voices in the ElevenLabs library. The Mini is calibrated to know the difference between a completed sentence and an "mm-hmm."

In its launch announcement, the company called the release "a new class of Speech to Speech models that removes those hurdles," meaning the months of tuning a homegrown voice stack demands. On X, the message was more direct: "Voice agents usually make you choose between quick replies or precise answers. Eleven Realtime gives you both."

Half a Second Is the Whole Game

Latency is no mere technicality in a voice conversation; it is what makes the difference between something that has the feel of a person and something like a phone tree. According to Artificial Analysis, an independent firm that puts AI models to the test, Eleven Realtime v1 Max needs on average 0.57 seconds for its first audio output, with the Mini at 0.62 seconds, so these are not figures ElevenLabs put out on its own.

Max is distinguished by its accuracy. On Big Bench Audio, a collection of 1,000 spoken reasoning questions on everything from logic and navigation to counting, Max was right 98.8 percent of the time, compared with 75.9 percent for the Mini. Where tool use is concerned, say looking up an order or making an appointment during a call, Max finished 87.1 percent of those tasks to the Mini's 60.3 percent. The smaller model did edge ahead on Full Duplex Bench, which measures turn taking, pauses and interruptions, with 95.8 percent against 94.1 percent.

To put it in perspective, when Artificial Analysis launched its Speech to Speech Index in June, OpenAI's GPT-Realtime-2, the highest placed model, needed 2.33 seconds for its initial audio. Deepslate Opal was the swiftest in the index at 0.44 seconds, but its score was well under the leaders. While the October results and the June index are not directly comparable, the gap explains why ElevenLabs is leading with speed and accuracy together.

The Swap That Takes Two Lines of Code

Plumbing is perhaps the most strategic element of the launch. Eleven Realtime comes in two forms: inside ElevenAgents, the company's own platform for building and running voice agents, or via a Realtime API that follows OpenAI's Realtime specification. An application put together with OpenAI's software development kit can be pointed at ElevenLabs with a change of endpoint and model name.

For the host of startups that built their first voice products on OpenAI, such compatibility is an open invitation. It reduces the cost of trying another supplier to an afternoon of testing, and it forces every other voice model provider to compete on price and speed rather than on lock in.

But there are constraints. The API is WebSocket only. Any app running on WebRTC, the go-to standard for real time audio in most browsers, will have to change how it sends and receives sound in order to make the switch. On WebSocket, the developer also has to handle capturing and playing back the raw audio. And since the API is separate from the ElevenAgents platform, phone and WhatsApp channels, call monitoring, guardrails and knowledge bases are reserved for those building on the platform.

What Developers Give Up

A managed agent involves a trade off. With ElevenLabs taking care of turn detection, interruptions and the orchestration of listening, thinking and speaking, much of the engineering work is done away with. The downside is that developers cannot swap out the speech or language models at the core, and they have fewer ways to work out what went wrong on a given call. Custom mode is an option for teams set on choosing their own models, though Custom agents cannot transfer calls to or from Realtime ones.

Then there is the matter of compliance. Mini lacks Zero Retention Mode and HIPAA configurations, which rules it out for a lot of health care applications. Max offers both in the United States and the European Union, but not in the India or Singapore data residency environments. Both are billed at the standard ElevenAgents per minute rate with token fees on top, Max being the pricier of the two per token. Whether Max's better tool use justifies the extra cost by reducing retries is something each team will have to measure on its own calls; benchmarks are no guide to how a model performs under the strain of production.

An $11 Billion Bet on the Voice Layer

Mati Staniszewski and Piotr Dąbkowski set up ElevenLabs in 2022, and the firm first made a name for itself with lifelike text to speech and voice cloning. In February the company raised $500 million at an $11 billion valuation, more than three times what it was worth a year earlier. The round was led by Sequoia Capital, with Andreessen Horowitz and Iconiq on board. The money was earmarked for expanding the technology, not least a platform for enterprise conversational agents in areas such as customer support, sales and training.

Eleven Realtime is the product that funding was pointing toward. There is a good business in selling voices to creators, but there is far more to be made from the agent that takes a bank's calls. With this launch ElevenLabs is in head to head competition with OpenAI, Google and xAI, all of which have their own speech to speech models, as well as the host of Silicon Valley startups that have put together voice agents from other companies' models.

What Happens When the Voice on the Line Is Never Off Script?

What is in store for the near future is a wave of voice agents good enough to put an end to hold music. The two sources of frustration with automated phone agents have been slow responses and unreliable tool calls; if both are now remedied at a reasonable cost, expect banks, airlines, clinics, utilities and small businesses to hand their routine calls to AI over the next two years. For Silicon Valley founders, OpenAI compatible endpoints make the voice model a swappable commodity, so the real competition moves to the data, trust and workflows built around it. Language apps, games and tutoring services could gain characters that carry on a real conversation in more than 90 languages.

The concerns are equally tangible. A voice agent that sounds like a person, never loses its temper and has 17,000 voices to call on is also well suited to fraud. The Federal Communications Commission determined in 2024 that the consent rules for artificial voices apply to AI generated voices in robocalls as well, yet enforcement has not been able to get ahead of the cheap tools available. There is the matter of privacy too: every call becomes audio and transcripts stored away somewhere, which makes the gaps in zero retention and HIPAA support an issue. Then there are the jobs. An agent capable of completing 87 percent of tool based tasks is aimed squarely at the work of the millions employed by call centers in the United States, India and the Philippines.

At the heart of the controversy is disclosure and consent. Critics say callers should be told when they are talking to a machine, and that voice cloning firms have outpaced the safeguards meant to check them. ElevenLabs has faced this kind of criticism since its early cloning tools were put to bad use, and it has added voice verification and detection tools in response. Supporters counter that most people would rather have their problem solved quickly by a capable agent in their own language at 3 a.m. than sit in a long hold queue. Both arguments will be put to the test as these agents go from demos to millions of real calls.

There are specific signals to watch. Will independent testing under production load bear out the 0.57 second latency and the tool use figures? With switching now so easy, does OpenAI answer with realtime models that are faster or cheaper? Will regulators in the United States and the European Union require AI voice agents to identify themselves at the start of a call? And do the first big enterprise rollouts see fewer transfers to human agents, or more complaints? The answers will determine whether Eleven Realtime is simply a better voice or the beginning of the end of the phone queue.

When the Machine Hears the Pause

At Sparknify we have been asking what happens when a machine can read you as well as talk with you. In When AI Understands Emotion, Does It Gain the Power to Control It? we considered the fine line that separates an AI which merely registers your feelings from one that sets out to influence them. A voice agent capable of telling a complete sentence apart from a tentative "mm-hmm" is a modest step in that direction. It listens not just to the words but to the way they are delivered, and that is where the technology's most useful and most unsettling applications begin.

The lag before a response used to be how you could tell you were speaking to a machine. ElevenLabs is betting that soon there will be no such giveaway.

Subscribe to Sparknify for Silicon Valley tech news and the people behind it.