I spent last month trying to give my text-based agent a voice. I told that story already, the four experiments and what I was chasing, in Giving My Agent a Voice. This essay is not that story. This is the map I drew afterward.
Because the surprising thing about building a voice agent is that the voice was never the hard part. The hard part was understanding the landscape well enough to know which kind of system I was even choosing between. Voice AI is not one technology. There are many, and they are so different from one another that a decision that is obvious for one use case is a trap for another. I worked with a provider-neutral cascade built from Deepgram and Piper, with Hume's empathic voice model, and with OpenAI's Realtime model, and only after standing on all three did the shape of the field come clear.
What follows is my foundational understanding of that field: the three architectures you can build on (as of today), and the two questions that decide which one is right for you.
The Architectures
At this early moment in the field, I see three kinds of voice architecture. As the technology grows there will almost certainly be more, but these are the three worth understanding today. They differ in a single respect, and it is worth stating that respect before the names: they differ in how much of a brain lives inside the voice system itself.
The cascade sandwich. This is the modular approach. You take a speech-to-text model on one end, a text-to-speech model on the other, and you put a reasoning agent in the middle. Speech-to-text is a recognizer: it turns spoken audio into written words, and nothing more. Text-to-speech is a synthesizer: it turns written words into natural-sounding audio, and nothing more. Neither is a language model. Neither reasons. They are specialized machine-learning components, each excellent at one narrow job and blind to everything else. The intelligence in a cascade is whatever the agent in the middle supplies. This is the sandwich I built with Deepgram and Piper.
The speech-language model. This is a language model, an LLM with genuine reasoning, whose senses have been extended to speech. The word "language" here does not mean English or French. It means the model has an LLM-style language faculty, the same kind of understanding GPT has, operating natively in a space where audio and text are the same kind of thing. One intelligence hears, understands, and speaks. And it can be trained so that emotion and prosody are first-class, understood on the way in and expressed on the way out. Hume's model is this.
The speech-to-speech model. Audio goes in, audio comes out, and it never has to become text. It thinks in speech itself, in continuous audio representations, which is why it can carry a sigh or a hesitation straight through the conversation without ever flattening it into words. OpenAI's Realtime model and Kyutai's Moshi live here.
The last two sound the same because they seem to do the same thing. The difference is what happens in the middle. A speech-to-speech model stays in audio the whole way: audio in, audio out, it never becomes text. A speech-language model does something different. It takes in audio, turns it into text, reasons, then turns text back into audio. These are separate models. And the speech-language model is not a recognizer bolted to a synthesizer. It is a single model built with two faculties at once, speech and text, the same way Claude is a single model built with vision and text.
The Two Foundational Axes
Now that you know the architectures, the real work begins: understanding them at a level deep enough to map any of them onto the system you are trying to build. Two questions do that work. I think of them as the two axes of voice AI, and every system on the market, today's and tomorrow's, can be placed on both.
The Intelligence Axis
The first axis is the one I have already been circling: how much intelligence is built into the voice system out of the box?
The cascade sits at the low end. Its parts are dumb by design. A recognizer and a synthesizer, with no understanding between them, which means every scrap of intelligence has to come from you.
The speech-language model and the speech-to-speech model sit at the high end. Both are real minds. They arrive already knowing how to listen, understand, and speak, and the best of them, like Hume, arrive understanding emotion as meaning rather than decoration. On this axis, the intelligent systems win easily. If intelligence were the only axis, the choice would be over.
It is not the only axis.
The Configurability Axis
The second axis is the one almost nobody names, and it is the one that decides whether you can actually ship. How much of the system are you allowed to touch?
I think of it as a single hard choice: configure or build. If a system exposes its controls, you configure it to fit your use case. If it seals them, you build the missing piece yourself, or you go without. These are two independent parameters, not one dial. Intelligence is how much the system already knows how to do. Configurability is how much of it you can shape. A system can be high on one and low on the other, or high on both, or neither. As it happens, today's systems tend to cluster: the dumber ones hand you everything, the smarter ones keep more sealed. But that is a pattern of the current market, not a law. The question is not where a system sits on some trade-off. It is how much of each parameter your problem actually needs.
The cascade is at the open end. You can touch everything. But because the parts hold no intelligence, "configure" quietly becomes "build," and you pay for that freedom in labor. The intelligent models are at the sealed end. They do the hard things beautifully, but the orchestration, the turn-taking, the pacing, the way words become sound, lives inside the model, past a wall you often cannot reach. You configure less because less is exposed, and you cannot easily build around what you cannot see.
So the useful question is never "which system is smartest" or "which is most open." It is narrower: does this system keep the specific controls my use case needs on my side of the wall?
The IC Matrix
Put the three architectures against the two axes and the picture becomes plain. I call it the IC Matrix, for Intelligence and Configurability, the two parameters every voice system should be measured on.
| Architecture | Intelligence | Configurability |
|---|---|---|
| Cascade sandwich (Deepgram + Piper) |
Low. Dumb parts; you supply all intelligence. | Total. You touch everything, which means you build everything. |
| Speech-language model (Hume) |
High. Emotionally aware, understands as it hears. | Low. The orchestration is sealed inside the model. |
| Speech-to-speech (OpenAI Realtime) |
High. A full mind in speech space. | Medium. Intelligent, and it exposed enough of the controls I needed. |
Read the table and you see two independent dials, not one. Each architecture sits somewhere on both. Today's options happen to spread out rather than pile into a corner that is high on both, which is exactly why the choice is real: you are deciding how much intelligence your problem needs, and how much control, and finding the architecture that gives you enough of each.
Stitching It All Together: The Seam
Here is what the two axes do not tell you. Even after you have sized both parameters and chosen your architecture, a gap remains between what the architecture hands you and a system that actually ships. That gap has a name. It is the seam.
My agent writes for the eye. Structure, bullet points, tables, headings, the occasional checkmark. None of that survives contact with the ear. Between my agent's answer and the spoken reply, that text has to be re-rendered into something a voice can say. And that re-rendering is not a pass-through.
What is the seam?
The seam is a translation between two representations of the same words: text optimized for reading, and text optimized for hearing. A list becomes a sentence. A checkmark becomes the word "done." A five-paragraph answer becomes three spoken sentences with an offer to go deeper. The voice model does not "ingest" my agent's output. It reads a version of it that I re-rendered for the ear.
The question of every voice architecture is: who owns that re-rendering seam? In a dumb cascade, it is entirely yours to build. In a sealed speech-language model, it is mostly the vendor's, hidden. The architecture does not just decide intelligence. It decides where the seam lives, and how much of it you build vs. delegate to an intelligent brain.
Every voice system will make you choose: how much of the seam you own, and how much you hand away. So I built Talkbox, an open-source harness that holds the seam, so a text-based agent can speak through any voice model without losing its mind in the handoff. The seam is the unfinished work of voice AI.
Written after building Talkbox, an open-source voice harness for text-based agents, and the four experiments in Giving My Agent a Voice.
Ideas, views, and opinions are my own and do not represent those of any organization I am affiliated with.