I had been running a personal agent for about five months. It remembered everything, checked my calendar, searched my notes, read my email, and answered with a personality that felt like a colleague rather than a tool. The input side was already solved, because I was dictating into it with Wispr Flow: my voice became text, text became context, context became answers.
But the answers came back as text. And I was still staring at my phone.
So I wanted to give it a voice. How hard could it be?
What began as bolting text-to-speech onto my agent turned into four failed architectures, three problems no one names, one working solution locked inside a company worth millions, an open one I built myself, and a conviction I did not expect: the future is not a multi-modal model, but a multi-modal harness.
If you just wanted to see it work, that was it. If you want the story of how I got there, and how to build one for your own agent, keep reading.
I was already turning my speech into text on the way in, so surely I could turn its text into speech on the way out, close the loop, and start talking to it.
It sounded simple, until it wasn't.
The moment I tried, three problems showed up that no text-to-speech engine can touch.
The format problem. My agent's answers are written to be read, not heard. Bullet points, headers, numbered lists, markdown tables. Spoken aloud, they sound like a robot reading a spreadsheet, because the way an answer is shaped for the eye is not the way it should arrive in the ear.
The silence problem. In text, you send a message and watch the indicator say "thinking," and that is fine. In voice, silence is violence. Ten seconds of nothing after a question does not feel like processing, it feels broken, because conversation carries a social contract of continuous engagement. Even a thinking human says "hmm" or "let me check."
The mute problem. When my agent runs tools, searches memory, and reads files, all of that work is actually happening, and you can even watch it scroll past in text: running bash, editing a file, writing to a note. It is not invisible. It is mute. Everything is there for the eye and nothing is voiced for the ear, as if the agent were doing the work with no way to say it out loud. You need someone narrating the work as it happens, like a sports commentator on a game you can hear but not see, not inventing answers, just keeping you with it.
None of these are solved by gluing a speech engine onto the output.
So I went digging, and I found the field split into two camps.
One camp is building voice-native agents: models built from the ground up to listen and speak. The trouble is that in this world the voice model is the agent, which means it has to do the real work itself, holding the tools, the memory, the reasoning. The other camp, quietly, is where the actual work happens: text-based, LLM-driven agents that already reason deeply, remember across a task, and call real tools. That is where the intelligence lives today, and where it will keep living for a while.
What almost nobody was building was the connector between the two. I wanted to keep the brain of the agent I already had and simply give it a voice, and I could not find an open way to do it. The one solution that came close, Vocal Bridge from Andrew Ng's AI Fund studio, did exactly this pattern, but it shipped as a paid, closed-source SDK. Someone had proven it was possible and then locked it away. So I decided to build the open one, and to get there, I ran experiments.
Four Architectural Experiments
I did not try four features. I tested four architectures for running an end-to-end voice system, in the order I actually ran them.
| Test | Architecture | Measured latency | Felt latency | Verdict |
|---|---|---|---|---|
| 1. Hume EVI | provider owns the voice loop | — | Inconsistent after turn one | Experimental adapter |
| 2. Provider-neutral runtime | swappable STT/TTS + instrumentation | (the measuring layer) | — | Benchmark foundation |
| 3. Deepgram + Piper | batch: STT → agent → TTS | ~12s, clean | Dead air. Walkie-talkie. | Baseline / fallback |
| 4. OpenAI Realtime | voice brain → ask_agent → agent |
~27s backend | Best. Silence filled. | Primary |
The first three all failed in the same way, for the same reason. Hume brought its own voice loop but took the controls away from me. So I built a provider-neutral runtime, swappable adapters and real instrumentation, to stop guessing and measure every approach on the same footing. The clean Deepgram-and-Piper sandwich then measured beautifully and felt broken, because you sit in silence until the whole answer arrives at once. None of these architectures could solve the format, silence, and mute problems, because in every one of them the voice layer was dumb plumbing and my agent was left holding a conversation it was never designed to have.
Then came OpenAI Realtime, and it changed the question entirely. Instead of a pipeline I had to orchestrate, it gave me a voice brain intelligent enough to own the conversation itself. The three problems I could name were the ones I had run into. The voice brain quietly handled the ones I could not, the whole messy category of what makes conversation feel human, without my having to solve any of it. And it did something the clean pipeline never could: it took more than twice as long on the backend, around twenty-seven seconds, and still felt faster, because it kept the conversation alive while my agent worked underneath it.
The slower path won. That gap between what the clock says and what the conversation feels like has a name worth keeping: perceived latency. It is the metric almost no benchmark reports, and it is the only one the user actually feels.
But even with an obvious winner in hand, I realized I wanted more than a winner. What the four architectures had really shown me was that I was not choosing between voice systems at all. I was standing between two entirely separate optimization engines, my reasoning agent and the voice brain, each excellent at its own job and neither built to do the other's. The winner told me which voice brain. It did not tell me how to make two different kinds of intelligence work as one. That question is where Talkbox begins.
(Hume EVI, Test 1, is a story of its own: a genuinely brilliant emotional-voice layer I ultimately could not build the core on. I wrote that up separately.)
What Talkbox Actually Is
Somewhere in the middle of those four experiments, the project quietly changed shape. I had started out hunting for a connector, a bit of glue to get a voice onto the agent I already had. What I actually walked away with was an understanding. I now knew what a voice model is really optimizing for, what my text agent was genuinely capable of, and exactly where the gap between them opened up. The winner was not just OpenAI Realtime. The winner was seeing the problem clearly.
And once I saw it clearly, the gap had a precise shape: no voice system, on its own, is built to pair with a powerful text-based agent. The voice-native ones want to be the agent. The dumb pipelines leave your agent solving conversation problems it was never designed for. Something has to sit between them and hold the line. What began as a bold hack turned into a principled one: not a patch over the gap, but a boundary designed for it.
That something is Talkbox, a voice harness whose heart is a single idea: a boundary and a set of policies that let a voice model behave like your agent without becoming your agent. The voice layer manages the social contract of conversation, the acknowledgement and pacing and filler, while your agent stays the sole owner of facts, memory, tools, and truth. Talkbox draws that line and holds it.
There is a deeper reason the boundary has to exist, and the experiments are what taught it to me. A large language model optimizes for reasoning, for holding the past and projecting the future and working a problem across time. A voice model optimizes for something else entirely: conversation, rapport, the here and now, the felt sense of being heard in this exact moment. These two are not orthogonal, but they are not the same, and the difference is temporal. Reasoning lives across time. Rapport lives in the present. Talkbox exists because you need both, and neither should have to pretend to be the other.
Everything Talkbox does comes from one principle: put each job in the right hands, and make the join invisible. The reasoning agent and the voice brain each do what they are good at, and the seam where they meet disappears. Every feature is one of those two jobs.
Putting each job in the right hands.
It gives the agent the truth and the voice the talking. Talkbox hands the voice model exactly one thing it can ask for, ask_agent, and one rule it cannot break: you may run the conversation, but you may never invent the answer. Facts come from the agent. Charm comes from the voice. Neither reaches into the other's job.
It lets the voice sound like your agent, without being it. Talkbox takes your agent's name, tone, and character and hands them to the voice, so the person hears your agent speaking, not a generic assistant reading your agent's words aloud. The identity belongs to the agent; the performance belongs to the voice.
It gives the voice something to say while the agent thinks. When the agent starts a long piece of work, Talkbox turns that work into things worth saying out loud, so you hear "let me check that, still pulling the details" instead of dead air. The agent does the thinking. The voice keeps you company while it happens.
Making the join invisible.
It keeps one conversation, not two. Whether you speak or type, it is the same thread with the same memory, so the handoff between voice and text never fractures into two disconnected chats.
It smooths the first second and the last inch. A greeting that knows your name, an answer reshaped so it lands in the ear instead of the eye. Small touches, all in service of one thing: you should never feel the seam between the two systems, even though it is right there, doing all the work.
The technical details of each of these live in the docs. What matters here is that not one of them is arbitrary. Every feature is Talkbox either handing a job to the system built for it, or hiding the place where the two systems meet.
How It Works
It is pluggable on both sides. Any voice provider through adapters, any text-based agent through adapters, because the harness does not care what sits on either end. Cal is the reference implementation, but the generic HTTP adapter works with any agent that accepts text and returns text.
Not a Multi-Modal Model. A Multi-Modal Harness.
Here is the insight I did not expect to find at the bottom of a voice project.
For a while the industry has been racing toward the multi-modal model: one giant model that sees, hears, speaks, and reasons, all at once. Building Talkbox left me doubting that is where the real leverage is. Reasoning and rapport optimize for different things, on different timescales, and forcing one model to be excellent at both may be asking a single mind to live in the past and the present at the same time.
What if the future is not a multi-modal model, but a multi-modal harness? A modality-specific harness for each modality, each one optimized ruthlessly for its own job, reasoning here, rapport there, vision somewhere else, all of them stitched together through a seam that lets them work as one. Not one mind pretending to be good at everything, but several, each excellent, coordinated.
Talkbox is one small attempt at that seam, the one between reasoning and rapport. Voice is not a feature you bolt onto an agent. It is a modality with its own physics, and it deserves its own harness. The pieces already exist: voice models that sound human, agents that reason deeply. What was missing is the protocol between them.
That's Talkbox. A voice harness for your text-based agents. Open source, MIT licensed, bring your own voice and bring your own agent.
In funk music, a talk box takes a guitar and makes it speak. This one takes your agent and gives it a voice.
Ideas, views, and opinions are my own and do not represent those of any organization I am affiliated with.