Why your voice agent stops mid-sentence — and goes quiet when you need it most
Part 1 of 8 in our series on Context Aware Turn Taking (CATT): how we are building voice agents that converse like people, on the sub-millisecond Psyclone AIOS platform.
Voice agents have become remarkably good at what they say. The models are fluent, the voices are warm, the answers are often right. And yet a few minutes into a conversation most people can tell they are not talking to anyone who is really listening. The problem is not the words. It is the timing.
We have spent the last months building and measuring a voice agent that runs on the Psyclone platform, in real conditions: at a desk, in a car, on a train, in a noisy stadium. Three failures come up again and again. None of them is about intelligence. All of them are about turn-taking — the split-second social machinery that decides who speaks when.

Failure 1: it can’t tell an interruption from a noise
You ask a question. The agent starts a thoughtful answer. Halfway through you say “mm-hm” to show you are following — and the agent stops dead. Or a door closes, someone coughs, the car indicator ticks, and the answer is gone.
This is not a rare edge case; it is how most stacks are built. Many voice pipelines treat any detected speech while the agent is talking as a barge-in. OpenAI’s Realtime API, for example, defaults to silence-based voice activity detection with interrupt_response switched on. LiveKit’s documentation openly describes “false interruptions” — the agent hears speech-like audio, stops, finds no words in the transcript, waits out a timeout (the documented example uses two seconds of silence) and then resumes from where it left off. Both are sensible engineering compromises. Neither is how a person behaves.
Humans do not stop talking every time the listener makes a sound. We distinguish, instantly and mostly unconsciously, between a backchannel (“yeah”, “right”, “mm”), an agreement, a nudge to hurry up, a soft interruption we can talk over, and a genuine “stop, that’s wrong”. The difference is in the timing, the energy and — above all — the context.
When we measured an earlier, purely acoustic interruption policy of our own against SID-Bench — a public set of real recorded interruptions and backchannels from the Qwen team (ICME 2026) — it stopped for nearly four out of five backchannels. That is the baseline the industry is quietly living with.
Failure 2: long silence, no sign of life
Ask an agent to do something real — check a calendar, search a document, book something — and it often goes quiet for five, ten, fifteen seconds. Did it hear me? Did it crash? Should I repeat myself? (And if you do repeat yourself, you have now interrupted it.)
A human assistant would never do this. They would say “let me check”, make a small sound of effort, give a progress cue, or simply look busy. The bolt-on fix many products use — a canned “one moment please”, or a looping thinking sound — helps a little but is not connected to what is actually happening, so it is easy to over-promise or to talk over the real answer when it finally arrives.
The deeper issue: in a typical cascaded pipeline (speech-to-text → language model → text-to-speech), the part that talks has no idea what the part that thinks is doing, how long it will take, or whether the user has already heard the last thing that was said.
Failure 3: it cuts you off when you pause to think
The classic one. Most systems still decide “your turn is over” from a stretch of silence. But silence is context-blind: it cannot tell “I’ve finished” from “I’m thinking” from “hang on, I’m looking for the number”. Set the silence timer short and the agent cuts people off; set it long and every exchange feels sluggish. Learned end-of-turn detectors from LiveKit, Deepgram and others have improved this considerably — and in our own paired re-scoring of LiveKit’s published turn-detection predictions, the best of them reach a 10 % cut-off rate after roughly 420–500 ms of silence, where a plain silence timer needs about 960 ms. But they answer only one question — has the user finished? — while real conversation keeps asking many others.
Why these are the same problem
All three failures come from the same design choice: turn-taking is treated as one yes/no decision driven by one signal — usually silence or voice activity — evaluated on one timescale.
Human turn-taking is nothing like that. Decades of conversation research, from the classic work of Sacks, Schegloff and Jefferson to layered real-time models of dialogue such as Kristinn R. Thórisson’s Ymir architecture, describe it as many processes running at once, at very different speeds: reflexes in tens of milliseconds, prosody and gaze over hundreds of milliseconds, meaning and intent over seconds — all shaped by the context of the conversation.
Our approach: Context Aware Turn Taking
CATT is a turn-taking layer for voice agents that uses hierarchical context to decide what a sound, a pause or an overlap actually means — and what the agent should do about it. It sits between any speech-to-text, language model and text-to-speech stack and the agent itself, so teams already committed to LiveKit, Pipecat, Vapi or a realtime model do not have to rip anything out.
Instead of one detector and one threshold, CATT runs many small detectors at many timescales, from sub-millisecond reflexes up to second-scale reasoning, and combines them through layered deciders into a turn decision that can be explained afterwards: why did the agent keep talking, why did it yield, why did it say “let me check”?
That only works if the platform underneath can move thousands of small messages between those detectors fast and predictably enough that the decision is made before the moment has passed. That platform is Psyclone AIOS, our open-source real-time orchestration runtime — and the reason we built CATT on it. (Our recent benchmark paper shows how Psyclone’s messaging compares with NATS, ROS 2, iceoryx and others: Psyclone AIOS benchmark report.)
What this series will cover
Over the next four weeks we will build up the picture, twice a week:
- Part 2 — Conversation is not ping-pong. What people actually do when they talk: backchannels, overlaps, urging-on, soft and hard interruptions — and why each needs a different response from the agent.
- Part 3 — “Did it hear me?” The uncanny valley of timing. The social side: why replying too fast can feel as wrong as too slow, how honest fillers and progress cues keep a conversation alive during long tasks, and how a fast “System 1” and a slower “System 2” can share one voice without talking over each other.
- Part 4 — The network is part of the conversation. What we measured on real mobile networks — in cars, on trains, in crowds — and why “silence” and “no audio has arrived yet” must never be confused.
- Part 5 — Many detectors, many timescales. How CATT is structured on Psyclone, and why a sub-millisecond message bus matters for something that happens at human speed.
- Part 6 — A digital twin of the listener. Latency-aware turn-taking: keeping a live model of each device and connection, including what the user has actually heard, and reasoning with it.
- Part 7 — Measuring turn-taking honestly. Our evaluation method, checked against a vendor’s own published report, and our results so far.
- Part 8 — The conversation, as audio. A recorded discussion that ties the series together.
If you are building voice agents and recognise these failures, we would like to hear from you — get in touch, or follow the series on LinkedIn.
