Act-Aware Turn-Taking

A while back, I wrote that the hard part of voice AI isn't latency or voice quality, it's behavioural science. Since then, I've been focused on one of the toughest problems in voice experiences today: interruptions.

Humans make noise to indicate that they are actively listening. For example: "Yeah." "Okay." "Mm-hmm." Linguists call these signals "backchanneling." In some languages these cues aren't optional: in Japanese they're called aizuchi, and skipping them reads as checked out, or even rude.

Voice agents hear all of this as interruption, leaving the politest, most engaged callers with the most broken experience.

Take the word "yes." When the agent is in the middle of explaining something, a caller's "yes" just means keep going. Right after a yes/no question, that same "yes" is the answer. The audio is identical; only the moment changed. To a voice agent, these moments are ambiguous, so it guesses. Guess wrong, and it either stops for a nod or talks straight over an answer.

Two weeks ago, OpenAI replaced ChatGPT's default voice mode with GPT-Live: a full-duplex model that listens while it speaks. I had hoped it would solve this backchanneling problem for good.

Early tests showed it handles a murmured "mm-hmm" fine. But a plain "ok" while it's answering, meaning nothing more than "I'm listening," can stop it mid-sentence.

Recently I had a breakthrough. Most backchannel and interruption solutions focus on what the caller says, in isolation. They study the energy and tone of the caller's audio to decide if something was a backchannel, and that helps a little. But the real signal is what the agent is saying at that moment, and what that speech invites.

Then I learned I wasn't the first. In 1974, the sociologists Sacks, Schegloff, and Jefferson published a study of how people take turns in conversation. Their finding: listeners don't wait for you to finish; they project what's coming from what you're saying. Schegloff later wrote an entire paper about "uh huh." He called these tokens continuers: signals that just mean keep going. Whatever you say to someone, you can roughly anticipate what comes back. Explain something, and you can expect small signs of listening. Ask a question, and you can expect an answer. The question-answer case has a name too: the adjacency pair. Every action invites a matching response.

So I've been working on a project called act-aware turn-taking. As our agent speaks, it tags what each moment invites. Mid-explanation: expect nods, talk through them. Right after a yes/no question: the next "yes" is an answer, stop and listen. Waiting on a booking reference: expect letters and numbers. The listening side stops guessing and starts anticipating.

Now, when that ambiguous "yes" arrives, the agent already knows which one it invited and what to do with it.