What actually goes wrong in AI chat

01

What goes wrong

I had a hunch I wanted to check: that conversational UI is oversold, that it is good at exactly one thing — capturing fuzzy intent when the user does not know the name of what they want — and bad at everything after that. Standard practitioner grumbling. I wanted to know whether the research backed it or whether I had just been repeating myself.

Short answer: mostly yes, but not for the reasons I assumed, and the interesting findings were in places I was not looking.

So this page is in three parts. What goes wrong when the interface is a conversation. What design can do about it. And what is left after design has done everything it can, because some of this sits in the model and some of it sits in people, and neither is mine to fix.

It is mostly not the model

Before the list of problems, the thing that let me sort them. Most of the issues below carry a layer tag, and the tags turned out to be the argument.

The thing that made everything else click was realising that “AI chat is bad” collapses several unrelated failures happening in different places. A message passes through seven stages.

The fact that reframed the whole thing for me: in ordinary voice chat, the model at L4 receives plain text and cannot tell a microphone was involved. Voice adds a layer before the model and changes a layer after it. It does not touch the model.

So “voice AI is worse” is true, but not because the AI is worse at voice. And once you tag each problem with a layer, only two of them live at L4, in the model itself: losing the thread across turns, and agreeing with whoever is talking. Everything else is capture, assembly, render or persistence, or it is in the person.

That is an interface problem, not a model problem. Which means it is mine.

The model loses the thread across turns

The finding first, then how I got to it. The same instruction scores about 65% when it arrives a piece at a time, and about 86% when the same words arrive in one message. Nothing else changed. The cause is the turns and not the length, and I spent a while believing the opposite.

The wrong turn that turned out to be the useful one

Early on I got fixated on a theory. If long AI conversations degrade, maybe it is because voice dictation produces longer, more rambling prompts than typing does — more tokens, fuller context window, worse performance. Voice as the culprit.

It is a clean theory. It is also wrong, and the paper that establishes the degradation had already tested it: LLMs Get Lost In Multi-Turn Conversation (Laban, Hayashi, Zhou & Neville, May 2025), from Microsoft Research and Salesforce Research. This is the one paper here I have read properly rather than skimmed, and it is worth the time.

Their setup: take instructions from existing single-turn benchmarks and shard them — break one complete instruction into pieces that each reveal one element, then feed them to the model one per turn. Compare against giving the whole thing at once.

They ran five conditions. Three matter:

• Full — the original complete instruction, one turn
• Sharded — the same information, revealed one piece per turn
• Concat — the same shards, in the same clumsy rephrased form, but all glued together into one message

Scale: 15 models across 8 families, 600 sharded instructions across 6 tasks (code, text-to-SQL, API calls, maths, data-to-text, multi-doc summary), 10 runs per combination. More than 200,000 simulated conversations, roughly $5,000 in compute.

That is a 25-point drop between the two averages in the table. The paper's own headline figure, the drop averaged per model, is −39%, and it happened to every model on every task.

And Concat is the kill shot for my theory. Same words. Same awkward sharded phrasing. Same token count. Delivered in one message instead of several: performance essentially intact. So it is not length, it is not wording, it is not context bloat. It is the turn structure.

So what is causing it

Their Appendix F identifies four mechanisms. This is the part I would have missed reading only the abstract:

1. 1Premature answer attempts. The model produces a complete solution on turn one, when it only has a fragment of the requirement. To do that it must invent the missing parts.
2. 2Answer bloat. It leans on its own previous, wrong attempts, so answers get longer and more encrusted with earlier assumptions.
3. 3Loss of middle turns. It over-weights the first and last turns of the conversation. Everything in between gets thinner. This is a conversational cousin of Lost in the Middle (Liu et al., TACL 2024), which found the same positional bias inside a single long context.
4. 4Verbosity. Long responses introduce assumptions that then compete with the user's actual words for the model's attention.

The bit that surprised me most: it shows up at two turns. They ran a gradual sharding experiment splitting instructions into between 2 and 8 pieces, holding task complexity constant. Degradation appeared from two shards onward, and the granularity barely mattered. Their conclusion is that providing all the information at once is the only effective method to improve reliability. Not fewer turns. One turn.

Where it does not happen

They also sharded a translation task, and it did not degrade at all. Their explanation is the useful part: translation is episodic — each turn adds sentences to translate and append, so it decomposes into independent subtasks.

They propose three properties that predict whether a task will break:

1. 1It is generative — editing and refining new content, not extracting or classifying
2. 2It is complex enough to have multiple explicit specifications
3. 3The solution is non-decomposable — a new requirement changes the whole answer, not just appends to it

That is a genuinely useful test. Most of what I would want to build conversationally in a banking context fails all three, which is to say it is exactly the vulnerable kind.

Their own caveats, which I should not skip past

• Users were simulated by GPT-4o-mini, not real. They say plainly these are not representative of human-AI conversation.
• The simulation is idealised — the conversation is guaranteed to end with enough information to solve the task. A real user might never get there.
• Their claim: the degradation is probably an underestimate of real-world conditions, because real conversations include misunderstandings, frustration and abandonment.
• English only, text only, analytical tasks only. Creative tasks untested and flagged as future work.

Whether any of this can be patched from outside the model is a separate question, and the answer is in part three.

The confusion I need to keep straight

I conflated two things and it took a while to untangle. Writing it down so I do not do it again.

Both are incomplete information at generation time. Different causes, different layers, non-transferable fixes. I was treating the second as a symptom of the first.

People do not say what they mean, and the model does not ask

Two separate causes, and both are documented.

Search habits. NN/g's diary study — 18 participants, 425 conversations across ChatGPT 4.0, Bard and Bing Chat over two weeks — found people carrying the search-engine model straight across, most of all when a bot was new to them. One of the six conversation types they catalogued is the search query: one prompt, no refinement. Across the whole study a conversation averaged 3.6 prompts, and Bard's, where the search habit ran strongest, were the shortest.

Saying more is genuinely hard. NN/g calls this the articulation barrier: the blank box demands you already know what the system does, pick the right capability, phrase it in a way that works, and judge the output — four independent failure points.

The academic version, which I like better, is Bridging the Gulf of Envisioning (Subramonyam et al., CHI 2024): a capability gap (can it do this?), an instruction gap (how do I say it?) and an intentionality gap (what should I expect, and how would I judge it?). Their point that forming a complete intention is itself cognitively demanding work is the one I would put in front of anyone who calls chat frictionless.

And Why Johnny Can't Prompt (Zamfirescu-Pereira et al., CHI 2023) shows what happens when people try anyway: participants were exclusively opportunistic, none ran more than one or two conversations or followed a single one to completion before jumping in to fix things, and a specific barrier was expectations carried over from instructing humans — assuming it would generalise from an example the way a colleague would.

Two numbers that are the sharpest version of this, both chased to their papers:

• Chang, Anderson and Hofman, in ChatBench (ACL 2025): given a fully specified benchmark question to put to a model, only 39.8% of people's opening prompts were a near restatement of it. The rest asked a related, often ambiguous question, leaving out something the answer needed.
• Shaikh, Mozannar, Bansal, Fourney and Horvitz, in Navigating Rifts in Human-LLM Grounding (ACL 2025): models were sixteen times less likely than humans to make a follow-up request, and three times less likely to ask for clarification.

The second is the whole design argument in one number: the fix for underspecification is asking, and the system does not ask.

That number has a name in a much older literature. Clark and Brennan called it grounding: two people in a conversation keep checking that they mean the same thing, and they do it constantly, with a nod, a “sorry, which one?”, a repeated word. It is work, and both sides do it. A model that follows up at a sixteenth of the human rate is not a worse conversational partner by degree. It has stopped doing half of what makes a conversation work.

And people do not adjust for that, because they cannot help bringing the habits with them. Nass and Moon showed in 2000 that people apply social rules to computers without deciding to: politeness, reciprocity, the lot. Luger and Sellen found the cost of it in 2016. The more human the assistant seemed, the more intelligence people assumed, and the further they fell when it was not there. So the user talks as if to a person, waits for the questions a person would ask, and gets a confident answer to something they only half said.

Chat has no surface

What NN/g found by just watching people

Accordion Editing and Apple Picking (Gibbons, Mugunthan & Nielsen, Sept 2023) — 8 participants, 90-minute sessions, their own real tasks. Small n, but it is observational and the behaviours are specific enough to be useful.

Accordion editing (L1/L5). Users repeatedly ask the model to shrink or expand to hit one goal. Shrinking via forced ranking or word count; expanding by asking it to elaborate or add something it missed. One participant needed a post under 280 characters and had to keep saying shorten — she described it as the tool generally not listening. Every one of those is a round trip doing the job of a slider.

Apple picking (L7). Referring back to part of an earlier response. There is no way to point, so people type or copy-paste a quote of the bit they mean. The key detail: users cannot hold accumulated output in working memory and have to scroll back, then get lost scrolling. Observed with all participants.

Editing outside the tool (L7). Most participants finished the job in a text editor — the model could not add current information, could not hit the right level of detail, could not format the way they wanted.

Voice: input side

Mishearing, silently (L2). The scariest one for anything transactional. Transcription is a separate system; it hands the model text; the model has no idea anything was garbled and answers confidently. In voice mode you never see your own words, so you cannot tell whether it misheard or misunderstood.

• A field study of LLM voice assistant use analysed 447 real queries — system errors and speech-to-text inaccuracies compromised 23% of them (arXiv:2408.16465).
• Hong and Findlater (CHI 2018) had sighted people listen to speech output of their own dictation: they caught only about half of the recognition errors. The follow-up with 12 blind and 12 sighted participants put blind screen-reader users at 40%, and not significantly different from sighted.

Say “transfer to my Cheltenham account”, get “Chatham”, get a fluent confirmation about the wrong account. Nothing flags it.

No draft state (L1). Typed input sits in a box you can revise before sending. Speech commits the moment you say it. I cannot find anyone who has studied this directly, which surprises me.

Endpointing (L1). Covered above. Long-established as a VUI problem (Murad et al., MobileHCI 2018) but I have found nothing quantifying it for LLM voice specifically.

Voice: output side — where I was most wrong about the literature

I had assumed this was uncharted. I told myself the output-side problems were my own analysis rather than documented findings. That was wrong, and it was wrong because I was searching the LLM literature when the answers were sitting in spoken conversational search and cognitive load theory — two fields that had worked this out before LLMs existed.

Speech cannot be scanned (L6/L7). The Dagstuhl Seminar 19461 report on Conversational Search — the field's own agenda-setting document — states it plainly: speech-only output is hard to process given cognitive and memory limits, and the temporally-linear, ephemeral nature of speech limits the ability to scan results. It calls for research into result summaries, tagging results for later reference, and highlighting which parts matter.

That middle one is apple picking, named as an open problem from the other direction.

Shortening loses content (L6). An N=139 study of an LLM-based voice agent compared long and short spoken responses across easy and hard search tasks. Short responses reduced cognitive load and improved satisfaction — on easy tasks. On hard ones, short responses made users ask more questions, which the authors read as hunting for what got cut. Their conclusion: no single response style works everywhere.

So voice answers are not repackaged, they are smaller — and the user pays it back in extra turns, which drops them straight into the multi-turn problem.

Formatting does not survive TTS (L6). Bold works because your eye sees it against non-bold. Speech has no “around it”. There is a whole document-to-audio research programme on this: current synthesis does not carry the meaning of visual elements, so complex structures come out ambiguous; prosody derived from how humans read tables aloud helps but is not adequate alone for large tables read linearly; adding earcons and spearcons improves things, notably listening effort (overview, prosodic mapping work).

There is a reason none of this improves with a better voice. That is in part three as well.

The cleanest voice-vs-text difference I have found

In text, the earlier answer is still on screen. Scrolling and copying is effortful but the referent exists. In voice, it does not. To reference the third of five spoken suggestions you would have to hold a long answer in memory, recall the exact wording, and say it back accurately enough to match. And there is no scrubbing back through audio the way you scroll text.

Trust drifts the longer you talk

Sycophancy escalates with conversation length. SycEval (Stanford) found sycophantic behaviour in 58.19% of cases across GPT-4o, Claude Sonnet and Gemini 1.5 Pro; regressive sycophancy — agreeing your way to a wrong answer — in 14.66%. Another paper tracks it rising from 22% to 71% across turns.

And it changes decisions. CHI 2026, N=106: opinion agreement reinforced people's initial decisions and made them less likely to change their minds even when the model presented opposing evidence.

That is a problem for any review or approval flow. The interface is biased toward confirming whatever the user walked in with, and gets more so the longer they talk.

Fluency suppresses checking. Longer, more elaborate explanations raised confidence without improving people's ability to tell right from wrong (Steyvers et al., Nature Machine Intelligence 2025, n=301). And only 26% of people correctly noticed when a system was overconfident; overconfident systems produced a misuse rate of 41% against 28% (arXiv:2402.07632).

Accessibility — flagging, not yet dug in

“I'M BLIND, ChatGPT” (CUI 2026) makes a point I had not considered: for visually impaired users, voice AI does not remove barriers, it rearranges them. The system's voice, the screen reader's voice and human speech all compete for one auditory channel, which makes timing, silence and overlap unusually consequential. And it is worse in LLM systems because they support long open-ended exchanges rather than short command-and-response.

PETRA 2025 inspected four generative AI tools with screen readers: accessibility limited in key respects across all four.

I want to come back to this properly. My instinct is that a conversation stream is a live region firing constantly with no landmark structure and no predictable focus order, and that most of these products would not survive a serious 2.1 AA audit on 4.1.3 and 2.4.3 alone — but that is my instinct, not something I have tested or found tested.

02

What design can fix

Each of these answers a problem from part one, roughly in the order they came up. Two things do not line up, and I would rather say so. The last bet, on saying what the thing is, answers the expectations problem that sat inside the section on people not saying what they mean. And accessibility gets no fix here. It comes back in part three, because what it runs into is memory.

Not conclusions. Just what I would bet on today, and I would change my mind on any of these with the right evidence.

Get the whole request in before the model answers

Concat scoring about 86% against Sharded's 65% is hard to argue with. And it points at forms, structured intake, or a system that asks before it answers — which is exactly what Shaikh et al. measured models not doing, at a sixteenth of the human rate.

Laban's paper ends on advice for users, not builders.

Their advice to users is almost comically low-tech and I think genuinely correct: start a new conversation, and before you do, ask the model to consolidate everything you have told it so far, then carry that into the fresh chat. They note Cursor users independently arrived at the same habit.

Put differently: if the model will not do the grounding, the interface has to. A form is grounding done in advance. A bot that asks three questions before it answers is grounding done out loud. Either way the whole request arrives in one piece, which is the one thing that paper found to work.

Conversation for capturing intent; structured UI for committing to anything

NN/g's own fix list is a list of ways to stop being a conversation:

• Compartmentalised responses — edit one part without regenerating everything
• Direct editing — change the text yourself
• Point-to-select — click the bit you mean

Every one of these needs a persistent surface to point at. Hold that thought; it becomes the whole voice argument later.

Speak the request, show the answer

This one matches the layer analysis: voice is fine at L1, and bad at L6 and L7.

A screen measurably helps (L7). Screen or No Screen? — real-world deployment with older adults, voice devices with and without a touchscreen. Voice was the preferred way to put information in; visuals were valuable for taking information out. Participants described starting to answer on the screenless device and forgetting the question. Seeing the rating options displayed made them easier to hold than hearing them.

That is the empirical backing for speak the request, show the answer, and it is the single most actionable finding I have got.

Never confirm anything consequential in audio alone

Half of transcription errors undetected, no way to re-read, no way to emphasise the number. In a banking flow that is three failures stacked on the one screen that matters most.

The fix is dull. Show people the words the system heard, and put the amount, the account and the date on a screen before anything happens.

Measure behaviour, not stated trust

Hedging changes behaviour but not stated trust. CUI 2026, financial domain, N=71: people rated hedged and unhedged systems as equally trustworthy, then were significantly less likely to follow the hedged advice. A trust survey would have detected nothing.

I keep coming back to that one. It is a methods warning as much as a finding.

Say what it is

The last one I am least sure of, because it rests on a paper I have only read the abstract of. That is why it is not in the list of five at the top. Pataranutaporn and colleagues (Nature Machine Intelligence, 2023) told people different things about the motives of the same chatbot before they used it. A positive or a neutral introduction steered what people believed about it, and with that how trustworthy, empathic and effective they rated it. Same system. So the introduction is a design material, whether or not anybody designed it.

My bet, and it is only that: an assistant introduced as what it is sets an expectation it can meet. Luger and Sellen's people were let down by the distance between the human manner and the machine underneath. Closing that distance from the manner side is cheap, and nobody has to retrain anything.

What this would change in the TD portal

Transactional flows are the weak spot. 200 real conversations with a transport chatbot: informational requests generally handled, transactional requests often understood but then handed to a human (Taylor & Francis). Which is to say the flows that justify the investment are the ones that work worst. Paying a loan is a transaction, so that is the case I have to design for.

The design work on this site is a portal where customers pay a car loan, and one of its research themes was whether customers wanted an app or a chat. The answer was a chatbot first. Read against everything above, that chatbot would be built to three rules I did not have words for at the time:

1. 1A chat entry point never commits. Whatever a customer starts in the chat, a payment or a due-date change, finishes on the review screen the portal already has, with the amount, the account and the date on it. Speak the request, show the answer. A confirmation spoken back, or typed back, would stack the half-missed-errors problem on the one screen that matters most.
2. 2The bot asks before it answers. Models make follow-up requests at a sixteenth of the human rate, and six people in ten do not put the whole question in their first prompt. So the chatbot's job is intake: the amount, the account, the date, then the handover. The 3.4-click payment path stays the place where anything is committed.
3. 3Test what people catch, not how they feel. The confirmation step IT wanted made nobody more confident in the prototype, and that is why it did not ship. The hedging study says a confidence question would have told us nothing either way. The next round seeds a wrong account or a wrong amount and measures who notices.

Look at what most of these have in common. They work by leaving the conversation, or by putting a screen next to it. That is fine until the task really is a conversation, or there is no screen. Part three is about that.

03

What remains

Everything in part two assumes there is something for design to push against. Some of what goes wrong has nothing there. I have sorted what is left into two piles, because they are not the same kind of problem. One pile belongs to the model, and might shrink. The other belongs to people, and will not.

What the model cannot do yet

Start with the multi-turn result, because the authors tried to fix it and wrote down what happened.

Concat had already ruled out length and wording. They checked two more explanations, and neither held:

• Temperature does not fix it. Dropping to T=0 for both the model and the simulated user still left about 30% unreliability in the sharded condition. In single-turn it cut unreliability by 50–80%. So it is not just sampling randomness — one token of divergence early in a conversation cascades.
• Reasoning models do not fix it. o3 and Deepseek-R1 degraded like everything else. The authors flag a possible reason: reasoning models produce responses about 33% longer, and longer responses contain more assumptions.

Mitigations they tested

• Recap — replay all prior user turns in one final message. Lifted GPT-4o from 59.1 to 76.6, against 93.0 for Full. But it requires knowing which turn is the last one, which you cannot know live.
• Snowball — restate everything revealed so far at every turn. Recovers only 15–20% of the loss.

Neither gets back to Full. Their conclusion is that you cannot paper over this in the orchestration layer.

So the only reliable fix for a conversation that degrades is to stop having one. That is what the first rule in part two does, and it works when the request can be gathered up front. It does not work for the tasks Laban's three properties pick out: writing, refining, anything where a new requirement changes the whole answer. Those are conversations because they have to be. For them I have nothing, and neither does the paper.

It agrees with you, and you prefer it

Part one had agreement rising with the length of the conversation. What makes it a limit is who wants it. Cheng and colleagues (Science, 2026; I have read the abstract, and the preprint for the method) measured how often eleven models took the side of the person asking for advice, against how people had answered the same questions. The models did it 49% more often. Then three preregistered experiments with 2,405 participants: even a short exchange with an agreeable model left people less willing to repair a conflict and more sure they had been right. They also rated that model as better, trusted it more, and preferred it.

So the behaviour that does the harm is the behaviour people choose. The authors make the point themselves: preference is the incentive to keep building it. They ask for changes to how models are trained and evaluated. That is design, but it is not mine. From the interface I can hedge an answer, and part two says how to check whether the hedge worked. I cannot make people want to be disagreed with.

I said this pile might shrink, and I should be careful with that. Multi-turn reliability is an engineering problem, and those get solved. Agreeableness is a commercial one, and those get solved when someone stops being paid for them.

What people cannot do at all

Memory

The mechanism (L6 plus human memory). The transient information effect from cognitive load theory. Persistent information lets working memory offload — you do not hold item 1 because it is still on screen. Transient information forces you to retain everything earlier while processing everything later. The spoken dialogue literature puts it as: in an audio channel you must remember what was said and reason about it simultaneously.

This is the strongest thing in these notes and it is not an AI finding at all. It is human memory, established decades ago. Nothing on any roadmap changes it.

Go back to formatting for a moment. Bold works on a page because the eye sees it against the text around it. Speech has no “around it”.

Which means this is not a synthesis-quality problem. Better voices do nothing. The best available fix is bolting a separate audio signalling system onto the side, and that has limits.

Then look at NN/g's fix list again — compartmentalised responses, direct editing, point-to-select. All three need something on screen. The entire repair path available to text chat is structurally closed to voice.

Part two's answer to all of this was a screen. Sometimes there is not one. A driver cannot look. A blind user has the screen reader, the assistant and the room all competing for one pair of ears, and caught 40% of dictation errors in Hong's study, about the same as sighted people listening. For them “never confirm anything consequential in audio alone” is not a rule. It is a description of the only channel they have. I do not know what the right design is. I know it is not a better voice.

The nearest thing to an answer I have seen is in the note on smart glasses: the assistant takes the easy questions, and one phrase hands the camera to a sighted volunteer for the ones that matter. It does not fix the channel. It puts a person at the end of it.

Knowing what you want

A form is grounding done in advance, I said. It is, when I know the fields. Amount, account, date. Subramonyam's point in part one was that forming a complete intention is work, and a form does that work for the user only where I have already done it for them. Ask someone what they want from a redesign, or a holiday, or a difficult email, and there are no fields. The chat box is honest about that at least. The gap it exposes is in the person's head, and it was there before the interface was.

Fluency

People take a long, confident answer as a sign of a right one. Part one has the numbers: longer explanations raised confidence without making people better at telling right from wrong, and only about a quarter noticed when a system was overconfident. Hedging moves behaviour a little, which is why it is in part two. It does not remove the reflex, and no amount of typography makes the other three quarters suspicious of a good sentence.

Put this next to the agreement a few paragraphs up, because they are not two problems. The model leans towards the answer I already hold. I take a well-made sentence as a true one. So the answer I am least likely to check is the one that agrees with me, said well, and that is the answer I am most likely to get. One half of that is the model's and may be fixed. The other half is mine and will not be. Each study here measured one half. I have not found one that measures the two together, so the compounding is my inference and not a finding.

Knowing it is a machine

This is the one I did not expect, and I have only the abstracts. Yin, Jia and Wakslak (PNAS, 2024) found that replies written by a model made people feel more heard than replies written by people. Then people learned the source, and the effect went. Rubin and colleagues (Nature Human Behaviour, 2025) ran nine studies with 6,282 people on the same question: identical responses, labelled as human or as AI. The human label won, across every variation they tried, and merely suspecting a machine had helped was enough to lose it.

So the words can be better and still be worth less. There is nothing to design there. The only way round the label is to lie about it. Most of what I would build is payments and dates, where nobody needs to feel heard. But loan servicing also has the conversation with someone who cannot pay this month, and that one should go to a person.

The same effect holds for pictures, and the note on retro design follows where it leads: identical images lose credit the moment a machine is named as the maker.

More of it is not better

Last, the measure everyone reaches for. Fang and colleagues ran a four-week randomised trial on ChatGPT with 981 people and more than 300,000 messages, in a three-by-three design: text, a neutral voice or an expressive voice, against open-ended, non-personal or personal topics. Whether people typed or talked, and what they talked about, made no significant difference to how lonely they were or how much they saw other people. How long they spent did. More daily use predicted more loneliness, less socialising, more emotional dependence and more problematic use.

Two caveats, because the paper has them. The conditions were not entirely inert: people given personal topics showed lower dependence, not higher. And this was ChatGPT, used by English speakers in the US on assigned topics, measured on wellbeing. It was not a bank's chatbot and nobody was paying a car loan. The same team also analysed over three million real ChatGPT conversations and found the markers of emotional dependence concentrated in the heaviest users, roughly the top tenth. That is the same trial reported twice plus one independent analysis, not two trials agreeing.

What carries over is narrower, and I think it still matters: more use is not evidence of more value. Put that next to Cheng, where people preferred the model that flattered them, and engagement stops being a number I would report as success. I would watch it as a warning, and count what people got done, what they caught, and whether they had to come back to fix it.

04

Where this leaves me

I started with a hunch: conversational UI is oversold, good at capturing a fuzzy intent and bad at everything after it. The research mostly backs that, and for better reasons than I had. Conversation is good at the first step because that is the step where the person does not yet know the fields. It is bad after that because committing to something needs a surface: something to see, check, point at and come back to. Most of what goes wrong is that missing surface, and most of it is not the model.

So the design answer is unglamorous. Let people talk to say what they want, then give them a screen to commit to it. That fixes more than I expected. It does not fix a model that agrees with whoever is talking. It does not fix a listener who cannot hold five spoken options in their head, or who trusts a good sentence. The first may improve. The second is us.

None of this is a reason not to build the thing. It is a list of what not to promise, and of where the handover to a person goes.

Open questions

Things I do not have answers to, roughly ordered by how much I want them.

1. 1Can you reference part of a spoken answer at all? Dagstuhl names it as needed research. Nobody has measured it. Cheapest study to run: same content, spoken versus displayed, measure whether people can act on item three.
2. 2Does the lack of a draft state in voice matter? Typed input can be revised pre-send; speech cannot. Genuinely cannot find anyone studying this.
3. 3How often does endpointing fire early, and does it correlate with task complexity? Known problem, unquantified for LLM voice.
4. 4Can users tell mis-hearing from mis-understanding? Hong and Findlater's figure hints, but does not test attribution.
5. 5Does any of this hold in a financial transaction? The existing studies are search tasks and appointment booking. Nobody has tested confirming a payment change, which is the thing I actually care about.
6. 6Does structured intake beat free-form chat with real users? The Concat result predicts yes. Never demonstrated in production.
7. 7Does the multi-turn effect hold for creative and open-ended work? Laban explicitly lists this as untested — analytical tasks only.

How much of this I have actually read

Being honest about depth, because it varies a lot and it matters.

Every figure in parts one and two has been checked against the paper it comes from. The figures in part three have been checked against each paper's abstract, and for Fang and Cheng against the text as well; Rubin is paywalled and I have not read past its abstract. Next reading, in order: the Dagstuhl report in full, since it is a research agenda and I have quoted one paragraph of it; Rubin in full, because the label effect carries more of part three than an abstract should; the N=139 response-length study for its actual measures; the CUI 2026 blind-users paper.

05

Sources

Multi-turn and the conversation pattern

• LLMs Get Lost In Multi-Turn Conversation — Laban, Hayashi, Zhou & Neville (2025)
• Lost in the Middle — Liu et al., TACL 12, 157–173 (2024)
• Bridging the Gulf of Envisioning — Subramonyam, Seifert et al., CHI 2024
• Why Johnny Can't Prompt — Zamfirescu-Pereira, Wong, Hartmann & Yang, CHI 2023
• Accordion Editing and Apple Picking — Gibbons, Mugunthan & Nielsen, NN/g (2023)
• Overcoming the Articulation Barrier — Mugunthan, NN/g (2023)
• Response Outlining with Generative-AI Chatbots — NN/g
• Information Foraging with Generative AI: A Study of 3 Chatbots — Budiu, Liu, Cionca & Zhang, NN/g (2023)
• The 6 Types of Conversations with Generative AI — Budiu, Liu, Cionca & Zhang, NN/g (2023)
• ChatBench: From Static Benchmarks to Human-AI Evaluation — Chang, Anderson & Hofman, ACL 2025
• Navigating Rifts in Human-LLM Grounding — Shaikh, Mozannar, Bansal, Fourney & Horvitz, ACL 2025

Speech, memory, presentation

• Conversational Search — Dagstuhl Seminar 19461 (2020)
• Effects of Response Length on User Search Experience in Spoken Conversational Search — N=139
• Reducing working memory load in spoken dialogue systems — Interacting with Computers 21(4), 276–287 (2009)
• Screen or No Screen?
• Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational Search (2024)
• Does Your Voice Assistant Remember? (2025)
• Human and LLM-Based Voice Assistant Interaction (2024)
• Document-to-audio and prosody research — overview and prosodic mapping

Trust and sycophancy

• SycEval — Fanous et al., Stanford (2025)
• Does Sycophancy Change Decisions? — CHI 2026
• Be Friendly, Not Friends — CHI 2026
• Measuring Opinion Bias and Sycophancy via LLM-based Persuasion
• ELEPHANT: Measuring social sycophancy
• Linguistic Uncertainty Markers for Trust Calibration — CUI 2026
• Understanding the Effects of Miscalibrated AI Confidence
• AI chatbots remain confident even when they're wrong — Cash, Oppenheimer et al., CMU (2025)
• What Large Language Models Know and What People Think They Know — Steyvers et al., Nature Machine Intelligence (2025)
• Sycophantic AI decreases prosocial intentions and promotes dependence — Cheng, Lee, Khadpe, Yu, Han & Jurafsky, Science 391(6792), 2026; preprint

People, labels and use

• Grounding in communication — Clark & Brennan, in Perspectives on Socially Shared Cognition (1991)
• Machines and Mindlessness: Social Responses to Computers — Nass & Moon, Journal of Social Issues 56, 81–103 (2000)
• Influencing human–AI interaction by priming beliefs about AI — Pataranutaporn et al., Nature Machine Intelligence (2023)
• AI can help people feel heard, but an AI label diminishes this impact — Yin, Jia & Wakslak, PNAS 121(14), 2024
• Comparing the value of perceived human versus AI-generated empathy — Rubin, Li, Zimmerman, Ong, Goldenberg & Perry, Nature Human Behaviour 9(11), 2345–2359 (2025)
• How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use — Fang, Liu, Danry, Lee, Chan, Pataranutaporn, Maes et al. (2025)
• Investigating Affective Use and Emotional Well-being on ChatGPT — Phang, Lampe, Ahmad, Agarwal, Fang et al. (2025)

Breakdown and repair

• Conversational Breakdown in a Customer Service Chatbot — ACM TOCHI
• “Have I Answered Your Question Satisfactorily?”
• Overcoming Breakdowns in Customer-Chatbot Interaction — MIS Quarterly 50(2)

Accessibility

• Identifying Speech Input Errors Through Audio-Only Interaction — Hong & Findlater, CHI 2018
• Reviewing Speech Input with Audio: Differences between Blind and Sighted Users — Hong, Vaing, Kacorri & Findlater, TACCESS 13(1), 2020
• “I'M BLIND, ChatGPT” — CUI 2026
• A Preliminary Evaluation of Generative AI Tools for Blind Users — PETRA 2025
• Age-Sensitive Usability in Conversational AI Agents — Kim & Choi (2025)

Background

• “Like Having a Really Bad PA” — Luger & Sellen, CHI 2016
• What Makes a Good Conversation? — Clark et al., CHI 2019
• Design guidelines for hands-free speech interaction — Murad, Munteanu, Clark & Cowan, MobileHCI 2018
• The Keyhole Effect: Why Chat Interfaces Fail at Data Analysis