What response latency should an AI companion toy target?

Under roughly 700 ms after the user stops speaking feels conversational; past 1.5 seconds users describe the product as laggy or broken. The target should be fixed in the requirements document before tooling.

Is the cloud LLM the main source of delay in AI toys?

Usually not. Wake-word detection, endpointing, noise suppression and speech recognition all run on the device before the network is touched, and they consume most of the budget. On-device compute — typically an NPU-equipped SoC — is what keeps this stage short.

Can an AI toy keep talking without a network connection?

From roughly 6 TOPS of NPU compute, a quantised small language model can run on the device as an offline fallback. It must be planned at silicon selection, not patched in later.

Why Your AI Companion Toy Feels Slow — the Latency Budget Explained

Users judge a conversational toy or companion robot in the first ten seconds, and they judge it on one thing: how long it stays silent after they stop talking.

Under roughly 700 milliseconds, the exchange feels like conversation. Past 1.5 seconds, reviews start saying “laggy”, “dumb”, “broken”. There is no comfortable middle.

The LLM is not where the time goes

Most brands assume the delay comes from the cloud model. In practice, most of the budget is spent before the request ever leaves the device.

A talking robot goes through this chain: wake-word detection → endpointing (deciding you have finished speaking) → noise suppression → speech recognition → network round trip → LLM generation → speech synthesis → playback.

The first four steps run on the device. Every millisecond they waste is taken from the model’s allowance. Endpointing is the usual offender: too aggressive and the toy interrupts the child mid-sentence; too conservative and half a second is gone before anything is sent.

This makes latency a hardware decision

Compressing the on-device chain into two or three hundred milliseconds takes local compute. The current standard approach is an application SoC with an NPU — 0.5 to 16 TOPS depending on what must stay local — running wake word, endpointing, noise suppression and speech recognition on the device, and sending only understanding and generation to a cloud LLM chosen for each market.

That is why two products calling the same LLM API can feel completely different. The difference is not the model. It is the half second in front of it.

Three questions to ask any ODM before tooling

Ask for a latency breakdown table — milliseconds per stage, on-device and network. A team that has shipped will produce it immediately; a team that has not will talk about the model instead.
Set the latency target at kickoff, in the requirements document, not after the first prototype disappoints.
Decide the offline story early. From roughly 6 TOPS a quantised small language model can run locally as a no-network fallback — but that is a silicon decision made at selection time, not a firmware patch later.

At 3CBAY we develop companion robots, in-car AI assistants and conversational toy cores with this budget fixed on day one — silicon selection, acoustics, firmware and enclosure handled by one team in Shenzhen. If you are planning a conversational device, tell us about it.