Watch two people talk and the listener is never still. They nod, they raise eyebrows, they start a gesture before their turn arrives and abandon it when the other person keeps going. Co-speech gesture systems, by construction, cannot do any of that: they generate motion from speech, so when the speech stops, the body stops.
ELLSA 2 — from Siyin Wang, Wenyi Yu, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu and Chao Zhang, posted 15 October — is an end-to-end full-duplex dialogue model that jointly optimises speech interaction and streaming gesture-expression generation.
The architectural move
An autoregressive, chunk-wise synthesiser combines pretrained acoustic features with distinct listening and speaking hidden states, drawn from both the dialogue backbone and the speech synthesiser.
That is the whole trick, and it is worth being precise about why it works. A full-duplex dialogue model already maintains different internal state depending on whether it currently holds the floor — that is what makes it full-duplex rather than turn-based. ELLSA 2 treats both of those states as conditioning for the non-verbal channel. So the body has something to be driven by even when the mouth has nothing to say, and the paper’s claim is exactly that: continuous non-verbal generation even when the model is verbally silent.
This is the natural successor to work we covered last month on predicting backchannel onset from a full-duplex model’s hidden states. That paper added an “uh-huh” at the right moment by reading the same kind of internal signal. This one adds the nod.
Separating whether from how
The second contribution addresses the problem that makes generated gesture feel hollow. Models trained to produce motion from audio converge on beat gestures — rhythmic hand movement that tracks prosody and means nothing. Semantic gestures, the ones that carry content, are rare in the data and get averaged away.
ELLSA 2 introduces contextual semantic calls and template guidance, which the authors describe as separating whether to invoke an available semantic gesture from how to realise it.
Decomposing it that way is sound. “Should there be a gesture meaning small here?” is a discourse-level decision the dialogue backbone is already equipped to make. “What does small look like, executed smoothly from the current pose at the current moment?” is a motion-synthesis problem. Collapsing the two is why generic systems produce either nothing specific or specific things at the wrong time. The paper reports that semantic template guidance significantly improves semantic gesture quality, which is the result that follows if the decomposition is doing real work.
Evaluation, including the awkward part
The authors establish a curated data and evaluation pipeline for dialogue-aware co-speech gesture generation, and then investigate using reference-based multimodal LLMs as automatic judges.
Using a language model to score a gesture model is the kind of thing that deserves scepticism, and the paper handles it the right way: it reports strong system-level correlations with human judgments and frames the scores as useful preliminary metrics rather than as a benchmark. System-level correlation means the automatic judge can rank models; it does not mean it can rank individual utterances. That distinction is the difference between a usable research tool and a misleading leaderboard.
On the results: ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models — i.e. adding the non-verbal channel does not cost it speech quality — and outperforms cascaded baselines across dialogue-aware gesture and expression metrics. The cascaded baseline is the obvious alternative: run a dialogue model, then run a gesture model on its output. Beating it is the case for end-to-end here.
The paper is 37 pages with five figures, and lists no public demo or code release.
What it is for
Anyone building an embodied agent — a kiosk, a museum guide, a virtual performer, a companion avatar — has had to choose between a body that moves constantly and meaninglessly, and a body that freezes between turns. Both read as broken, in different ways. Freezing is worse, because it tells the user the system has stopped paying attention at precisely the moment they are speaking.
A listening-state-driven non-verbal channel is the fix for that specific failure. It is also, for the same reason, a fix that will be most visible in the interactions people complain about least articulately: not “the gestures were bad” but “it felt like it wasn’t listening.”