Creative Coding

CrowdDirector Lets You Steer a Unity Crowd Scene by Just Telling It What to Do

A graph neural network drives hundreds of agents in real time, calling an LLM only when a human issues a new natural-language instruction — not on every tick — so direction stays live without repeated model calls slowing the scene down.

Directing a crowd scene — a plaza full of pedestrians, a stadium of extras — has traditionally meant either hand-authoring behavior rules per agent type or accepting whatever a generic crowd simulator defaults to. CrowdDirector, a new research system, proposes a third option: tell the crowd what you want in plain language, and let the system work out how every individual agent should act.

How the pieces split the work

The key design choice is keeping the expensive part — the language model — out of the real-time loop. An LLM is called only when a human adds or changes an instruction, not on every simulation tick. When it is called, it doesn’t move agents directly; it produces a structured instruction, which a scene validator checks against what’s actually in the scene (do the referenced entities and actions exist?) before accepting it. Once validated, the instruction stays active, and a graph neural network — trained offline on simulated scenarios with LLM guidance — picks an action and a target for each agent at every tick, while ordinary Unity systems handle the resulting movement, collision, and animation.

Why the validator matters as much as the LLM

The research reports that the scene validator’s accept/reject calls matched human judgments about feasibility more often than the LLM’s own self-assessment did — in other words, checking a proposed instruction against the literal state of the scene caught problems the language model itself didn’t reliably flag. The system is also reported to follow instructions more closely than handwritten behavior rules, without needing a fresh LLM call on every frame to maintain that responsiveness.

Why this matters beyond games

Crowd simulation shows up anywhere a scene needs populated background life without hand-animating every figure — games, architectural visualization, film previsualization, urban-planning walkthroughs. A workflow where directing that crowd is closer to giving verbal direction on a film set than writing behavior-tree code is a meaningful shift in who can actually art-direct a crowd scene, not just who can program one.