Every projection-mapped installation starts with the same unglamorous week: survey the room, pick the projector position, build the mask, lock it all down. The geometry is fixed because the projector is fixed, and the content is authored against that fixture.
LICOA, posted to arXiv on 15 October by Yuchen Deng, Shunkai Zhou, Ruixuan Zhu, Haibin Ling and Bingyao Huang, asks what happens if you stop fixing it. Their framing: make projected light an embodied action. Given a language instruction and current visual observations, the system must decide how to move, where to place the projection, and at what scale.
The platform
Two control loops at two timescales, which is the right decomposition.
- A differential-drive chassis provides room-scale approach and repositioning. Slow, large moves.
- A steerable mirror with two degrees of freedom, plus projection scale, gives rapid local control of the beam. Fast, small moves.
So the robot drives to roughly the right place and then aims. Steering a mirror rather than the whole projector is the mechanically sensible choice — you are moving grams instead of kilograms, and the optical path stays calibrated.
Perception is a shared interface mapping synchronised external and onboard RGB cameras plus robot control state onto physical optical actions, designed so that pretrained action models can be dropped in against it.
The data problem, and how they solved it
Instruction-conditioned optical action did not previously exist as a task, so there was no training data and no evaluation protocol. The authors built both:
- A matched simulation embodiment — the same robot, in sim.
- A corpus of 150 tasks and 7,500 rendered episodes.
Then the honest part: they train models only on that corpus and evaluate in previously unseen real-world environments. No real-world fine-tuning in the headline numbers.
The numbers
Across 480 complete closed-loop real-world rollouts on 12 tasks:
| Success rate | Success weighted by path length | |
|---|---|---|
| Best adapted action model | 57.50% | 40.64% |
| Same architecture, trained from scratch | 8.33% | 5.15% |
A further 150 real-world ablation rollouts establish that both camera views are needed, and that chassis and mirror have to be controlled in coordination rather than independently.
Read the comparison carefully, because it is the paper’s real claim. The gap is not between two architectures — it is the same architecture with and without pretraining. 8.33% is roughly what you would expect from a policy that has learned the action space but not the world. Starting from a pretrained action model and adapting it gets you to 57.5%. That is a statement about transfer, not about this particular robot.
And 57.5% is not a product. Every second attempt fails. The weighted figure of 40.64% says that even the successes often take an inefficient route to the target.
Why this matters outside robotics
Three things follow if the projector can move itself.
Authoring changes from geometry to intent. The current workflow encodes “project this onto that wall, from here.” An instruction-conditioned system encodes “put the label on the part being assembled,” and the placement is solved at runtime. That is the same shift XR interface authoring is going through — and compare AdapTest, where designers had to be moved into the headset because the conditions could not be reproduced at a desk. Here the conditions are not even stable.
Surfaces stop being chosen in advance. A fixed projector maps one surface well and everything else badly. A mobile one can treat the room as a set of candidate surfaces and pick.
The evaluation protocol is reusable. The paper contributes a platform, a shared perception-and-action interface, a corpus and a real-world protocol. For a field where most projection work is bespoke and unreproducible, that is arguably the most durable part.
What it does not yet do is anything an artist would call content. The 12 tasks are placement tasks. Nobody has asked this system to do a show.