Interactive Art

ControlAnything Lets You Point at a Device in Mixed Reality and Just Start Using It

A single-author research project has a coding agent write the integration code on the spot — connecting a Meta Quest 3 wearer's hands to whatever nearby device they select, with no pre-written driver.

Connecting a hand gesture in mixed reality to a real, physical device normally means someone wrote that integration ahead of time — the transport protocol, the mapping from gesture to command, all fixed before the headset goes on. A new research project called ControlAnything, posted to arXiv November 5 by researcher Bektur Ryskeldiev, defers that work to the moment you actually need it, and hands it to a coding agent.

How it works

On a Meta Quest 3, the wearer selects a physical object that’s been outlined using Meta’s SAM 3.1 segmentation model, then simply states what their hands should control. An AI agent built on Meta’s Muse Spark model then writes a connector program on the wearer’s laptop in real time, testing it against the live device over hand-written MIDI, Bluetooth, or key-press transports as needed. The output is what the paper calls generative UI — but instead of just generating the interface, the agent also writes and tests the code binding that interface to a real-world object.

This is an unreviewed preprint, posted directly to arXiv without a stated conference or journal venue — worth flagging up front, since the evaluation below is small and author-run rather than independently replicated.

What the evaluation actually showed

The reported numbers come from a single-session technical evaluation with the author as the only wearer: nine of ten requests across five target devices reached a working, “ready” state, with the wearer manually stopping one. The median time to ready was 32.5 seconds (range 24.9–57.8 seconds, speech transcription time excluded), and sequential AI model calls accounted for a median of 89.9% of that time — meaning the bottleneck is largely the agent thinking, not the headset or the hardware link. A notable efficiency detail: 91.7% of input tokens came from a prompt cache rather than being processed fresh each time.

The paper is honest about the limits of its own evidence: the system can confirm a transport-level acknowledgement from the device (a message got through), but that’s not the same as confirming the physical effect actually happened. A user study with people other than the author is planned as the next step.

Why it matters for interactive work

For anyone building interactive installations or performance rigs that touch real hardware — lighting, sequencers, motorized props — the appeal of ControlAnything’s approach is obvious: it collapses the usual “write a driver first” step that makes ad-hoc, one-off device integrations expensive to build. Whether that holds up outside a single researcher’s own living-room setup is exactly what the planned user study will need to show.