Most interfaces for steering generative AI assume the user will describe what they want in words — a text prompt, maybe a slider or two bolted on afterward. A new paper from researchers including Chu Li, Nicolai Marquardt, Hugo Romat, Ken Hinckley, Nathalie Riche, Kwon Ko and Fanny Chevalier (several affiliated with Microsoft Research) proposes something structurally different: MaterialVellum, an interaction approach that replaces prompt-writing with manipulating materials that behave like physical constraints.
Constraints as materials, not sentences
The core idea: in domains like urban design, zoning bounds can be defined by how far you stretch an elastic band, building density can be redistributed by molding a patch of virtual clay, and functional relationships can be organized by the pull of magnets. Rather than typing “make this zone denser” and hoping the model interprets it correctly, a user directly manipulates a material-like object whose physical behavior — elasticity, malleability, attraction — is the constraint being communicated to the generative pipeline.
The paper frames this as mapping material affordances onto spatial, relational and proportional design constraints, operationalized through direct-manipulation multi-touch gestures — stretching, sculpting, clustering — whose effects become control parameters feeding a generative AI system, rather than natural-language instructions the model has to interpret.
Tested across three design domains
The team demonstrated the approach through technology probes spanning three different scenarios: an urban design task, a flower arrangement design task, and a graphical design scenario — a deliberately varied set, testing whether a single material-manipulation vocabulary generalizes across domains with very different notions of “constraint.”
A qualitative study with 10 participants (N=10) found that material-based interaction improves accessibility of constraint expression and supports exploratory workflows, particularly for users without technical design backgrounds who might otherwise struggle to translate an intention into either code or a precise prompt. The caveat the paper is careful to include: effectiveness depends on alignment with users’ mental models — stretching an elastic band reads intuitively as “zoning bounds” to some users and not to others, meaning the metaphor has to be chosen well, not just applied uniformly.
Where this sits in the generative-AI interface conversation
MaterialVellum is part of a broader shift in HCI research away from treating natural-language prompting as the default interface to generative systems, instead asking what non-expert users already understand intuitively — physical material behavior, in this case — and building the interaction layer around that instead. The paper carries a related DOI pending registration, suggesting publication alongside a venue presentation.