Research & Innovation

A New Design Framework Wants You to Edit an AI's Thoughts, Not Just Its Prompts

Neural Representation Interaction treats the internal activations of a neural network as something a person can directly inspect, steer, and edit — not just a black box you talk to through text in and text out.

Almost every interaction with an AI system today happens through the same narrow channel: a prompt goes in, an output comes out, and whatever the model is actually doing internally stays completely opaque. A new paper out of MIT Media Lab, Neural Representation Interaction (NRI), argues that channel is needlessly thin — the concepts and internal states a neural network already encodes are rich enough to be a direct interaction material, not just a hidden process you infer from its outputs.

Treating the inside of a model like an interface

NRI’s core move is to situate a model’s internal representations alongside other interaction paradigms HCI already has names for — graphical interfaces, tangible interaction, bio-computer interfaces — and ask the same design questions of them: what can a person see, steer, and edit here, directly, rather than negotiated entirely through text? The paper draws on mechanistic interpretability, the subfield of AI research focused on reverse-engineering what’s actually happening inside a trained network, and reframes its methods as potential interaction primitives rather than purely diagnostic tools for researchers.

Mapping where the interfaces actually are

To organize the space, the team uses General Morphological Analysis to construct a design space spanning four dimensions: Interaction (what a person can do), Target (which part of the model’s internals), Referent (what that internal state is understood to represent), and Purpose (why a person would want to touch it at all). Mapping existing interfaces onto that space surfaces a clear gap: nearly everything that exists today is built for expert auditors doing interpretability research, and almost nothing is built for ordinary users who might benefit just as much from seeing what’s actually driving a model’s behavior.

What this could look like in practice

The paper sketches speculative applications rather than shipped products: in education, a student could inspect which internal features a tutoring model is actually drawing on to explain a concept, rather than trusting an explanation blind. In personal AI modeling, a user adjusting a companion app’s “personality” could see how their customization actually shifted the model’s internal state, rather than guessing from trial and error — closely related to the team’s earlier “neural transparency” work, which projected a chosen AI companion’s custom system prompt onto internal activation directions for empathy, honesty, or sycophancy and visualized the predicted result as a sunburst diagram, finding that users guessed the resulting personality wrong on 11 of 15 traits tested. In hiring audits, an auditor could inspect what an AI screening tool’s internal representations are actually keying on, rather than relying solely on its stated outputs.

The authors are upfront that this remains a design framework and a set of principles rather than a deployed interface, and they discuss the real sociotechnical risks of giving ordinary users more direct access to a model’s internals — including the possibility that exposing more “thinking” doesn’t straightforwardly translate into better human decisions about an AI system’s behavior.