Creative Coding

A New Model Opens a Box of Loose LEGO and Figures Out How It Was Supposed to Go Together

BrickFlow takes an inventory of parts and a single photo of the finished build, then infers the exact 6D pose of every brick — including the ones hidden inside the structure — by learning the physical grammar of how LEGO actually connects.

Imagine opening a box of LEGO to find a pile of loose pieces and no instructions — just the picture on the front of the box. You can identify the parts visible on the surface, but the photo only shows the outside; everything the model is built from on the inside has to be inferred. That’s the exact problem BrickFlow, a new paper accepted to NeurIPS 2026 from researchers at Inria, École Normale Supérieure, CNRS, and the Max Planck Institute for Intelligent Systems, sets out to solve: given an inventory of known parts and a single image of the assembled object, recover the precise 6D pose — position and orientation — of every piece.

Why this is harder than ordinary 3D reconstruction

Normal 3D reconstruction from an image tolerates a reasonable amount of slop — a surface that’s slightly off still looks fine. LEGO doesn’t work that way. Every piece has to land on a discrete, connector-defined site: studs have to mesh with anti-studs, Technic pins have to align, and a small pose error doesn’t just look slightly wrong, it produces a physically invalid structure. That turns what sounds like a reconstruction problem into a combinatorial one, where the model has to respect hard physical constraints rather than just approximate a surface.

Learning the grammar of how bricks fit together

To learn a prior over valid assemblies at all, the team built ProcBrick, a procedural generator that starts from a random seed piece and iteratively attaches new parts at compatible connector sites — rejecting any placement that would collide — to synthesize 100 million structures of 100 parts each. None of these assemblies are human-designed or semantically meaningful; they exist purely to give the model dense, constraint-rich examples of the physical grammar LEGO actually obeys.

On top of that dataset, BrickFlow trains a set-conditioned SE(3) flow-matching model: a diffusion-transformer backbone that predicts the translational and angular velocity of every part simultaneously, with no positional encoding, since the input set of parts has no inherent order. The team then fine-tunes this pretrained assembly prior to add image conditioning — using only 1% as much data as the original pretraining run — to turn “denoise this pile into a valid structure” into “denoise this pile into the structure shown in this specific photo.”

The missing piece: enforcing connectivity at inference time

Here’s the catch the paper is actually built around: the learned flow captures the global structure of plausible assemblies well, but it doesn’t strictly enforce that every piece ends up connected to the rest. To close that gap, BrickFlow adds an analytic connectivity-field guidance step applied at test time — a direct, non-learned correction that actively steers predictions toward fully connected assemblies rather than ones that merely look right from a distance. The team reports that build quality improves consistently with model size, with pretraining on the full ProcBrick dataset, and with this guidance step added at the end.

The researchers have released their models and the ProcBrick data publicly for research use, alongside an interactive demo on the project page that visualizes the flow process denoising a pile of pieces into a finished structure in real time.