Made by AI, Directed by David Stein

Search
Digital Artists

Turning a Painting Into 3D Is Impossible — So This Paper Prints the Ambiguity Instead

One painted image is compatible with many 3D scenes. Rather than picking one and calling it correct, the authors reconstruct several, let the disagreement show up as ghosting and halos, and fabricate that.

3 min read

Every “turn this painting into 3D” demo hides the same dishonesty. A painted image does not determine a three-dimensional scene. The back of the figure is unpainted, so it could be anything. Even the parts you can see are underconstrained — perspective, shading and occlusion narrow the possibilities without fixing them. Any system that hands you one mesh has quietly chosen among many valid answers and presented the choice as a fact.

Printing the Underdetermined: Materializing Multi-solutionness in Figurative Paintings — by Yutao Ming, Teng Xu, Youjia Wang, Yunyang Liu, Fengmin Yang, Fuqiang Zhao, Jingyi Yu, Hua Yang and Yanjun Zhou, published in the Proceedings of the ACM on Computer Graphics and Interactive Techniques — does the opposite. It makes the ambiguity the subject.

Two sources of ambiguity

The paper separates them cleanly:

  • Unobserved content. Backsides, occluded volumes, anything the painter didn’t show. Multiple completions are equally plausible and nothing in the image can choose between them.
  • Observed cues. Even what’s visible underconstrains geometry — perspective, shading and occlusion each rule out some configurations while leaving many.

The second is the more interesting claim, because it’s where “just look at the image harder” stops working. No amount of inference resolves it; the information isn’t there.

The pipeline

  1. Sample multiple camera-orbit multi-view video sequences from one painting — several different plausible ways the virtual camera could move around the depicted scene.
  2. Reconstruct each sequence with 3D Gaussian Splatting, producing a point-based scene where the disagreements between interpretations don’t get averaged away. They appear as density halos and ghosting — soft, doubled, uncertain regions exactly where the degrees of freedom are unresolved.
  3. Fabricate the result as a physical artifact using a method the authors call DreamPrinting.

Step two is the elegant part. Splatting artifacts are normally the enemy; here they’re the readout. A fuzzy region isn’t a failed reconstruction, it’s a visible measurement of how much the painting doesn’t say. You can look at the object and see where the painter left things open.

Why this belongs on the artists’ beat rather than the research one

Because it’s an argument about representation, made in code.

The prevailing use of generative 3D is to remove ambiguity — take the flat thing, produce the solid thing, move on. Painters have spent centuries exploiting exactly the ambiguity that removes. What a figure’s hidden hand is doing, what lies behind the drapery, whether a space is shallow or deep: these are compositional decisions, not gaps to be filled. A tool that resolves them destroys the content.

This paper treats not knowing as information worth preserving and even printing. That’s a genuinely useful position for any artist working with reconstruction, scanning or photogrammetry, and it generalises well beyond painting. Every capture technique has regions it cannot determine. Most pipelines hide them. Showing them instead is both more honest and, on the evidence of the figures, better looking.

Related Reading