From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches is by Ahmet Neçirvan Doğan, Eren Acar Başaran, Metin Sezgin and Zahra Montazeri, posted to arXiv on 15 October across cs.GR, cs.CV and cs.HC.
It identifies a flaw in sketch-to-3D pipelines that is obvious once stated and that almost all of them share.
The flaw
The paper’s framing of why sketching matters is the standard one, and correct:
Unlike the text prompts that most generative systems rely on today, a sketch captures spatial structure and intent directly, including where a designer wants highlights and shadows to appear.
And then the problem:
Yet current sketch-to-3D pipelines have no input for that intent: they bake the drawn shading into the texture as dark paint, and the light cannot be moved.
That is the whole issue. A designer hatches an area to indicate that it falls into shadow. The pipeline interprets the hatching as the thing being dark rather than the thing being unlit. You get a 3D garment with a dark patch painted onto it, which looks correct from exactly one lighting setup and wrong from every other — and the moment the asset enters a scene with its own lights, the painted shadow fights the real one.
Shading in a drawing is a statement about the scene’s light. Treating it as albedo throws away the statement and keeps the pixels.
Two symbols, chosen by asking
The system takes three inputs: a rough garment sketch, sparse annotations, and a short text prompt describing the texture. The annotations are two marks:
- A star for a highlight
- A moon for a shadow
What makes this a good design decision rather than an arbitrary one is how the symbols were picked. The authors ran an elicitation study with art students and took the two most frequently proposed marks. That is a small piece of methodology with an outsized effect: a notation that users invent is a notation they do not have to learn, and the failure mode of annotation interfaces is almost always that the symbol set made sense to whoever built it.
What the model actually predicts
From the marked sketch, the system predicts two things, separately:
- The scene’s directional light — inferred from where the star and moon marks are relative to the form
- A displacement field that bends the surface so that moon-marked regions fall into shadow under that light
The second is the clever part and the part worth dwelling on. The naive approach to “make this region dark” is to darken it. The approach here is to change the geometry so that the region is genuinely occluded or angled away from the inferred light. The darkness then emerges from the render rather than being asserted by the texture.
This has a consequence the paper names in its title: the result is a scene that can be relit and edited after generation, rather than an image-shaped artefact. Move the light and the shadows move, because they were never painted — they are a function of surface normals that the displacement field put there on purpose.
It also means the designer’s shading marks become a geometry authoring channel. Hatching a fold to indicate shadow produces an actual fold. That is closer to what the designer meant than any texture operation could be.
Why this matters outside fashion
Garments are a sensible domain to prove this on — they are drape-dominated, folds are where the shading information lives, and there is a large existing pipeline for turning garment designs into 3D. But the principle generalises to any sketch-driven asset pipeline, and the useful way to state it is as a separation:
A drawing contains at least three kinds of information — form, material, and lighting — and most pipelines collapse all three into texture.
Text prompts are good at material and poor at form. Sketches are good at form. Nothing in a standard pipeline accepts lighting intent at all, which is why the drawn version gets flattened into the one channel that will take it.
A practical caveat for anyone reading this as a production tool: a displacement field that bends geometry to satisfy a shading instruction is editing the silhouette of the thing it is shading. Where shading intent and form intent disagree — a designer marks a shadow in a place the garment’s actual drape would not produce one — something has to give, and what gives is the shape. For illustration that is fine. For a garment that will be graded and cut, it may not be.
The paper is a preprint; there is no published code or dataset release yet, and the elicitation study’s population is art students rather than practising pattern designers.
Sources
- From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches — arXiv:2610.21865
- arXiv listing, cs.GR
- Koç University Intelligent User Interfaces Laboratory
- Zahra Montazeri — University of Manchester