Every Gaussian-splatting renderer built so far, from the original paper onward, shares one step: a rasterizer that projects each Gaussian onto the image plane and blends the overlapping ones by hand-coded rules. A new preprint, currently under review at TMLR, asks a more structural question — can a neural network learn to do that whole job itself, with no rasterizer in the loop at all?
Replacing the rasterizer with attention
GaussianRenderFormer, from researchers Shahaf Valencia Levy and Sagie Benaim, is a transformer that takes a set of Gaussians and a camera and outputs the rendered image directly. Each Gaussian becomes one token fed into a view-independent encoder; a separate view-dependent decoder then turns small patches of camera rays into pixels by attending back to those Gaussian tokens. That’s a genuinely different computational structure from rasterization — instead of geometrically projecting and alpha-blending Gaussians in a fixed, hand-designed order, the model learns whatever internal process produces a correct-looking pixel from the relevant Gaussians’ tokens.
How close it gets, and where it falls short
Trained on 26,820 Objaverse objects, each pruned down to 20,000 Gaussians, GaussianRenderFormer renders 300 held-out, unseen objects at 38.08dB PSNR when compared against the full, unpruned splat — while simply rasterizing that same pruned input reaches 44.02dB. That’s a meaningful quality gap, but the authors are careful to localize it rather than wave it away: their diagnostics point to a bottleneck specifically in the view decoder. Explicitly telling the decoder the projected footprint of each Gaussian — information it could in principle infer on its own from the tokens — recovers 1.4dB of that gap by itself, which suggests the network is spending some of its capacity re-deriving geometry a rasterizer gets for free, rather than failing at the rendering task more broadly. The authors also note that under-training hasn’t been ruled out as a contributing factor.
Why skip the rasterizer at all
A hand-coded rasterizer is fast and well understood, so replacing it isn’t an obvious win on its own — the case for a learned renderer is about what becomes possible once rendering is differentiable and learnable end-to-end alongside everything else in a pipeline, rather than a fixed, separate stage bolted onto a neural scene representation. That framing puts GaussianRenderFormer in company with other recent attempts to fold traditionally hand-engineered graphics stages into a trainable neural network rather than ambitious raw image-quality claims.
As with any preprint not yet through peer review, the 38.08dB figure and the “bottleneck is the view decoder” diagnosis are the authors’ own reported results — the paper’s stated open question, what a better-resourced view decoder could close, is exactly the kind of claim that review or follow-up replication will test. Code, trained models, and demonstration videos are published alongside the preprint.