TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
THE QUICK READ
The paper proposes Test-Time Baking (TTB), which learns to bake global scene information from cross-view attention into a lightweight MLP for fast, high-fidelity novel-view decoding. The authors say they repurpose TTT's fast-weight MLP to explicitly learn the cross-view attention mapping, and disentangle context tokens into camera-only rendering inputs and image-content mapping targets to address modality misalignment. The abstract claims up to 23x rendering speedup with quality on par with state-of-the-art methods across several benchmarks.
KEY POINTS
- TTB bakes the cross-view attention mapping into a lightweight MLP instead of replacing attention entirely as TTT does.
- Context tokens are split into camera-only rendering inputs and image-content mapping targets to reduce modality misalignment.
- The abstract claims up to 23x rendering speedup with quality on par with state-of-the-art methods across several benchmarks.
READING LIMITS
Reading only the title and abstract, I cannot verify benchmark details, comparison fairness, the conditions behind the 23x speedup, or whether the claimed quality holds.
Source abstract [EN]
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
AI summaries use public titles and abstracts only. Research claims belong to the authors; consult the original paper.