Unbundling Multimodal Vision Encoding
THE QUICK READ
This paper examines how multimodal language models should encode visual input. The authors note that most models use a pretrained vision encoder, while some recent recipes train it from scratch or remove it. They build a controlled framework that pretrains all models on common data and compares them at matched lifetime compute. The authors report that a visual prior is unnecessary, dedicated vision capacity matters, and that capacity need not precede multimodal fusion.
KEY POINTS
- The authors separate vision encoders into three bundled choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity.
- Under common-data pretraining and matched lifetime compute, the authors report that a jointly trained from-scratch encoder is on par with a pretrained one, so the visual prior is not necessary.
- The authors report that dedicated vision capacity is what matters: an encoder-free model trails on visual understanding even with an enlarged backbone, while vision-only modules added in-fusion perform comparably and can lower time to first token.
READING LIMITS
Reading only the title and abstract, I cannot verify the controlled setup, benchmark choices, or statistical significance, and the reported results are the authors' claims rather than independently confirmed findings.
Source abstract [EN]
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
AI summaries use public titles and abstracts only. Research claims belong to the authors; consult the original paper.