拆解多模態視覺編碼的設計選擇
Unbundling Multimodal Vision Encoding
研究速讀
這篇論文探討多模態語言模型該如何編碼視覺輸入。作者指出,多數模型採用預訓練視覺編碼器,但近期也有從頭訓練或完全移除編碼器的做法。他們建立受控框架,在相同資料與計算量下比較不同設計,主張視覺先驗並非必要,真正關鍵的是專屬視覺容量,且該容量不必置於融合之前。
值得留意
- 作者將視覺編碼器拆解為三項設計選擇:上游預訓練帶來的視覺先驗、專屬視覺容量,以及容量置於多模態融合之前的位置。
- 在共同資料上從頭預訓練並對齊終身計算量後,作者報告從頭聯合訓練的編碼器與預訓練編碼器表現相當,視覺先驗並非必要。
- 作者報告專屬視覺容量才是關鍵:無編碼器模型即使放大主幹參數仍在視覺理解上落後,而置於融合早期的視覺模組可與前置編碼器相當,並可能降低首字延遲。
閱讀限制
僅閱讀標題與摘要,無法確認受控框架的實驗細節、基準選擇與統計顯著性,也無法獨立驗證作者結論。
原始摘要 [EN]
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
AI 摘要僅根據公開標題與摘要整理;研究結論來自作者,請以原文為準。