Multimodal models need dedicated vision capacity, not a pretrained encoder
Multimodal model builders can skip separate vision pretraining, but not dedicated visual capacity: encoder-free models need about 3× the compute to match an encoder’s average VQA score across 1.7B to 6.4B backbones. The study trains each recipe from scratch on the same data and finds a scratch-trained encoder performs on par with one pretrained separately. Extra encoder pretraining improves ImageNet accuracy, not VQA, while vision modules inside early backbone layers perform comparably to a pre-fusion encoder.