Our team is headed to CVPR 2026!
In our new paper, we investigate Masked Image Modeling (MIM) and uncover a critical bottleneck: current MIM objectives cause learned representations to retain non-semantic information. This essentially acts as visual clutter, which we found ultimately degrades zero-shot performance during inference. To properly measure this phenomenon, we introduce a novel, model-agnostic score for semantic invariance utilizing Principal Component Analysis (PCA) on both real and synthetic non-semantic images.
Building on that metric, we introduce Semantic Orthogonal Projection (SOaP), a straightforward method to directly suppress this non-semantic information within patch representations. Implementing SOaP leads to consistent, measurable improvements in zero-shot performance across a variety of MIM-based models. The best part of this approach is its efficiency—SOaP is a completely post-hoc suppression method that requires absolutely zero additional training and can be cleanly attached to any existing model as a single linear head.
Look for us at CVPR 2026 for more!