Our paper for the ICLR 2026 Workshop on Multimodal Intelligence has been approved!
Our work examines whether vision models can serve as a single interface for non-visual data, such as physiological signals and text. We introduce MIMI, a framework for emotion recognition that converts electrodermal activity (EDA), photoplethysmography (PPG), and participant text descriptions of their emotions into RGB images. These images are then processed with a frozen ImageNet-pretrained ViT-Tiny backbone, and we only train small projection heads to align the different data types.
When we compare MIMI to a pipeline that uses an MLP for signals and a language model for text, we see steady improvements in the classification of emotional states. MIMI increases accuracy by 2 to 4 percentage points and F1 scores by 5 to 11 points for classifying arousal, valence, and engagement, using a leave-one-participant-out protocol to test how well it generalizes to new people. Our model works best for emotions that are easy to describe in words, such as valence, but is less effective for aspects like arousal and engagement.
Besides sharing our findings, our paper also addresses a key question for the workshop: when should we use a single multimodal backbone, and when are specialized encoders still needed? We see this as a trade-off that depends on the data and the domain. We suggest hybrid architectures that combine frozen vision backbones with lightweight, modality-specific pathways.
We look forward to discussing this approach at the ICLR 2026 Workshop on Multimodal Intelligence.