Green Informatics Network Presents Baby Language Models - a Talk by David Samuel

This shared task limits training data to just 10–100M words – roughly what children hear while learning language. This constraint makes it easier to explore fundamental questions about language learning without the massive computing resources and trillions of tokens that typical language model research demands. We present findings from multiple challenge submissions. These include a GPT-BERT hybrid that combines masked and causal language modeling, and a layer-selective transformer that lets each layer choose its inputs from previous layers. The results suggest promising directions for building more efficient models, while demonstrating how constrained settings can advance our understanding of language learning.