AI is ever more pervasive: predicting the shape of folded proteins, thrashing humans at games, steering cars, and generating ex nihilo novel text and images. However, we lack a theoretical understanding of why deep learning works, and naïve calculations even suggest it should be impossible! At the same time, empirical “neural scaling laws” (e.g. Kaplan 2020) predict how neural network performance improves as model size, training data, and available compute are scaled up. These scaling laws are useful for more than converting venture capital into data centres – the fact that they apply to very different models in rather different domains hints at unified mechanisms underlying deep learning.
In this talk, I will argue that the Physicists’ approach to theory building, focusing on minimal toy models, symmetries, and scaling arguments, can offer useful insight into the surprising success of deep learning. Since neural scaling laws suggest that many details of the neural networks themselves are irrelevant for learning, our group focuses on the input side of the problem: what structure does data need to have to be learnable? I will describe our favourite “spherical cow in a vacuum” toy model, the Random Hierarchy Model (RHM). The RHM is the minimal model capturing both hierarchy and compositionality in data, two features suggested by linguists and neuroscientists to be essential to learning. We use this model to understand how much data is necessary for learning to be possible and suggest ways to make deep learning more data efficient. In parallel, our predictions about how neural networks encode salient features of the world at different levels of abstraction suggest new tools for probing the structure of data, which we are doing for text and images.
