In the last few years, radical increase in the scale of deep neural
language models (both in terms of the size of the training data and the
size of the models themselves) has led to impressive achievements in
various natural language processing tasks. "Celebrity" models, like the
recently announced ChatGPT, are already sometimes described to as
"approaching artificial intelligence", although the reality can differ
from over-hyped media coverage.
In this talk, I will describe the foundations of the technology behind
large-scale language models. Two most important components behind their
success are 1) state-of-the-art deep learning architectures (in
particular, Transformer) and 2) the availability of tremendous amount of
textual data used to train such models. The interaction of these two
poses intricate theoretical and practical questions, also linked to
issues with unequal distribution of computing resources and biases in
training data. Can we actually reach AI by simply training ever larger
language models?"