The Large Hadron Collider (LHC) at CERN is the world’s most powerful particle accelerator, colliding protons at very high energies. One of its main experiments, the ATLAS detector, records the particles produced in these collisions to test the Standard Model and search for new physics.
For many years ATLAS and other high energy physics experiments have used machine learning to identify different kinds of particle physics processes taking place in the detector. For example a machine learning algorithm may be trained to distinguish between the decay of a top quark, or a Higgs Boson, from background processes which have superficially similar properties. Several important discoveries have been made using machine learning, including the top quark and the decay of the Higgs Boson into a pair of W bosons. More than half of the physics publications from the LHC now use machine learning in one form or another.
Typically the algorithms are either neural networks or Boosted Decision Trees (BDTs) and are trained on simulated data. For example, an algorithm might be trained by labelling simulated Higgs decays as “signal” and other simulated data as “background”. Once trained and validated the algorithm can be exposed to real data. This can be referred to as “narrow AI” because the algorithms are trained to do a specific task using specific training samples.
With the advent of Large Language Models (LLMs) the concept of “foundation models” has been widely discussed. Foundation models are usually based on the Transformer architecture, and in contrast to narrow AI, they are trained on a huge corpus of data (written human language in the case of LLMs). A second training step known as “fine tuning” is then used to train the algorithm to do a specific task (responding to human queries in the case of chat-bots). Whereas a narrowly trained model may require very large volumes of dedicated training data to reach a given level of performance, foundation models may need far fewer training examples in this second step. Although the initial training may require vast quantities of data, the flexibility of foundation models means that the same model can be used for a wide variety of tasks. Consequently many applications can profit from the same base training.
In contrast to an LLM, a tabular foundation model is trained on a huge corpus of tabular (usually numerical) data of all kinds rather than human language. Using real and simulated ATLAS data, this project will explore whether such models can be fine-tuned to do a specific analysis task in high energy physics more effectively than a narrowly trained boosted decision tree. “Effectiveness” in this context means how well the model performs in its task, how much dedicated simulated physics data is needed in the fine tuning to reach the same performance as the BDT, the computational performance (training time, inference time, memory usage) and whether it is practically more straightforward to use.
Below are some milestones of the proposed project:
M1: Perform a focused literature review of open source tabular foundation models and identify candidates that can be used in the project
M2: Download one or more models and understand how to use them in terms of fine tuning and inference. As part of this step you will need to learn how to use the relevant computational platforms including GPU resources. Initially we will use local resources but may need to use national HPC facilities later on in the project
M3: Identify a physics analysis task to focus on. We will try to pick an analysis task which is either similar to or the same as a task being worked on by other master’s students, to enable sharing of data samples and code.
M4: Train a boosted decision tree to perform the task and set baseline performance metrics
M5: Fine-tune the foundation model(s) to do the same task; aim to reach the same level of physics performance as the BDT.
M6: Compare and contrast the performance in terms of the volume of training data needed, the time required to train, the memory usage of the models and the practical complexity of running the training and inference. If you have more than one tabular foundation model, compare them against one another.
M7: try to push the performance of the foundation models beyond that of the BDT and see how much better it is able to do
M8: investigate how a dedicated particle physics foundation model might be built (it is unlikely you will have time to do this for real).
M9: Summarise the overall findings and draw conclusions about whether tabular foundation models have a role to play in high energy physics data analyses
M10: submit thesis
Courses relevant from the bachelor's degree:
Recommended courses for theoretical curriculum in the master's programme:
(If not taken during bachelor)