Modern AI systems increasingly rely on distributed computing to train and run large models. Training may involve many GPUs or computing nodes that continuously exchange data, gradients, and model parameters. The Message Passing Interface (MPI) standard and its software implementation are widely used "under the hood" to enable communication among processes running across multiple computing nodes.
This communication is critical for distributed AI workloads. However, MPI-based applications can experience communication faults, process failures, delays, or unexpected interruptions. Such software runtime faults may slow down computation, interrupt synchronization, cause training jobs to fail, or affect the behaviour and reliability of the overall AI system.
Goal of the master project
The goal of this thesis is to investigate the runtime faults of the communication software in MPI-based distributed AI systems. The project will focus on understanding the impact of typical runtime errors of the MPI communication layer and systematically reproducing them under controlled conditions. The project will also investigate the possibility of synthesising new runtime faults in the MPI layer. These faults will then be analysed with respect to their effects on an AI system and model, including training behaviour, accuracy, execution time, communication performance, availability, and recovery.
Steps:
- Learn about the basic components and steps involved in distributed AI systems, especially the data communication between computing nodes;
- Learn about the basic knowledge of MPI programming and its usage in distributed AI system implementation;
- Learn about existing methods for synthetic generation of MPI runtime faults faults;
- Development of additional fault synthesis methods;
- Measurement and analysis of the behaviour of distributed AI systems, in different MPI runtime fault scenarios;
- Investigation of the vulnerability of distributed AI systems with respect to accuracy, latency, throughput, availability, and recovery.
Learning outcome
This project gives you the opportunity to work at the intersection of AI, high-performance computing, and distributed systems. You will learn how AI models are trained and executed across multiple computing nodes, how MPI enables communication among distributed processes, and how communication faults can be synthesized and analysed experimentally. Moreover, you will also develop more MPI runtime fault scenarios and synthesis methods, for the purpose of making distributed AI systems more reliable.
Qualifications
You do not need to be an expert in AI, MPI, or high-performance computing to start this project. The most important qualifications are curiosity, an interest in understanding how modern distributed AI systems work, and motivation to experiment with real systems.
Some experience with Python, C/C++, Linux, parallel programming, distributed systems, or machine learning is beneficial, but not mandatory. You will receive support in learning MPI, selecting AI workloads, designing fault-injection experiments, introducing communication faults, and analysing their effects.
References
- Guangba Yu, Gou Tan, Haojia Huang, Zhenyu Zhang, Pengfei Chen, Roberto Natella, Zibin Zheng, and Michael R. Lyu. 2025. A Survey on Failure Analysis and Fault Injection in AI Systems. ACM Trans. Softw. Eng. Methodol. 35, 1. https://dl.acm.org/doi/10.1145/3732777
- Wirewalk Advisory. 2026. An MPI troubleshooting playbook: symptom to cause.
- K. Feng, M. G. Venkata, D. Li and X. -H. Sun. Fast Fault Injection and Sensitivity Analysis for Collective Communications. 2015 IEEE International Conference on Cluster Computing, Chicago, IL, USA, 2015, pp. 148-157. https://doi.org/10.1109/CLUSTER.2015.31