Modern AI systems increasingly depend on communication networks. Training a large AI model may require many GPUs or computing nodes to continuously exchange data, gradients, and model parameters. During inference, AI applications may also be distributed across devices, edge servers, and cloud infrastructure, requiring data and intermediate results to move between different parts of the system.
This means that communication is becoming an essential part of both AI training and inference. However, real networks are not always reliable. They may experience packet loss, congestion, variable latency, limited bandwidth, temporary disconnections, corrupted messages, or unavailable nodes. During training, such problems may slow down computation, interrupt synchronization, or cause training jobs to fail. During inference, they may increase response time, affect prediction quality, interrupt a distributed pipeline, or make an AI service temporarily unavailable.
Goal of the master project
The goal of this thesis is to investigate the impact of faults in InfiniBand network communication on distributed AI systems. The project will focus on identifying some common communication faults and systematically reproducing them under controlled conditions. These faults will then be synthetically introduced during distributed AI training and inference to study their effects on the AI workload and model, including training behaviour, accuracy, latency, throughput, availability, and recovery.
Steps:
- Learn about the basic components and steps involved in distributed AI systems, especially the data communication between compute nodes;
- Learn about the basic knowledge of InfiniBand network communication;
- Learn about typical InfiniBand network faults and the detection mechanisms;
- Synthetic generation of network faults;
- Measurement and analysis of the behaviour of distributed AI systems, in different fault scenarios;
- Investigation of the vulnerability of distributed AI systems with respect to accuracy, latency, throughput, availability, and recovery.
Learning outcome
This project will give you the opportunity to work at the intersection of AI, computer networks, and distributed systems. You will learn how AI models are trained, evaluated, and used for inference, as well as how these operations can be distributed across multiple computing nodes. Moreover, your research will contribute to systematically understanding the impact of network faults on distributed AI systems, which can lead to making distributed AI systems more robust.
Qualifications
You do not need to be an expert in AI or computer networking to start this project. The most important qualifications are curiosity, an interest in understanding how modern AI systems work, and motivation to experiment with real systems.
Some experience with Python, Linux, networking, distributed systems, or machine learning is beneficial, but not mandatory. You will receive support in understanding the existing framework, selecting AI models and workloads, designing experiments, introducing communication failures, and analysing the results.
References
- Guangba Yu, Gou Tan, Haojia Huang, Zhenyu Zhang, Pengfei Chen, Roberto Natella, Zibin Zheng, and Michael R. Lyu. 2025. A Survey on Failure Analysis and Fault Injection in AI Systems. ACM Trans. Softw. Eng. Methodol. 35, 1. https://doi.org/10.1145/3732777
- Murad Bayoun. 2024. Troubleshooting InfiniBand Networks: A Detailed Guide.