Advancements in PCIe hardware have enabled the development of low-latency, high-bandwidth commodity interconnect technologies such as PCIe 6.0 and CXL 2.0. These advances have opened up new research opportunities, such as composable disaggregated infrastructure (CDI), which utilizes shared memory for memory-mapped I/O devices, allowing them to appear in the host’s local device tree even though they are physically remote. In the deployment of I/O-bound services, this capability provides system engineers with increased flexibility, enabling higher I/O utilization, reduced power consumption, lower operating costs, and improved failover mechanisms.
Thesis Goals
The primary goal of this thesis is to explore and gain insight into the performance benchmarking of composable infrastructures and the traffic profile of PCIe interconnects. This can be achieved through a combination of the following objectives:
- Literature review. Conduct a comprehensive survey of existing approaches to benchmarking intra-host PCIe, CXL, and RDMA networks, as well as prior attempts to profile PCIe as an interconnect.
- Tool development. Adapt an existing approach or implement a new benchmarking tool and deploy it as a proof-of-concept on Dolphin ICS software and hardware.
- Benchmark design. Design a range of application-independent microbenchmarks and/or find macrobenchmarks relevant to a specific domain.
- Benchmark execution. Successfully execute benchmarking runs, collect data, and document their results.
- Analysis and recommendations. Discuss the findings, analyze, draw conclusions, and derive specific system design tips (such as topology setups, device and workload allocation, etc.) based on the observations.
Proposed Research Areas
The overarching goal of this thesis is to demonstrate the relevance of the PCIe stack as an interconnect in composable infrastructures. To achieve this, the work will focus on the following key areas:
- Lessons from previous attempts: Which approaches exist, and what can be learned from them? What are their strengths, limitations, and gaps? Determine how these insights can inform the design of the benchmarking tool.
- Benchmarking tool composition and implementation: What are the foundational tools needed to trace PCIe traffic? Is custom hardware, such as FPGAs and PCIe tracers, necessary? Do we intercept traffic to collect data, or is it sufficient to collect end-to-end metrics?
- PCIe as an interconnect and intra-host communication: Does PCIe behave differently in composable infrastructures than in host-isolated domains?
- Application-specific traffic: How do different workloads affect traffic? What can be derived from observations to improve workload performance? What are the performance bottlenecks? What about bandwidth fairness?
- Architectural distinctions: Does the underlying hardware (micro-)architecture affect flow behavior? What about different device vendors?
- Theoretical modeling: Can the observations be extrapolated to design a predictive model of the traffic by hand? Can we utilize advanced tools such as artificial intelligence for traffic prediction and system optimization?
Learning outcome
Upon successful completion of this thesis, the student will have gained:
- Advanced knowledge of modern interconnect technologies, including CXL, PCIe, and RDMA, and familiarity with various I/O device classes such as NVMe storage, NICs, GPUs, and FPGAs.
- Expertise in operating systems and hardware development: Hands-on experience in designing, implementing, and optimizing device drivers. Practical knowledge of low-level interactions between hardware and software.
- Problem-solving in cutting-edge technologies: Experience in tackling open research problems that span applications and hardware. The ability to analyze system behavior from device-level to overall system performance.
- Strong research and analytical skills. The ability to conduct independent research, critically evaluate scientific literature, design and execute complex experiments, analyze data, and present findings clearly and concisely.
- Software development: Practical experience with Dolphin ICS’s hardware and driver stack, as well as other open- and closed-source hardware and software platforms.
Qualifications
The scope of this topic is both broad and deep, and a thesis is not expected to cover all the questions above. Therefore, the scope of the project can and should be narrowed down. However, candidates should possess:
- Required:
- BSc or equivalent in Computer Science, Electrical Engineering, Applied Physics, or a related field.
- Strong motivation for hands-on experimental work with hardware.
- Proficiency in C programming and experience with developer tools such as Git.
- Experience in working with computer architecture and operating systems.
- Basic understanding of networking concepts such as routing and flow control.
- General understanding of computer system principles.
- Highly desired (but can be learned during the thesis or scoped out):
- Prior exposure to device drivers and Linux kernel development.
- Experience with FPGA development or CUDA development.