Developing Systematic Techniques for Multi-Agent LLM System Development

Large Language Models (LLMs) have evolved from being single-shot tools to becoming core components of multi-agent systems, where several LLM-powered agents collaborate to solve complex problems. While there is a growing body of work on orchestration frameworks (e.g., AutoGen, LangGraph), there is still limited research on systematic development techniques for designing, coordinating, and optimizing such multi-agent LLM systems. 

Research Problem

The development of multi-agent LLM systems today is often ad hoc:

• There are no well-defined design patterns or architectural guidelines for when to use multiple agents versus a single powerful model.
• Most orchestration frameworks focus on execution but provide little support for testing, debugging, or optimizing communication flows between agents.
• System efficiency and sustainability are rarely considered explicitly, even though multi-agent setups can significantly increase energy consumption and latency.

Objectives

The objective of this thesis is to create a structured methodology for designing and developing multi-agent LLM systems. The student will propose, implement, and evaluate development techniques that guide practitioners in:

• Agent Role Definition & Coordination: Formalizing how agents are decomposed into roles (planner, executor, critic, etc.) and how they interact.
• Workflow Optimization: Designing and optimizing communication and decision-making protocols to minimize redundancy, improve response accuracy, and reduce energy usage.
• Systematic Testing: Introducing testing and debugging methods (e.g., metamorphic testing, output agreement, adversarial perturbations) to improve robustness.
• Energy-Aware Development: Measuring energy consumption and proposing strategies to make the system more sustainable (e.g., caching strategies, model selection, turn-taking optimization).

Methodology & Tasks

The work will be divided into these phases:

1. Literature Review: Survey multi-agent LLM frameworks, orchestration tools, and current development practices. Identify gaps in testing, optimization, and sustainability.
2. Conceptual Framework: Define a reference model and set of design patterns for multi-agent LLM systems, including guidelines for agent roles, communication, and error-handling.
3. Prototype Implementation: Build a small but representative multi-agent system (e.g., for code generation + review or log analysis) using open-source frameworks (AutoGen, LangGraph, etc.).
4. Evaluation: Measure performance, accuracy, robustness, and energy consumption of different configurations (single agent, dual-agent, multi-agent) and analyze trade-offs.
5. Validation: Demonstrate the methodology on a real-world use case (e.g., software engineering task, IoT log analysis) and document lessons learned.

Expected Contributions

• A methodology and set of best practices for systematic multi-agent LLM development.
• A reference implementation with modular and reproducible code.
• A comparative evaluation study on accuracy, robustness, latency, and energy efficiency.
• Insights into the trade-offs between complexity and performance in agentic workflows.

Expected Outcome

A structured, validated framework that allows developers and researchers to design, implement, and optimize multi-agent LLM systems in a repeatable and energy-aware way. The work is expected to lead to a publishable result at venues like ICSE (SE4AI track), AAMAS, or NeurIPS DMLR/LLM-Agent workshops.