The work will explore how specialized LLM agents—responsible for tasks such as data ingestion, transformation, validation, and orchestration—can collaboratively generate, configure, and optimize end-to-end pipelines. The study should include the design of a multi-agent workflow, implementation using an existing LLM-agent framework, and evaluation against manually designed pipelines. Evaluation criteria will focus on correctness, efficiency, scalability, and energy consumption. The expected contribution is a framework and empirical insights into the effectiveness of multi-agent LLMs in automating data pipeline creation. Work to be done:
- Literature Review: Study existing research on multi-agent LLM systems, pipeline orchestration frameworks (e.g., Airflow, Kubeflow, SIM-PIPE), and AI-assisted programming.
- Problem Analysis: Identify challenges in current pipeline creation workflows (e.g., manual specification, error-prone configurations).
- Design: Define agent roles (e.g., Data Ingestion Agent, Transformation Agent, Validation Agent, Orchestration Agent) and specify their interactions.
- Implementation: Build a prototype using an existing multi-agent framework (e.g., AutoGen, LangChain Agents, CrewAI).
- Evaluation Setup: Select benchmark tasks (e.g., ETL pipeline, ML training pipeline, big data workflow).
- Experiments: Compare LLM-generated pipelines against manually engineered ones in terms of correctness, performance, scalability, and energy usage.
- Analysis: Assess strengths, limitations, and potential improvements of multi-agent LLM-driven pipeline generation.