Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly used to extract, connect, and summarize information from large collections of documents. While powerful, these systems often function as “black boxes,” making it difficult to understand why a particular answer was produced. This lack of transparency and interpretability limits trust, especially in sensitive contexts such as medical workflows or trend analysis in research and policy. To make such systems more trustworthy, mechanisms for explainable AI (XAI) are needed — highlighting evidence used, tracking provenance, and providing metadata on how results were derived.
This project will design and implement XAI-enhancements for an existing LLM/RAG-based pipeline that extracts data from textual sources. The focus will be on developing methods that improve transparency, interpretability, and trustworthiness of system outputs. Work to be done:
- Evidence extraction: Show what text passages or documents support a given answer.
- Provenance tracking: Record what processing steps were applied to which sources.
- Version control & metadata: Track dataset versions, prompts, and pipeline changes to ensure reproducibility.
- Interpretability mechanisms: Explore approaches such as structured explanations, simplified “reasoning chains,” or metadata-based visualizations.
The system will be demonstrated in one of two contexts:
- A medical workflow (e.g., extracting findings and linking to evidence in clinical guidelines).
- A trend detection scenario (e.g., analyzing research reports or policy texts for emerging topics).