Rare diseases are often difficult to study because of the limited number of patients and the scattered nature of clinical data. Modern hospitals and research centers generate large amounts of multimodal information, including medical images (MRI, CT, X-ray) and clinical data (electronic health records, lab results, structured and unstructured notes). Managing, linking, and making sense of these heterogeneous data sources is a major challenge. Recent advances in Retrieval-Augmented Generation (RAG) systems and large language models (LLMs) open the possibility of developing research assistants that can connect different data modalities, annotate medical images with meaningful metadata, and help generate new insights for disease understanding.
The goal of this project is to design a RAG system that links medical images (MRI, CT, or X-rays) with clinical data (EHR notes, lab results, structured/unstructured text). By using clinical knowledge and ontologies to tag images and connect them with patient records, the system aims to uncover new patterns and support deeper disease understanding. The project will also explore the ethical and legal challenges of working with sensitive health data. The goal is to prototype an AI assistant that doesn’t just manage multimodal data, but helps make it meaningful for rare disease research. The system should demonstrate how LLMs and retrieval components can:
- Ingest and manage multimodal data: one or two imaging modalities (MRI, CT, or X-ray) combined with clinical data (lab results, clinical notes, structured/unstructured EHR data).
- Link imaging and clinical data: identify relationships between image findings and corresponding clinical parameters.
- Tag images with ontology-based labels: use existing biomedical ontologies (e.g., SNOMED CT, RadLex, or Orphanet) and extracted clinical features to annotate images.
- Generate insights: explore how integrated multimodal data can support better understanding of rare disease mechanisms, presentation, or diagnosis.
The project should also include an exploration of ethical and legal considerations in handling medical data, such as patient privacy, data protection regulations, and responsible use of AI.