Large‑language models (LLMs) are increasingly integrated into critical applications (e.g., code generation, decision support, customer service). Studies have demonstrated the inherent risk of such integration, where adversarial prompting can potentially produce harmful, biased, or confidential leaking outputs.
What if we develop threat-hunting methods to proactively analyze LLM activities? This work will explore threat hunting methods in an LLM context.
Project Goal:
This work will evaluate existing methods for identifying adversarial behaviors within LLM prompts and propose a threat hunting framework. The goal of the framework is to identify and explain "what", "how" and "why". What activity occurred, how the activity occurred and why the activity was/wasn't successful. The framework must align with techniques (some not every single one) in the MITRE ATLAS framework
https://atlas.mitre.org/matrices/ATLAS
Note: This project is available as a group project for 2 students. We can discuss the roles and responsibilities.
Requirements: Basic Python knowledge