As Large Language Models (LLMs) and general-purpose AI systems become increasingly prevalent, they introduce new and complex security challenges. In this context, AI security is defined as the protection of these systems against intentional, malicious threats from adversarial actors. This project will explore the frontier of AI red teaming, adapting classic cybersecurity methodologies to address these threats within the unique context of open-source AI models.
You will contribute to the development of a holistic framework for proactive AI security testing. This involves simulating adversarial attacks to discover and mitigate vulnerabilities before they can be exploited by threat actors. You will gain hands-on experience with cutting-edge offensive security techniques and an understanding of the adversarial mindset crucial for effective defense.
Project Goal: The primary goal is to advance the methodology of AI security red teaming by systematically identifying and exploiting vulnerabilities in open-source general-purpose AI systems. The project will culminate in the proposal of a novel framework for security assessment, tailored to the unique threat landscape of these systems.
Your work will involve:
- Conducting a literature review on red teaming for cybersecurity and existing adversarial attacks (e.g., prompt injection, data poisoning) and their applications against large language models.
- Designing and executing reproducible experiments to evaluate the effectiveness of adversarial techniques (e.g., prompt injection, data poisoning, model extraction attacks) against a predetermined set of open-source LLMs.
- Documenting and categorizing the discovered vulnerabilities that are new to general purpose open-source AI systems, compared to general cybersecurity vulnerabilities.
- Proposing a novel AI red teaming framework for the security domain that is based on established cybersecurity principles but specifically adapted for the unique threat landscape of open-source general purpose AI systems.