This research aims to explore the potential of Large Language Models (LLMs) for automating the analysis of privacy policies, focusing on identifying and flagging high-risk data practices such as biometric data usage, Personally Identifiable Information (PII) handling, profiling, location tracking, and private conversation monitoring. The study will involve comparing different LLMs (e.g., GPT-4o, Google Gemini, etc...) to assess their effectiveness in identifying privacy risks and evaluating how each model performs in terms of accuracy, sensitivity, and comprehensiveness.
Objectives:
- Prompt engineering: identify relevant privacy risks and create a prompt which is effective for the selected model. Find a solution to manage the contex window constraints, token limits of the models, which might require policy segmentations in case of smaller models.
- Automated Privacy Policy Analysis: Investigate the ability of LLMs to analyze company privacy policies and flag high-risk areas such as the collection and usage of sensitive data (e.g., biometrics, PII, profiling, location data).
- Risk Classification: Develop a framework for categorizing privacy risks into high-risk (e.g., invasive data practices), moderate-risk (e.g., standard PII collection), and low-risk categories.
- Model Comparison: Perform a comparative analysis of multiple LLMs (e.g., GPT-4o, Google Gemini, etc... and evaluate their performance in accurately identifying and highlighting the outline risks.
- Human vs. AI Analysis (optional): Compare LLM performance against human experts in privacy law and policy analysis to assess the reliability and practical usefulness of LLMs in real-world applications.
Methodology:
- Dataset Creation:
- Collect a diverse set of company privacy policies from various industries (e.g., healthcare, finance, social media, e-commerce). The policies should cover a range of data practices and complexities.
- Evaluation:
- Annotate these policies with privacy risks based on established privacy frameworks like GDPR, CCPA, and industry best practices. Categories will include risks related to biometrics, PII, profiling, location data, and more.
- LLMs Selection: Use GPT-4o, Google Gemini, and potentially other models (e.g., Claude by Anthropic, Meta’s LLaMA) for the analysis.
- Human vs. AI Comparison (optional):
- Conduct a parallel privacy policy analysis by human privacy experts. Compare the results to the AI-generated risk reports to evaluate how closely the AI aligns with human judgment.
- Qualitative analysis will be used to highlight areas where the AI models either outperform or fall short compared to human reviewers.