Background and Motivation
Spam emails have long been a challenge, leading to cybersecurity risks, and increased operational costs. Traditional spam filters, such as rule-based systems or Bayesian filters, have been effective to a degree but often fail to adapt to the increasingly sophisticated nature of spam messages that are now enhanced thanks to LLMs.
The emergence of AI and machine learning (ML) provides a promising direction for tackling this challenge. Many large organizations rely on advanced ML-powered spam filters, which require vast amounts of data and computing resources. However, smaller enterprises and individuals cannot afford such computationally expensive models. Therefore, this research aims to explore whether smaller, less resource-intensive LLMs can offer an effective spam-filtering solution.
Research Questions and Objectives
The proposed research will focus on answering the following questions:
- How effective are smaller Smaller Transformer-based Models in detecting spam when trained on a limited dataset?
- Does increasing the size and complexity of a model improve the detection accuracy, and by how much?
- What trade-offs exist between computational cost and detection efficacy when employing models of different sizes?
- Can AI models adapt to evolving spam strategies more efficiently than traditional methods?
The core objectives are:
- Collect a well-labeled dataset consisting of both spam and ham emails. Open-source datasets such as the Enron Email Dataset, SpamAssassin. We should be also able to get our hands on spam emails from UiO. Preprocessing steps will include tokenization, stemming, lemmatization, and optionally removal of unnecessary features such as email metadata (headers) if not relevant. (spacy)
- To train and evaluate a smaller LLM (7 billion) for spam detection using both spam and non-spam (ham) examples.
- To compare its performance against a larger (un)trained model to identify the trade-offs.
- To propose an optimal solution for spam filtering that balances accuracy with computational efficiency given the resources of the end user
Literature Review and Related Work
To provide context and theoretical grounding, you will explore the following areas:
- Traditional Models as a Baseline: Before jumping into complex transformer models, it’s wise to first explore and understand the related literature for traditional machine learning models. This allows you to establish a baseline in terms of performance (accuracy, precision, recall, etc.) and resource usage (training time, memory, etc.).
- Spam detection systems based on deep learning models (e.g., Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Transformer models).
- A review of open-source datasets such as Enron spam corpus and others that are widely used for training spam filters.
- Other relevant paper doing similar things
Requirements: Programming languages: strong Python foundations In case of no machine learning background, it is advised to spend time in the second semester to take: https://www.uio.no/studier/emner/matnat/ifi/IN5550/index.html.
The thesis can be potentially be a join thesis for two students.