Evaluation of Transformer-Based Large Language Models for Email Spam Detection Using BERT, Phi, and Gemma
Keywords:
Large Language Models, Spam Detection, Fine-Tuning, Binary Classification, Cybersecurity
Abstract
In this paper, we study how LLMs based on the transformer architecture work and the possibility of adjusting these models to use only the body of email messages to classify them as spam or ham. The models studied are BERT, Gemma, and Phi. All of them underwent quantization stages, fine-tuning with a real dataset, and evaluation with metrics commonly used in binary classification problems. The Gemma model achieves over 99% accuracy in detecting spam, standing out as the best among the compared models.References
1. Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, L. Sifre, M. Rivi` ere, M. S. Kale, J. Love, P. Tafti, L. Hussenot, et
al., Gemma, Kaggle, 2024. Available at: https://www.kaggle.com/m/3301.
2. M. Javaheripi, and S. Bubeck, Phi-2: The surprising power of small language models, Published online,
December 12, 2023. Accessed June 10, 2024. Available at: https://www.microsoft.com/en-us/research/blog/
phi-2-the-surprising-power-of-small-language-models/.
3. S. Gunasekar, Y. Zhang, J. Aneja, C. Cesar, T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O.
Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y. T. Lee, Y. Li, Textbooks are all you
need, Published online, June 2023. Available at: https://www.microsoft.com/en-us/research/publication/
textbooks-are-all-you-need/.
4. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, arXiv preprint,
arXiv:2305.14314, 2023.
5. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, LoRA: Low-Rank Adaptation of Large Language
Models, arXiv preprint, arXiv:2106.09685, 2021.
6. P. Singhvi, Spam Email Classification Dataset, Kaggle, 2023. Available at: https://www.kaggle.com/datasets/
purusinghvi/email-spam-classification-dataset.
7. V. Metsis, I. Androutsopoulos, and G. Paliouras, Enron-Spam Datasets, Published online, 2006. Available at: https://www2.
aueb.gr/users/ion/data/enron-spam/readme.txt.
8. G. V. Cormack, and T. R. Lynam, TREC 2007 Public Corpus, University of Waterloo, 2007. Available at: https://plg.
uwaterloo.ca/˜gvcormac/treccorpus07/about.html.
9. Google, Google Colaboratory - Perguntas frequentes, Published online, 2024. Accessed March 17, 2024. Available at: https:
//research.google.com/colaboratory/intl/pt-BR/faq.html.
10. HuggingFace, Hugging Face - Quantization Guide, Published online, 2024. Accessed June 10, 2024. Available at: https:
//huggingface.co/docs/peft/v0.11.0/en/developer_guides/quantization.
11. S. Talebi, QLoRA — How to Fine-Tune an LLM on a Single GPU, Published
online, 2024. Accessed July 2, 2024. Available at: https://towardsdatascience.com/
qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32.
12. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J.
Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine Learning in Python, Journal
of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
13. Y. Wu, S. Si, Y. Zhang, J. Gu, and J. Wosik, Evaluating the Performance of ChatGPT for Spam Email Detection, arXiv preprint,
arXiv:2402.15537, 2024.
14. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention Is All You Need,
arXiv preprint, arXiv:1706.03762, 2023.
15. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language
Understanding, arXiv preprint, arXiv:1810.04805, 2019.
16. Gmail, Gmail Tweet, October 26, 2018. Accessed May 31, 2024. Available at: https://x.com/gmail/status/
1055806807174725633.
17. R. Broadhurst, and H. Trivedi, Malware in spam email: Risks and trends in the Australian Spam Intelligence Database, Trends and
Issues in Crime and Criminal Justice, Australian Institute of Criminology, Barton, ACT, no. 603, 2020. DOI: 10.52922/ti04657.
18. M. Alazab, and R. Broadhurst, Spam and criminal activity, Trends and Issues in Crime and Criminal Justice, Australian Institute of
Criminology, Barton, ACT, no. 526, 2016. DOI: 10.1007/978-3-319-32824-9 13.
19. ITU, Measuring digital development Facts and Figures 2023, ITU Publications, International Telecommunication
Union Development Sector, Geneva, 2023. Available at: https://www.itu.int/hub/publication/d-ind-ict_
mdd-2023-1/.
20. P. Graham, A Plan for Spam, Published online, 2002. Accessed June 11, 2024. Available at: https://paulgraham.com/
spam.html.
21. N. Kumaran, Understanding Gmail’s spam filters, Google, May 27, 2022. Accessed June 11, 2024. Available at: https:
//workspace.google.com/blog/identity-and-security/an-overview-of-gmails-spam-filters.
Stat., Optim. Inf. Comput. Vol. x, Month 202x
22. H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, A Comprehensive Overview of
Large Language Models, arXiv preprint, arXiv:2307.06435, 2024.
23. S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf, Transfer Learning in Natural Language Processing, in Proceedings of the
2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, A. Sarkar, and M.
Strube, Eds., Association for Computational Linguistics, Minneapolis, Minnesota, pp. 15–18, 2019. DOI: 10.18653/v1/N19-5004.
al., Gemma, Kaggle, 2024. Available at: https://www.kaggle.com/m/3301.
2. M. Javaheripi, and S. Bubeck, Phi-2: The surprising power of small language models, Published online,
December 12, 2023. Accessed June 10, 2024. Available at: https://www.microsoft.com/en-us/research/blog/
phi-2-the-surprising-power-of-small-language-models/.
3. S. Gunasekar, Y. Zhang, J. Aneja, C. Cesar, T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O.
Saarikivi, A. Salim, S. Shah, H. S. Behl, X. Wang, S. Bubeck, R. Eldan, A. T. Kalai, Y. T. Lee, Y. Li, Textbooks are all you
need, Published online, June 2023. Available at: https://www.microsoft.com/en-us/research/publication/
textbooks-are-all-you-need/.
4. T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, QLoRA: Efficient Finetuning of Quantized LLMs, arXiv preprint,
arXiv:2305.14314, 2023.
5. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, LoRA: Low-Rank Adaptation of Large Language
Models, arXiv preprint, arXiv:2106.09685, 2021.
6. P. Singhvi, Spam Email Classification Dataset, Kaggle, 2023. Available at: https://www.kaggle.com/datasets/
purusinghvi/email-spam-classification-dataset.
7. V. Metsis, I. Androutsopoulos, and G. Paliouras, Enron-Spam Datasets, Published online, 2006. Available at: https://www2.
aueb.gr/users/ion/data/enron-spam/readme.txt.
8. G. V. Cormack, and T. R. Lynam, TREC 2007 Public Corpus, University of Waterloo, 2007. Available at: https://plg.
uwaterloo.ca/˜gvcormac/treccorpus07/about.html.
9. Google, Google Colaboratory - Perguntas frequentes, Published online, 2024. Accessed March 17, 2024. Available at: https:
//research.google.com/colaboratory/intl/pt-BR/faq.html.
10. HuggingFace, Hugging Face - Quantization Guide, Published online, 2024. Accessed June 10, 2024. Available at: https:
//huggingface.co/docs/peft/v0.11.0/en/developer_guides/quantization.
11. S. Talebi, QLoRA — How to Fine-Tune an LLM on a Single GPU, Published
online, 2024. Accessed July 2, 2024. Available at: https://towardsdatascience.com/
qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32.
12. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J.
Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine Learning in Python, Journal
of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
13. Y. Wu, S. Si, Y. Zhang, J. Gu, and J. Wosik, Evaluating the Performance of ChatGPT for Spam Email Detection, arXiv preprint,
arXiv:2402.15537, 2024.
14. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention Is All You Need,
arXiv preprint, arXiv:1706.03762, 2023.
15. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language
Understanding, arXiv preprint, arXiv:1810.04805, 2019.
16. Gmail, Gmail Tweet, October 26, 2018. Accessed May 31, 2024. Available at: https://x.com/gmail/status/
1055806807174725633.
17. R. Broadhurst, and H. Trivedi, Malware in spam email: Risks and trends in the Australian Spam Intelligence Database, Trends and
Issues in Crime and Criminal Justice, Australian Institute of Criminology, Barton, ACT, no. 603, 2020. DOI: 10.52922/ti04657.
18. M. Alazab, and R. Broadhurst, Spam and criminal activity, Trends and Issues in Crime and Criminal Justice, Australian Institute of
Criminology, Barton, ACT, no. 526, 2016. DOI: 10.1007/978-3-319-32824-9 13.
19. ITU, Measuring digital development Facts and Figures 2023, ITU Publications, International Telecommunication
Union Development Sector, Geneva, 2023. Available at: https://www.itu.int/hub/publication/d-ind-ict_
mdd-2023-1/.
20. P. Graham, A Plan for Spam, Published online, 2002. Accessed June 11, 2024. Available at: https://paulgraham.com/
spam.html.
21. N. Kumaran, Understanding Gmail’s spam filters, Google, May 27, 2022. Accessed June 11, 2024. Available at: https:
//workspace.google.com/blog/identity-and-security/an-overview-of-gmails-spam-filters.
Stat., Optim. Inf. Comput. Vol. x, Month 202x
22. H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, A Comprehensive Overview of
Large Language Models, arXiv preprint, arXiv:2307.06435, 2024.
23. S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf, Transfer Learning in Natural Language Processing, in Proceedings of the
2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, A. Sarkar, and M.
Strube, Eds., Association for Computational Linguistics, Minneapolis, Minnesota, pp. 15–18, 2019. DOI: 10.18653/v1/N19-5004.
Published
2024-12-26
How to Cite
C. Grassmann, A. C., C. Feitosa, J., F. Brega, J. R., & A. P. da Costa, K. (2024). Evaluation of Transformer-Based Large Language Models for Email Spam Detection Using BERT, Phi, and Gemma. Statistics, Optimization & Information Computing, 13(2), 459-473. https://doi.org/10.19139/soic-2310-5070-2267
Issue
Section
Research Articles
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
