College
College of Behavioral and Community Sciences
Mentor Information
George Burruss, Ph.D.
Description
Senior citizens are disproportionately targeted by cybercrime, yet many lack accessible resources for prevention and recovery. The SeniorSafeAI project explores how large language model (LLM) chatbots can provide reliable guidance to older adults experiencing or at risk of online fraud. We developed a chatbot trained on a “ground truth” dataset consisting of 589 question–answer pairs covering common cybercrime categories affecting seniors, including identity theft, romance scams, investment fraud, and fake tech support scams. Undergraduate researchers played a central role in constructing this dataset by identifying evidence-based sources and drafting responses grounded in government reports, academic research, and cybersecurity best practices.
To determine the most effective chatbot model, we evaluated nine open- and closed-source LLMs, including base models and LoRA fine-tuned variants. Model outputs were compared against ground truth responses using a mixed-methods evaluation approach combining automatic metrics with human ratings assessing clarity, relevance, accuracy, and usefulness. Finally, we conducted a hallucination analysis to identify instances where models generated unsupported or inaccurate information. Human coders compared chatbot responses against verified ground truth answers to detect factual errors or omissions.
Together, these analyses provide insights into how domain-specific training data, model selection, and evaluation can improve the reliability of AI chatbots designed to support cybercrime prevention and victim assistance.
SeniorSafe AI: Developing an AI-Powered Chatbot to Support Older Adults Who Are Victims of Cybercrime
Senior citizens are disproportionately targeted by cybercrime, yet many lack accessible resources for prevention and recovery. The SeniorSafeAI project explores how large language model (LLM) chatbots can provide reliable guidance to older adults experiencing or at risk of online fraud. We developed a chatbot trained on a “ground truth” dataset consisting of 589 question–answer pairs covering common cybercrime categories affecting seniors, including identity theft, romance scams, investment fraud, and fake tech support scams. Undergraduate researchers played a central role in constructing this dataset by identifying evidence-based sources and drafting responses grounded in government reports, academic research, and cybersecurity best practices.
To determine the most effective chatbot model, we evaluated nine open- and closed-source LLMs, including base models and LoRA fine-tuned variants. Model outputs were compared against ground truth responses using a mixed-methods evaluation approach combining automatic metrics with human ratings assessing clarity, relevance, accuracy, and usefulness. Finally, we conducted a hallucination analysis to identify instances where models generated unsupported or inaccurate information. Human coders compared chatbot responses against verified ground truth answers to detect factual errors or omissions.
Together, these analyses provide insights into how domain-specific training data, model selection, and evaluation can improve the reliability of AI chatbots designed to support cybercrime prevention and victim assistance.
