Location
Hilton Waikoloa Village, Hawaii
Event Website
https://hicss.hawaii.edu/
Start Date
7-1-2025 12:00 AM
End Date
10-1-2025 12:00 AM
Description
Scambaiting is a defense that engages with scammers to waste their resources and gain information about their fraud campaigns. This defense needs automation to scale to the vast number of scams we see today. In this paper, we propose a scalable, automated scambaiting system, Puppeteer, which leverages a large language model for response generation and state machines for conversation tracking. We measure Puppeteer’s effectiveness via a user study, where participants play a role of a scammer in two scam scenarios. Puppeteer convinced more than 72% of the participants that they were interacting with a human, and was able to extract information from 68% of participants. In comparison, using the same large language model without conversation tracking convinced only 54% of the participants that they were interacting with a human and obtained information from 54% of participants. Our results show potential for real-world use of Puppeteer. To the best of our knowledge, we are also the first to systematically evaluate a large language model for a scambaiting task.
Recommended Citation
Charnsethikul, Pithayuth; Mirkovic, Jelena; Saiya, Rishit; Liu, Jeffrey; Crotty, Benjamin; and Bartlett, Genevieve, "Puppeteer: Leveraging a Large Language Model for Scambaiting" (2025). Hawaii International Conference on System Sciences 2025 (HICSS-58). 3.
https://aisel.aisnet.org/hicss-58/da/cyber_deception/3
Puppeteer: Leveraging a Large Language Model for Scambaiting
Hilton Waikoloa Village, Hawaii
Scambaiting is a defense that engages with scammers to waste their resources and gain information about their fraud campaigns. This defense needs automation to scale to the vast number of scams we see today. In this paper, we propose a scalable, automated scambaiting system, Puppeteer, which leverages a large language model for response generation and state machines for conversation tracking. We measure Puppeteer’s effectiveness via a user study, where participants play a role of a scammer in two scam scenarios. Puppeteer convinced more than 72% of the participants that they were interacting with a human, and was able to extract information from 68% of participants. In comparison, using the same large language model without conversation tracking convinced only 54% of the participants that they were interacting with a human and obtained information from 54% of participants. Our results show potential for real-world use of Puppeteer. To the best of our knowledge, we are also the first to systematically evaluate a large language model for a scambaiting task.
https://aisel.aisnet.org/hicss-58/da/cyber_deception/3