SophosAI at DEF CON: Orchestrating large-scale scams using text, audio and image generative AI

Sophos X-Ops usually focuses on how artificial intelligence technologies can be used to enhance cybersecurity. But for a presentation at DEF CON’s AI Village on Saturday, August 12, SophosAI’s Ben Gelman and Younghoo Lee have taken a turn toward the dark side—demonstrating how generative AI can be used to automate the creation of campaigns to steal users’ credentials and other personal information.

As Gelman and Lee note in their abstract for the talk, large language model (LLM) text generation (such as OpenAI’s ChatGPT, Google’s Bard, and Meta’s Llama 2), speech AI (such as Google’s WaveNet) and image generative technologies (such as DALL-E, Stable Diffusion, and Midjourney) have made it possible to create “a new level of diverse, synthetic content” that can deceive on an unprecedented scale.”  Scammers have already been observed using LLMs as part of their toolkit, to generate text in conversations with victims where fluency in the target’s language may be a barrier, but LLMs can also be used to generate other content—including not just text, but code for websites. Combined with image and audio creation capabilities of other generative AI tools, it’s possible to use AI to generate entire scam campaigns that blur the boundary between reality and fiction.

Gelman and Lee will present on a proof-of-concept for automatically orchestrating credential-stealing scam campaigns—using AI to create code, text, images, and audio to build dozens of websites, product catalogs, testimonials, and social media advertisements. They will also discuss how the barriers to entry have decreased for criminals with a minimal knowledge of AI, and the scale of automation that can be achieved with AI tools—as well as the limitations of the tools that still require human intervention to complete these tasks.

The talk, entitled The Sinister Synergy of Advanced AI: Automatically Orchestrating Large-Scale Scam Campaigns with Large Generative Models, will be presented at 1:30 PM Pacific Time at the AI Village in Caesar’s Forum.