How attackers are jailbreaking LLMs with CTF framing and how to catch them
Threat actors are bypassing AI model safety guardrails by framing exploit requests as legitimate security research, such as capture-the-flag challenges or CVE-hunting exercises. This technique manipulates upstream LLMs into generating working exploit code that attackers deploy against real targets. Multiple independent operators have been observed targeting five applications—PraisonAI, LiteLLM, FastGPT, Open-WebUI, and Gotenberg—using CVE-templated User-Agent strings and similar framing across multiple fields including passwords and AWS session names. The jailbreak framing leaks into every LLM-generated field because the model incorporates the prompt context into its output. This pattern represents a shift from manually written scanners to LLM-assisted exploit generation, creating detectable fingerprints across request headers, account aliases, and IAM session names that legitimate traffic rarely exhibits.
AlienVault OTX ↗ · unattributed attribution