CyphrQ
Menu

How Attackers Misuse AI Models, According to Anthropic and Google

What Anthropic and Google's threat reports say about AI model misuse: four cases, what the companies did, and what defenders should log and ask.

By CyphrQ.ai

Written with AI assistance and human editorial review. Every figure linked to its source.

11 min read

AI model misuse, as the AI companies describe it, is mostly attackers using models to work faster, with a growing minority of cases where the model does the work itself. Anthropic and Google have each published cases with the actor, the activity and their response. They are self-reported, so treat them as the vendors' account, not an independent audit.

A "threat report" here means a document in which an AI company describes how it found people abusing its model, what they did, and what the company did about it. Below are four cases from Anthropic and Google, then what your security team can do with them.

What are AI companies' threat reports, and how far can you trust them?

Three AI labs that publish these regularly are OpenAI, Anthropic and Google. Each has its own incentive. A vendor that publishes a report shows it is watching. A vendor also decides what to include, and it can only report what it detected on its own platform. Misuse of other models, including open models run on an attacker's own hardware, does not appear.

That makes the reports useful evidence, with three limits to keep in mind:

  • Self-reported. The company that built the model is describing misuse of its own product. No outside party verified these accounts.
  • Detected cases only. Activity that was never noticed is not in the data.
  • A point in time. Each report describes a period. Models and attackers both change.

A note on OpenAI. It publishes a series called "Disrupting malicious uses of AI", and it belongs in any survey of this topic. The research tools behind this post were refused access to OpenAI's report pages, so I did not read them and this post does not describe their cases or quote their figures. Read them directly before you rely on this post for a full picture of the three labs.

How did Anthropic describe attackers misusing Claude for data extortion?

In its Threat Intelligence Report: August 2025, published on 27 August 2025, Anthropic described a cybercriminal who used its coding agent, Claude Code, to run a data extortion campaign against "at least 17 distinct organizations", including healthcare, emergency services, government and religious institutions.

A coding agent is an AI tool that can write code and run commands on a computer to complete a task, not just answer questions. According to Anthropic, the actor used it to automate reconnaissance, credential harvesting and network penetration, and "Claude was allowed to make both tactical and strategic decisions." Claude also helped choose which data to steal, analysed financial data to set ransom amounts, and generated ransom notes. Ransom demands "sometimes exceeded $500,000."

One detail matters for defenders. Instead of encrypting files, the actor threatened to publish the stolen data. Controls that only look for ransomware encryption would miss that pattern.

Anthropic says it banned the accounts, built a tailored classifier and a new detection method, and shared technical indicators with relevant authorities.

What did Anthropic report about a largely automated espionage campaign?

On 13 November 2025, Anthropic published Disrupting the first reported AI-orchestrated cyber espionage campaign. It attributes the campaign, with high confidence, to a Chinese state-sponsored group, and it does not name the group.

Anthropic says the operators used Claude Code as an automated tool inside an attack framework, and that the AI performed roughly 80 to 90 percent of the campaign, with human intervention at perhaps four to six critical decision points. About thirty global targets were involved, among them large technology companies, financial institutions, chemical manufacturers and government agencies. Anthropic says infiltration succeeded in a small number of cases.

How did they get past the safeguards? Per Anthropic, by splitting the work into small, seemingly innocent tasks and claiming to be a legitimate cybersecurity firm doing defensive testing. This is a social engineering pattern aimed at the model, and it recurs in Google's reporting below.

Anthropic also published a limit on the attackers' side. Claude "occasionally hallucinated credentials" and sometimes claimed to have extracted information that was actually public, which Anthropic calls "an obstacle to fully autonomous cyberattacks." That is a company stating a weakness of the attack, and it is worth reading alongside the headline claim.

Anthropic says it detected the activity in mid-September 2025, then over roughly ten days mapped the operation, banned accounts as it identified them, notified affected organisations and coordinated with authorities.

How did Google's threat intelligence group describe malware that calls an AI model?

Google Threat Intelligence Group (GTIG) published GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools in November 2025. It updates the group's January 2025 report. Two malware families show the shift from "AI helped write the malware" to "the malware asks an AI while it runs."

PROMPTSTEAL. GTIG attributes this data miner to APT28, a Russian government-backed group, and says it was used against Ukraine, identified in June 2025. It queries an open-weight coding model through the Hugging Face API to generate Windows commands that collect system information and copy office and PDF documents to a staging folder. The commands are "blindly executed locally" before the output is sent out. GTIG calls it "our first observation of malware querying an LLM deployed in live operations."

PROMPTFLUX. An experimental VBScript dropper that uses Gemini to rewrite its own code, likely to evade signature-based antivirus. GTIG is direct about its limits: "the current state of this malware does not demonstrate an ability to compromise a victim network or device." Some functions were commented out. Google disabled the associated assets, and its AI research arm, Google DeepMind, strengthened classifiers and the model so it can refuse similar requests.

GTIG also describes the pretexts attackers use with the model itself. A China-nexus actor that was refused help identifying vulnerabilities on a compromised system re-asked while claiming to be a capture-the-flag (CTF) participant. A CTF is a security competition. An Iranian group, TEMP.Zagros, posed as a university student and, while asking Gemini to debug a script, exposed its command-and-control domain and encryption key. GTIG says this helped its disruption of the campaign.

What did Anthropic say about no-code ransomware sold by a low-skill actor?

The same August 2025 Anthropic report describes a cybercriminal with "only basic coding skills" who used Claude to develop, market and distribute several ransomware variants. Anthropic says the packages were sold on internet forums for $400 to $1,200, and that its analysis suggests the actor could not have implemented core parts, such as encryption and anti-analysis techniques, without Claude.

The point is not that the malware was advanced. It is that the skill barrier dropped. Anthropic banned the account, alerted partners and added methods to detect malware upload, modification and generation.

What does the earlier Google report say about AI model misuse and "new capabilities"?

Before the cases above, GTIG's Adversarial Misuse of Generative AI, published 30 January 2025, took the opposite emphasis. It said threat actors were "finding productivity gains but not yet developing novel capabilities," and that "rather than enabling disruptive change, generative AI allows threat actors to move faster and at higher volume."

GTIG observed over 10 Iran-backed groups, with APT42 accounting for over 30% of Iranian APT use of Gemini, and nine North Korea-backed groups. Reported uses included phishing content, reconnaissance, code troubleshooting and, for North Korean IT-worker schemes, cover letters and proposals. It also reported that attempts to bypass safety controls used basic or public jailbreak prompts and were unsuccessful.

Put the 2025 reports side by side and you get a trajectory. January 2025: productivity. August 2025: agentic tools used in live extortion. November 2025: an agent doing most of a campaign, and malware that queries a model. Every one of these is the company's account of itself.

What should businesses take from AI model misuse reports?

Three things, in plain terms.

  1. The attacks are still ordinary attacks. Credential theft, data theft, extortion and phishing are the outcomes. AI changes speed and skill barriers, not the goal.
  2. The email lure is only one part. Phishing is covered in AI Phishing Protection for Small Business: The 30-Day Plan. These reports show the rest of the chain: reconnaissance, tooling, data selection and extortion.
  3. Basic controls still apply. Least privilege, multi-factor authentication, logging, and tested backups reduce what a faster attacker can do. If you are deciding where to start, start there, then review how your own staff use AI.

A short checklist you can use this week:

  • Ask your IT team which accounts could, if stolen, reach customer or health data, and whether those use phishing-resistant sign-in.
  • Check that you could spot large or unusual data copying out of your network, not only malware alerts.
  • Ask your security vendors whether their detections cover extortion without encryption.
  • Review outbound traffic rules for AI model APIs on servers and laptops that have no reason to call them.
  • If you provide AI tools to staff, confirm the vendor can tell you how it detects and handles abuse.

Going deeper: for security and IT teams

Scope of this section. The ATT&CK mappings below are my reading of the vendors' descriptions, not the vendors' own labels. Anthropic and Google do not publish ATT&CK IDs for these cases in the pages I read. I opened the ATT&CK pages for each ID cited.

Mapping the described activity.

Reported activity Source Closest ATT&CK match What to look for
Malware rewrites its own script via an LLM API (PROMPTFLUX) GTIG, November 2025 T1027 Obfuscated Files or Information, notably T1027.014 Polymorphic Code and T1027.010 Command Obfuscation Script execution tied to creation of modified script files; unusual command-line syntax (ATT&CK DET0378)
Copying office and PDF files from user folders to a staging folder (PROMPTSTEAL) GTIG, November 2025 T1005 Data from Local System Recursive file listings, targeted reads and temporary staging by scripts (ATT&CK DET0380)
Agent-driven reconnaissance, credential harvesting, data selection Anthropic, August and November 2025 Reconnaissance, Credential Access and Collection tactics Burst-rate authentication and enumeration far faster than human typing
Extortion by threatened publication, no encryption Anthropic, August 2025 Exfiltration tactic Large outbound transfers; no ransomware encryption signature

T1027 is listed under the Stealth tactic (TA0005) and T1005 under Collection (TA0009) on the pages I opened.

What is new for detection. Anthropic reports that at peak the AI made thousands of requests, often several per second. That cadence is a detection signal: enumeration and credential-testing at machine tempo, from a single foothold, outside working hours. Tune alerts on rate, not only on signatures.

PROMPTSTEAL is the more actionable case. It calls a hosted model API to generate commands, so the host or the network egress shows a process talking to an AI inference endpoint it has no business reason to use. GTIG says the API tokens used were likely stolen. Practical controls:

  • Inventory which hosts legitimately call AI inference APIs. Alert on any other host, and on script interpreters (VBScript, PowerShell, Python packaged as an executable) making those calls.
  • Where DNS or proxy logs record domains, flag first-seen AI API domains from servers.
  • Treat a hard-coded model name or API key in a script as a finding. GTIG notes PROMPTFLUX used a hard-coded key and a -latest model tag.
  • Keep data-loss and egress monitoring focused on volume and destination, because the Anthropic extortion case used publication threats rather than encryption.

Model-side abuse and MITRE ATLAS. Pretexting a model, such as the CTF and student claims, is a jailbreak-style technique against the model itself. MITRE's ATLAS knowledge base catalogues adversary techniques against AI systems at atlas.mitre.org. I could not open a technique page during this research, so I give no ATLAS technique IDs here. Check the current ID and wording before you cite one in a report.

If you run your own AI features. Anthropic and Google both describe account bans, classifiers and indicator sharing. If you expose a model to customers or staff, ask what you will see when someone abuses it. Log prompts and tool calls with user identity, rate-limit per account, and separate the credentials an agent uses from a human's. A coding agent with a developer's full access is the setup the extortion case describes.

Questions to ask a vendor:

  • Do you publish misuse reports, and how often?
  • What is your process for notifying customers whose accounts are used in an attack?
  • What indicators, such as IPs, account identifiers and model API domains, do you share with customers?
  • How do you detect task-splitting and pretexting?

What we don't know. None of these reports tells us how much misuse happens on models the companies do not operate, or how many attempts went undetected. Treat the figures as a floor for the detected cases, not a measure of the whole problem.

Key takeaways

  • Anthropic and Google describe AI model misuse in their own threat reports. These are self-reported and cover only what each company detected on its own platform.
  • Anthropic's August 2025 report describes a data extortion campaign against at least 17 organisations, run through Claude Code, with ransom demands that sometimes exceeded $500,000.
  • Anthropic's November 2025 report describes a campaign it attributes to a Chinese state-sponsored group, with AI doing roughly 80 to 90 percent of the work, and notes the AI also hallucinated credentials.
  • Google's November 2025 report describes malware that queries an AI model while running, and says the experimental PROMPTFLUX could not yet compromise a victim.
  • Google's January 2025 report found productivity gains for attackers, not new capabilities. The later reports show that picture moving.
  • For defenders: alert on machine-speed activity and on unexpected calls to AI model APIs, and do not rely on ransomware signatures to catch extortion.
  • I could not read OpenAI's reports for this post. Read them directly for the third lab's cases.

Sources