In early 2024, an employee at the engineering firm Arup joined a video call with the company's CFO and several colleagues, then made 15 transfers totalling about US$25 million. Everyone else on the call was a deepfake. Arup said none of its internal systems were compromised.
No firewall failed and no CVE was involved. The attackers simply made a request look legitimate.
That story is a good starting point because it captures what AI has done to security. Most of the underlying problems are old: social engineering, supply chain attacks, over-privileged services, untrusted input. AI makes them cheaper to exploit, and it adds one new component to our systems that can't reliably tell instructions from data.
This post covers what has changed on both sides, and what you can do about it if you ship software with an LLM in it.
Attackers got cheaper
In November 2025, Anthropic reported disrupting an espionage campaign that it attributed to a Chinese state-sponsored group. The operators targeted about 30 organizations, a small number successfully, and used an AI coding agent wired to ordinary penetration-testing tools. According to the report, the AI carried out 80 to 90 percent of the tactical work, with humans stepping in at key decision points.
The detail I find most useful is that the model sometimes overstated results and made up findings, so the operators had to validate its output. Attackers have the same reliability problem you do. It slows them down, but it doesn't stop them.
Then there's the Nx npm compromise from August 2025. Malicious package versions looked for AI command-line tools already installed on the developer's machine and ran them with flags like --dangerously-skip-permissions and --yolo, asking them to help find secrets. The malware didn't need to know where your credentials were, because it could ask a capable assistant you had already installed and logged in.
If you use a coding agent locally, it is now part of your attack surface.
Defenders got better tools, and more noise
The same capability works for defense. At DARPA's AI Cyber Challenge final in 2025, teams built autonomous systems that found and patched vulnerabilities in open source code. The winning system, from Team Atlanta, also found 18 previously unknown vulnerabilities in real-world software.
The cost shows up elsewhere. curl ended its bug bounty at the end of January 2026 after a flood of low-quality reports, many of which appeared to be AI-generated. Its maintainer, Daniel Stenberg, has also acknowledged receiving strong reports from a researcher who used AI as an aid.
Both things are true. AI in skilled hands finds real bugs, and AI in careless hands burns maintainer time. If you run a security inbox, the new requirement is a working proof of concept, because a plausible-sounding paragraph now costs nothing to produce.
Prompt injection is a design problem
Prompt injection has held the top spot in the OWASP Top 10 for LLM Applications, and it's worth being precise about why it's so stubborn.
SQL injection was solved by separating code from data with parameterized queries. LLMs have no equivalent. The system prompt, the user's message, and the contents of a web page all arrive as one stream of tokens. A model can be trained to favor some of those over others, but that's a statistical preference and not a boundary.
Security people have an old name for what happens next: the confused deputy. A program holding legitimate authority gets tricked into using that authority for someone else. Your agent has your OAuth token, and it will read whatever text you hand it.
EchoLeak (CVE-2025-32711) showed this working against a mainstream product. Researchers at Aim Labs found that a crafted email could cause Microsoft 365 Copilot to leak organizational data with no click from the victim. Microsoft fixed it server-side and reported no customer impact, but the pattern is what matters: untrusted content entered the context, the assistant had access to private data, and there was a route out.
Simon Willison named that combination the lethal trifecta:
- Access to private data
- Exposure to untrusted content
- A way to communicate externally
Meta generalized it into the Agents Rule of Two: until injection can be reliably detected and refused, an agent should have no more than two of these three properties in a session without human approval: processing untrustworthy input, accessing sensitive data, and changing state or communicating externally. Meta is upfront that this doesn't make an agent safe on its own, but it gives you something you can actually enforce.
Enforce it outside the model
The important part is where the check lives. A line in the system prompt saying "never send private data externally" is a request, and injected text can make a competing request. The check has to be ordinary code that the model can't talk its way past.
Here's a minimal version:
from dataclasses import dataclass, field # Declared by developers, never derived from model output. TOOLS = { "search_tickets": {"private"}, "fetch_url": {"untrusted"}, "send_email": {"acts"}, "update_record": {"acts"}, } DANGEROUS = {"private", "untrusted", "acts"} class ApprovalRequired(Exception): pass @dataclass class Session: seen: set = field(default_factory=set) def authorize(session: Session, tool: str) -> None: """Call before every tool execution. Unknown tools raise KeyError.""" after = session.seen | TOOLS[tool] if DANGEROUS <= after: raise ApprovalRequired(f"{tool} would complete the trifecta") session.seen = after The agent can search tickets and send an email, or fetch a URL and send an email. Once it has done both reads, any action stops and waits for a person:
session = Session() authorize(session, "search_tickets") # ok authorize(session, "fetch_url") # ok authorize(session, "send_email") # raises ApprovalRequired A real system would track this per data flow and treat tool results as untrusted by default, including results from your own database if users can write to those fields. The principle stays the same at any scale: permissions come from the session's history, and the model doesn't get a vote.
The exits you forgot about
Teams often lock down the obvious tools and leave quieter channels open. Anything that causes a request to a URL the model influenced is an outbound channel:
- Rendered Markdown images.
sends data the moment the browser loads it. Disable remote images in model output or proxy them through an allowlist. - Link previews and unfurls. Chat clients fetch URLs automatically.
- Generic HTTP tools. A
fetch_urltool can exfiltrate through its query string, so restrict it to known domains where you can. - Allowlisted domains with open redirects or user content. An allowlist is only as strict as the most permissive site on it.
Your dependencies might be hallucinated
LLMs invent package names, and they do it consistently. A study of 576,000 generated code samples found that about 20 percent of recommended packages didn't exist, though the rate varied widely by model. When researchers re-ran the prompts, 43 percent of the invented names came back in all ten runs.
That repeatability is what makes "slopsquatting" work. An attacker can collect the names models tend to invent and register them. Few of those names look like typos of real packages, so a typo check won't catch them.
Some habits help:
# When was this package created? Days old is a warning sign. npm view some-package time.created # Install exactly what the lockfile says. npm ci # Don't run install scripts unless you have decided to. npm config set ignore-scripts true The ignore-scripts setting can break packages that need a build step, so expect to allow a few explicitly. On the agent side, don't let a coding agent add dependencies unreviewed. A new line in a manifest deserves the same attention as new code.
A checklist for your next AI feature
Before you ship
- List every tool the agent can call and label it: reads private data, ingests untrusted content, changes state or sends data out.
- Check that no session can hit all three without a human approving.
- Give the agent the user's permissions, not a service account that can see everything.
- Prefer narrow tools.
get_invoice(id)is much easier to secure thanrun_sql(query). - Treat retrieved documents, emails, web pages, and tool output as attacker-controlled.
At the approval step
- Show the person the exact action and its parameters, not the model's summary of them.
- Keep approvals rare. People click through dialogs they see fifty times a day.
In production
- Log every prompt, tool call, argument, and result. Without this, you can't investigate anything afterward.
- Alert on unusual tool sequences, such as a read from a sensitive source followed by an outbound call.
- Budget tokens per user. Unbounded consumption is a cost problem and an availability problem.
- Include injection cases in your test suite, and rerun them when you change models, because behavior shifts between versions.
For your people
- Verify money movement and credential resets over a separate channel you initiated. After Arup, a familiar face on a video call doesn't count as authentication.
- Scope the credentials on developer machines. Coding agents and malware can both read
~/.aws.
What won't save you
A stronger system prompt. It helps with accidental misbehavior and does little against a determined attacker.
A classifier in front of the model. Guard models are a reasonable layer and they catch a lot. But a filter that stops 99 percent of attacks still fails against an adversary who gets unlimited attempts. Use one to reduce noise, and don't make it the thing standing between an attacker and your customer data.
Waiting for better models. Newer models do resist injection better, and you should take those gains. Your architecture still shouldn't depend on the model never being fooled.
The mindset that holds up
Think of an LLM as a capable, fast new hire who believes everything they read. You wouldn't give that person production credentials and an unmonitored inbox from the public internet. You'd limit what they can touch, review the consequential things, and keep records.
The good news is that most of this is familiar work. Least privilege, input distrust, egress control, audit logs, and supply chain hygiene are decades old. What's new is that the component in the middle can be persuaded, so the controls need to sit around it.
What's the riskiest tool you've given an agent, and what stops it from being misused? I'd like to hear how you've handled approvals without everyone clicking through them.
This article was written with AI assistance and reviewed before publishing.
