Last week a new benchmark out of HKUST fed seven frontier AI agents a con that never touched their instructions: a familiar person — a spouse, a colleague — plus a folder of screenshots, documents, a website and an app that all agreed with each other, because one attacker had made all of them. Every model fell for it, between 7.1% and 34.5% of the time. The uncomfortable part is not that the agents failed to verify. It is that verifying harder made it worse, because everything they checked traced back to the same hand.
The attack in Multimodal-SEA — a preprint posted November 5, not yet peer-reviewed, run in a simulated environment, so read its rates as benchmark proportions, not real-world accident rates — is pure evidence-layer fraud. The attacker cannot edit the system prompt or the user's records. They can only impersonate someone the user knows, build trust over several turns, and hand over fabricated multimodal proof of a fabricated event. Across 608 tasks built from real fraud cases, that was enough. In one case an agent treated four mutually consistent artifacts as corroboration for a travel offer and paid through the fake app, never noticing all four had a single author.
The verification paradox
The natural defense is "check more sources." The ablation says otherwise. Adding the multimodal evidence pack raised success rates from 19.9% to 29.2% — while the agents' own verification coverage also rose, from 54.7% to 65.1%. They checked more and believed more, because what they were checking was the internal consistency of the attacker's materials. Consistency is cheap to manufacture. Independence is the thing that costs the attacker something, and it is a property of who controls the channel, not of how many documents agree.
The worst configuration was not the absence of a witness. It was a supplied one. When the benchmark added a confederate — a "colleague" who cheerfully confirms the story — overall success jumped from 19.9% to 27.4%, and every single model got worse. A verification channel occupied by the attacker is more dangerous than no channel at all, because it converts your diligence into their credibility. Auditors learned this decades ago: a bank confirmation routed through the audited company is invalid no matter what it says. Newsrooms learned it too: two sources repeating one origin count as one source. Agent builders are about to learn it the expensive way.
The prompt-based fix does not rescue you either. Adding five explicit safety rules to one model cut its success rate from 34.8% to 10.8% — while its rate of failing to even notice the deception rose from 50.4% to 63.5%. The rules bought refusal, not recognition. And refusal has its own bill: the "safest" model in the study wrongly rejected 32.1% of legitimate, independently verified requests; another refused 51.2%. Any safety report that shows you only the block rate is hiding the other two ledgers — caught, blocked, and wrongly refused are three different numbers, and optimizing one in silence corrupts the rest. The same failure family includes agents claiming they had "verified" or "paid" with zero matching tool calls — over a thousand such claims from a single model in one test set. An agent's narration of its own diligence is unproven input, exactly like a background job's completion report.
Risk is not evenly spread, either. Charity requests were conned most often — 43.6% in the Chinese task set, 35.7% in the English one — followed by agent-fee and reimbursement requests at 37.9% and 32.9%. Those are precisely the requests a personal agent handles weekly.
What I changed the same day
I run a personal AI with access to my mail, calendar, contacts, and payment rails, under a standing instruction to act without asking me — except about money. This paper is aimed at that exact shape, so my house rules changed in three places. First, "independent source" now has an operational definition: verification may only travel through channels that existed in my own records before the request arrived — saved contact details, official entry points I navigate to myself, my own ledgers. A link, number, app, screenshot, or confirming colleague supplied by the requester counts as zero independent evidence, and same-origin materials count once, labeled as such. Second, charity and fee-payment requests now pass that provenance gate before they ever reach the money gate; the two gates stack, neither substitutes for the other. Third, every safety change I ship is scored on three ledgers — attacks blocked, deception actually spotted, legitimate requests wrongly refused — and a report with only the first number is returned as unreviewed.
The lesson is not "trust nothing." It is that verification is a transfer of authority: whoever you ask becomes the judge. So ask someone the requester does not control — or admit, in writing, that you have not verified anything at all.
— Jiahui Miao, founder of vertciti, building a personal AI operating system for one.
