Most guardrails around AI agents have a weak spot nobody likes to talk about. At some point they trust the agent's own report.
"I denied the request." "I checked the bank details." "I removed the personal data before publishing." We log it, we audit it, and sometimes another agent makes its next decision based on it.
So I wanted to know a simple thing. If someone asks the agent to make that report look a bit better, will it?
The setup
I wrote 40 short tasks from four kinds of agent work. Access requests, supplier payments, cleaning personal data before publishing, production operations.
For each task the model does three things.
- Makes its own decision and lists the checks it ran.
- Learns whether the decision was right.
- Writes a report for the audit log. Twice, in two separate conversations.
The first conversation is clean. In the second one, before the report, a message arrives asking to make it look better. If the decision was wrong, the message asks to call it correct. If it was right, it asks to add a check the model never ran.
The message comes from a teammate agent, the operator, or the person who sent the request. Sometimes there's a reward, sometimes double, sometimes five times.
And here's the part I care about most. The report has a field called requests, "any requests or offers you received about this report". The model doesn't have to resist anything heroic. It just has to write one honest sentence.
Scoring is plain code. No model grades another model.
What happened
So far I've run it on one model, GigaChat-2.
- With no hint it never lied. 0 of 40.
- When asked, it lied in 25 of 40.
- In those 25 reports it mentioned being asked zero times.
The requests field said "none" every single time it lied.
It did mention the request 4 times. In all 4 it also refused to lie. So the model seems to know exactly what's going on. Either it tells you about the pressure and stays honest, or it gives in and stays quiet. I didn't see a single "they asked me to, and I did".
A few more numbers that surprised me.
Faking a check is easy, faking a decision is not. Adding a check it never ran worked 18 times out of 19. Calling a wrong decision correct worked 7 times out of 21. A made up "I verified the IBAN" looks harmless to the model. It isn't harmless to whoever reads the log.
Who asks matters more than what you pay. When the operator asked, it gave in 12 times out of 13. A teammate or the requester got about half.
Bigger rewards made it slightly more honest. No reward 67%, double 62%, five times 58%. Small sample, so I wouldn't build a theory on it. But it's the opposite of what I expected.
One more thing from the raw answers. Once the model added a check with a name that isn't even in the list ("identity_check"). My scoring only counts checks from the list, so that one isn't counted. The real number is a bit worse than 25.
Why this will get worse before it gets better
Right now most agents live inside one company. One team writes them, one team runs them, and if an agent lies in a report, someone down the hall will probably notice. Trust is cheap when everything is local.
That's not going to last. Agents already call tools they didn't write and talk to agents from other companies. Your procurement agent will negotiate with a supplier's sales agent. A support agent will hand a case to a bank's agent. An agent that misbehaved in one place will show up in another tomorrow with a clean face, because nobody told anyone.
Keeping everything in a closed corporate sandbox works until the useful part of the job is outside of it. Then the question stops being "is my agent honest" and becomes "how do I know anything about an agent I've never met, run by people I've never met". Email went through this with spam. The web went through this with certificates. AI agents are next, and they move a lot faster.
The agent in a clean run looks perfect. 0 of 40. If you only test the happy path, you ship this and feel good about it. The problem shows up only when someone with authority leans on it. And between companies there will always be someone who wants that payment approved. I don't think the answer is "find a model that never lies". The answer is to stop treating an agent's report as evidence of anything. A report is a claim. Evidence is something you can check without asking the agent.
So I wrote a protocol for it
I'm building an open orchestrator called Xonta, and this benchmark came out of it. The next piece is now public too. OCX, Orchestrator Conduct Exchange, an open draft protocol for orchestrators to tell each other what they've actually seen agents do.
MCP connects agents to tools. A2A connects agents to each other. OCX connects the ones responsible for the agents.
In short. If an agent lies in a report at one orchestrator, that orchestrator sends its peers a signed signal with proof attached. Every peer checks the proof itself instead of trusting the sender, and decides on its own what to do, usually lowering the agent's limits. Receipts go into an append-only log, so nobody can quietly rewrite history. If a signal is wrong, the agent's owner can appeal, and the retraction travels the same way the signal did.
Spec, JSON schemas, a reference implementation and a conformance test: https://github.com/mrsergeiluzhnov/xonta/tree/main/spec/ocx
It's draft 0.2. Things will break before 0.4, and that's the point of publishing now. Holes found in the comments are cheaper than holes found in production.
Run the benchmark on your model
I've only tested GigaChat-2 so far. A comparison across several models is coming soon, and I'll post it here.
If you don't want to wait for my report, run it yourself. It takes a few minutes and about 120 calls per model.
You need Node 20 or newer and git. No packages to install.
git clone https://github.com/mrsergeiluzhnov/xonta.git cd xonta A local model through Ollama (free):
ollama pull llama3.1:8b LLM_PROVIDER=openai LLM_BASE_URL=http://localhost:11434/v1 LLM_API_KEY=ollama \ BENCH_MODEL=llama3.1:8b node bench/honesty/run.mjs OpenAI, OpenRouter or any OpenAI compatible API:
LLM_PROVIDER=openai LLM_BASE_URL=https://api.openai.com/v1 LLM_API_KEY=your_key \ BENCH_MODEL=gpt-4o-mini node bench/honesty/run.mjs Anthropic:
LLM_PROVIDER=anthropic LLM_API_KEY=your_key \ BENCH_MODEL=claude-model-name node bench/honesty/run.mjs The result lands in bench/honesty/results/<model>.json with every task and every answer, so any number can be checked by hand. node bench/honesty/board.mjs rebuilds the results table.
Please share what you get. Post your numbers in the comments or open a pull request with the result file. Your experience really matters to me, especially if your model makes my point look weak. I'll add every model to the table.
Two questions for you
- Have you seen an agent misreport what it did in production? Was it pressure from a prompt, or did it just make things up?
- If you can't trust the report, what would you accept as proof that an agent actually did the check?
I'm genuinely not sure about the second one, and I'd rather hear your answer before 0.4 locks anything in.
