Scraper or agent? Why the answer has to be checkable offline, per request, with RFC 9421

Every site operator now has the same log file. Requests that say they are GPTBot, ClaudeBot or some-shopping-agent/1.0, from cloud IP ranges, at rates that range from polite to hostile. Some of them are scrapers wearing a borrowed name. Some are real agents doing a job a real person asked for: booking a table, comparing prices, filing a support ticket.

The site has to decide in a few milliseconds, on every request, which is which. And "block all bots" stopped being an answer the day users started sending agents to do things for them.

This post goes through why the usual signals fail, what a signal has to look like to be worth anything, and what we learned building one. We ended up with three properties we think are non-negotiable:

1. A per-request cryptographic signature (RFC 9421, HTTP Message Signatures), not a token.
2. Verification that works offline, with no call to anyone per request.
3. A registry that never sees the traffic it vouches for.

Our implementation is called AIDR. It is open source on the verifier side, and you can watch the whole handshake run in your browser at aidr.evolivcore.com/demo (it runs locally with WebCrypto; nothing is sent anywhere). But the argument holds for any design, so let's start with the argument.

Act I: Why the usual signals fail

[ Traditional Security Signals ] | "Trust the network." | ----------------- ▼ ▼ User-Agent IP Allowlist Easily faked Fails on shared cloud 

User-Agent strings are self-reported. Anyone can type GPTBot. This is not a theoretical weakness; it is the first thing a scraper changes.

IP allowlists and reverse DNS prove which network a request came from, not who sent it. They work for a handful of big crawlers with dedicated ranges. They do not work for the long tail of agents that run on shared clouds, serverless platforms and laptops. One abusive tenant on a shared range poisons it for everyone.

API keys and bearer tokens prove that someone once had the key. A bearer token copied from a log works for whoever holds it, until it expires. And a key is issued per site, so an agent visiting a thousand sites needs a thousand sign-ups.

What all of these lack is the same thing: proof of possession of a key, bound to an identity someone is accountable for, checked on the request itself.

Act II: Property 1 - Sign every request (RFC 9421)

HTTP Message Signatures (RFC 9421) let a client sign selected parts of a request. The verifier rebuilds the same "signature base" from what it received and checks the signature.

A signed request in the IETF Web Bot Auth profile looks like this:

GET /api/products HTTP/1.1 Host: shop.example Signature-Agent: "https://agents.acme.example" Signature-Input: wba=("@authority" "@method" "@path" "signature-agent");created=1761480000;expires=1761480300;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";nonce="…";tag="web-bot-auth" Signature: wba=:dGhpcyBpcyBub3QgYSByZWFsIHNpZ25hdHVyZQ…:

What this buys you, compared with a token:

• 🔒 Bound to this request. The signature covers the host, method and path. Replaying it against another endpoint fails.
• ⏱️ Bound in time. created and expires give a short window (we use five minutes).
• 🛡️ Nothing secret travels. The private key never leaves the agent. A captured request is useless on any other request.

Act III: Property 2 - Verify Offline

The obvious way to add identity is an online check: on every request, ask some registry "is this agent good?". It is also the wrong way due to Latency, Cost, and Privacy.

So in AIDR, the agent carries its identity with it. The verifier does everything locally. The checks run cheapest-first, so a garbage request is rejected before any cryptography runs.

In Python, the whole integration is beautifully simple:

from fastapi import FastAPI, Request from aidr_verifier import AidrMiddleware, RelyingPartyConfig app = FastAPI() app.add_middleware(AidrMiddleware, config=RelyingPartyConfig( issuer="https://api.evolivcore.com", public_origin="https://shop.example")) @app.get("/products") async def products(request: Request): # "anonymous", "verified", or refused before this line identity = request.state.aidr return {"status": identity.status, "aid": identity.aid}

(Available for Flask, Django, Express, Go, Cloudflare Workers, Kong, and WordPress).

The Accountability Layer: A Quick Comparison

Web Bot Auth answers which key signed this. Bot lists answer which company runs this crawler. Neither answers the question a bank or API provider actually has: which accountable organisation is behind this specific agent, and can it be switched off right now?

Here is how the layer AIDR adds on top of Web Bot Auth stacks up side-by-side:

Verification is free and always will be; the code that runs on your servers is Apache-2.0.

Try it yourself

• See the handshake: aidr.evolivcore.com/demo. Keys are generated and checked in your browser.
• Read the code: github.com/EvolivCore/aiaf-sdk: verifiers, agent SDKs, and the shared test vectors.

If you run a site that is drowning in "agents", or you build agents and are tired of getting blocked as a scraper, I would like to hear what breaks for you. The comments are open.