This week, Amazon Web Services released AWS Emulator, a local copy of AWS that coding agents can test against. It covers more than 200 AWS services and was written entirely by agents. The most useful part of the launch is a story Swami Sivasubramanian tells in the announcement.
About 20 years ago, Sivasubramanian helped build the replication engine inside Alf, a Paxos-based lock service that still powers AWS systems such as S3 and DynamoDB. The first build took about four months. When he later asked Kiro, AWS’s agentic coding tool, to rebuild it, he supervised the agent for four nights before he saw the problem: The agent had no way to check its own results. Once it had a spec and a test environment, the rebuild took about two hours.
Same agent, same task. Four nights became two hours because the feedback loop changed.
“An agent is only as good as its verification loop, and a verification loop is only as good as what it runs against.”
As Sivasubramanian puts it, distributed systems are hard to test. That matches what I see across teams running coding agents on production systems. An agent is only as good as its verification loop, and a verification loop is only as good as what it runs against. AWS Emulator solves that for the AWS APIs your code calls. It does not solve it for the services your team owns, and that is where most agent-written changes break.
AWS puts infrastructure inside the agent’s loop
A coding agent works in a loop. It writes code, runs it, reads the result, and decides what to change next. Sivasubramanian’s point is that in distributed systems, that loop runs inside infrastructure. If the services the code calls do not behave like the real ones, the agent learns the wrong lesson and keeps going.
AWS Emulator puts AWS into that loop on a laptop. Point existing AWS tools and code at it with one environment variable, and tests run against emulated services without changes.
Autonomous agents wrote every line of the emulator, working from AWS’s public API models. Each behavior is pinned by a fidelity test that first passes against real AWS, then against the emulator, and every change passes a merge gate where security agents review it. Sivasubramanian’s conclusion is that when agents write the code, the engineer’s job is to build the specs, tests, and merge gates that decide when the work is correct.
The first reactions on Hacker News went straight to trust. One commenter counted roughly 30,000 commits in three weeks and questioned the quality. That is a fair question aimed at the wrong thing. Who wrote the code tells you little. What it was verified against tells you a lot, and AWS’s answer is real AWS.
“Slow or unrealistic feedback was a tax on people. For agents, it decides whether the loop works at all.”
What changed is who runs the loop. A developer checks a change a few times a day, and an agent checks it on every iteration. Slow or unrealistic feedback was a tax on people. For agents, it decides whether the loop works at all.
Every loop runs against mocks, an emulator, or the running system
An agent can verify its work against one of three things, and each has a different weakness.
Mocks are the weakest option, and AI has made them cheap. One Hacker News commenter argued that it is now faster to have AI mock your APIs, adding that “AI has been a boon for test coverage.” But an agent that writes its own mocks writes them to match its own assumptions. The tests pass because they were built to pass, so coverage goes up while confidence should not.
Emulators are a real step up. AWS Emulator is checked against real AWS, so its behavior comes from the service rather than from the agent.
An emulator is still a model, though, and AWS is open about that. The repository keeps a folder of behaviors the agents found that do not match real AWS yet, and operations that are not implemented return an error instead of a fake success. An IDC analyst noted that at launch the emulator does not simulate throttling, outages, or per-Region availability, which is where retry and failover logic gets exercised.
“An agent that writes its own mocks writes them to match its own assumptions. The tests pass because they were built to pass, so coverage goes up while confidence should not.”
Sivasubramanian says as much about his own work on Alf. Simulation ironed out most of its concurrency bugs before it ran on real servers. Testing on real servers then turned up different problems, because “the real world exposes things a simulation cannot.”
You can’t emulate your own system the way AWS emulated its own
In the same HN thread, a self-described Amazonian outlined the approach most teams take with agents today: Close off as much of the system as possible, mock or simulate the dependencies that cannot be closed off, then let the agent loose. That works when the part you cannot close off is a cloud API with a published spec. It breaks down when that part is your own system.
AWS Emulator works because AWS had two things most teams will never have for their own software: a complete, machine-readable spec of every operation and a stable reference to check against in real AWS.
Now picture a payments platform with 60 microservices. The contracts between those services live in code, in a few OpenAPI files that drifted months ago and in the heads of the engineers who wrote them. There is no single spec, and the reference implementation is the running system itself, which changes every time someone merges. With agents opening pull requests (PRs) multiple times a day, it changes faster than any model of it could keep up with.
That is also where agent-written application code tends to break. The failures are rarely about how a managed queue handles a visibility timeout. They are a field renamed in one service that a consumer two teams away still reads, an event payload a downstream service does not expect, a query that is fast on fixtures and slow on production-sized data, or configuration, feature flags and secrets that only exist in the cluster. No emulator of a cloud provider will ever see them.
The environment belongs on the engineer’s list
Sivasubramanian lists specs, tests, and merge gates as the engineer’s job now. I would add the environment, because it decides what the agent can catch.
A realistic loop has six properties. The change runs in a production-like environment, next to the services, data, and configuration it will ship into. Each change gets its own isolated environment, so 10 agents working in parallel do not step on each other. The environment starts in seconds, because an agent that waits 20 minutes stops using it and the loop collapses back to unit tests. And the agent creates the environment and reads results itself, without filing a ticket.
The loop also runs the integration and end-to-end suites the team already trusts, not tests the agent wrote to agree with itself. And it produces evidence a reviewer or merge gate can act on before the change merges.
The hard part is getting production-like fidelity and fast startup at the same time. Cloning a full stack for every change buys fidelity and loses on speed and cost. A team of 10 engineers running multiple parallel agent sessions might produce 100 PRs a day or more that need to be verified. No team can justify the cost of 100+ full copies of the system.
Emulate what you don’t own, and run what you own for real
Emulators and production-like environments are not competing ideas. They are layers in the same loop.
For managed cloud services with published contracts, an emulator gives agents fast, free feedback on a laptop. If a service writes to S3 and reads from a queue, AWS Emulator catches API misuse in seconds without touching an AWS account.
“Emulators and production-like environments are not competing ideas. They are layers in the same loop.”
For the system you own, deploy only what the change modifies instead of cloning the stack. Requests and messages tagged for that change’s environment reach the new versions, and everything else is served by the shared stable environment that is already running. Databases and queues get lightweight per-environment schemas or topics only where the change needs data isolation. That is how an environment starts in seconds and still behaves like production. Uber and Lyft built internal systems on this model for the same reason.
Then make the merge gate ask for both. A PR that passed against an emulator and against the system it is joining is one a reviewer can trust.
Every agent will have a loop, and realism is what separates them
Sivasubramanian’s agent went from four nights to two hours once it could check its own work. For application teams, the next jump depends on what it checks against. An emulator covers the cloud APIs you call, and only the running system covers the services you own.
Having a verification loop is now expected. The realism of that loop is what decides how much you can trust the code that comes out of it. If you want your agents verifying every change in a production-like environment before merge, we built Signadot to make that the default.
The post AWS Emulator proves agents need a verification loop. Realism is the hard part. appeared first on The New Stack.
