Raindrop wants developers to know what else will change when they edit an agent’s prompt, swap its model, or give it a new tool.

The company, which started out as a monitoring startup for agents (“Sentry for agents” was its early motto), is making its Simulations platform available in public beta on Thursday.

An agent’s prompt can depend on feature flags, account details, and the user’s plan, says Ben Hylak, Raindrop’s co-founder and CTO.

“You have to put the agent in motion in order to really evaluate it,” he says.

With Simulations, Raindrop reconstructs an agent’s production environment, including its tools, APIs, and databases, so developers can see how a change affects its behavior before shipping it, the company says. That change could be a model swap, a prompt edit, or a new tool — any of which could change an agent’s behavior in production.

“Every time agents run, they go different paths.” — Raindrop CTO Ben Hylak.

Hylak argues that predefined tests and evals can miss failures developers didn’t anticipate, especially as agents take on more consequential work.

“Every time agents run, they go different paths,” Hylak says.

Rebuilding the environment

Raindrop describes its simulation environment as a set of stateful synthetic services that the customer’s real agent code runs against. It uses production traces to reconstruct the services the agent interacts with, filling gaps with generated data.

The agent runs its real prompts, models, and agent loop in an isolated sandbox without access to production services or credentials, according to Raindrop. For each tool call, Raindrop generates a response grounded in production traces, validates it against the types the code expects, and keeps the simulated environment consistent throughout the run.

Running the test against live services wouldn’t recreate the original conditions, either, Hylak says.

“The state of production has changed since you were running that agent,” he says.

Hylak says Raindrop sets up the agent harness and test environment automatically, using sandboxes both to build and run the simulations. The agent under test may also create sandboxes of its own.

“It’s like sandboxes all the way down,” he says.

Hylak says Raindrop runs its production issue detection over the simulations to find failures and unexpected changes in behavior. The company is looking for failures a developer didn’t anticipate, as well as checking the scenarios the developer already wrote down.

Filling in missing data

Simulating the environment is one thing, but the agents also need data to work with. A prompt edit or model swap can reuse context from earlier runs, Hylak says. Adding a Salesforce or HubSpot tool is harder because Raindrop may never have seen the data that it returns.

Raindrop can generate synthetic data to fill in the holes, but Hylak says, “We know that that will be a less faithful simulation than the real world.”

To give its users confidence in those simulations, Raindrop gives them a simulation fidelity score and flags what’s lowering it, Hylak says. The score estimates how closely the test reflects production. Customers can then add a file or connect a data source through the Model Context Protocol and rerun the test under more realistic conditions.

A missing feature-flag value can also leave the simulator without information that shaped the original prompt. Hylak says Raindrop can suggest pull requests that improve the agent’s telemetry so that context gets recorded.

Models can often pick up on inconsistencies in synthetic data and realize they’re being tested, Hylak says. That can change their behavior during the test. The team tries to measure this testing awareness as part of its simulations.

To account for normal variation between runs, Raindrop also runs multiple simulations and compares the findings with production history, Hylak says.

Testing different agents

OpenAI already uses production conversations to test candidate models, including coding agents whose tools are simulated using historical repository state and recorded tool interactions. Sierra, meanwhile, automatically generates simulation suites for agents on its platform using standard operating procedures, knowledge bases, historical coaching transcripts, and conversation flows.

Hylak says Raindrop is building for customers’ own agents, across different frameworks and codebases, with results delivered as part of the pull request. During early access, the team had to get the simulations working with agents embedded in large enterprise codebases.

During early access, customers and Raindrop’s own team ran more than 33,000 simulations across more than 1,700 pull requests, according to the company.

OpenAI’s account of the Hugging Face incident describes agents exploiting interconnected systems during cybersecurity evaluations with reduced safeguards. More recently, it documented internal coding agents weakening monitoring while carrying out software changes.

Raindrop first announced Simulations in September, alongside a Series A that brought its total funding to $50 million.

Pricing will be based on pull requests, Hylak says, with charges for reconstructing the environment and running the simulations. The agent’s own model usage is initially billed through Raindrop as well, so the bill will depend in part on how long the agent runs.

The post Raindrop simulates your agent’s world to catch what your changes break appeared first on The New Stack.