I let Claude, GPT and Gemini grade each other's AWS architectures

CloudArchitect Lite turns a plain-English description of an app into an AWS architecture diagram, then checks it with 25 deterministic rules. Version 0.3 lets you bring a key from Anthropic, OpenAI or Google. Before I let users pick a model, I wanted numbers, so I benchmarked six models on the same tasks and had a panel of three AI reviewers, one from each provider, score the results blind.

This post covers how the review works, what it found and where the evidence is thin. The full report with every number is in the repository.

The contestants

Two models per provider, matched by price rather than by "flagship" status:

Every model runs through the same pipeline: a requirements step on the provider's economical model, one clarification round, then a design constrained to a catalog of 38 AWS services by structured output.

Step 1: automated checks

Each model generated designs for 22 scenarios, from a blog to a regulated portal (economical models twice per scenario, best-quality models once on the 15 base scenarios). A design passes when it contains the services the scenario needs and no error-level rule fires. Every output, from every model, matched the schema.

Step 2: a blind panel of three providers

A model grading its own provider's work is the obvious bias, so the panel has one reviewer from each provider: Claude Sonnet 5.5, GPT-6.1 Sol and Gemini 4 Argon.

• Designs are rendered into one neutral text format with renumbered ids and no model or provider names.
• Each design is paired with the same scenario's design from the same-tier Claude model, in random A/B order.
• Reviewers score both designs on a 7-dimension rubric (0–3), list defects with AWS documentation links, and pick a preference.
• A design's published score is the median of the three reviewers.

63 designs, 51 pairs, 17 packets.

Results

* At Gemini 4 Argon's price from 1 December; it costs half that until then.

Head to head:

• GPT-6.1 Sol vs Claude Sonnet: Sol 9 of 9. The OpenAI and Gemini reviewers both said 9–0; the Claude reviewer still preferred Sol, 4–3 with one tie.
• Sonnet vs Argon: Sonnet 5–4. Sonnet vs Haiku: Sonnet 6–3.
• Haiku vs Luna: Haiku 11–1. Haiku vs Flash-Lite: Haiku 12–0.

So the best-reviewed designs came from GPT-6.1 Sol, but Claude Sonnet was the only best-quality model that passed every automated check, and it was twice as fast. In the cheap tier Claude Haiku was clearly ahead.

Which rules each model breaks

One overall score hides how a model goes wrong. A reader on dev.to suggested publishing rule hits per model, and the eval already stored them. Here are the warnings that fired most, as "runs with the finding / runs":

Each model fails in its own way:

• Sonnet passed every check, yet in 8 of 15 designs it left a requirement without any node or edge for it. No other model does that.
• Sol got the best blind score, yet in 6 of 15 designs it put a public API in front without saying how callers authenticate. The reviewers liked the designs overall; the rules caught the gap.
• Luna almost never says how data is encrypted at rest (32 of 44).
• Argon left the fewest findings of all.

No final design from any model has an error-level finding. The cheap models get one repair turn that sends the findings back to the model, and it ran in 34–38 of their 44 runs. I only logged findings after that turn, so the next benchmark will also show what each model gets wrong in its first draft.

Who gets into the app

I set a listing bar: at least 85% of designs passing the checks, at least 85% without a critical defect, and a blind score of at least 2.0. Four models meet it and are offered in the app: Sol, Sonnet, Haiku and Argon. GPT-6 Luna and Gemini 3.5 Flash-Lite still work if you enable them by configuration, but the editor doesn't offer them.

Where the evidence is thin

• Small samples. 9–12 reviewed designs per model. One design moves a rate by several points.
• Own-provider lean. On the pair closest to its own provider, each reviewer leaned slightly toward it (the Gemini reviewer preferred Argon over Sonnet 5–4; the other two preferred Sonnet). On other pairs the lean disappears.
• An incomplete reviewer. In three packets the Claude reviewer answered only one scenario of three, even after a retry, so 17 of 63 designs have a two-reviewer median.
• Model reviewers. They saw no AWS documentation during review and are not a substitute for a human architect.

Things that broke along the way

The adapters were the easy part to write and the hard part to get right, because "structured output" means something different for each provider. OpenAI's strict mode rejected a description next to a $ref. Anthropic rejected minimum/maximum on integers, so the 0–3 score range moved into my own validation. And the Claude reviewer sometimes returned perfectly valid JSON that reviewed one design twice and another not at all. The fix in every case was the same: one source schema, converted per provider, and every answer re-validated in code.

Cost

Smoke tests, six benchmark runs and the review panel together cost about $12 in API calls across the three providers, including retries.

Try it

You need Docker and an API key from Anthropic, OpenAI or Google:

curl -fsSLO https://raw.githubusercontent.com/aliaksandr-liapin/cloudarchitect-lite/v0.3.0/compose.yaml && docker compose up 

There's also a clickable demo with no install. If you run your own prompts and a model gets an architecture wrong, please open an issue with the prompt. Those cases go straight into the next benchmark.