model-radar is two small commands I run for my AI agents:

  • model-radar lists the free models on OpenRouter and actually asks each new one to make a tool call. Loads of free models say they support tools and then answer in prose.
  • model-canary watches the paid model my agents use, and the credit balance, and fails over to a cheaper model or a local one when either runs out.

Neither asks a model for an opinion. They measure and act. Setup is about ten minutes, and the guide is run on a fresh Ubuntu machine every week.

model-radar on a fresh GitHub runner with no API key: Python check, help, a dry run that skips both sources, and the canary refusing with exit 1

Install

Python 3.11 or newer, nothing to pip install.

git clone https://github.com/casareanderson/model-radar.git && cd model-radar cp config.example.toml config.toml ./bin/model-radar --help

Check it fails safely first

Before adding a key, run both with no credentials:

./bin/model-radar --dry-run # both sources "no credential, skipped", empty baseline ./bin/model-canary --dry-run # "no OPENROUTER_API_KEY -- nothing to check", exit 1

That exit 1 matters. If the canary can't check anything, a timer should notice, not shrug.

The real run 

export OPENROUTER_API_KEY=sk-or-... ./bin/model-radar --dry-run

One line per model: PASS, FAIL or BUSY. BUSY means rate-limited or timed out, and it's never treated as a verdict, it just gets tried next run. On 8 October that gave me 4 passes out of 8 probed, two 403s and two busy. One free model I'd tried earlier answered every request and never once called the tool.

Run it once without --dry-run to save a baseline. From then on it only probes models it hasn't seen.

The canary

Point it at whatever your agents use (a JSON file, or hermes-agent profiles), add a paid fallback ladder and a local model in config.toml, and run it every few hours from cron. Two hard failures in a row and it walks the ladder. Timeouts and 429s never count.

What CI can't prove

Anything with a key. The weekly run has none, so it proves install, config, help and the no-key behaviour. The probes and the canary's failover need your OpenRouter account.

Full guide: github.com/casareanderson/model-radar/blob/main/docs/SETUP.md

How do you pick models for your agents, and how do you find out when one quietly stops working?

🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.