Most teams switch model providers on vibes, then discover the new tier breaks their parser at 3 a.m.
This case study walks through one small nightly job, the three-file harness I built before moving it onto a free model tier, and the decision table that turned an argument into a gate. The artifact, not the tier, is the point: once you can measure contract cleanliness and tokens per accepted output, swapping backends stops being a leap of faith.
The short version
- The job is a dependency-advisory summarizer that runs every 30 minutes over a handful of repos.
- The harness runs identical canned
npm auditpayloads through two interchangeable backends and diffs the results. - The gate is binary: any payload that fails the output contract fails the whole run, and CI exits non-zero.
- The useful metric is tokens per accepted summary, because rejected output costs you twice.
- Model IDs and base URLs live in environment variables on purpose, so nothing here goes stale when a free model list rotates.
Background: a nightly job nobody wanted to babysit
Our release manager wanted a short human-readable line per advisory instead of a wall of npm audit --json output. The job itself is boring: read the audit JSON, ask a model for a five-key summary object, append it to a digest, and post the digest to a chat channel.
It ran fine on a paid key for a few weeks, which is exactly when the interesting question shows up. Could this run on a free tier instead, and what evidence would justify the move? I did not want the answer to be "it looked okay in a notebook for ten minutes."
One candidate for that move is MonkeyCode, which offers free model access and a free server option as operator-supplied availability at the time of writing.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
I am treating both availability claims as things to verify myself rather than as guarantees, and I deliberately keep every endpoint, model ID, and quota detail in configuration instead of prose. Free tiers change their model lists and limits regularly, so any hardcoded identifier in a blog post is a bug waiting to happen.
Goal and non-goals
The goal was narrow: decide, with a reproducible test, whether a given backend can serve this specific job without a human cleaning up its output. The test had to run in under a minute and cost effectively nothing to re-run while I iterated on the prompt.
Non-goals matter just as much. I was not benchmarking general model quality, not ranking providers, and not claiming any number generalizes to your workload. A pass rate measured once, on one payload set, ages badly, so this article publishes the gate rather than a score.
The artifact: three files and one command
The whole harness is adapter.py, cases.json, and parity.py. I wrote the HTTP client on the standard library so the harness installs nothing and can run in a bare CI container.
File 1 — adapter.py
One protocol, two interchangeable implementations, and a factory that reads credentials from the environment.
# adapter.py — one narrow interface, two interchangeable backends. from __future__ import annotations import json import os import time import urllib.error import urllib.request from dataclasses import dataclass from typing import Protocol @dataclass class Result: text: str input_tokens: int output_tokens: int latency_ms: int retries: int = 0 class Summarizer(Protocol): name: str def summarize(self, system: str, payload: dict) -> Result: ... class ChatCompletions: '''Minimal OpenAI-compatible client. base_url, model and key all come from env.''' def __init__(self, name: str, base_url: str, model: str, api_key: str, json_mode: bool = True, timeout: int = 45, max_retries: int = 2): self.name = name self.url = base_url.rstrip('/') + '/chat/completions' self.model = model self.api_key = api_key self.json_mode = json_mode self.timeout = timeout self.max_retries = max_retries def _post(self, body: dict) -> dict: req = urllib.request.Request( self.url, data=json.dumps(body).encode(), headers={'Content-Type': 'application/json', 'Authorization': 'Bearer ' + self.api_key}, ) with urllib.request.urlopen(req, timeout=self.timeout) as resp: return json.loads(resp.read()) def summarize(self, system: str, payload: dict) -> Result: body = { 'model': self.model, 'temperature': 0, 'messages': [ {'role': 'system', 'content': system}, {'role': 'user', 'content': json.dumps(payload)}, ], } if self.json_mode: body['response_format'] = {'type': 'json_object'} started = time.monotonic() retries = 0 while True: try: data = self._post(body) break except urllib.error.HTTPError as err: # A hard cap, not infinite backoff. Your retry policy still applies. if err.code not in (429, 500, 502, 503) or retries >= self.max_retries: raise retries += 1 time.sleep(min(2 ** retries, 8)) usage = data.get('usage') or {} return Result( text=data['choices'][0]['message']['content'], input_tokens=int(usage.get('prompt_tokens', 0)), output_tokens=int(usage.get('completion_tokens', 0)), latency_ms=int((time.monotonic() - started) * 1000), retries=retries, ) def from_env(prefix: str, name: str) -> ChatCompletions: '''Build a backend from MC_BASE_URL / MC_MODEL / MC_API_KEY style vars.''' return ChatCompletions( name=name, base_url=os.environ[prefix + '_BASE_URL'], model=os.environ[prefix + '_MODEL'], api_key=os.environ[prefix + '_API_KEY'], json_mode=os.environ.get(prefix + '_JSON_MODE', '1') == '1', ) Two details earn their keep. The json_mode flag exists because not every backend honours response_format, and silently ignoring it is how you get prose instead of JSON. The retries field is recorded per call, because a rising retry count is the cheapest early warning that a tier is under load.
File 2 — cases.json
A dozen trimmed advisory payloads that cover the shapes you actually see: a critical with a fix, a moderate without one, a package with a long via chain, and one advisory with odd casing in the package name. Keep them small, commit them, and never let the harness fetch live data.
[ { "id": "case-01", "audit": { "name": "left-pad", "severity": "high", "via": [{"title": "Prototype Pollution", "cvss": {"score": 7.5}}], "fixAvailable": {"name": "left-pad", "version": "1.3.1"} } }, { "id": "case-02", "audit": { "name": "EigenDependency", "severity": "moderate", "via": [{"title": "Inefficient Regex"}], "fixAvailable": false } } ] File 3 — parity.py
The runner validates the output contract, computes tokens per accepted summary, and exits non-zero when anything is unclean.
# parity.py — run identical golden payloads through both backends and gate on the diff. import json import statistics import sys from adapter import from_env SYSTEM = '''You summarize a dependency advisory for a release manager. Return ONE JSON object with exactly these keys: repo, package, severity, summary, action. severity must be one of: low, moderate, high, critical. summary: one sentence, 30 words max. action: imperative, 15 words max. If fixAvailable is false or missing, action must start with "no fix available".''' REQUIRED = ('repo', 'package', 'severity', 'summary', 'action') SEVERITIES = {'low', 'moderate', 'high', 'critical'} def validate(text: str) -> list[str]: '''Return contract violations. An empty list means the output is usable.''' try: obj = json.loads(text) except json.JSONDecodeError: return ['not valid json'] if not isinstance(obj, dict): return ['not a json object'] problems = [f'missing key {k}' for k in REQUIRED if k not in obj] if obj.get('severity') not in SEVERITIES: problems.append(f'bad severity {obj.get("severity")!r}') if len(str(obj.get('summary', '')).split()) > 30: problems.append('summary too long') return problems def run(backend, cases): rows = [] for case in cases: res = backend.summarize(SYSTEM, case['audit']) rows.append({'id': case['id'], 'problems': validate(res.text), 'tokens': res.input_tokens + res.output_tokens, 'latency_ms': res.latency_ms, 'retries': res.retries}) return rows def report(name, rows): accepted = [r for r in rows if not r['problems']] tokens = sum(r['tokens'] for r in rows) per_ok = tokens / len(accepted) if accepted else float('inf') latency = statistics.median(r['latency_ms'] for r in rows) print(f'\n{name}: {len(accepted)}/{len(rows)} contract-clean') print(f' median latency {latency:.0f} ms') print(f' tokens total {tokens}') print(f' tokens/accepted {per_ok:.0f}') for row in rows: if row['problems']: print(f' FAIL {row["id"]}: ' + '; '.join(row['problems'])) return len(accepted) == len(rows) if __name__ == '__main__': cases = json.load(open(sys.argv[1] if len(sys.argv) > 1 else 'cases.json')) ok = True for prefix in ('PAID', 'MC'): try: backend = from_env(prefix, prefix.lower()) except KeyError as missing: print(f'skipping {prefix}: {missing} not set') continue ok &= report(prefix, run(backend, cases)) sys.exit(0 if ok else 1) The command is deliberately dull, and that is what makes it usable in CI:
export PAID_BASE_URL=... PAID_MODEL=... PAID_API_KEY=... export MC_BASE_URL=... MC_MODEL=... MC_API_KEY=... python3 parity.py cases.json echo "exit=$?" Where the free tier and the free server actually helped
Prompt iteration is the token sink nobody budgets for, and that is precisely where free model access pays off. Each pass over a twelve-case golden set costs almost nothing in wall-clock time, so you can tighten the system prompt, add a failing case, and re-run twenty times in an afternoon without watching a spend graph.
The free server option fits the other half of the problem: this job needs a small always-on box running cron, not a managed platform. Putting the digest runner on a free server keeps the paid credential out of the loop for development passes, and the same adapter.py works unchanged in both places.
Before you plan around any of this, read the current terms yourself. Free tiers adjust quotas, model lists, and acceptable-use rules, and the number someone quotes you in a blog post is not the number you will see on the pricing page.
Reading the report: a decision table, not a vibe
The report gives you four numbers per backend, and the thresholds below are the ones I use. They are decisions, not measurements, so treat them as a starting point you will argue with.
| Signal | Threshold I set | What it means | Action |
|---|---|---|---|
| Contract-clean rate | 100% of golden cases | Any rejection is a human's problem later | Block the move, fix prompt or parser |
| Median latency | Under 8 s | Cron every 30 min tolerates slow calls | Raise the timeout, not the retry count |
| Tokens per accepted summary | Within 1.5× of current | Overspend hides in retries and rejects | Compare on accepted output only |
| Retry rate | Under 5% of calls | Capacity pressure shows here first | Re-check before scheduling a rollout |
The single most useful column is tokens per accepted summary, because it prices failure. A backend that answers in half the tokens but fails one case in five costs you more than the cheap-looking number suggests, and the harness makes that visible on the first run.
Running it on the free server without babysitting it
A digest job on a small box needs exactly two guards: a lock so overlapping cron ticks cannot stack, and a log you can tail after the fact. Thirty seconds of shell is enough for both.
*/30 * * * * cd /srv/advisory-digest && \ flock -n .run.lock python3 parity.py cases.json >> logs/run.log 2>&1 If you prefer systemd, a timer plus Requires= on the service gets you the same exclusivity with better journald output. Either way, keep the golden set in the repo and re-run the gate after every prompt change, because the prompt is code and it deserves the same regression protection.
Limitations, and who should skip this
This harness proves contract compatibility on a small fixed set, and it proves nothing about the long tail of real advisories. It also says nothing about data handling, so check how any provider treats your payloads before you send internal package names to it.
Skip this approach if your job is latency-sensitive, contractually tied to a specific model version for reproducibility, or subject to rules that require capacity guarantees. It is also the wrong tool for anyone unwilling to maintain a validation gate, because a parity harness with no owner decays into a script nobody runs.
Finally, do not read a passing run as a permanent verdict. Backends change underneath you, and the whole point of keeping model IDs in environment variables is that next month's swap is a config diff, not a rewrite.
Lessons learned
- Contract validation beats eyeballing output, and it costs about forty lines of Python.
- Price the tier per accepted output, not per request, or you will undercount the expensive failures.
- Keep credentials and identifiers out of the source, so a rotated quota never becomes an outage.
- Run the gate before the migration, not after, and make CI fail loudly when it regresses.
If you want to try the harness against MonkeyCode's free model access and free server option, fill in the environment variables above and let the gate tell you whether that tier fits your job before you trust it with a schedule.
