48-Hour Field Notes: My Readiness Check Returned 200, Then the Batch Hit a Cold Model Server

Why does a service that answers /healthz in a few milliseconds still fail the very first real request? I spent two days on that question, and the answer was less about the model and more about what I had agreed to call "ready".

This is the field-notes version: what I tried, what broke, and the three small scripts I now keep in the repo so the next batch does not repeat the lesson. If your job runner fires requests at a shared endpoint the moment a container reports healthy, you have probably met the same cold-start gap and blamed something else for it.

What I was actually trying to ship

My task was small and boring: take a folder of application log files, group lines by failure signature, and write one short summary per group. A language model is a reasonable fit for the clustering step, so I wired a batch script to an OpenAI-compatible endpoint and ran it on a free server option during off-hours. I used MonkeyCode's free model access for the model side, because the whole thing was a rehearsal for an internal tool that had not been approved yet.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The script did the four obvious things: read files, build prompts, call the endpoint, write JSON. It also did one non-obvious thing wrong — it treated a successful HTTP probe as proof that the model path was warm.

The 48-hour timeline, honestly

• Hours 0–4: I wrote the batch loop and a /healthz check, then declared the worker "production shaped" and went to lunch.
• Hours 4–8: The first real run produced a handful of good summaries and then a wall of timeouts, which I initially blamed on my own concurrency settings.
• Hours 8–20: I lowered concurrency, added retries, and made the failure slower and more expensive without making it rarer.
• Hours 20–30: I logged time-to-first-token per attempt instead of averaging everything, and the pattern finally became visible.
• Hours 30–40: I replaced the probe with one that has to produce an actual token, added a deadline instead of a retry count, and keyed every job so a retry could not duplicate work.
• Hours 40–48: The batch completed, and more importantly, it failed in a way I could explain in one sentence.

Liveness, readiness, and the lie of a 200

There are three different claims hiding inside the word "up". The socket accepts connections, which only means a process exists. The HTTP handler returns 200, which only means the web framework is scheduled. The model can produce a token for the prompt shape you are about to send, which is the only claim your batch actually depends on.

My probe checked the second claim and I acted on the third. That is not a subtle bug; it is a category error, and it explains why every retry at the same moment failed the same way. A cold worker that is already accepting traffic is worse than a closed port, because your own health check keeps telling you the truth you want.

Artifact 1: a readiness probe that has to produce a token

This is the skeleton I now run before a batch starts. It is not a library, and you should adapt the endpoint and model id to your own environment rather than copying my strings.

import os, time, requests BASE_URL = os.environ["LLM_BASE_URL"].rstrip("/") MODEL_ID = os.environ["LLM_MODEL_ID"] def model_is_ready(deadline_s: float = 120.0, max_tokens: int = 1) -> bool: """Readiness = this endpoint returned a token for this model id.""" start = time.monotonic() while time.monotonic() - start < deadline_s: try: r = requests.post( f"{BASE_URL}/chat/completions", json={ "model": MODEL_ID, "messages": [{"role": "user", "content": "ok"}], "max_tokens": max_tokens, "temperature": 0, }, timeout=10, ) if r.status_code == 200: body = r.json() if body.get("choices") and body["choices"][0].get("message"): return True except requests.RequestException: pass time.sleep(1.5) return False 

A /models endpoint is not a substitute here, because a catalogue can answer before the weights behind an entry are servable. The only honest probe is the same request shape your batch will send, with the smallest possible completion budget.

Artifact 2: retry with full jitter and a wall-clock deadline

My first retry policy counted attempts. Counting attempts is a way of hiding time from yourself, because ten attempts with exponential backoff can span minutes or hours depending on where the counter starts. I switched to a deadline, and I added full jitter so a batch of workers does not re-synchronise into a second storm.

import random, time, requests def call_with_deadline(payload: dict, deadline_s: float = 90.0): start = time.monotonic() attempt = 0 while True: try: r = requests.post(f"{BASE_URL}/chat/completions", json=payload, timeout=30) if r.status_code == 200: return r.json() if r.status_code in (408, 429, 500, 502, 503, 504): raise requests.HTTPError(f"retryable {r.status_code}") r.raise_for_status() # 4xx that retries cannot fix  except (requests.RequestException, requests.HTTPError): attempt += 1 remaining = deadline_s - (time.monotonic() - start) if remaining <= 0: raise TimeoutError(f"gave up after {attempt} attempts") # full jitter: sleep uniform in [0, cap], capped by remaining time  cap = min(30.0, 0.5 * (2 ** min(attempt, 6))) time.sleep(random.uniform(0, min(cap, remaining))) 

The important change is not the formula, it is the question the loop answers: "do I still have time to finish this job", not "have I tried enough times to feel thorough".

Artifact 3: idempotency so a retry cannot duplicate work

Retries are only safe if the second attempt cannot create a second side effect. For a summariser this is easy, and for anything that writes rows it is essential. I hash the stable inputs and use that as the output filename, so a repeated attempt overwrites instead of appending.

import hashlib, json, pathlib def job_key(file_bytes: bytes, prompt: str, model_id: str) -> str: h = hashlib.sha256() h.update(file_bytes) h.update(prompt.encode("utf-8")) h.update(model_id.encode("utf-8")) return h.hexdigest()[:16] def write_once(out_dir: str, key: str, payload: dict) -> pathlib.Path: p = pathlib.Path(out_dir) / f"{key}.json" tmp = p.with_suffix(".json.tmp") tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2)) tmp.replace(p) return p 

Note the atomic write via rename. A process that dies mid-write should leave a tmp file, not a half-parsed result that your next run silently trusts.

When shared free capacity is the right tool

Free model access plus a free server option is not a smaller version of a paid production deployment, and treating it as one produces exactly the confusion above. Here is the decision table I now use before starting a batch.

Limitations I would tell a teammate out loud

• I do not control when shared capacity is warm, so every design decision has to assume the first attempt may fail.
• Availability of free model access and the free server option is a current product offering, not a guarantee, and I plan for it changing.
• My readiness probe costs one tiny request per start; on a large fleet that is still traffic you should count.
• A sentinel completion proves the path works once; it does not prove the path stays warm for your entire batch.
• This pattern reduces visible failures, it does not create a latency SLA, and nobody should quote one on my behalf.

Who should skip this approach entirely

If your batch touches customer data, stop and get the data path reviewed before optimising anything. If your job must finish inside a fixed window with contractual consequences, pay for capacity you can reason about instead of building ever more elaborate gates. And if you have never measured time-to-first-token on your own workload, you do not yet know whether the cold start matters, so measure before you engineer.

What I would repeat on Monday

I would write the probe first, before the prompt, because the probe is what tells me whether the prompt ran at all. I would keep the deadline rather than the attempt counter, and I would keep the idempotency key even though it feels like overkill for a script that only writes JSON files. Most of all, I would log per-attempt timing from the beginning instead of averaging it into a number that hides the shape of the failure.

Isn't that always the pattern? The expensive part was never the model call, it was my confident assumption about what a green checkmark meant.

If you want to rehearse this failure mode before it finds you in production, the free model access and free server option are a cheap place to point the readiness gate above and watch what actually happens.