A lot of teams get to the same place with their first production model: a cron job retrains it every week, and every week the new version replaces the old one. It feels like automation. It's actually a coin flip with a deploy button.

Retraining is the safe part. Promotion is where a model gets worse without anyone noticing. Here's the small set of checks we'd put between "a new model exists" and "users are getting its predictions", with code you can lift.

1. Promote against a frozen eval set, not against last week

The new model should earn its place by beating the current one on the same evaluation set. If the eval set changed, the comparison means nothing, and the gate should say so instead of guessing.

def promote(champion: dict, challenger: dict, min_gain: float = 0.005, max_latency_ms: float = 120) -> tuple[bool, str]: """Promote only if the challenger beats the champion on the same frozen eval set and stays inside the latency budget.""" if challenger["eval_set"] != champion["eval_set"]: return False, "different eval sets: not comparable" gain = challenger["auc"] - champion["auc"] if gain < min_gain: return False, f"gain {gain:+.4f} below {min_gain}" if challenger["p95_latency_ms"] > max_latency_ms: return False, f"p95 latency {challenger['p95_latency_ms']} ms over budget" return True, f"promote: gain {gain:+.4f}" champ = {"eval_set": "eval-2026-10", "auc": 0.861, "p95_latency_ms": 85} print(promote(champ, {"eval_set": "eval-2026-10", "auc": 0.864, "p95_latency_ms": 90})) print(promote(champ, {"eval_set": "eval-2026-10", "auc": 0.874, "p95_latency_ms": 95})) print(promote(champ, {"eval_set": "eval-2026-11", "auc": 0.902, "p95_latency_ms": 80})) 
(False, 'gain +0.0030 below 0.005') (True, 'promote: gain +0.0130') (False, 'different eval sets: not comparable')

The third case is the one that bites. A challenger scored on a newer, easier eval set looks like a big win. It isn't evidence of anything.

Two choices matter more than the code:

  • min_gain is a product decision. A gain smaller than your metric's run-to-run noise is noise. Measure that noise once (retrain the same model with different seeds) and set the threshold above it.
  • Latency is part of "better". A model that's 1% more accurate and blows the p95 budget is a regression for the people waiting on it.

2. Watch the inputs, and agree on thresholds before anything pages anyone

Drift alerts are only useful if someone agreed in advance what counts as drift and who gets woken up. A common, simple input check is the Population Stability Index (PSI) per feature:

import numpy as np def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float: """Population Stability Index between a reference sample and a live sample of one feature.""" edges = np.quantile(expected, np.linspace(0, 1, bins + 1)) edges[0], edges[-1] = -np.inf, np.inf e = np.histogram(expected, edges)[0] / len(expected) a = np.histogram(actual, edges)[0] / len(actual) e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None) return float(np.sum((a - e) * np.log(a / e))) rng = np.random.default_rng(7) ref = rng.normal(100, 15, 20_000) print(round(psi(ref, rng.normal(100, 15, 5_000)), 3)) # same distribution print(round(psi(ref, rng.normal(112, 18, 5_000)), 3)) # shifted after a market change 
0.002 0.542

A widely used rule of thumb reads PSI under 0.1 as stable and over 0.25 as a real shift. Treat those as a starting point, not a law: tune them per feature, write them down, and name the owner of each model who receives the alert. An alert that goes to a shared channel goes to nobody.

3. Ship with traffic you can take back

Even a model that passed the gate can fail on live traffic. Two patterns keep that cheap:

  • Shadow: the new model scores real requests, but its predictions aren't used. You compare offline.
  • Canary: a small slice of traffic gets the new model, with an automatic stop if error rates jump.

Both need one release path for every model, usually through a model registry, so "which version is live?" always has a single answer.

4. Rehearse the rollback before you need it

Every release path needs a way back to the last good version, and that way back needs to have been run at least once before a real model depends on it. A rollback nobody has tried is a hope: the old artifact got garbage-collected, the feature schema moved on, or the one person who knew the command is on holiday.

A practical version: in the same week you set up promotion, deploy a deliberately worse model to staging, let the canary stop fire, and roll back. Time it. Write the steps down.

5. Size the platform to the models you have

You don't need a full Kubeflow install to do any of this. With a handful of models, MLflow (or any registry) plus a scheduled pipeline usually covers it. Start by automating the riskiest manual step in how your models ship today, and add pieces only when a release actually needs them.

A checklist to steal

  • [ ] Retraining can run on a schedule; promotion runs through a gate.
  • [ ] The gate compares champion and challenger on the same frozen eval set, with a minimum gain above measured noise and a latency budget.
  • [ ] Drift thresholds are written down per feature, with one named owner per model.
  • [ ] New versions go out as shadow or canary, with an automatic stop.
  • [ ] The rollback has been rehearsed and timed.

How we scope this kind of platform work, what our MLOps engineers are screened on, and the stack that comes up most often are at bromena.com/ai/mlops.

— Bromena team