Last week one API token expired. Nothing crashed. Every scheduled job started on time, checked the token, got a 401, wrote a polite "skipped, credential invalid" and finished green. Four days later I counted more than 40 runs that had done nothing, and three daily data sources with a four-day hole that cannot be backfilled.

My thesis after this week: for agent pipelines, the dangerous failures are not the crashes. They are the jobs that stay green while doing nothing. Most "agent reliability" advice is about retries, timeouts and hallucinations. The thing that actually stopped us was a credential, and the design that made it worse was ours.

Disclosure: I build data tools for AI agents as Fast Data on Apify. The expired token was our own Apify token; this is not a complaint about Apify, which behaved exactly as it should. It is about how I wired things.

The setup

A handful of scheduled agent routines, each a prompt plus some scripts:

Routine Runs per day Goes through
Build and publish a new data tool 5 the platform API (one token)
Research + post news to X and Bluesky 5 the platform (research proxy and posting actor)
Daily data archive (3 public sources) 1 the platform (fetch proxy)
Write and test an agent skill 1 the platform (test runs)
Engagement shortlist 3 the platform (search actor)
Publish a dev.to article 3 dev.to API directly

Five of six routines depended on the same token, some for obvious reasons (building tools on the platform), some only out of convenience (I routed web fetches and social posting through it because it was already there).

What happened

Kept working Silently stopped
dev.to articles yes, separate key, direct API
X posting of videos yes, after I switched to the official X API directly
Engagement shortlist partly, after I switched its X search to the official API Bluesky part
News posts to X/Bluesky yes, research ran through the platform
Data archive 1 of 4 sources (the one fetched directly) 3 sources, 4 days lost
Tool building, skill testing yes

Two patterns are obvious in hindsight:

  1. Everything that talked to its service directly survived. Everything that took a detour through a shared platform credential died with it.
  2. The "skip politely" logic hid the outage. Each run did the responsible thing in isolation: no token, no spending, no half-finished work, a short status line. Twenty identical status lines a day are not an alert; they are wallpaper.

Four rules I now apply

1. One integration, one credential, as direct as possible. If a job needs to fetch a public RSS feed, it should fetch the feed, not ask a platform to fetch it. Detours are fine for things only the platform can do. We rewrote the archive to try every source directly first and only fall back to the platform for the rest.

remaining = {} for name, url in SOURCES.items(): try: resp = client.get(url, headers={"User-Agent": "FastDataArchive/1.0"}, timeout=60) resp.raise_for_status() save(name, parse(name, resp.text)) except Exception: remaining[name] = url # only these go through the platform fallback 

2. A credential check is a health check, not a precondition. Run it on its own schedule, record the result, and alert on the transition from OK to failing, once. Not inside every job, where it turns into 40 skips.

def check_credentials(state): ok = requests.get("https://api.example.com/v2/users/me", timeout=20).status_code == 200 if state.get("ok", True) and not ok: alert("Platform token stopped working - dependent jobs are paused") # once if not state.get("ok", True) and ok: alert("Platform token works again") state["ok"] = ok return state 

3. A skipped run is a failed run for monitoring purposes. Let the job skip safely, but count it as red on the dashboard. "Did nothing for a good reason" and "did its job" must not look the same.

4. Data you cannot backfill gets a second path. Content and tools can wait a few days. A daily snapshot of a feed that only shows the last few weeks cannot. Those jobs get a fallback path and the strictest alerting.

What I would argue with myself about

  • "Just renew the token faster." Yes, and that is the fix for this incident. But a human renewing a token is the same single point of failure with a delay, and it will happen again with a different credential.
  • "Shared platforms reduce complexity." They do, until the shared part fails. I still think routing everything through one platform was the right call to start; it was the wrong call to keep once the setup grew.
  • "Agents should just handle this." An agent that notices a 401 and stops is behaving correctly. Deciding that 20 identical stops per day are an incident is a system-level judgement. Today that belongs in plain monitoring code, not in each agent's prompt.

Try it

The archive script with the direct-first fallback is in our private repo, but the pattern above is all there is to it. Our public snippets (dev.to stats, X video upload, Eurostat parser, canvas-to-video) are in data-recipes.

Question: how do you monitor scheduled agent jobs that can legitimately decide to do nothing? I have not found a clean way to tell "nothing to do today" from "could not do anything" without making every job report a reason code.