Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run
You check on a Kubernetes CronJob that should run every minute and find that it hasn't created a Job in hours. Somewhere in the controller logs or the CronJob's events there's a message like this:
Cannot determine if job needs to be started: too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew. On newer clusters the same situation shows up as a warning event with the reason TooManyMissedTimes. Either way, the message is telling you exactly what to do, but not why it happened. Here's both.
What the controller is counting
Each time the CronJob controller looks at a CronJob, it works out which scheduled times have passed since the job last ran (.status.lastScheduleTime, or the CronJob's creation time if it has never run). Normally that's zero or one.
If the controller wasn't able to start the job for a while, the count grows. For a job scheduled every minute, two hours of not starting means 120 missed times. Above 100, the controller logs this message.
What happens next depends on your Kubernetes version:
- The original CronJob controller (the default in older clusters, before the rewritten controller became the default around Kubernetes 1.21) stops scheduling the job entirely at that point. It keeps logging the error and creating nothing, until someone intervenes. That's the "my CronJob silently stopped" version of the problem.
- The rewritten controller used by current versions records a
TooManyMissedTimeswarning event and still schedules the most recent missed run, so the job usually recovers. The warning is still worth fixing, because it means runs were missed and something caused it.
Step 1: find out how it got there
kubectl describe cronjob my-job -n my-namespace kubectl get events -n my-namespace --field-selector involvedObject.name=my-job kubectl get cronjob my-job -n my-namespace -o jsonpath='{.status.lastScheduleTime}{"\n"}' Compare lastScheduleTime with the current time. A large gap means something stopped runs from being created. The usual causes:
The CronJob was suspended, then resumed
suspend: true stops new runs. When you set it back to false days later, the controller counts every scheduled time during the suspension as missed. For an every-minute job, a day of suspension is 1,440 missed times.
The controller wasn't running
If kube-controller-manager was down, restarting, or overloaded (for example during a control plane upgrade, or on a struggling self-managed cluster), no CronJob was evaluated. On managed clusters this is rare but not impossible during upgrades.
Clock skew
If the clock on the control plane node jumped, or differs a lot from the time in lastScheduleTime (recorded earlier, possibly by another controller instance), the controller can compute a large number of missed slots. Check that NTP is running on control plane nodes. On a managed cluster, this one is on your provider.
The schedule is very frequent and runs were blocked
With concurrencyPolicy: Forbid, a run that hangs blocks every later run. If an every-minute job hangs for two hours, the controller counts 120 skipped schedules. Fix the hang too: set activeDeadlineSeconds on the Job template so a stuck run is killed.
Kubernetes CronJobs aren't the only way to get a reliable schedule. delay.run runs HTTP cron jobs with retries and start logs, without depending on a controller's missed-start budget.
Step 2: set startingDeadlineSeconds
The fix the message suggests is the right one. startingDeadlineSeconds tells the controller: "If you're more than this many seconds late for a scheduled run, skip it." With that set, the controller only counts missed times within the deadline window, so the count can't reach 100 unless the window is huge.
apiVersion: batch/v1 kind: CronJob metadata: name: my-job spec: schedule: "* * * * *" startingDeadlineSeconds: 120 # skip runs that are more than 2 minutes late concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: spec: activeDeadlineSeconds: 300 # kill a run that takes more than 5 minutes backoffLimit: 2 template: spec: restartPolicy: Never containers: - name: worker image: registry.example.com/worker:1.4.2 args: ["run-once"] How to pick the value:
- It must be comfortably larger than the controller's evaluation interval (around 10 seconds). Very small values, under about 10 seconds, can make the job never run at all.
- For frequent jobs where only the latest run matters (sync, cleanup, polling), a value close to the schedule interval is fine: a late run is replaced by the next one.
- For daily or weekly jobs that must not be skipped, use a larger value, such as a few hours, so a short control plane outage still results in a late run rather than a skipped one.
Step 3: if the job is stuck on an old cluster
On a cluster with the legacy controller, adding startingDeadlineSeconds is enough: the next evaluation only looks inside the window, finds at most a couple of missed times, and starts scheduling again. If you need it to run right now, trigger one manually:
kubectl create job --from=cronjob/my-job my-job-manual-$(date +%s) -n my-namespace Deleting and recreating the CronJob also resets the count, but you lose its history and status, so prefer the deadline.
Step 4: decide whether missed runs matter
The error is a symptom of runs not happening. Ask whether that's acceptable for this job:
- If a missed run means missed invoices, reminders or backups, you need monitoring that notices when a job didn't run, independent of the CronJob controller. A heartbeat ("the job pings a URL on success; alert if no ping in N minutes") is the simplest version.
- If the job must catch up after an outage, make it work out what it missed (from a "last processed" timestamp in your own data) instead of relying on the scheduler to replay every slot.
Schedulers outside the cluster, such as delay.run, avoid the dependency on the control plane for timing, but the heartbeat and the catch-up logic are worth having wherever the schedule lives.
Related reading
- Case study (hypothetical): the provider disabled our webhook endpoint after a weekend of 500s, and nobody noticed for 9 days
- Cron in Docker FAQ: 11 questions from everyone whose job worked locally and does nothing in the container
- What happens to the cron job that's running when you deploy? Graceful shutdown for scheduled work
- The server was down at 02:00. Should the 02:00 job run late, or not at all?
Have you hit "too many missed start times"? Was it a resumed suspension, a controller outage, or something stranger?

