InfraGuard Engineering Devlog, part 2 of 6.

On August 18, a weekly backup was repeatedly creating a roughly 5 GB full archive. The cause was a chain of process, reporting and scheduling failures.

Five links in the failure chain

  1. PM2 restarted the backup runner every minute through cron_restart.
  2. When the runner died, its child backup shell lost its output pipe.
  3. A subsequent log write could terminate the shell with SIGPIPE, before it reported a result.
  4. With no report, the application did not advance the next-run timestamp.
  5. The next runner tick still saw an overdue job and started it again.

Redirecting errors or adding || true did not solve signal-driven termination. Swallowing an exit status and surviving a signal are different things.

Change the scheduling trade-off explicitly

We changed eligibility to a phase-based minute slot derived from the start time and interval, rather than an indefinitely overdue next-run value. A runner missing its eligible minute skips that occurrence instead of repeatedly catching up.

That trades automatic recovery of a missed slot for avoiding this runaway replay path. Slot eligibility is not a general exactly-once guarantee across multiple runners; concurrency control remains separate.

The scripts ignore SIGPIPE and attempt failure reports for handled termination signals such as TERM, INT and HUP.

Successful work, rejected report

A second incident produced a 2.1 GB incremental archive but zero execution records. A preview script had been rendered before the job existed, then saved with JOB_ID=0. Its report was rejected with HTTP 422.

Without --fail, curl can exit successfully after an HTTP error response. We had confused transport completion with application acceptance.

The fix was to render with the real ID before saving, use curl --fail, and log rejected reports. Retrying an unchanged invalid payload would not make it valid.

Tool-specific semantics matter

GNU tar's exit code 1 can mean files changed during archiving. We treated that documented case as a warning, while retaining failure handling for a nonzero database-dump result. A single generic exit-code policy was wrong.

The manifest workflow also moved preparation ahead of the expensive transfer and used a rename to finalize the validated result. Previously, completed archives had failed to establish their expected incremental baseline.

Takeaway: A backup is a schedule, a process tree, a baseline and an accepted result record, not just an archive. Test the boundaries between them.

About this series: Adapted from InfraGuard engineering records. Measurements describe the recorded environment, not product-wide benchmarks. AI was used to draft and translate this article.

Disclosure: Solution Market develops and sells InfraGuard. This is our engineering devlog, not an independent review. InfraGuard is a self-hosted infrastructure operations and security platform. Project · Documentation · Solution Market