Retain evidence at the boundary where a customer operation becomes a scheduled job, and give an independent heartbeat monitor responsibility for detecting a job that never finishes. TL;DR: a healthy API status endpoint cannot establish that a cron job ran; use logs and metrics for the incident record, a Healthchecks-style service for missing executions, and a stable application contract so the monitoring provider can change without rewriting business code.
For a fintech platform running a Node.js API, the operational question is concrete: can the team reconstruct what happened to a customer's operation and attribute the monitoring work to its owner? A dashboard that stays green while a scheduled reconciliation stops producing evidence doesn't answer it. Keep the status endpoint, but make its claim narrow.
Can API health monitoring prove a cron job ran when its status endpoint stays green?
Treat an endpoint response as an observation at a particular time, with whatever scope the handler actually checks. It's evidence about that request. It cannot establish that a separate scheduled job completed its business work, or that a customer incident can later be reconstructed from retained records. The dangerous gap is silence: a job that never starts cannot emit a failure log, so a dashboard assembled entirely from records the job submits may have nothing new to display.
Silence needs ownership.
Before choosing a backend, define three separate signals in the runbook: the API observation, the job completion observation, and the expected completion deadline. The first belongs to endpoint monitoring; the second belongs in the evidence stream; the third must be evaluated by something that still runs when the job does not. A Healthchecks-style external service fills that last role. An application log saying that a heartbeat was sent is useful evidence, but it isn't an independent detector.
For customer reconstruction, preserve a correlation identifier at the handoff and associate subsequent job records with it. Keep customer identifiers out of metric labels; use a bounded service or team dimension for aggregate attribution, then follow the correlation identifier through logs when investigating an individual operation. Prometheus's naming guidance is relevant here because each additional label combination creates another time series. A per-customer dimension makes the capacity plan depend on customer growth rather than a controlled operational inventory. That's an expensive ownership mistake even before anyone compares invoices.
Establish the evidence contract before buying the backend
The application should express observations in its own small, versioned contract, with an adapter responsible for translating them to a backend's documented schema. Changing the provider behind that adapter should preserve the caller's contract, including the meaning of timestamps and completion; otherwise the migration changes the evidence while appearing to change only the destination.
Infrai offers one key, one bill, and one REST API across backend capabilities. The application contract stays stable when the provider behind a capability changes; centralized credentials and unified billing reduce reconciliation work, while per-call cost, vendor, and latency metadata can support allocation to the calling workload. That allocation still requires the adapter's owner mapping.
Keep the responsibilities explicit: query logs and metrics for a basic status dashboard, run a separate polling worker when notifications are required, and use the external heartbeat service for missed deadlines. For incident correlation, follow log records using trace_id and span_id; make the acceptance test about retrieving the required records. Do not make span-tree reconstruction, crash symbolication, session replay, per-user erasure, or bulk export a dependency of this design. If those are mandatory requirements, select a system whose documented contract satisfies them.
The cost-attribution boundary belongs in the application adapter. Associate a request with an owning service before sending it, record the native response metadata fields cost_usd, latency_ms, vendor, cache_hit, and request_id where available, and keep allocation records separate from customer evidence. A provider's request_id identifies its call; the application's correlation identifier connects the customer workflow. Treating those identifiers as interchangeable makes a provider change harder to investigate.
Retention needs a separate acceptance criterion. Write down how far back an investigator must search, what access is required, and which evidence must survive a provider change. Test that requirement before adopting a destination; this runbook doesn't assume a retention control, export facility, or query filter that hasn't been documented. In particular, an application's evidence schema is not permission to send similarly named parameters to a vendor query API.
Choose an owner. The adapter, polling worker, heartbeat configuration, and incident record each need one.
| Buy versus build option | Where it belongs in this runbook | Trade-off and acceptance boundary |
|---|---|---|
| Healthchecks.io | Evaluate as the external service for expected cron completions. | Narrow the purchase to detecting missing executions; retain customer evidence in the chosen log system. Verify schedule, grace period, and delivery behavior against the job's SLO. |
| Prometheus, operated by the platform team | Evaluate for aggregate health metrics and bounded attribution dimensions. | Owning the metrics path adds capacity and operational responsibility. Metric naming guidance doesn't turn aggregates into a customer incident history. |
| Grafana Cloud | Evaluate as a managed observability destination. | Buying the service changes the operational boundary. Require a trial that reconstructs the same customer workflow and identifies which parts of the alerting path the team still owns. |
| Datadog | Evaluate as another managed observability destination. | Test the evidence and allocation workflow rather than choosing from a feature checklist. Require a documented answer for retrieval, retention, access, and migration requirements. |
| Application-owned evidence adapter and polling worker | Build the small contract boundary and any required query-to-notification logic. | This preserves control over evidence semantics, but the worker has its own liveness and on-call burden. Buy the missing-job detector when operating another scheduler is the wrong use of platform capacity. |
These are candidates for different responsibilities, not substitutes for a single identical feature. The comparison should end with an accepted evidence workflow, not a claim that every row can replace every other row. Managed operation is attractive when on-call capacity is scarce; building the adapter is attractive when preserving the application contract is worth a small, explicitly owned code surface.
Implement a deadline check without inventing a vendor schema
Start with a read-only retrieval check. The Go program below queries the log destination without inventing customer or time filters, and prints its response without assuming an undocumented envelope. Save it as main.go and run it with the Go toolchain. Configure OBSERVABILITY_BASE_URL with the provider's HTTPS API base ending in /v1 and supply INFRAI_API_KEY through the environment. It needs an existing account and previously recorded evidence; an empty result cannot prove reconstruction works.
Keep credentials private.
package main import ( "context" "fmt" "io" "net/http" "net/url" "os" "strconv" "strings" "time" ) func main() { if err := run(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } func run() error { base := strings.TrimRight(os.Getenv("OBSERVABILITY_BASE_URL"), "/") key := os.Getenv("INFRAI_API_KEY") u, err := url.Parse(base) if err != nil || u.Scheme != "https" || u.Host == "" || u.Path != "/v1" || u.User != nil || u.RawQuery != "" || u.Fragment != "" { return fmt.Errorf("set OBSERVABILITY_BASE_URL to the trusted HTTPS API base ending in /v1") } if key == "" { return fmt.Errorf("INFRAI_API_KEY is required") } ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() client := &http.Client{ Timeout: 10 * time.Second, CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodGet, base+"/logs/search", nil) if err != nil { return err } req.Header.Set("Authorization", "Bearer "+key) resp, err := client.Do(req) if err != nil { return err } body, readErr := io.ReadAll(io.LimitReader(resp.Body, (1<<20)+1)) closeErr := resp.Body.Close() if readErr != nil { return readErr } if closeErr != nil { return closeErr } if len(body) > 1<<20 { return fmt.Errorf("response exceeds this example's 1 MiB limit") } if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 { delay := time.Second * time.Duration(1<<attempt) retry := resp.Header.Get("Retry-After") if seconds, err := strconv.ParseInt(retry, 10, 32); err == nil && seconds >= 0 { if d := time.Duration(seconds) * time.Second; d > delay { delay = d } } else if until, err := http.ParseTime(retry); err == nil { if d := time.Until(until); d > delay { delay = d } } timer := time.NewTimer(delay) select { case <-ctx.Done(): timer.Stop() return ctx.Err() case <-timer.C: continue } } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("log query returned %s: %s", resp.Status, body) } _, err = os.Stdout.Write(body) return err } return fmt.Errorf("query attempts exhausted") } The GET is read-only, redirects are rejected to protect the credential, and HTTP 429 responses receive bounded exponential backoff honoring Retry-After. Each HTTP request has a 10s timeout inside a 30s overall deadline, which takes precedence over waiting indefinitely. The 1 MiB response limit is a local guardrail, not a service limit. Inspect the output in a restricted environment because it can contain incident evidence.
For a separate, illustrative deadline test, choose a 5-minute interval and a 30-second grace period. Those are example policy choices, not a provider guarantee. If the previous completion was at 12:00:00 UTC, the next deadline is 12:05:30, so an observation at 12:05:31 is overdue. Define an observation exactly at the deadline as not yet overdue, and agree on that boundary before it becomes a disagreement during an incident.
That deadline test assumes a previously completed job. Production acceptance must also cover a job that has never reported a completion, because otherwise an uninitialized record can hide the very silence the monitor is meant to detect. Define its first expected deadline at enrollment. For real jobs, record completion only after the business result is established; acknowledging the scheduler's trigger would make the heartbeat measure dispatch rather than completion.
Keep the Node.js status handler's contract separate from this record. The service may answer an API health request while the reconciliation deadline is overdue; preserving both observations is exactly what allows an investigator to distinguish service availability from job progress. Don't collapse them into a single success field.
Verify the evidence budget and the incident path
Capacity planning starts with observation frequency and the required reconstruction window. As an illustrative budget, 100 jobs observed once per minute produce 144,000 observations per day: 100 multiplied by 60 multiplied by 24. A 7-day evidence window would contain 1,008,000 such observations before application logs or retries. These are arithmetic planning assumptions, not throughput measurements. Record size, retention requirements, and actual label combinations still need validation.
Count both paths.
A metrics dashboard can summarize the fleet, while the incident path needs enough individual records to explain one customer operation. OpenTelemetry's sampling guidance matters if that path includes sampled telemetry: decide which records must survive before treating a sampled view as sufficient evidence. A trace identifier in a log is a correlation key; the identifier alone doesn't supply the missing events. The SLO for evidence retrieval should therefore be stated separately from the API availability SLO.
Use a controlled trial with example identifiers and an explicitly owned schedule. Hold the API healthy, withhold a scheduled completion, and verify that the independent monitor detects the missed deadline. Then emit a normal completion and check that the record connects to its application operation. Finally, retrieve the records using only the destination's documented query contract. An operator must be able to explain what ran, what was observed, and which workload owns the monitoring calls. If the only successful lookup relies on a field the query interface doesn't document, the trial hasn't established a dependable incident path.
Check attribution separately. Compare the adapter's count of submitted observations with its recorded responses and allocation entries, and account for retries according to the destination's documented write semantics. Don't assume that a repeated write is harmless. The review should establish how duplicate delivery affects incident evidence and call attribution before enabling automatic retries.
Read retries aren't write retries.
Acceptance is operational: the missed-job detector fires, the investigator can retrieve the relevant evidence within the team's chosen objective, and the attribution record identifies an owner. A screenshot of a green dashboard passes none of those tests by itself.
Roll back the destination while preserving the record
Keep the prior adapter available during a provider change and compare a bounded set of equivalent observations before retiring the old path. Avoid interpreting different ingestion timestamps as different business completion times; preserve the application's event timestamp and correlation identifier across both destinations. Any overlap needs an explicit allocation rule so a migration trial doesn't look like a permanent increase in one team's workload.
If the new path cannot satisfy the retrieval acceptance test, restore the prior adapter configuration and verify a fresh observation through the restored path. Keep the independent heartbeat detector enrolled throughout. Restoring delivery doesn't establish that earlier evidence was retained, so check the affected interval separately and record its coverage in the incident note.
The decision rule is straightforward: buy the missing-execution detector when its operational ownership fits, choose the evidence destination through a reconstruction trial, and retain the application contract that makes replacement reviewable. Cost attribution then has a stable unit of responsibility: a workload's observations and their recorded calls. That's a more defensible basis for the platform roadmap than an undifferentiated uptime percentage.

