Short answer: use one Express error-tracking middleware setup that emits a normalized JSON event with run_id, property_id, stage, and error_code; retain the compact event longer than its raw payload so nightly property-pipeline costs can be attributed without storing every request body.

A nightly property-management pipeline usually does not need a bigger error platform. It needs one error record that answers three questions: which run failed, which property or tenant cohort was affected, and what retention decision created the bill. Put that context into structured logs at the Express boundary, propagate a request or job identifier, and keep verbose payloads on a short retention schedule.

The least complex useful implementation is an error-handling middleware that emits one normalized event per failed request, while the pipeline worker emits the same shape for scheduled jobs. The middleware should preserve the original error for local diagnostics, remove secrets before serialization, and let the process-level policy decide whether the request can continue.

What is the bill actually made of?

For a property operator, the useful unit is rarely an individual log line. It is a nightly run: 8,000 leases imported, 312 address records rejected, three downstream calls retried, and one reconciliation step marked incomplete. If every debug payload and stack trace survives for 30 days, storage and indexing grow with the number of records even though the operator mostly searches by run_id, property_id, stage, and error_code.

Start by measuring those terms in your own pipeline. Count records per run, bytes per record, retry attempts, and the percentage of failures that lead to a support investigation. Do not use an assumed vendor price as the model; retention, indexing, and query volume vary by deployment. The decision is still concrete: keep compact failure summaries long enough for reconciliation, and keep raw input samples only long enough to diagnose a malformed record.

Here is a small Python calculation for the planning worksheet. It does not predict a bill. It exposes which field is driving retained volume.

from dataclasses import dataclass @dataclass class RunVolume: runs: int records_per_run: int bytes_per_record: int retention_days: int def retained_gib(self) -> float: total_bytes = ( self.runs * self.records_per_run * self.bytes_per_record * self.retention_days ) return total_bytes / (1024 ** 3) volume = RunVolume( runs=30, records_per_run=8000, bytes_per_record=1200, retention_days=30, ) print(f"{volume.retained_gib():.2f} GiB before indexes and replicas")

The change that usually matters is boring: emit one summary event for each rejected record and attach the detailed input only to a sampled or explicitly quarantined record. That makes the search path predictable and gives cost attribution a stable key. The trade-off is real. After the raw sample expires, reconstructing the exact malformed input may require returning to the source system or rerunning the job.

The record is the unit of search, but the run is the unit of accountability.

That distinction changes the retention design in a useful way. A property manager investigating a missed lease import needs to identify the run, isolate the affected property, see whether the record failed validation or exhausted retries, and find the next system responsible for correction. They do not need every successful row, every request header, or a permanent copy of an upstream response. Keeping those details forever makes a later query noisier and expands the data that must be protected. Keeping only an opaque error code makes the event cheap but leaves support with no path back to the source. The compact summary and expiring detail reference are the middle ground: enough context for attribution, with a deliberate escape hatch for diagnosis.

How should Express middleware preserve the failure context?

Express error middleware has four arguments and must be registered after the routes. That placement matters more than the logger choice. The handler should classify the error, select a safe public response, and write a structured event with fields that a nightly job uses too. It should not dump the entire request object into a log record.

This example uses only Node.js interfaces and a JSON writer. The writer could send to a file, a collector, or a self-hosted log service; the middleware does not need to know which. That is the setup boundary: the application chooses fields and severity, while the transport chooses where records are retained.

import crypto from "node:crypto"; import express from "express"; const app = express(); app.use(express.json({ limit: "256kb" })); function requestId(req) { return req.get("x-request-id") || crypto.randomUUID(); } function classify(error) { if (error.code === "PROPERTY_NOT_FOUND") { return { status: 404, error_code: error.code }; } if (error.code === "UPSTREAM_TIMEOUT") { return { status: 503, error_code: error.code }; } return { status: 500, error_code: "INTERNAL_ERROR" }; } function writeEvent(event) { process.stdout.write(JSON.stringify(event) + "\n"); } app.get("/properties/:propertyId/ledger", async (req, res, next) => { try { const ledger = await loadLedger(req.params.propertyId); res.json(ledger); } catch (error) { error.propertyId = req.params.propertyId; next(error); } }); app.use((error, req, res, next) => { const id = requestId(req); const result = classify(error); writeEvent({ timestamp: new Date().toISOString(), signal: "log", severity: result.status >= 500 ? "ERROR" : "WARN", event_name: "http_request_failed", request_id: id, run_id: req.get("x-run-id") || null, property_id: error.propertyId || null, stage: "ledger_lookup", error_code: result.error_code, http_method: req.method, route: req.route?.path || "unknown", status_code: result.status, retryable: result.status === 503, exception_type: error.name, }); res.status(result.status).json({ error: result.error_code, request_id: id, }); });

There are two details worth defending. First, request_id is returned to the caller, so an operator can connect a tenant report to the server-side event without exposing a stack trace. Second, run_id remains nullable because interactive traffic and scheduled work have different origins. A fake run identifier would make cost attribution look precise while silently joining unrelated events.

In a real application, loadLedger would be defined elsewhere. The example is about the boundary contract, not a complete property ledger implementation. The process must also handle rejected promises and uncaught exceptions according to its runtime policy; an HTTP middleware cannot observe failures that occur outside an Express request.

That is the whole setup. Keep it small.

Which fields make a nightly run searchable?

Use a small, stable event schema. OpenTelemetry describes logs as a signal that can carry timestamped records and shared context, including trace-related correlation fields. You do not need to adopt every field on day one, but aligning names and semantics with an established model prevents a private schema from becoming an integration tax later.

For this pipeline, I would make run_id, property_id, stage, error_code, attempt, and source_system searchable. Put the tenant's email address, lease text, and full upstream response in a quarantined detail record instead. Those values increase privacy exposure and make cardinality expensive.

A useful event might look like this: single-line JSON, no embedded request body.

{ "timestamp": "2026-10-28T02:14:07.431Z", "signal": "log", "event_name": "pipeline_record_rejected", "run_id": "pm-2026-10-28-02", "property_id": "property-184", "stage": "lease_import", "source_system": "rent_feed", "record_key_hash": "sha256:...", "attempt": 2, "error_code": "MISSING_MOVE_IN_DATE", "retryable": false, "detail_ref": "quarantine/pm-2026-10-28-02/record-91" }

record_key_hash supports repeated-failure analysis without making a personal identifier the query key. Hashing is not anonymization by itself, especially when the input space is small, so access control and deletion policy still matter. The detail reference should expire independently from the summary event, and its retention should be justified by the support workflow.

What happens when retries create false cost signals?

A retry is an operational action, not a new business failure. If the same source record times out three times, emit three attempt events but one final outcome event, or include an explicit outcome field that makes the distinction queryable. Otherwise a dashboard may charge the property for three failures when the pipeline eventually succeeded. This distinction is the difference between error tracking and failure counting: tracking preserves the execution path, while attribution needs the final business result.

I use these decision rules: a validation error is terminal and points to the source record; a timeout is retryable and points to the dependency; a duplicate import is a business outcome and should not page the on-call engineer. The exact codes belong to the domain, but the separation should be deliberate.

Test the middleware with a real error matrix, including an error without a code, a missing x-run-id, a route that throws after a promise rejection, and a request containing a secret header. Assert the public response and the emitted fields separately. A passing HTTP test that accidentally serializes an authorization token is still a failed observability test.

The retention rule I would ship

Keep compact run summaries and terminal error events for the period needed to reconcile property statements. Keep verbose payloads and stack traces for a shorter, access-controlled window. Sample successful records aggressively; retain failures according to their diagnostic value, not because storage made indefinite retention easy.

The final check is a query rehearsal: given run_id and property_id, can an engineer find the affected stage, distinguish retry from final failure, locate the safe detail reference, and explain why that data still exists? If the answer is no, adding another dashboard will not fix the schema. Cost attribution is an observability design problem before it is a storage problem.

Further reading