When an alert says an experiment failed but its error field is empty, the first question is not which dashboard to open. It is which stage lost the field. For a media experiment comparing tenant cohorts, I treat the error as a path: response, parser, event, collector, rule. The first missing value is the useful boundary.
Short answer: preserve a typed error envelope, classify the response as present, missing, or malformed, and alert on context quality separately from failure volume.
Three states are enough for the first decision:
-
present: the response has a usable error code and message. -
missing: the expected field is absent. -
malformed: the body or content type cannot support the expected shape.
The practical fix is to preserve that envelope at every boundary, then alert on field presence and signal quality separately. That gives operators two honest outcomes: a failure with usable context, or a failure whose context is itself degraded. The second one should never masquerade as the first.
Keep those states visible.
How can diagnosing missing or malformed fields improve an error response?
Think of the pipeline as a relay. The service hands over an HTTP response. The parser hands over an error object. The logger hands over a record. The alert rule reads selected attributes. If message exists in the response but disappears from the log, the alert system did not cause the loss; it exposed it.
OpenTelemetry describes a log record with fields such as time, body, attributes, severity, and optional trace and span identifiers. That vocabulary is useful even when the collection stack is different. Keep the human-readable body for inspection, and keep stable attributes for grouping: tenant_id, cohort, experiment_id, error_code, and field_state.
I start with a before/after comparison. Before: every failure becomes { error: response.error }, so an HTML error page, an empty body, and a structured error all collapse into confusing output. After: the boundary classifies the body, records why classification failed, and emits one stable event shape.
Step 1: classify the response before extracting fields
Do not cast an unknown response to an error type. Check the status, content type, and body shape first. This example is deliberately small enough to paste into a test file.
type ErrorEnvelope = { code: string; message: string; fieldState: 'present' | 'missing' | 'malformed'; }; type ParsedFailure = { status: number; error: ErrorEnvelope; }; function parseFailure(status: number, contentType: string | null, body: unknown): ParsedFailure { const isObject = typeof body === 'object' && body !== null; const record = isObject ? body as Record<string, unknown> : {}; const code = typeof record.code === 'string' ? record.code : 'unknown_error'; const message = typeof record.message === 'string' ? record.message : ''; if (!isObject || !contentType?.includes('application/json')) { return { status, error: { code, message: 'Response was not a JSON error envelope', fieldState: 'malformed' } }; } if (!message) { return { status, error: { code, message: 'JSON error envelope omitted message', fieldState: 'missing' } }; } return { status, error: { code, message, fieldState: 'present' } }; } The decision rule is simple: present can support a diagnostic alert; missing and malformed should add a data-quality signal. The exact threshold depends on traffic, but the classification should not. A 502 with an HTML body is still an upstream failure. It is also a malformed error payload for this pipeline.
Step 2: carry context, test the boundary, and keep alerts honest
A media experiment often has a large tenant count and a smaller number of cohorts. Alerting on every tenant creates a wall of nearly identical incidents. Dropping tenant context creates an alert that cannot answer which cohort moved.
The 3-state distinction is deliberately small. A larger taxonomy can wait until an operator can name the decision it improves.
I keep high-cardinality identifiers in the event for investigation, then group the alert by the dimensions that represent the decision: experiment and cohort. The event can still retain tenant_id as an attribute for drill-down. In practice, that means a spike affecting one tenant does not page as if the whole experiment failed, while the raw event still lets an operator inspect that tenant later. If a cohort boundary is missing, I would rather delay the comparison than attach a precise-looking alert to the wrong population. This is a signal-quality trade-off, not a formatting preference.
type FailureEvent = { timestamp: string; body: string; attributes: { tenant_id: string; cohort: string; experiment_id: string; error_code: string; field_state: ErrorEnvelope['fieldState'] }; severity: 'error' | 'warn'; }; function toFailureEvent(tenantId: string, cohort: string, experimentId: string, failure: ParsedFailure): FailureEvent { const { error } = failure; return { timestamp: new Date().toISOString(), body: error.message, attributes: { tenant_id: tenantId, cohort, experiment_id: experimentId, error_code: error.code, field_state: error.fieldState }, severity: error.fieldState === 'present' ? 'error' : 'warn' }; } Notice the warning severity for degraded context. That is intentional. It prevents an observability defect from inflating the same failure count used to compare cohorts. A separate rule can page when malformed or missing fields cross a chosen rate over a chosen window.
Testing belongs at this same boundary, not only on the happy path.
A unit test for a valid JSON error proves very little. Add cases for an empty JSON object, a non-JSON body, a valid code without a message, and a response whose content type is absent. Then assert both the value and its classification.
const cases: Array<[string, ParsedFailure]> = [ ['valid', parseFailure(500, 'application/json', { code: 'ENCODE_FAILED', message: 'Transcode failed' })], ['missing', parseFailure(500, 'application/json', { code: 'ENCODE_FAILED' })], ['malformed', parseFailure(502, 'text/html', '<html>bad gateway</html>')], ['empty', parseFailure(500, 'application/json', {})] ]; for (const [name, result] of cases) { if (result.error.fieldState === 'present' && !result.error.message) { throw new Error(`${name}: present errors need a message`); } } The useful test assertion is not that every failure has a friendly sentence. It is that every failure has a predictable field_state. That one attribute lets the rule distinguish application trouble from telemetry trouble.
Which signal should page the team?
Use two questions. First: are failures increasing for one cohort or across all cohorts? Second: is the error context usable? A cohort-specific increase with field_state: present is a product or service investigation. A broad increase in missing or malformed is a boundary or upstream contract investigation. Mixing them creates noise precisely when the experiment needs a clean comparison.
The alert payload should include the experiment, cohort, error code, field state, observation window, and a link or identifier for the underlying records. It should not attempt to serialize an arbitrary response body into a notification. Bodies can be large, sensitive, or structurally unstable.
One objection is that this adds fields to every event. It does. The trade-off is explicit context at the point where the failure is observed, instead of a later guess based on a truncated notification. Another objection is that a warning might hide a serious upstream outage. It should not: severity describes context quality here, while a separate status-based failure metric captures outage volume. Keep those measurements independent.
I initially treated an empty error message as a parsing nuisance. Later, I treated it as a signal-quality state instead. The sharper lesson is that it is a measurement. Once missing and malformed are first-class states, the alert can say what it knows and what it does not. That is the difference between a noisy experiment dashboard and a useful operating signal.
