{ "Attributes": { "ApproximateNumberOfMessages": "4819", "ApproximateNumberOfMessagesNotVisible": "0" } } 

That was the output of aws sqs get-queue-attributes on prod-merchant-payouts-dlq when our finance lead finally asked why Tuesday's Stripe Connect batch was missing $62,400 in disbursements.

Every single REST API contract test in our monorepo had passed that morning. Our Playwright suite was green. Our Go service's unit tests were green. Our TypeScript worker's unit tests were green.

None of them caught the bug, because the bug did not live on an HTTP route. It lived in the invisible gap between a Go service calling sqs.SendMessage and a Node.js worker calling JSON.parse(record.Body) as PayoutRequestedEvent.

Why REST Contract Tests Miss Half Your Architecture

Engineering teams spend huge amounts of energy locking down synchronous HTTP boundaries. We generate TypeScript clients from OpenAPI specs, run Supertest against Express routers, and fail CI if a response field changes from int to string.

Then we take an asynchronous SQS queue, a Kafka topic, or an internal SNS fan-out—which often carries the actual money-moving side effects—and treat it like a lawless JSON dropbox.

In our case, two separate teams owned the two ends of prod-merchant-payouts:

  • Producer (ledger-go): Maintained by the core billing team in Go.
  • Consumer (payout-worker): Maintained by the platform team in TypeScript/Node 22.

In payout-worker, the consumer had a type annotation that looked safe in code review and meant literally nothing at runtime:

// Looks typed in your IDE. Completely unchecked at runtime. const payload = JSON.parse(message.Body!) as { merchantId: string; payoutCents: number; currency: string; }; 

When ledger-go refactored its decimal serialization to avoid float rounding and started emitting "payoutCents": "14900" as a string instead of 14900 as an integer, JSON.parse happily returned a string. Downstream, Math.round(payload.payoutCents) did not throw—Wait: actually, Math.round("14900") works in JavaScript!

What did throw was the strict Postgres BIGINT / Stripe SDK validation two functions later (Invalid integer: "14900.00" when a fractional fee was included), crashing the worker, exhausting maxReceiveCount: 3, and dumping 4,819 messages into prod-merchant-payouts-dlq.

Nobody was watching the DLQ metric in Datadog because that queue had sat at 0 for eight months.

What Unit Tests Lied About on Both Sides

When we pulled the two repos side-by-side after replaying the DLQ, both teams had written tests for that exact flow:

  • ledger-go asserted that its MarshalJSON() helper produced valid JSON containing "payoutCents". It passed.
  • payout-worker had a Jest test with a hardcoded fixture file fixtures/payout-message.json where payoutCents was still 14900. It passed too.

Static fixtures in consumer unit tests are a slow-acting poison. The day you write fixtures/payout-message.json, it matches reality. Six months later, after three producer refactors, your consumer tests are certifying a payload shape that no longer exists anywhere in production.

One Shared JSON Schema Contract That Breaks Both Repos.

We did not want to deploy a heavy Confluent Schema Registry just for 14 SQS queues. We wanted something boring that worked across Go and TypeScript inside GitHub Actions with zero new infrastructure.

So we made JSON Schema (Draft 2020-12) the single source of truth inside a lightweight shared directory (contracts/queues/*.schema.json), paired with:

  1. Runtime validation at the consumer boundary using ajv (or zod compiled to JSON Schema) so malformed messages fail immediately at line 1 of the handler with a structured SchemaDriftError log instead of half-way through a Stripe call.
  2. A CI cross-repo contract check so if ledger-go changes a struct tag or field type, it has to run against the exact same .schema.json file that payout-worker consumes.

Here is contracts/queues/payout-requested.v1.schema.json:

{ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://contracts.internal/queues/payout-requested.v1.schema.json", "type": "object", "additionalProperties": false, "required": ["eventId", "merchantId", "payoutCents", "currency", "occurredAt"], "properties": { "eventId": { "type": "string", "format": "uuid" }, "merchantId": { "type": "string", "minLength": 4 }, "payoutCents": { "type": "integer", "minimum": 1 }, "currency": { "type": "string", "enum": ["USD", "EUR", "GBP"] }, "occurredAt": { "type": "string", "format": "date-time" } } } 

And here is the TypeScript consumer boundary wrapper (src/queues/validateMessage.ts) that replaced JSON.parse(...) as ...:

import Ajv from 'ajv'; import addFormats from 'ajv-formats'; import payoutSchema from '../../contracts/queues/payout-requested.v1.schema.json'; const ajv = new Ajv({ allErrors: true, strict: true }); addFormats(ajv); const validatePayout = ajv.compile<{ eventId: string; merchantId: string; payoutCents: number; currency: 'USD' | 'EUR' | 'GBP'; occurredAt: string; }>(payoutSchema); export function parsePayoutMessage(rawBody: string | undefined) { if (!rawBody) { throw new Error('SQS_EMPTY_BODY: Missing message body'); } const parsed: unknown = JSON.parse(rawBody); if (!validatePayout(parsed)) { const details = ajv.errorsText(validatePayout.errors); throw new Error(`SCHEMA_DRIFT_ERROR: ${details}`); } return parsed; } 

Notice additionalProperties: false and "type": "integer" on payoutCents. If ledger-go ever serializes "14900" as a string again, parsePayoutMessage rejects it in under 0.1 milliseconds with an unmistakable SCHEMA_DRIFT_ERROR instead of letting it corrupt downstream state.

Testing the Real SQS Loop With LocalStack in CI.

Validating a schema in memory is only half the story. You also want to prove that the Go producer's actual serialized bytes survive SQS attributes, base64/JSON encoding, and the TypeScript consumer's parsePayoutMessage() function.

We added a 55-line integration test (tests/contracts/sqs-payout-contract.test.ts) that spins up LocalStack via Testcontainers (or a GitHub Actions service container), runs the Go producer binary's emit-sample command to push real messages onto a local queue, and lets the TypeScript consumer pull and validate them:

import { execFileSync } from 'node:child_process'; import { SQSClient, CreateQueueCommand, ReceiveMessageCommand } from '@aws-sdk/client-sqs'; import { parsePayoutMessage } from '../../src/queues/validateMessage'; const sqs = new SQSClient({ region: 'us-east-1', endpoint: process.env.LOCALSTACK_URL || 'http://127.0.0.1:4566', credentials: { accessKeyId: 'test', secretAccessKey: 'test' }, }); test('Go producer payload passes TypeScript SQS consumer contract', async () => { const { QueueUrl } = await sqs.send( new CreateQueueCommand({ QueueName: `payouts-contract-${Date.now()}` }) ); // Run the actual Go producer serialization path against LocalStack execFileSync('./bin/ledger-emit-sample', ['--queue-url', QueueUrl!], { env: { ...process.env, AWS_ENDPOINT_URL: process.env.LOCALSTACK_URL || 'http://127.0.0.1:4566' }, }); const received = await sqs.send( new ReceiveMessageCommand({ QueueUrl, MaxNumberOfMessages: 1, WaitTimeSeconds: 3, }) ); expect(received.Messages).toHaveLength(1); const validated = parsePayoutMessage(received.Messages![0].Body); expect(validated.payoutCents).toBeGreaterThan(0); }); 

That single test runs in 6.4 seconds in GitHub Actions. No mocked JSON fixtures. If a Go developer changes int64 to string on PayoutCents—even by adding ,string to a Go struct tag—that test blows up on their PR before payout-worker ever sees it in production.

Never Let a Dead-Letter Queue Sit Silent Again

The other embarrassing half of the $62K incident was not technical—it was operational. A queue had 4,819 failed payout messages sitting in it for 19 hours because nobody had wired an alarm to ApproximateNumberOfMessagesVisible > 0 on *-dlq.

We fixed that across all 14 queues in one Terraform module (infra/modules/sqs_dlq_alarm/main.tf) so every new queue gets a PagerDuty/Slack alert on message #1 in the DLQ, not message #4,819:

resource "aws_cloudwatch_metric_alarm" "dlq_not_empty" { for_each = toset(var.dlq_names) alarm_name = "${each.value}-has-messages" comparison_operator = "GreaterThanThreshold" evaluation_periods = 1 metric_name = "ApproximateNumberOfMessagesVisible" namespace = "AWS/SQS" period = 60 statistic = "Maximum" threshold = 0 treat_missing_data = "notBreaching" dimensions = { QueueName = each.value } alarm_actions = [var.ops_alerts_sns_topic_arn] } 

Zero tolerance on DLQ depth sounds aggressive until you realize a dead-letter queue is literally a bucket of failed customer promises. If one message lands there, you want to know within 60 seconds, not when finance reconciles Stripe payouts on Wednesday morning.

Handling In-Flight Messages During Schema Upgrades

Putting a strict JSON Schema validator on a queue consumer introduces a subtle trap during rolling deployments: in-flight messages.

When Kubernetes rolls payout-worker to a new version, there can easily be 800 messages still sitting in SQS serialized by the previous version of ledger-go. If you rename a required field or tighten an enum in payout-requested.v1.schema.json, your new consumer pods will immediately reject valid in-flight messages as SCHEMA_DRIFT_ERROR.

We learned that the hard way on our second schema update. Now we enforce two rules in CI:

  1. Never mutate a published v1 schema in a breaking way. Adding an optional property is fine. Removing a field, changing a type, or adding a new required key means creating payout-requested.v2.schema.json and supporting both v1 and v2 in the consumer until v1 traffic drains to zero.
  2. Automate the backwards-compatibility check. We added a 45-line CI script (scripts/check-schema-backcompat.ts) that diffs every modified *.schema.json file against origin/main and fails the PR if a required field was added or a property's type changed:
import { execFileSync } from 'node:child_process'; import fs from 'node:fs'; interface JsonSchema { required?: string[]; properties?: Record<string, { type?: string }>; } const changedFiles = execFileSync( 'git', ['diff', '--name-only', 'origin/main...HEAD', '--', 'contracts/queues/*.schema.json'], { encoding: 'utf8' } ) .trim() .split('\n') .filter(Boolean); for (const file of changedFiles) { let oldRaw = ''; try { oldRaw = execFileSync('git', ['show', `origin/main:${file}`], { encoding: 'utf8' }); } catch { // Brand new schema version (e.g., v2) — always allowed continue; } const prev = JSON.parse(oldRaw) as JsonSchema; const next = JSON.parse(fs.readFileSync(file, 'utf8')) as JsonSchema; const prevReq = new Set(prev.required || []); for (const key of next.required || []) { if (!prevReq.has(key)) { console.error(`BREAKING: Added required field "${key}" to ${file}. Bump to a new version instead.`); process.exit(1); } } for (const [prop, def] of Object.entries(prev.properties || {})) { const nextDef = next.properties?.[prop]; if (!nextDef || nextDef.type !== def.type) { console.error(`BREAKING: Property "${prop}" changed or removed in ${file}. Bump schema version.`); process.exit(1); } } } console.log('All modified queue schemas are backwards-compatible.'); 

Replaying 4,819 Dead-Letter Messages Without Double-Paying

One last thing nobody tells you until you actually have to drain a 4,819-message financial DLQ: you cannot just click "Start DLQ Redrive" in the AWS Console and walk away.

During the 19-hour outage, our finance team had already manually wired payouts for 140 urgent enterprise merchants via the Stripe dashboard. If we had blindly redriven all 4,819 messages back into the main queue, those 140 merchants would have been paid twice.

Before replaying a single message, we added an INSERT INTO processed_queue_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING idempotency guard backed by Postgres, pre-seeded it with the 140 manually settled eventIds, and ran a small redrive script that patched "payoutCents" from string to integer on the fly while logging every skipped duplicate.

We tested the redrive script with --dry-run on the first 50 DLQ messages, verified the Stripe idempotency keys matched, and then drained the remaining 4,769 messages in batches of 100. The whole replay finished in 4 minutes and 40 seconds across 20 concurrent workers, with zero double payouts and zero manual spreadsheet reconciliation afterward.

That idempotency table stayed in payout-worker permanently—because SQS is at-least-once delivery anyway, and sooner or later every distributed queue delivers the same message twice.

The Quiet Side Effect Nobody Mentions

Once we put JSON Schema contracts and LocalStack round-trip tests on our SQS queues, we expected fewer production outages. And yes, we have had zero schema-drift DLQ incidents in the five months since.

What surprised me was how much faster onboarding got. New engineers stopped having to grep through three Go repositories to figure out what shape an SQS event actually had—they just opened contracts/queues/*.schema.json and read the exact contract in ten seconds.

If your test suite checks every REST route twice while your SQS, Kafka, or BullMQ workers still rely on JSON.parse(...) as MyType and hand-written test fixtures, you are only testing half your system. The other half is one struct tag away from a very quiet, very expensive pileup.