Setting up a dead-letter queue with CDK takes three lines: deadLetterQueue: { queue, maxReceiveCount }, done. The real work starts afterwards. Who notices when the DLQ fills up? What happens to a message that will never work? And who moves the others back once the billing API is running again after an hour-long outage? In many projects the honest answer is: somebody, at some point, by hand.
In my first article on the SQS dead-letter queue I built this with the Serverless Framework: queue, DLQ, alarm and a redrive script. Today we go one step further. You wrap everything in a reusable AWS CDK construct, separate transient from permanent errors and let the DLQ move back automatically on a schedule. To keep this from turning into an endless loop, the most important ingredient comes before the schedule: clean error classification in the consumer.
In short
A custom CDK construct bundles queue, DLQ, alarms and a scheduled redrive via EventBridge Scheduler, so every new queue gets the same safety nets. Automatic redrive is only safe, however, if the DLQ contains nothing but transient failures. That is why the consumer parks permanently broken messages in a separate queue; otherwise they circle until their retention period runs out.
What you will learn in this article
- Wrap queue, DLQ and alarms in a custom CDK construct
- Separate transient and permanent errors in the Lambda consumer
- Schedule the DLQ redrive with EventBridge Scheduler and a universal target
- Protect the construct with Vitest and template assertions
- Recognize when automatic redrive does more harm than good
Why build a dead-letter queue with CDK as a construct?
In the AWS CDK, a construct is a building block that combines several resources into one unit. You call it with a few parameters and get a finished package back. That sounds like convenience, but for queues it is mostly a matter of discipline. The rules for a good DLQ are well known, and yet the third queue in a project suddenly lacks the alarm, or its DLQ keeps messages just as briefly as the source queue.
A construct for the dead-letter queue with CDK defines these rules once. I would rather write 150 lines of tested TypeScript than copy the same 40 lines of configuration five times. You can also tighten the rules centrally later, for example by adding a new alarm, and every queue gets it on the next deployment.
Our example is a fictional bike rental. Every return puts an event into an SQS queue, and a Lambda function sends the rental duration to a billing API. The construct is called QueueWithDlq and provides:
- a source queue with a suitable visibility timeout and redrive policy,
- a DLQ with 14 days of retention,
- a parking queue for permanently broken messages,
- three CloudWatch alarms, optionally with SNS notifications,
- on request, a scheduled redrive via EventBridge Scheduler.
Separating transient and permanent errors
The classic DLQ makes no distinction between two very different situations. In the first case, the billing API is down for an hour. Every message is fine; it just arrived at the wrong time. In the second case, the message lacks a bikeId. That message will not work in three days either, no matter how often you try.
As long as both sit in the same DLQ, you cannot redrive automatically. A redrive would take the broken messages along; they fail five times, move back to the DLQ and wait for the next redrive. This is known as a “poison message loop”, and it costs Lambda invocations, clutters logs and hides real problems.
The solution lives in the consumer. For every error it decides whether a later attempt can change anything. If not, it sends the message to a parking queue and reports it as processed successfully, so SQS deletes it from the source queue. If yes, it reports it as failed via the partial batch response. That is a response from the Lambda function listing only the failed messages of a batch, so the successful ones are not retried.
The consumer decides: retry via the DLQ or park right away.
How do you recognize a transient error?
A simple rule of thumb: anything that depends on the state of another system is transient. Anything that depends only on the message itself is permanent. For HTTP calls, that means:
| Error | Type | Destination |
|---|---|---|
| Timeout, network error | transient | retry, then DLQ |
| HTTP 429 (too many requests) | transient | retry, then DLQ |
| HTTP 500 to 599 | transient | retry, then DLQ |
| Invalid JSON, required field missing | permanent | parking queue |
| HTTP 400, 404, 422 | permanent | parking queue |
There are still edge cases. An HTTP 409 (conflict) can mean that another process is working on the same record right now, in which case waiting helps. It can also mean that the booking already exists. Such codes are best clarified with the team that runs the API instead of guessing.
How does the automatic redrive work?
Since SQS offers the StartMessageMoveTask API, you no longer need a custom script that reads messages and sends them again. You pass the ARN of the DLQ, and SQS moves the messages back asynchronously. Without a destination ARN they land in the queue they came from. With MaxNumberOfMessagesPerSecond you throttle the pace; the upper limit is 500 messages per second. In addition, only one move task can run per queue at a time.
The call is made by EventBridge Scheduler. This service starts targets on a schedule and, through so-called universal targets, can call almost any AWS API directly without you writing a Lambda function in between. The target ARN follows the pattern arn:aws:scheduler:::aws-sdk:sqs:startMessageMoveTask, and the parameters go into the input as JSON.
🎬 Automatic redrive of an SQS DLQ – this animated diagram only runs on my site: watch it on sascha-ms.de
So the cycle looks like this: a message fails five times, moves to the DLQ, and the alarm goes off. At most an hour later, the scheduler starts the move task, the message lands in the source queue again, and the Lambda function tries once more. If the API is back, the matter is settled without anyone having to get up at night.
Step by step: build a dead-letter queue with CDK
Prerequisites
- Node.js 22 and npm
- An AWS account with credentials configured for the AWS CLI
- An account and region prepared with
npx cdk bootstrap - Basic knowledge of TypeScript and SQS (the first part explains the terms)
I tested the code with aws-cdk-lib 2.276.0, CDK CLI 2.1148.1, TypeScript 7.0.2 and Vitest 5.1.0. Because TypeScript 7 no longer supports ts-node, the CDK app starts via tsx.
Step 1: Create the project
mkdir rental-returns && cd rental-returns mkdir bin lib src test npm init -y npm install aws-cdk-lib constructs @aws-sdk/client-sqs npm install -D aws-cdk typescript tsx esbuild vitest @types/node @types/aws-lambda Add "type": "module" to your package.json along with the scripts "test": "vitest run" and "build": "tsc --noEmit". Then add two small configuration files:
{ "compilerOptions": { "target": "ES2022", "module": "nodenext", "moduleResolution": "nodenext", "strict": true, "skipLibCheck": true, "noEmit": true, "types": ["node"] }, "include": ["bin", "lib", "src", "test"] } { "app": "npx tsx bin/app.ts" } Step 2: Write the QueueWithDlq construct
This is the heart of your dead-letter queue with CDK. Read the code first; I explain the important decisions right below it.
// lib/queue-with-dlq.ts import { Duration } from 'aws-cdk-lib'; import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch'; import * as cwActions from 'aws-cdk-lib/aws-cloudwatch-actions'; import * as iam from 'aws-cdk-lib/aws-iam'; import * as lambda from 'aws-cdk-lib/aws-lambda'; import { SqsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources'; import * as scheduler from 'aws-cdk-lib/aws-scheduler'; import * as targets from 'aws-cdk-lib/aws-scheduler-targets'; import * as sns from 'aws-cdk-lib/aws-sns'; import * as sqs from 'aws-cdk-lib/aws-sqs'; import { Construct } from 'constructs'; export interface AutoRedriveOptions { /** When the redrive starts, e.g. ScheduleExpression.rate(Duration.hours(1)) */ readonly schedule: scheduler.ScheduleExpression; /** Throttling for the redrive (1 to 500 messages per second) */ readonly maxMessagesPerSecond?: number; } export interface QueueWithDlqProps { /** The Lambda function that processes the queue */ readonly consumer: lambda.Function; /** Function timeout, used to calculate the visibility timeout */ readonly consumerTimeout: Duration; /** Attempts before a message moves to the DLQ (default: 5) */ readonly maxReceiveCount?: number; /** Age of the oldest message that triggers the backlog alarm (default: 15 minutes) */ readonly maxMessageAge?: Duration; /** Optional: SNS topic for all alarms */ readonly alarmTopic?: sns.ITopic; /** Optional: automatic redrive, meant for transient errors only */ readonly autoRedrive?: AutoRedriveOptions; } export class QueueWithDlq extends Construct { public readonly queue: sqs.Queue; public readonly deadLetterQueue: sqs.Queue; public readonly parkingQueue: sqs.Queue; public readonly alarms: cloudwatch.Alarm[] = []; constructor(scope: Construct, id: string, props: QueueWithDlqProps) { super(scope, id); const maxReceiveCount = props.maxReceiveCount ?? 5; const rate = props.autoRedrive?.maxMessagesPerSecond ?? 10; if (maxReceiveCount < 2) { throw new Error('maxReceiveCount must be at least 2, otherwise a single timeout sends a message to the DLQ.'); } if (rate < 1 || rate > 500) { throw new Error('maxMessagesPerSecond must be between 1 and 500.'); } // DLQ: longest retention, because the age from the source queue carries over this.deadLetterQueue = new sqs.Queue(this, 'Dlq', { retentionPeriod: Duration.days(14), encryption: sqs.QueueEncryption.SQS_MANAGED, enforceSSL: true, }); // Parking spot for permanently broken messages: no redrive, analysis only this.parkingQueue = new sqs.Queue(this, 'Parking', { retentionPeriod: Duration.days(14), encryption: sqs.QueueEncryption.SQS_MANAGED, enforceSSL: true, }); this.queue = new sqs.Queue(this, 'Queue', { // Recommendation for Lambda: at least six times the function timeout visibilityTimeout: Duration.seconds(props.consumerTimeout.toSeconds() * 6), retentionPeriod: Duration.days(4), encryption: sqs.QueueEncryption.SQS_MANAGED, enforceSSL: true, deadLetterQueue: { queue: this.deadLetterQueue, maxReceiveCount }, }); props.consumer.addEventSource( new SqsEventSource(this.queue, { batchSize: 10, reportBatchItemFailures: true, }), ); this.parkingQueue.grantSendMessages(props.consumer); props.consumer.addEnvironment('PARKING_QUEUE_URL', this.parkingQueue.queueUrl); this.addAlarm('DlqNotEmpty', { metric: this.deadLetterQueue.metricApproximateNumberOfMessagesVisible({ period: Duration.minutes(1), statistic: 'Maximum', }), threshold: 1, alarmDescription: 'Messages in the DLQ: processing failed repeatedly.', }, props.alarmTopic); this.addAlarm('ParkingNotEmpty', { metric: this.parkingQueue.metricApproximateNumberOfMessagesVisible({ period: Duration.minutes(5), statistic: 'Maximum', }), threshold: 1, alarmDescription: 'Permanently broken messages parked: please take a look.', }, props.alarmTopic); this.addAlarm('QueueBacklog', { metric: this.queue.metricApproximateAgeOfOldestMessage({ period: Duration.minutes(1), statistic: 'Maximum', }), threshold: (props.maxMessageAge ?? Duration.minutes(15)).toSeconds(), evaluationPeriods: 5, alarmDescription: 'The oldest message has been waiting too long: consumer too slow or failing.', }, props.alarmTopic); if (props.autoRedrive) { new scheduler.Schedule(this, 'AutoRedrive', { description: 'Moves the DLQ back to the source queue on a schedule', schedule: props.autoRedrive.schedule, target: new targets.Universal({ service: 'sqs', action: 'startMessageMoveTask', // Without DestinationArn, messages go back to their source queue input: scheduler.ScheduleTargetInput.fromObject({ SourceArn: this.deadLetterQueue.queueArn, MaxNumberOfMessagesPerSecond: rate, }), retryAttempts: 0, policyStatements: [ new iam.PolicyStatement({ actions: [ 'sqs:StartMessageMoveTask', 'sqs:ReceiveMessage', 'sqs:DeleteMessage', 'sqs:GetQueueAttributes', ], resources: [this.deadLetterQueue.queueArn], }), new iam.PolicyStatement({ actions: ['sqs:SendMessage'], resources: [this.queue.queueArn], }), ], }), }); } } private addAlarm( id: string, props: Omit<cloudwatch.AlarmProps, 'evaluationPeriods'> & { evaluationPeriods?: number }, topic?: sns.ITopic, ): void { const alarm = new cloudwatch.Alarm(this, id, { comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_OR_EQUAL_TO_THRESHOLD, treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING, ...props, evaluationPeriods: props.evaluationPeriods ?? 1, }); if (topic) { alarm.addAlarmAction(new cwActions.SnsAction(topic)); alarm.addOkAction(new cwActions.SnsAction(topic)); } this.alarms.push(alarm); } } Some values are deliberately not configurable. The construct sets the visibility timeout to six times the function timeout, as AWS recommends for SQS with Lambda. The buffer covers cases where Lambda is throttled while a batch is still running. The DLQ keeps messages for 14 days, the SQS maximum. With standard queues, a message keeps its original timestamp when it moves to the DLQ. A DLQ with the same retention as the source would therefore lose messages sooner than you think.
Also note policyStatements on the universal target. Without it, CDK grants the scheduler role only the sqs:StartMessageMoveTask permission. That is not enough for a redrive: according to the SQS documentation, the role also needs ReceiveMessage, DeleteMessage and GetQueueAttributes on the DLQ plus SendMessage on the destination queue. If you encrypt with your own KMS key, KMS permissions come on top. The construct uses SQS-managed encryption and skips that part.
retryAttempts: 0 prevents the scheduler from retrying a failed call several times. If, for example, a move task from the last hour is still running, SQS rejects the new one, and that is fine. The next slot comes anyway.
Step 3: Consumer with error classes
The Lambda function distinguishes two kinds of errors. A custom PermanentError class marks everything that would fail the same way on the next attempt. Any other error counts as transient.
// src/return-consumer.ts import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs'; import type { SQSBatchResponse, SQSEvent, SQSRecord } from 'aws-lambda'; const sqs = new SQSClient({}); /** Won't turn out differently on the next attempt: park instead of retrying */ export class PermanentError extends Error {} interface RentalReturn { rentalId: string; bikeId: string; minutes: number; } export function parseReturn(body: string): RentalReturn { let data: unknown; try { data = JSON.parse(body); } catch { throw new PermanentError('Not valid JSON'); } const r = data as Partial<RentalReturn>; if (typeof r.rentalId !== 'string' || typeof r.bikeId !== 'string') { throw new PermanentError('rentalId or bikeId missing'); } if (typeof r.minutes !== 'number' || r.minutes <= 0) { throw new PermanentError('minutes must be a positive number'); } return { rentalId: r.rentalId, bikeId: r.bikeId, minutes: r.minutes }; } export async function chargeRental(ret: RentalReturn): Promise<void> { const res = await fetch(process.env.BILLING_URL!, { method: 'POST', headers: { 'content-type': 'application/json', // Same key on every attempt: the API charges only once 'idempotency-key': ret.rentalId, }, body: JSON.stringify(ret), signal: AbortSignal.timeout(5000), }); if (res.ok) return; // 429 and 5xx are transient, other 4xx stay wrong on the next attempt too if (res.status === 429 || res.status >= 500) { throw new Error(`Billing temporarily unavailable: HTTP ${res.status}`); } throw new PermanentError(`Billing rejected: HTTP ${res.status}`); } async function park(record: SQSRecord, reason: string): Promise<void> { await sqs.send( new SendMessageCommand({ QueueUrl: process.env.PARKING_QUEUE_URL!, MessageBody: record.body, MessageAttributes: { reason: { DataType: 'String', StringValue: reason.slice(0, 256) }, originalMessageId: { DataType: 'String', StringValue: record.messageId }, }, }), ); } export async function handler(event: SQSEvent): Promise<SQSBatchResponse> { const batchItemFailures: SQSBatchResponse['batchItemFailures'] Source：https://dev.to/sascha-ms/dead-letter-queue-with-cdk-alarms-and-automatic-redrive-1cg3 Related posts
Buburu: A Women's Safety App Built in 48 Hours, Born From a Walk We Never Forgot.
We built a talking stethoscope in a weekend. The hardest bug was the word "Great!"
Turning noisy bathroom-scale data into a projected date: SMA vs. EMA vs. linear regression
Ctrl+A Only Copied a Third of the Page, So I Built a Chrome Extension
ไฟล์ตั้งค่า 104,185 ไฟล์เปิดสาธารณะบน GitHub: Claude Code รั่วความลับอย่างไร?
