With the Step Functions Distributed Map you process huge amounts of data from S3 in parallel without building your own queues, workers or progress counters. I care about this topic because I keep seeing the same improvised setup in serverless projects: one Lambda reads a large CSV file, pushes the rows to SQS, a worker handles them, and in the end nobody really knows whether every row made it.
That is exactly the gap the Distributed Map closes. You only describe which file to read, how big the chunks are and how many of them may run at once. Step Functions takes care of everything else, including a failure threshold and a result report in S3. In this article you build the whole thing once: with the Serverless Framework, TypeScript, tests and an example you can run in your own AWS account in about half an hour.
In short
For every batch of a large S3 file, the Distributed Map starts its own child workflow and thus processes hundreds of thousands of rows in parallel. The most important setting is MaxConcurrency, because it protects your database and third-party systems from your own parallelism. The most common pitfall is missing IAM permissions for starting the child workflows and for the results folder.
What you will learn in this article
- When the regular Map is no longer enough and the Distributed Map makes sense
- How ItemReader, ItemBatcher, MaxConcurrency and ResultWriter work together
- A complete CSV import with the Serverless Framework and TypeScript
- Which IAM permissions the state machine really needs
- How to keep failures, costs and reruns under control
When is the regular Map no longer enough?
Step Functions offers the Map state in two flavors. The inline Map runs inside the execution that started it. That is convenient but comes with tight limits: the input has to fit into the execution payload (256 KiB), every step ends up in one shared execution history capped at 25,000 entries, and you cannot go beyond 40 parallel iterations.
The Distributed Map removes these limits. Every iteration is a separate child workflow execution with its own history. The data comes straight from S3, for example from a CSV, JSON, JSON Lines or Parquet file or from a list of S3 objects. Without a limit, Step Functions starts up to 10,000 child workflows at the same time.
AWS names three signals that tell you it is time to switch. I use them as a rule of thumb:
- Your dataset is bigger than 256 KiB.
- A single execution history would grow past 25,000 entries.
- You need more than 40 iterations running at once.
If none of these applies, stick with the inline Map. It is easier to read, and you avoid the extra IAM permissions we will get to in a moment.
How the Step Functions Distributed Map works
Before we write code, let’s look at the terms. You will see them all the time in the console, in the docs and in error messages.
| Term | What it means |
|---|---|
| Parent workflow | The state machine that contains the Map state. It has to be of type Standard. |
| Child workflow | A single run for one item or one batch. It can be Standard or Express. |
| Map Run | All child workflows of one Map invocation, with its own ARN and its own view in the console. |
| ItemReader | Defines where the data comes from, for example s3:getObject for a CSV file. |
| ItemBatcher | Groups several items into a batch so that not every row starts its own child workflow. |
| ResultWriter | Writes the results of all child workflows to S3 in bulk instead of putting them into the output. |
Express workflows are a cheaper, faster flavor of Step Functions for short-lived processes. They may run for at most five minutes and are billed by count and duration. That is why they almost always fit child workflows that only hand a batch over to a Lambda function.
Catalog import: the Distributed Map reads the CSV from S3 and distributes batches to up to 20 child workflows.
The diagram shows the setup we are about to build. The parent workflow reads the file, splits it into batches and starts at most 20 child workflows at once. Each one calls the same Lambda function, which validates its 500 rows and writes them to DynamoDB. At the end, the ResultWriter stores a report in S3.
🎬 Flow of a Step Functions Distributed Map – this animated diagram only runs on my site: watch it on sascha-ms.de
The example: a catalog import for a city library
Picture a city library that receives an export from its union catalog every night. The file is a CSV that uses semicolons as delimiter, the way many Excel exports do, and it contains a few hundred thousand rows with ISBN, title, author, publication year and number of copies.
Every row needs to be validated: Is the ISBN check digit correct? Is there a title? Is the year plausible? Valid rows go into a DynamoDB table, invalid ones should show up in the report so someone can fix them. A single typo in the export must never abort the whole import.
A quick note up front: if you just want to dump data from S3 into an existing table without any logic, take a look at DynamoDB Bulk Put Items. The Distributed Map shines as soon as every row needs its own code, such as a validation, an enrichment or a call to a third-party system.
Prerequisites
- An AWS account and configured credentials for the AWS CLI
- Node.js 22 or newer and npm
- Serverless Framework v4 (
npm i -g serverless), logged in withserverless login - Basic knowledge of Lambda and Step Functions, for example from the article on the saga pattern
Step by step to a Step Functions Distributed Map
Step 1: Create the project
Create an empty folder and install the dependencies. For Step Functions I use the serverless-step-functions plugin, which lets you describe the state machine directly in serverless.yml.
mkdir katalog-import && cd katalog-import npm init -y npm i @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb npm i -D typescript vitest tsx @types/node @types/aws-lambda serverless-step-functions You will find the complete package.json and tsconfig.json in the example repository linked at the end of the article.
Step 2: Validate a row
The business validation lives in its own module without any AWS dependencies. The reason is simple: you can test it in milliseconds. Step Functions hands over every CSV row as an object whose values are all strings. The column names come from the first line of the file.
// src/validate-row.ts // Validates a CSV row from the catalog export and turns it into a record. // Step Functions delivers each row as an object with string values. export interface RawRow { isbn?: string; title?: string; author?: string; year?: string; copies?: string; } export interface Book { isbn: string; title: string; author: string; year: number; copies: number; } export type ValidationResult = | { ok: true; book: Book } | { ok: false; reason: string }; // ISBN-13: strip hyphens and spaces, then verify the check digit export function normalizeIsbn(value: string): string | null { const digits = value.replace(/[-\s]/g, ''); if (!/^97[89]\d{10}$/.test(digits)) return null; const sum = digits .slice(0, 12) .split('') .reduce((acc, d, i) => acc + Number(d) * (i % 2 === 0 ? 1 : 3), 0); const check = (10 - (sum % 10)) % 10; return check === Number(digits[12]) ? digits : null; } export function validateRow(row: RawRow, currentYear = new Date().getFullYear()): ValidationResult { const isbn = normalizeIsbn(row.isbn ?? ''); if (!isbn) return { ok: false, reason: 'invalid ISBN' }; const title = (row.title ?? '').trim(); if (title.length === 0) return { ok: false, reason: 'missing title' }; const year = Number(row.year); if (!Number.isInteger(year) || year < 1450 || year > currentYear + 1) { return { ok: false, reason: 'implausible publication year' }; } const copies = Number(row.copies); if (!Number.isInteger(copies) || copies < 0) { return { ok: false, reason: 'copies is not a number ≥ 0' }; } return { ok: true, book: { isbn, title, author: (row.author ?? '').trim(), year, copies }, }; } The return type matters here. Instead of throwing an exception, the function returns either a finished record or a reason for rejecting the row. That way the caller decides what happens to broken rows.
Step 3: The Lambda function for one batch
With ItemBatcher enabled, the Lambda function does not receive a single item but an object with an Items field. On top of that, a batch can carry fixed values in BatchInput. The function validates all rows, writes the valid ones to DynamoDB in chunks of 25 items and returns the rejected ones.
// src/import-batch.ts // Called once per batch by a child workflow of the Distributed Map. import { DynamoDBClient } from '@aws-sdk/client-dynamodb'; import { BatchWriteCommand, DynamoDBDocumentClient } from '@aws-sdk/lib-dynamodb'; import { validateRow, type Book, type RawRow } from './validate-row.js'; const TABLE_NAME = process.env.TABLE_NAME ?? ''; const doc = DynamoDBDocumentClient.from(new DynamoDBClient({})); // This is what the input looks like when ItemBatcher is enabled export interface BatchEvent { Items: RawRow[]; BatchInput?: { source?: string }; } export interface BatchResult { written: number; rejected: { isbn: string; reason: string }[]; } const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); // BatchWriteItem accepts at most 25 items; resend unprocessed items async function writeChunk(books: Book[], source: string): Promise<void> { let pending: Record<string, unknown>[] = books.map((book) => ({ PutRequest: { Item: { ...book, source, importedAt: new Date().toISOString() } }, })); for (let attempt = 0; pending.length > 0; attempt++) { if (attempt >= 5) { // Hands the batch back to Step Functions: retry or failure threshold applies throw new Error(`${pending.length} items not written after 5 attempts`); } if (attempt > 0) await sleep(Math.random() * 100 * 2 ** attempt); const response = await doc.send( new BatchWriteCommand({ RequestItems: { [TABLE_NAME]: pending } }), ); pending = (response.UnprocessedItems?.[TABLE_NAME] ?? []) as Record<string, unknown>[]; } } export const handler = async (event: BatchEvent): Promise<BatchResult> => { const source = event.BatchInput?.source ?? 'unknown'; const valid: Book[] = []; const rejected: BatchResult['rejected'] = []; for (const row of event.Items) { const result = validateRow(row); if (result.ok) valid.push(result.book); else rejected.push({ isbn: row.isbn ?? '', reason: result.reason }); } for (let i = 0; i < valid.length; i += 25) { await writeChunk(valid.slice(i, i + 25), source); } return { written: valid.length, rejected }; }; Two details matter to me here. First, BatchWriteItem may return some items as unprocessed when the table throttles. The loop resends them with a growing, randomized delay, as described in the article on exponential backoff with jitter. Second, business errors (a broken ISBN) do not cause an exception, while technical errors (DynamoDB unreachable for good) do. Only the second kind should make a child workflow fail.
Step 4: The state machine in serverless.yml
Now for the core. The definition looks longer than it is, because most of the lines are resources and permissions. Read through it calmly once; the explanation follows right after.
# serverless.yml service: katalog-import frameworkVersion: '4' plugins: - serverless-step-functions provider: name: aws runtime: nodejs24.x architecture: arm64 region: eu-central-1 memorySize: 512 timeout: 60 environment: TABLE_NAME: !Ref BooksTable iam: role: statements: - Effect: Allow Action: dynamodb:BatchWriteItem Resource: !GetAtt BooksTable.Arn functions: importBatch: handler: src/import-batch.handler stepFunctions: validate: true stateMachines: katalogImport: id: KatalogImportStateMachine name: ${self:service}-${sls:stage}-katalog-import role: Fn::GetAtt: [StateMachineRole, Arn] definition: Comment: Reads a catalog CSV from S3 and imports it in batches StartAt: KatalogImportieren States: KatalogImportieren: Type: Map Label: KatalogImport MaxConcurrency: 20 ToleratedFailurePercentage: 5 ItemReader: Resource: arn:aws:states:::s3:getObject ReaderConfig: InputType: CSV CSVHeaderLocation: FIRST_ROW CSVDelimiter: SEMICOLON Parameters: Bucket: Ref: ImportBucket Key.$: $.key ItemBatcher: MaxItemsPerBatch: 500 BatchInput: source: verbundkatalog ItemProcessor: ProcessorConfig: Mode: DISTRIBUTED ExecutionType: EXPRESS StartAt: BatchSpeichern States: BatchSpeichern: Type: Task Resource: arn:aws:states:::lambda:invoke Parameters: FunctionName: Fn::GetAtt: [ImportBatchLambdaFunction, Arn] Payload.$: $ OutputPath: $.Payload Retry: - ErrorEquals: - Lambda.TooManyRequestsException - Lambda.ServiceException - Lambda.AWSLambdaException - Lambda.SdkClientException IntervalSeconds: 2 MaxAttempts: 4 BackoffRate: 2 JitterStrategy: FULL End: true ResultWriter: Resource: arn:aws:states:::s3:putObject Parameters: Bucket: Ref: ImportBucket Prefix: ergebnisse End: true resources: Resources: ImportBucket: Type: AWS::S3::Bucket Properties: PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true LifecycleConfiguration: Rules: - Id: ErgebnisseAufraeumen Status: Enabled Prefix: ergebnisse/ ExpirationInDays: 30 BooksTable: Type: AWS::DynamoDB::Table Properties: BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: isbn AttributeType: S KeySchema: - AttributeName: isbn KeyType: HASH StateMachineRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: states.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: katalog-import PolicyDocument: Version: '2012-10-17' Statement: # Start and observe the child workflows of the Distributed Map - Effect: Allow Action: states:StartExecution Resource: arn:aws:states:${aws:region}:${aws:accountId}:stateMachine:${self:service}-${sls:stage}-katalog-import - Effect: Allow Action: states:DescribeExecution Resource: - arn:aws:states:${aws:region}:${aws:accountId}:execution:${self:service}-${sls:stage}-katalog-import:* - arn:aws:states:${aws:region}:${aws:accountId}:execution:${self:service}-${sls:stage}-katalog-import/* # Restart failed runs (redrive) - Effect: Allow Action: states:RedriveExecution Resource: arn:aws:states:${aws:region}:${aws:accountId}:execution:${self:service}-${sls:stage}-katalog-import/KatalogImport:* # Read the CSV - Effect: Allow Action: s3:GetObject Resource: !Join ['', [!GetAtt ImportBucket.Arn, '/eingang/*']] # Write the results - Effect: Allow Action: - s3:PutObject - s3:GetObject - s3:ListMultipartUploadParts - s3:AbortMultipartUpload Resource: !Join ['', [!GetAtt ImportBucket.Arn, '/ergebnisse/*']] - Effect: Allow Action: lambda:InvokeFunction Resource: - !GetAtt ImportBatchLambdaFunction.Arn - !Join ['', [!GetAtt ImportBatchLambdaFunction.Arn, ':*']] Outputs: ImportBucketName: Value: !Ref ImportBucket StateMachineArn: Value: !Ref KatalogImportStateMachine The most important fields of the Map state in a nutshell:
-
ItemReaderreads the file withs3:getObject.CSVHeaderLocation: FIRST_ROWuses the first line as column names, andCSVDelimiter: SEMICOLONmakes sure the Excel export is split correctly. -
ItemBatcherbundles 500 rows per child workflow. In addition, a batch may be at most 256 KiB. If it gets bigger, Step Functions automatically puts fewer rows into it. -
MaxConcurrency: 20caps the parallelism. Without this line, Step Functions starts up to 10,000 child workflows at the same time, and your table or a third-party system will feel it. -
ToleratedFailurePercentage: 5keeps the run going as long as at most 5% of the items fail. Above that, the Map Run stops withStates.ExceedToleratedFailureThreshold. -
ResultWriterwrites all results toergebnisse/in the same bucket instead of cramming them into the execution output.
The retry deliberately sits in the task inside the child workflow and not on the Map state itself. According to the docs, a retry on the Map state would start a completely new Map Run and repeat all child workflows, including the successful ones.

