I've just open-sourced dynamoat, a server that speaks the DynamoDB API and stores everything in PostgreSQL. You point an AWS SDK at it by changing one environment variable:

export AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8000

and your DynamoDB code runs against a Postgres database you control. This post is about why I built it, how DynamoDB's data model maps onto Postgres, and how I know the answers it gives match the real thing.

Why

We run a hosted product on DynamoDB. It creates a table per tenant, each with several secondary indexes, and the application code is written against the DynamoDB API. Three things pushed me to build an alternative backend:

  • Local development. DynamoDB Local works, but it's a black box: you can't look at the data with normal SQL tools, and it diverges from the real service in places.
  • Customers who can't use AWS. Some want the product on their own infrastructure. Rewriting the data layer for them wasn't an option.
  • Lock-in. Having a second backend that works in the same way means the API is the dependency, not the cloud vendor.

Rather than add a storage abstraction to the application, I made the storage look like DynamoDB. The application doesn't know the difference.

What it is

One Go binary. It serves the DynamoDB wire protocol (JSON over HTTP, routed by the X-Amz-Target header) and stores data in Postgres 15–18. No extensions, no superuser features, so it works on any managed Postgres. The process is stateless, so you run several copies behind a load balancer and let the database do the hard part.

Request signing (SigV4) is verified by default, so existing AWS credentials and SDK configuration work unchanged. There's a none mode for local use.

Covered: the data plane (Get, Put, Update, Delete, Query, Scan, the batch operations), tables with local and global secondary indexes, condition and update expressions, transactions, PartiQL, tags, TTL, on-demand backups, point-in-time recovery and the Streams API.

How DynamoDB maps onto Postgres

The obvious design, a Postgres table per DynamoDB table, doesn't survive contact with a table-per-tenant application. Creating seven indexes' worth of DDL every time a customer signs up doesn't scale, and neither does a database with tens of thousands of tables.

So a DynamoDB table is a row in a catalog table, and every item of every table lives in one items table:

column meaning
table_id which DynamoDB table this item belongs to
pk_hash a 32-bit hash of the partition key
pk, sk the partition and sort keys, encoded as bytea
item the whole item as DynamoDB's typed JSON
ttl_at the TTL attribute as a timestamp, for the expiry sweeper

Global secondary indexes are key-only rows in a second table, joined back to items for the projection. That's roughly how DynamoDB itself does it.

Getting the sort order right

The interesting problem is the sort key. DynamoDB sorts strings by UTF-8 byte value, binary by raw bytes, and numbers numerically, with 38 digits of precision. Postgres compares bytea byte by byte. So the key encoding has to be order-preserving: encoding two values and comparing the bytes must give the same answer as DynamoDB comparing the originals.

Strings and binary are easy: the bytes already sort correctly. Numbers aren't. DynamoDB numbers are decimals with up to 38 significant digits and an exponent from -130 to +125, which no native Postgres type matches exactly. dynamoat normalises each number to 0.d1d2...dn × 10^E and encodes it as a sign byte, the exponent, then each digit shifted by one so no digit encodes as 0x00, then a terminator. Negative numbers flip every component so they sort in reverse. The result is a bytea that sorts in numeric order, and Query with BETWEEN or begins_with on a sort key becomes an ordinary index range scan.

Writes and conditions

Each write is one Postgres transaction. It takes an advisory lock on the item's partition, reads the current item, evaluates any condition expression, writes the item, updates the index entries, and writes a stream record, all committed together. TransactWriteItems takes its locks in sorted key order, which is what stops two transactions deadlocking on each other.

Stream records get a little care: a record becomes visible to consumers only once every transaction that could have preceded it has finished. Without that, a consumer checkpointing the "last position seen" could skip a write that committed late.

How I know it matches

"Compatible" is the claim everyone will test first, so this is where most of the effort went.

The conformance suite is a set of TypeScript cases written against the AWS SDK v3. Each case runs against real DynamoDB with recording turned on, and the responses are normalised (timestamps zeroed, request IDs and pagination tokens replaced by placeholders, ARNs stripped) and saved as golden files. Then the same cases run against dynamoat, and every response is compared with the recording. The suite runs in CI on Postgres 15, 16, 17 and 18.

Recording against the real service taught me things the documentation doesn't say. A few examples:

  • A new table's stream has four shards.
  • Tags are eventually consistent: a TagResource takes about six seconds to show up, and a second tag operation during that window gets ResourceInUseException.
  • Enabling point-in-time recovery straight after CreateTable can fail with ContinuousBackupsUnavailableException. Both restorable times are set immediately, equal to the enable time.
  • Whether the last page of a Query carries a LastEvaluatedKey is nondeterministic.
  • Error messages have an exact shape, down to a 1 validation error detected: prefix on some of them, and the SDKs' retry logic depends on the __type field being spelled exactly right.

Each of those became either a change to dynamoat or a rule in the normaliser, and the recordings are in the repo, so anyone can see exactly what was compared.

What's missing

  • Global tables are bookkeeping. DescribeTable reports replicas so IaC tools and desktop clients are happy, but nothing is copied between regions. Point every instance at the same database, or replicate the database.
  • The 10 GB item collection limit isn't enforced. On a table with local secondary indexes, DynamoDB rejects a write when all the items sharing a partition key exceed 10 GB. dynamoat lets them grow.
  • No S3 import/export, Kinesis destinations, contributor insights or resource policies. PartiQL covers SELECT, INSERT, UPDATE and DELETE only.

Try it 

git clone https://github.com/toclocoinc/dynamoat cd dynamoat docker compose -f deploy/docker-compose.yml up -d aws --endpoint-url http://localhost:8000 --region us-east-1 dynamodb list-tables

That starts dynamoat and a Postgres 17 database on port 8000. There are also binaries for Linux and macOS on the releases page, a multi-arch image at ghcr.io/toclocoinc/dynamoat, and a Helm chart for Kubernetes.

It's Apache-2.0. If you try it against code you have and something differs from DynamoDB, an issue with the operation and the request is the most useful thing you can send me.