The short version

  • OpenAI runs ChatGPT on a single PostgreSQL primary that takes every write, plus nearly 50 read replicas across regions, and serves 800 million users without sharding Postgres.
  • Keep load off the primary: send reads to replicas and move shardable, write-heavy workloads to a sharded database. OpenAI no longer adds new tables to its Postgres.
  • On a cache miss, let one request per key take a lock and refill the cache from the database while the others wait, so a drop in the hit rate doesn't flood Postgres.
  • Changing a column's type rewrites the whole table. On a million rows it took 3.3 seconds, while adding a column with a default took 6 ms.
  • Set lock_timeout on migrations. An ALTER TABLE waiting on an open transaction blocks every query behind it, and in my test a one-row lookup waited 6.5 seconds.

OpenAI scales PostgreSQL for ChatGPT with one primary server that takes every write and nearly 50 read replicas spread across regions, and that handles millions of queries per second for 800 million users. Most of the work goes into keeping load off the primary, and into limits at every layer so one spike can't snowball into an outage.

Below is each problem they hit and how they fixed it, checked against OpenAI's own engineering post from January 2026, plus the parts I'd copy in a normal app, with SQL I tested on Postgres 18. The video above covers the same ground.

One primary, nearly 50 replicas

This came as a shock to me. I had no idea a single Postgres write instance could go that far. The primary is an Azure Database for PostgreSQL flexible server, and OpenAI says its Postgres load grew more than 10x in the past year.

They haven't sharded it, because sharding the existing workloads would mean changing hundreds of application endpoints and could take months or years. Instead they've drawn a line: no new tables go into this Postgres deployment, and new features default to sharded systems like Azure Cosmos DB.

Writes are the weak spot. Postgres uses multiversion concurrency control (MVCC), so updating even one field copies the whole row into a new version. Under heavy writes that means write amplification, dead rows that queries have to skip past, table and index bloat, and autovacuum work. Reads scale out across replicas. Writes all land on one machine.

How Postgres falls over

OpenAI says its Postgres outages usually follow the same loop. Something upstream causes a sudden jump in load. The cache layer fails and every request misses, or a new feature launch floods the database with writes. A burst of expensive multi-table joins eating the CPU can do it too. Queries slow down and start timing out. Clients retry, the retries add even more load, and it keeps getting worse until the whole service is down.

Every fix below breaks that loop somewhere.

Keep work off the primary

Reads go to replicas wherever possible. The only reads left on the primary are the ones inside write transactions, and those get tuned so they stay fast.

For writes, OpenAI moved the shardable, write-heavy workloads to Cosmos DB. The harder-to-shard ones are still being moved. Then they went through their own code. They fixed bugs that wrote the same data twice and added lazy writes to spread spikes out. Backfills got strict rate limits, so filling in a column can't hammer the primary (more on that below).

You can hunt for duplicate writes in a small app the same way. Turn on the pg_stat_statements extension, sort by how often each query runs, and look for writes that run more often than they should.

Fix the expensive queries

OpenAI found a query that joined 12 tables, and spikes in it caused some of their worst incidents. Their advice is to avoid multi-table joins where you can, and to break up the ones you need and do the join in application code instead.

They also say many of the bad queries came from ORMs. Prisma, Drizzle, SQLAlchemy and Django's ORM all make it easy to write code that turns into slow SQL without you noticing, so read the SQL your ORM generates for your hottest endpoints.

Then set a timeout for transactions that sit open doing nothing. OpenAI uses idle_in_transaction_session_timeout, because those sessions stop autovacuum from cleaning up dead rows:

ALTER DATABASE postgres SET idle_in_transaction_session_timeout = '60s'; 

Swap postgres for your database name. New connections pick it up. When I tested it with a 3-second value, a session that ran BEGIN and then sat idle got cut off with FATAL: terminating connection due to idle-in-transaction timeout.

Plan for the primary going down

A single writer is a single point of failure. OpenAI runs the primary in high-availability mode with a hot standby, a replica kept in sync that can be promoted quickly if the primary dies or needs maintenance. In the video I called this zero downtime, but the post only promises to keep downtime short.

The bigger fix is that most critical requests only read data, and those reads now go to replicas. If the primary goes down, writes fail but ChatGPT keeps answering reads, so it's no longer their worst severity of incident. They also run more than one replica in every region, so losing one replica doesn't take a region down.

Keep noisy workloads apart

Sometimes one workload, like a new feature with inefficient queries, eats most of the CPU and slows everything else on the same instance. OpenAI splits requests into high-priority and low-priority tiers and routes each tier to its own instances. A heavy low-priority job can then max out its own servers without touching high-priority traffic. They do the same between products, so one product's traffic can't hurt another.

Pool connections with PgBouncer

Each Azure PostgreSQL instance allows at most 5,000 connections, and OpenAI has had incidents where a connection storm used them all up. (Plain Postgres has no fixed number. max_connections is a setting, 100 by default.)

Their fix is PgBouncer in statement or transaction pooling mode, so many clients share a few real connections. In their benchmarks the average time to get a connection dropped from 50 ms to 5 ms. PgBouncer runs in the same region as the clients and the replica it sits in front of, and every read replica has its own Kubernetes deployment running several PgBouncer pods behind one Kubernetes Service.

To be fair, I haven't hit this limit myself, because I'm not serving hundreds of millions of users. But connections leak in ordinary apps too, through idle ones you forgot to close or one connection per request, so a pooler is worth having long before you're at OpenAI's size.

Stop cache-miss stampedes

OpenAI serves most reads from a cache. The danger is a sudden drop in the hit rate, because every miss turns into a database query at the same moment.

Their fix is a lock per cache key. When many requests miss on the same key, one takes the lock, reads from Postgres and refills the cache. The rest wait for the cache instead of all going to the database. I think this is genius, because the data only needs to come from the database once. If you're caching LLM answers rather than rows, I wrote about semantic caching separately.

Replicas cost the primary too

The primary streams its write-ahead log (WAL), the record of every change, to each replica. With nearly 50 replicas that's a lot of network and CPU on one machine, and replica lag gets worse as you add more.

The fix is cascading replication: the primary sends WAL to a few intermediate replicas, and they pass it on to the rest. OpenAI says this could take them past a hundred replicas. In the video I talked about it as if it's already running, but it isn't. They're still testing it with the Azure team, because it makes failover harder.

Rate limits everywhere, and slow retries

OpenAI rate-limits at the application, the connection pooler, the proxy and the query layer. They also changed their ORM so it can rate-limit or fully block a specific query pattern, which lets them cut off a runaway expensive query fast.

They also warn against short retry intervals, because those cause retry storms. The post doesn't give numbers, so here's what I'd do: wait 5 or 10 seconds before the first retry, back off to 30 or 60 seconds if it keeps failing, add some randomness so clients don't retry in sync, and after a few failures tell the user something's wrong and alert yourself.

Schema changes that don't take the site down

Some schema changes, like changing a column's type, make Postgres rewrite the whole table. OpenAI only allows lightweight changes, such as adding or removing certain columns. Creating and dropping indexes concurrently is allowed. Every schema change has a strict 5-second timeout. When they backfill a column, the rate limit is so tight that it sometimes takes over a week.

A week sounds absurd when you can change a schema in seconds on a small app. But it never causes downtime, and at their scale I think that's the right trade.

Check whether a change rewrites the table

Here's how to see it for yourself. Start with a table of a million rows:

CREATE TABLE users ( id bigint PRIMARY KEY, email text NOT NULL, age integer ); INSERT INTO users (id, email, age) SELECT g, 'user' || g || '@example.com', 20 + g % 50 FROM generate_series(1, 1000000) AS g; 

Then compare the table's file number before and after each change. A new number means Postgres wrote a whole new copy of the table:

\timing on SELECT pg_relation_filenode('users'); -- Metadata only: no rewrite ALTER TABLE users ADD COLUMN plan text DEFAULT 'free'; SELECT pg_relation_filenode('users'); -- Changing the type rewrites every row ALTER TABLE users ALTER COLUMN age TYPE bigint; SELECT pg_relation_filenode('users'); 

Run it with psql -f. Adding the column took 6 ms and the file number stayed at 16384. Changing age to bigint took 3.3 seconds and the number changed to 16394. That was on a million small rows on my machine. On a big production table it can run for minutes, and the table is locked the whole time.

Put a timeout on every migration

OpenAI doesn't say which timeout setting they use. I'd set two at the top of every migration:

SET lock_timeout = '2s'; SET statement_timeout = '5s'; ALTER TABLE users ADD COLUMN email_domain text; 

lock_timeout matters even for instant changes. An ALTER TABLE needs an exclusive lock, so it waits behind any open transaction on the table, and every query that arrives after it queues behind the ALTER. In my test, one transaction sat open on users and an ALTER TABLE without a timeout went into the queue. A one-row lookup by primary key then took 6.5 seconds. With the timeouts above, the migration gave up after 2 seconds with canceling statement due to lock timeout, and the same lookup took 1.7 seconds. You run the migration again later.

Backfill in batches

To fill the new column, update a few thousand rows at a time and pause between batches:

#!/usr/bin/env bash # Fill users.email_domain 5,000 rows at a time, with a pause between batches. set -euo pipefail max_id=$(psql "$DATABASE_URL" -qtAc "SELECT max(id) FROM users") for ((start = 1; start <= max_id; start += 5000)); do psql "$DATABASE_URL" -qc " UPDATE users SET email_domain = split_part(email, '@', 2) WHERE id >= $start AND id < $start + 5000 AND email_domain IS NULL" sleep 0.2 done echo "done up to id $max_id" 

Run it with your connection string in DATABASE_URL:

DATABASE_URL="postgresql://user:password@localhost:5432/postgres" ./backfill.sh 

It printed done up to id 1000000 after 72 seconds, 40 of them spent in the pauses. A single UPDATE did the same million rows in 9 seconds, but it locks every row until it commits and writes all its WAL in one burst that every replica then has to replay. Raise the sleep if your database is busy. OpenAI's backfills sometimes take over a week.

The results

OpenAI reports low double-digit millisecond p99 latency on the client side and five nines of availability. In the past 12 months they had one SEV-0 Postgres incident, during the ChatGPT ImageGen launch, when write traffic jumped more than 10x as over 100 million new users signed up in a week. They're still moving the remaining write-heavy workloads off Postgres, and they haven't ruled out sharding it later.

What I'd take from this

If you're a backend developer, this post is gold. Most of us reach for a new database the moment an app gets a few thousand users, and OpenAI is running ChatGPT on one Postgres primary. Most of their fixes work on a small app too. Reading the SQL your ORM writes and putting a timeout on every migration cost you an afternoon, and neither needs a new database. Go read their post too, because they explain all of it better than I did in the video.

Originally published on hassandev.me.