Getting MySQL to DuckDB usually starts with mysqldump and a DuckDB file that gets copied around. It works for a first look. The file is stale the moment it is written, every copy drifts from the others, and an update made in MySQL never reaches any of them.

Here are the common ways to put MySQL data in front of DuckDB, each with a real use:

  • The DuckDB MySQL extension. DuckDB logs in to MySQL and reads the current rows on every query. Nothing is copied, but every notebook that runs it puts load on the production server.
  • mysqldump or SELECT ... INTO OUTFILE, then COPY into DuckDB. A script and a scheduler rebuild a DuckDB file. Freshness is hours to daily, and only full reloads are possible, so updates and deletes never arrive as changes.
  • Debezium and Kafka, with a consumer that writes Parquet. Changes land within seconds. You run Kafka, Kafka Connect, Debezium and the consumer, and you write the merge logic for updates and deletes yourself.
  • Managed extract, load, transform (ELT) into a warehouse, with DuckDB reading the warehouse. Nothing to host, and minutes of freshness depending on the plan. You pay per row, per GB or per connector for a second copy you also pay to store.

This post walks through a fifth route. OLake Go, an open source (Apache 2.0) replication tool, reads the MySQL binary log (binlog) and commits change data capture (CDC) as Apache Iceberg tables into Apache Polaris. Polaris is an open source Iceberg Representational State Transfer (REST) catalog you host yourself, and DuckDB attaches it with one statement.

I ran this on 2026-10-29 with MySQL 8.4, OLake Go v1.1.0, Polaris 1.8.0 on MinIO and DuckDB 1.5.5. Every number below comes from that run.

What the MySQL to DuckDB pipeline looks like

  • Source: MySQL 8.4 with the binlog in ROW format, a database called shop, and two tables, customers and orders.
  • Pipeline: OLake UI from Docker Compose. The first sync copies the tables in parallel chunks, and every later sync replays only the binlog since the saved position.
  • Catalog and storage: Polaris and MinIO in Docker. On AWS, a plain S3 bucket and an Identity and Access Management (IAM) role take MinIO's place.
  • Readers: DuckDB first, then ClickHouse 26.8.36 and Trino 484 on the very same tables.

You need Docker and about 40 minutes. DuckDB must be 1.4 or later; it installs the iceberg and httpfs extensions itself on first use.

Step 1: Turn on the binlog and create a reader user

Put these settings in my.cnf (or the parameter group on a managed server) and restart MySQL. log_bin is the one that cannot change without a restart.

server-id = 1 log_bin = mysql-bin binlog_format = ROW binlog_row_metadata = FULL binlog_expire_logs_seconds = 604800 

Then create the user OLake Go connects as, and the two sample tables if you want to follow along with the same shape:

CREATE USER IF NOT EXISTS 'olake'@'%' IDENTIFIED BY '<password>'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'olake'@'%'; FLUSH PRIVILEGES; CREATE TABLE customers ( id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100), email VARCHAR(200), city VARCHAR(100), tier VARCHAR(20), created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP); CREATE TABLE orders ( id INT AUTO_INCREMENT PRIMARY KEY, customer_id INT, product VARCHAR(100), amount DECIMAL(10,2), status VARCHAR(20), created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, FOREIGN KEY (customer_id) REFERENCES customers(id)); 

Load your own rows. My run started with 8 customers and 40 orders, and the change SQL in step 7 updates customers named Liam Carter and Zara Ali, so include those two if you want it to match.

Now the step most people skip:

ANALYZE TABLE customers, orders; SHOW VARIABLES WHERE Variable_name IN ('log_bin', 'binlog_format', 'binlog_row_metadata', 'server_id'); SHOW GRANTS FOR 'olake'@'%'; 

OLake Go plans the parallel snapshot from table statistics. A freshly loaded table has none, and the first sync stops until you run ANALYZE TABLE.

Step 2: Start Polaris and MinIO, then create the catalog

services: minio: image: minio/minio hostname: lake-minio command: server /data --console-address ":9001" environment: { MINIO_ROOT_USER: minioadmin, MINIO_ROOT_PASSWORD: minioadmin } ports: ["9000:9000", "9001:9001"] polaris: image: apache/polaris:latest depends_on: [minio] ports: ["8181:8181", "8182:8182"] environment: AWS_REGION: us-east-1 AWS_ACCESS_KEY_ID: minioadmin AWS_SECRET_ACCESS_KEY: minioadmin POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,<root-secret> polaris.realm-context.realms: POLARIS 

Polaris keeps its state in memory in this setup, so it starts empty after every restart. Create the bucket, a catalog called olake, and a principal called olake with one grant:

P=http://localhost:8181 aws --endpoint-url http://localhost:9000 s3 mb s3://lakehouse TOKEN=$(curl -s $P/api/catalog/v1/oauth/tokens -H 'Polaris-Realm: POLARIS' \ -d grant_type=client_credentials -d client_id=root -d client_secret=<root-secret> \ -d scope=PRINCIPAL_ROLE:ALL | jq -r .access_token) H=(-H "Authorization: Bearer $TOKEN" -H 'Polaris-Realm: POLARIS' -H 'Content-Type: application/json') curl -s "${H[@]}" $P/api/management/v1/catalogs -d '{"catalog": { "name": "olake", "type": "INTERNAL", "readOnly": false, "properties": {"default-base-location": "s3://lakehouse/olake"}, "storageConfigInfo": {"storageType": "S3", "allowedLocations": ["s3://lakehouse/olake"], "endpoint": "http://<minio-host>:9000/", "endpointInternal": "http://lake-minio:9000", "pathStyleAccess": true, "region": "us-east-1"}}}' curl -s "${H[@]}" $P/api/management/v1/principals -d '{"principal": {"name": "olake"}}' curl -s "${H[@]}" $P/api/management/v1/principal-roles -d '{"principalRole": {"name": "olake_role"}}' curl -s "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles -d '{"catalogRole": {"name": "olake_catalog_role"}}' curl -s -X PUT "${H[@]}" $P/api/management/v1/principals/olake/principal-roles -d '{"principalRole": {"name": "olake_role"}}' curl -s -X PUT "${H[@]}" $P/api/management/v1/principal-roles/olake_role/catalog-roles/olake -d '{"catalogRole": {"name": "olake_catalog_role"}}' curl -s -X PUT "${H[@]}" $P/api/management/v1/catalogs/olake/catalog-roles/olake_catalog_role/grants -d '{"type": "catalog", "privilege": "CATALOG_MANAGE_CONTENT"}' 

💡 The principal response holds clientId and clientSecret, and Polaris shows the secret only once. Save it now; OLake Go and DuckDB both need it.

Keep the MinIO hostname free of underscores. Polaris parses addresses with Java's URI parser, which rejects lab_minio and fails every table creation.

Step 3: Start OLake UI and add the MySQL source

curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d 

Open http://localhost:8000 and sign in as admin / password. Go to Sources, click Create Source, pick MySQL, and leave Enable CDC on.

In my run MySQL sat in another container, so the host was host.docker.internal, the database shop and the user olake. Initial Wait Time stayed at 10, and SSL was off for the local test.

OLake UI MySQL source form with Enable CDC on, initial wait time 10, host host.docker.internal, database shop and user olake

Set up source tests the connection and checks binlog_access, log_bin, binlog_format and binlog_row_metadata before saving. A failing check shows the current value next to the recommended one.

Step 4: Add the Polaris destination

Go to Destinations, click Create Destination, pick Apache Iceberg, then Polaris as the catalog type and OAuth2 as the authentication type. The values from my run:

Field Value
REST Catalog URI http://host.docker.internal:8181/api/catalog
Polaris Catalog Name olake
REST Auth URI http://host.docker.internal:8181/api/catalog/v1/oauth/tokens
REST Credential <client_id>:<client_secret>
REST Scope PRINCIPAL_ROLE:ALL
S3 Endpoint http://host.docker.internal:9000
AWS Region us-east-1

OLake UI Polaris destination form with OAuth2, REST catalog URI on port 8181, catalog name olake and the MinIO S3 endpoint on port 9000

Clicking Create exchanges the credential for a token, creates a test table through Polaris, writes to the bucket and drops the table again. A failure here is nearly always the secret, the grant or the catalog name.

Step 5: Create the job and run the first sync

Go to Jobs, click Create Job, name it mysql_to_duckdb and pick the source and destination. Under Advanced Settings, set Target Query Engines to DuckDB, plus Trino if it will read the tables too.

OLake Go then offers only the delete formats every listed engine can read. I picked Positional Deletes, the one DuckDB documents.

On the Streams page, tick customers and orders with Full Refresh + CDC and Upsert. Then click Create Job and choose Sync Now from the job menu.

OLake UI Streams page with customers and orders selected, Full Refresh + CDC, Upsert and Positional Deletes on Iceberg V2

The first sync copied 48 records (2.10 KB) in 17 seconds, container start included. The namespace is the job name plus the source database, mysql_to_duckdb_shop, and the Iceberg DB field at the top of the Streams page renames it before the first run.

Step 6: Query the tables in DuckDB

INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs; CREATE SECRET polaris_secret ( TYPE iceberg, CLIENT_ID '<client_id>', CLIENT_SECRET '<client_secret>', OAUTH2_SERVER_URI 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens', OAUTH2_SCOPE 'PRINCIPAL_ROLE:ALL' ); CREATE SECRET store_secret ( TYPE s3, KEY_ID '<access key>', SECRET '<secret key>', ENDPOINT '<minio-host>:9000', URL_STYLE 'path', USE_SSL false, REGION 'us-east-1' ); ATTACH 'olake' AS polaris ( TYPE iceberg, ENDPOINT 'http://<polaris-host>:8181/api/catalog', SECRET polaris_secret, ACCESS_DELEGATION_MODE 'none' ); SHOW ALL TABLES; SELECT id, name, city, tier FROM polaris.mysql_to_duckdb_shop.customers ORDER BY id; 

The S3 secret and ACCESS_DELEGATION_MODE 'none' exist because of the laptop. Polaris vends the MinIO address as host.docker.internal, which only resolves inside Docker. Against real S3, drop both and DuckDB reads with the credentials Polaris vends.

Step 7: Change rows in MySQL and sync again

INSERT INTO customers (name, email, city, tier) VALUES ('Maya Patel', 'maya@example.com', 'Pune', 'silver'), ('Lucas Silva', 'lucas@example.com', 'São Paulo', 'bronze'); INSERT INTO orders (customer_id, product, amount, status) VALUES (9, 'Laptop', 1299.00, 'placed'), (9, 'Mouse', 25.50, 'placed'), (10, 'Monitor', 349.99, 'placed'), (3, 'Keyboard', 89.00, 'placed'), (7, 'Webcam', 79.90, 'placed'); UPDATE customers SET tier = 'gold' WHERE name = 'Liam Carter'; UPDATE customers SET city = 'Abu Dhabi' WHERE name = 'Zara Ali'; UPDATE orders SET status = 'delivered' WHERE id IN (1, 4, 7); 

Those are 7 inserts and 5 updates. Sync Now read 12 records (563 bytes) from the binlog and finished in 9 seconds.

OLake UI job history for mysql_to_duckdb showing the first sync in 17 seconds and the change sync in 9 seconds

For customers the commit added one data file with 4 rows, the 2 inserts and the new versions of the 2 updated rows. One positional delete file retired the 2 old versions, and the log says so: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2.

OLake UI sync log filtered to the Iceberg commit report with addedDataFiles 1 and addedPositionalDeleteFiles 1

Run the same SELECT again, with no refresh. DuckDB returns 10 rows, one per key: Liam Carter is gold, Zara Ali is in Abu Dhabi, and the two new customers are there.

DuckDB 1.5.5 shell returning 10 customer rows from the Polaris catalog after the change sync

The _op_type column tells you how each row got there: r from the snapshot, c for an insert, u for an update and d for a delete. OLake Go also adds _olake_id, _olake_timestamp and _cdc_timestamp to every table.

Read the same tables from ClickHouse and Trino

This is where writing Iceberg beats exporting a file. ClickHouse mounts the catalog as one database:

SET allow_database_iceberg = 1; CREATE DATABASE polaris ENGINE = DataLakeCatalog('http://<polaris-host>:8181/api/catalog') SETTINGS catalog_type = 'rest', warehouse = 'olake', catalog_credential = '<client_id>:<client_secret>', oauth_server_uri = 'http://<polaris-host>:8181/api/catalog/v1/oauth/tokens', auth_scope = 'PRINCIPAL_ROLE:ALL', vended_credentials = true, region = 'us-east-1'; SELECT id, name, city, tier FROM polaris.`mysql_to_duckdb_shop.customers` ORDER BY id; 

clickhouse-client reading polaris.mysql_to_duckdb_shop.customers through DataLakeCatalog and returning the same 10 rows

Trino needs one etc/catalog/polaris.properties file with iceberg.catalog.type=rest, the OAuth2 credential, and iceberg.rest-catalog.vended-credentials-enabled=true. Trino 484 returned the same 10 rows.

An orders roll-up matched MySQL to the cent in all three engines: 16 delivered orders for 8659.68, 18 placed for 10522.26 and 11 shipped for 6397.25.

Tips from the run

  • decimal arrives as DOUBLE. The totals above still matched to the cent, but round at query time when exact decimal math matters. datetime and timestamp become TIMESTAMP WITH TIME ZONE in UTC.
  • json, enum and time arrive as VARCHAR. Parse json with DuckDB's json extension.
  • Watch binlog retention. A job paused longer than binlog_expire_logs_seconds cannot resume and needs a fresh snapshot.
  • Hot tables can be cached. DuckDB reads Parquet over the network on each query, so CREATE TABLE local AS FROM polaris... once for a table you query all day.
  • Plan compaction. Each sync adds a small data file and a delete file per table, and nothing compacts them for you. Spark's rewrite_data_files works through the same Polaris endpoint.

How the MySQL to DuckDB methods compare

Method Freshness Updates and deletes What you run
DuckDB MySQL extension Live, per query Reads the current row A MySQL login for DuckDB
mysqldump or INTO OUTFILE, then COPY Hours to daily Full reloads only A script and a scheduler
Debezium, Kafka and a Parquet consumer Seconds Yes, with your own merge logic Kafka, Kafka Connect, Debezium, a consumer
Managed ELT to a warehouse Minutes, by plan Varies by tool Nothing; priced per row, GB or connector
OLake Go to Iceberg on Polaris CDC on a schedule down to one minute Yes, as Iceberg row-level deletes One Docker Compose stack plus Polaris

When one analyst queries one small table, the MySQL extension or a nightly COPY is less to run than a catalog. Tables without a primary key are another poor fit: inserts replicate, but updates and deletes cannot be matched to a row.

FAQ

Why does the DuckDB secret need ACCESS_DELEGATION_MODE set to none?

On a laptop, the MinIO address Polaris hands out is a Docker-internal name that DuckDB cannot resolve. Turning delegation off makes DuckDB use your own S3 secret with a reachable endpoint instead. On AWS S3 the vended address works everywhere, so both lines go away.

Can DuckDB write back to these tables?

No. The attached Polaris catalog is read-only from DuckDB, and OLake Go is the only writer. Copy a table into a local DuckDB table if you need to transform it in place.

How do I add a column without breaking the job?

Add it in MySQL as usual. OLake Go adds it to the Iceberg table on the next sync, and DuckDB shows it after the next ATTACH. Widening a type such as int to bigint works too, while a narrowing change fails the sync instead of corrupting the table.

Does the change sync rescan the whole table?

No. It opens the binlog at the position saved by the last run and reads only what changed. In my run that was 12 records and 563 bytes, against 48 records on the first sync.

Where to go from here

  • Point the source at your real MySQL. Amazon Relational Database Service (RDS), Aurora, Cloud SQL and Azure set the binlog options through a parameter group or flag instead of my.cnf.
  • Move Polaris off in-memory state and onto Postgres before anyone depends on it.
  • The full guide, with the data type table, the Trino properties and the troubleshooting list, is on olake.io.

If a step does not match what you see, say so in the comments and I will update the post.