Operational databases are excellent at serving application requests. They are not always the best place to run large analytical queries, build historical datasets, or feed multiple warehouses. Moving that data safely into a lakehouse usually introduces a pipeline made of connectors, message brokers, stream processors, and table-maintenance jobs.
OLake Go is an open-source ingestion engine designed to reduce that infrastructure. It replicates databases, Kafka, and S3 data into Apache Iceberg tables or plain Parquet files. In the November 22 GitHub trending archive, it ranked #1 in the Go daily list, with a reported daily gain of 93 stars.
The repository is the renamed continuation of datazip-inc/olake; it is not a brand-new project. The Go implementation is the current repository to inspect, while table maintenance is handled separately by OLake Fusion.
The problem OLake targets
A common architecture looks like this:
PostgreSQL / MySQL / MongoDB | Full load or CDC | OLake Go | Iceberg tables / Parquet | Spark, Trino, DuckDB, BI The source database remains the system of record. OLake reads initial data and subsequent changes, applies configured transformations, handles schema mapping, and writes to an analytical destination.
The repository lists PostgreSQL, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, and S3 among its sources. Destinations include Apache Iceberg and plain Parquet. The Iceberg path can use Glue, Hive, JDBC, or REST catalogs; the local development documentation uses MinIO as an S3-compatible object store.
Credit: OLake's official documentation architecture diagram.
What stands out?
Full load and change data capture. A pipeline can perform an initial copy and then keep the destination synchronized as source records change. The exact CDC mechanism depends on the source connector and its configuration.
Schema discovery and evolution. OLake describes automatic schema discovery and evolution, which reduces the amount of hand-written mapping required when source tables change.
Iceberg-native output. Iceberg gives analytics engines a table format with metadata and snapshot semantics rather than a loose collection of files. Plain Parquet remains useful when a table layer is unnecessary.
A UI and CLI. The web UI manages jobs, sources, destinations, and configurations. The Go repository also contains the ingestion engine and connector code.
Connector capability is not identical across sources:
| Source | Current documented mode |
|---|---|
| PostgreSQL | Full load, CDC via pgoutput, incremental |
| MySQL | Full load, binlog-based CDC, incremental |
| MongoDB | Full load, oplog-based CDC, incremental |
| Oracle | Full load and incremental; CDC is marked WIP |
| DB2 | Full load and incremental; CDC is not listed |
| Kafka | Bounded incremental sync |
The project publishes performance benchmarks, but those are maintainer-reported measurements under specified workloads. Treat them as a starting point for evaluation, not as an independent guarantee that OLake will outperform every managed or self-hosted alternative in your environment.
Run the local UI
The official quickstart requires Docker, Docker Compose, at least 4 GB of RAM available to Docker, and an unused local port 8000.
The documented one-command setup is:
curl -sSL https://raw.githubusercontent.com/datazip-inc/olake-ui/master/docker-compose-v1.yml | docker compose -f - up -d Then open http://localhost:8000. The quickstart creates a default administrator account with the initial credentials admin / password.
Important: those credentials are for a local demonstration. Change them before any shared or network-accessible deployment, keep the UI behind appropriate access controls, and do not expose port 8000 directly to the public internet. For production, inspect the Compose file and pin the image versions you deploy instead of blindly following a moving master branch.
Credit: OLake's official getting-started documentation.
Create a first ingestion job
For a safe first test:
- Start the local stack and sign in on localhost.
- Choose a supported source such as the local PostgreSQL service from the development environment guide.
- Configure the source connection and CDC settings.
- Choose an Iceberg or Parquet destination.
- Run an initial load and inspect the resulting files or table metadata.
- Make a test insert or update in the source and verify that the next sync is reflected in the destination.
The official development guide provides a Docker Compose setup for PostgreSQL, MySQL, or MongoDB plus a local Iceberg/Parquet destination. It includes sample data so you can test the pipeline without connecting a production database. The CLI guide uses source.json and destination.json files, and documents a discover step before sync:
./build.sh driver-postgres discover --config "$(pwd)/source.json" That command writes stream-discovery files for review. Follow the Commands and Flags guide for the selected connector's sync flags; do not run a production sync until you have checked the selected streams and destination configuration.
Where OLake fits, and where it does not
OLake is attractive when you want to move operational data into an open lakehouse format without assembling a large streaming stack. It is especially relevant for data teams that want Iceberg tables available to several query engines.
It is not automatically the right choice for every ingestion problem. Connector maturity, source-specific CDC semantics, schema changes, catalog configuration, and recovery behavior all need to be tested with your actual workload. A benchmark on one database and hardware profile is not proof of performance for another.
| Alternative | Consider it when |
|---|---|
| Airbyte | You need a broad connector ecosystem and a familiar ELT workflow |
| Debezium + Kafka/Flink | You need a more composable event-streaming architecture |
| Managed replication services | You prefer a vendor-operated pipeline and accept its pricing and control model |
| OLake Go | You want an open-source path into Iceberg or Parquet with a comparatively small local stack |
The Go repository is Apache-2.0 licensed. Review the licenses and operational requirements of the source databases, object store, catalog, and query engines you connect to as well.
A short project walkthrough
The official OLake documentation includes a project walkthrough embedded in its setup material:
This is an OLake project video linked from its own documentation, rather than an unrelated generic CDC tutorial.
Sources and further reading
- OLake Go repository
- OLake Go release and tag history
- Official UI quickstart
- Development environment and local pipeline
- Architecture documentation
- Commands and flags reference
- OLake Fusion table maintenance
If you are evaluating a lakehouse ingestion engine, which matters most in your first proof of concept: source coverage, CDC correctness, schema evolution, or operational simplicity?


