Run the official quickstart, connect Spark, and create your first Iceberg table

If you work with Apache Iceberg, you already know a catalog is what tells engines where a table's current metadata lives. Apache Polaris is an open source Iceberg catalog that implements the Iceberg REST API, so Spark, Trino, Flink, and other engines can share the same tables. It also handles access control and hands clients short-lived storage credentials, so you don't have to give every engine a permanent key to the bucket.

This walkthrough is for developers who want to run that locally and create a table, without setting up PostgreSQL or a cloud account first. I've been contributing to Polaris since June, and this is the path I recommend when someone asks how to try it. As of mid-November 2026, the latest release is 1.9.0, and the official quickstart uses the apache/polaris:latest image, which currently points at that release.

The first time you run this, pulling the images can take longer than ten minutes. After that, starting the stack and creating a table is usually quick. This setup is for evaluation only, not for production.

What you need

You need Docker and Docker Compose v2 (docker compose, not the older docker-compose). You also need Apache Spark 3.5.x built for Scala 2.12, with spark-sql on your PATH, because the packages in the quickstart are iceberg-spark-runtime-3.5_2.12:1.12.0 and iceberg-aws-bundle:1.12.0. Those are the versions the current compose file prints, and 1.12.0 is still the latest Iceberg Spark 3.5 runtime on Maven Central. Spark 3.5 works with Java 17.

You don't need an AWS account. The quickstart starts RustFS, an S3-compatible store, next to Polaris.

Start Polaris

From any directory, run the command from the quickstart guide:

curl -s https://raw.githubusercontent.com/apache/polaris/refs/heads/main/site/content/guides/quickstart/docker-compose.yml \ | docker compose -p polaris-quickstart -f - up -d 

That starts Polaris and RustFS, creates an S3 bucket named bucket123, and bootstraps:

  • Realm: POLARIS
  • Catalog: quickstart_catalog, stored at s3://bucket123
  • Principal: quickstart_user, with the CATALOG_MANAGE_CONTENT privilege

The Polaris REST API listens on http://localhost:8181. Health and metrics are on http://localhost:8182 (/q/health and /q/metrics). RustFS is on port 9000, and its console is at http://localhost:9001.

Give the setup container a moment to finish before you look for credentials.

Read the credentials from the logs

docker compose -p polaris-quickstart logs 

You can also follow just the setup service with docker compose -p polaris-quickstart logs polaris-setup. Wait until you see Polaris Quickstart Setup Complete!. The log then prints two sets of credentials.

The root credentials default to client ID root and client secret s3cr3t. Those belong to the bootstrap admin principal and are the ones you use with the management CLI later.

The user credentials belong to quickstart_user and are generated at setup time. The client ID and secret will be different every time you start a fresh stack. Copy those two values. The Spark command in the same log already has them filled in, so the easiest path is to copy that whole command.

Don't commit those secrets, and don't reuse this compose file as-is in a shared environment. The root secret is a well-known default.

Connect Spark

Here's the spark-sql command the setup log prints, with the generated user credentials left for you to paste:

spark-sql \ --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.12.0,org.apache.iceberg:iceberg-aws-bundle:1.12.0 \ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ --conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \ --conf spark.sql.catalog.polaris.type=rest \ --conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \ --conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \ --conf spark.sql.catalog.polaris.credential=<client_id>:<client_secret> \ --conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \ --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials \ --conf spark.sql.defaultCatalog=polaris 

A few of those lines are doing the important work.

--packages pulls the Iceberg Spark 3.5 runtime and the AWS bundle, which Spark needs to talk to S3-compatible storage with the credentials Polaris returns. The versions have to match your Spark release and Scala line. This command is for Spark 3.5 and Scala 2.12.

spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog and type=rest tell Spark to use Iceberg's standard REST catalog client, not a Polaris-specific plugin. Any REST-compatible catalog would be configured this way.

warehouse=quickstart_catalog is the Polaris catalog name, not a filesystem path. If you typo this, Spark won't find the catalog the setup script created.

uri=http://localhost:8181/api/catalog is the Iceberg REST base URL. The service itself lives under /api/catalog/v1, and the client adds the versioned paths.

credential is the quickstart_user client ID and secret, separated by a colon. Spark exchanges them for an access token using the OAuth client-credentials flow.

scope=PRINCIPAL_ROLE:ALL asks the token to include every principal role granted to that user, which in the quickstart is the role that has catalog management rights.

header.X-Iceberg-Access-Delegation=vended-credentials is the line people most often skip. It tells Polaris to return short-lived storage credentials, plus the endpoint and region, in the table-load response. Without it, Spark has no reason to know that the data lives on RustFS at localhost:9000, and reads or writes will fail even though the catalog call succeeded.

defaultCatalog=polaris means you can write CREATE TABLE db1.table1 instead of CREATE TABLE polaris.db1.table1.

The first launch may spend a minute downloading the packages from Maven.

Create a namespace and a table

Once you have a spark-sql prompt, the example from the Polaris README is enough:

CREATE DATABASE db1; SHOW DATABASES; CREATE TABLE db1.table1 (id INT, name STRING); INSERT INTO db1.table1 VALUES (1, 'a'); SELECT * FROM db1.table1; 

In Spark with Iceberg, CREATE DATABASE and CREATE NAMESPACE are the same idea. After the insert, SELECT should return the row (1, a).

If the create or insert fails with a storage or credentials error, go back and check that the vended-credentials header is present and that you used the user credentials from the latest setup log, not an old pair from a previous run.

Look at the files

Open http://localhost:9001 and sign in with the RustFS keys from the compose file: access key polaris_root, secret key polaris_pass. Browse bucket bucket123. You should see objects for db1.table1, including Iceberg metadata files and at least one data file for the insert.

If you prefer the command line and have the AWS CLI installed:

AWS_ACCESS_KEY_ID=polaris_root AWS_SECRET_ACCESS_KEY=polaris_pass \ aws s3 ls s3://bucket123 --endpoint-url http://localhost:9000 --recursive 

The catalog didn't store the row. It stored a pointer to the table's current metadata, and Spark wrote the data and metadata into the bucket using credentials Polaris issued for that table.

A quick look at RBAC (optional)

The quickstart_user can do almost anything in this catalog, because it was granted CATALOG_MANAGE_CONTENT. To see the role model, you can add a read-only principal with the Python CLI:

pip install apache-polaris 

The CLI talks to localhost:8181 by default. Use the root credentials from the logs (the defaults are shown below) and pass the realm, since otherwise the CLI won't send the Polaris-Realm header:

export CLIENT_ID=root export CLIENT_SECRET=s3cr3t polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ principals create analyst polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ principal-roles create analyst_role polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ principal-roles grant --principal analyst analyst_role polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ catalog-roles create --catalog quickstart_catalog catalog_reader polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ catalog-roles grant --catalog quickstart_catalog --principal-role analyst_role catalog_reader polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ privileges catalog grant --catalog quickstart_catalog --catalog-role catalog_reader TABLE_READ_DATA polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ privileges catalog grant --catalog quickstart_catalog --catalog-role catalog_reader TABLE_READ_PROPERTIES polaris --client-id "$CLIENT_ID" --client-secret "$CLIENT_SECRET" --realm POLARIS \ privileges catalog grant --catalog quickstart_catalog --catalog-role catalog_reader NAMESPACE_LIST 

principals create should print a new client ID and secret for analyst. Point a second spark-sql session at those credentials, keep the vended-credentials header, and SELECT * FROM db1.table1 should work. An INSERT should be denied, because this role never received TABLE_WRITE_DATA.

Privileges are not granted directly to the user. They go to a catalog role, the catalog role is granted to a principal role, and the principal role is granted to the principal. That split is what lets a catalog admin define "readers" without knowing who the people are.

If listing namespaces or tables fails in the analyst session, grant the missing list or read-properties privilege and try again. I didn't run this second session against a live stack while writing this, so treat it as a starting point and adjust if the server asks for another privilege.

What just happened

Spark never had a long-lived key for the bucket. It authenticated to Polaris as a principal, asked for table metadata over the Iceberg REST API, and because of the delegation header, Polaris returned temporary credentials scoped to that table's location. Spark then used those credentials to write and read objects in RustFS.

That's the difference between a catalog that only stores pointers and a catalog that also enforces access at the storage layer. If every engine has a wide-open bucket policy, table-level privileges in the catalog are only advisory. I've spent a fair amount of time in the Polaris code that scopes those credentials, and the details around paths and endpoints matter more than they first appear.

The in-memory metastore used by this quickstart will forget the catalog when the container stops. The files in RustFS may still be there until you remove the containers, but a new up will bootstrap a new catalog and new user credentials.

Cleanup

docker compose -p polaris-quickstart down 

That stops and removes the quickstart containers. If you want to make sure the RustFS data is gone too, remove any leftover containers or volumes for the polaris-quickstart project with docker compose -p polaris-quickstart down -v.

Common problems

The logs have no credentials yet. The Polaris container can be healthy before polaris-setup has finished creating the catalog and user. Wait for Polaris Quickstart Setup Complete!, or inspect logs polaris-setup specifically.

A port is already in use. The compose file binds 8181, 8182, 9000, 9001, and also 5005 for a JVM debugger. Stop whatever is using the port, or change the mapping in a local copy of the compose file.

docker-compose is not found, or the one-liner fails. You need Compose V2, invoked as docker compose. The curl command has to be passed to -f - so Compose reads the file from stdin.

Spark can't resolve the Iceberg packages, or the session fails to start. Confirm you are on Spark 3.5 with Scala 2.12. A Spark 3.4 install or a Scala 2.13 build won't match iceberg-spark-runtime-3.5_2.12.

Create or insert fails with a storage, endpoint, or access-denied error. The usual cause is a missing X-Iceberg-Access-Delegation=vended-credentials header, or using stale credentials from a previous docker compose up. Copy the command from the latest setup log.

warehouse not found or unauthorized. The name must be quickstart_catalog, and the credentials must be the ones for quickstart_user (or another principal that has been granted access), not a typo or a mix of root and user values.

CLI commands return 401 or talk to the wrong realm. Pass --realm POLARIS and the root client ID and secret from the logs.

Next steps

When you're ready to leave the in-memory store behind, the docs cover a binary distribution, PostgreSQL as the metastore, and an official Helm chart. There's also a Spark getting-started guide that runs Spark in a Jupyter notebook next to Polaris, if you'd rather not install Spark on your machine.

For day-to-day administration, keep using the CLI, and read the access control and vended credentials pages before you point this at a real bucket. If you want to change the project itself, the CONTRIBUTING guide and ./gradlew run are the place to start.

Links