DDBD: free and open source database on Linux and macOS

DDBD: the database behind Skapi, now free and open source on Linux and macOS

Hello, and thank you for being here. Today we are releasing something we have been building toward since the first day of Skapi: DDBD, a schemaless record database with writer-based access control (WBAC). It is the record design that runs Skapi's database, packed into a server you can run yourself. It is free and open source under the Apache-2.0 license, and it runs on Linux and macOS. Everything about it is at ddbd.skapi.com.

This article explains the problems we set out to solve from day one, how DDBD solves them, how it performs against PostgreSQL, MongoDB and SQLite, and how to install it in a minute.

The problems we set out to solve from day one

When we started Skapi, we wanted four things from a database, all at the same time.

Records without a schema. You should be able to save a JSON record and query it right away, without designing tables, writing migrations or setting up indexes first. In DDBD, the user who saves a record names its collection, its access level, its tags and its indexes in the same request. There is nothing to set up beforehand.

Access rules that belong to the data. Most databases ask an administrator to write and maintain access policies for every user and every case. We wanted the opposite: the person who writes a record decides who can read it, and the database enforces it on every read. We call this writer-based access control (WBAC). RBAC asks who you are. WBAC asks who wrote it. A record can be private to its writer, shared with users at an access level, or readable by specific users, and no admin has to keep a policy up to date.

Fully serverless, scaling at low cost. We wanted a backend with no database server to keep running around the clock, where the cost of a request follows the size of its answer, not the size of the database. That only works if every query is cheap by construction, with no query planner deciding at read time how to combine indexes.

Safer AI agent development. Agents now read and write data on their own. An agent should get only what its user may see, within limits the user sets, and everything it writes should say which agent wrote it. Nobody should have to hand an agent a password, or an admin key.

The core technology that runs Skapi

The answer to all four is one design. When a record is saved, its writer's choices (collection, access level, tag, index value) are turned into sorted keys and written together with the record, in one atomic step. A query then reads one range of those keys for each access level the reader may read, merged in order. It starts at the first match and stops when the page is full.

The access rule is part of the key. A reader's query never reaches records it may not read, so there is nothing to filter out after the fact.

This is what lets Skapi run with no database server to keep running: every getRecords() call on Skapi works this way, as one range read on a sorted key store, so it runs in short-lived functions that cost nothing while idle and scale with the traffic. Access control costs nothing extra at read time, because it is already in the key. And schemaless records cost nothing extra either, because the writer, not a schema, says what to index.

DDBD is this same record design as a standalone database server, on RocksDB. It speaks HTTP and JSON, so any language with an HTTP client can use it without a client library, and it comes with a user shell and a web admin.

How fast it is

We measured DDBD 0.1.2 against PostgreSQL 16.2, MongoDB 8.0.4 and SQLite 3.46.0 with the same million products in each. Every product has a price, one of 20 tags (sale on 5% of them) and an access level. The query asks for the 50 cheapest products tagged sale above a random price, the kind of question an app or an agent asks all day, for a user who may read only part of the records.

With 1,000,000 records and a user who may read 1% of them, DDBD answered in 2.7 ms: 5.8× faster than PostgreSQL, 6.7× faster than MongoDB and 24× faster than SQLite.

When the user may read only 0.1% of the records, the gap is similar: DDBD 1.8 ms, PostgreSQL 10.8 ms, MongoDB 4.0 ms, SQLite 65.8 ms.

The more important result is what happens as the database grows. From 10,000 to 1,000,000 records, a hundred times more data, the same query took 6.8× longer on DDBD, while it took 24× longer on MongoDB, 30× longer on PostgreSQL and 98× longer on SQLite. Part of DDBD's growth is the answer itself: at 10,000 records only 4.1 products matched on average, at 1,000,000 the page was full with 46.3.

Query time grows with the answer, not with the database.

All results at 1,000,000 records

The fastest time in each row is in bold.

The table shows where each design does its work. With a tag and an access level in the query, a SQL database or MongoDB has to combine the tag index, the price order and the access filter at query time, and the less a user may read, the more it has to skip. DDBD wrote a key that already holds all three when the record was saved. A plain price range with no tag, for a user who may read 10% or more of the records, is exactly what a single price index serves best, and there PostgreSQL and SQLite answer faster. When the user may read every record there is no access filter to combine, and PostgreSQL answers the tag query faster too. That first kind is the everyday work of an app with many users, or of agents that each read their own slice of a shared store.

How it was measured. An Intel Core i7-1160G7 laptop (4 cores, 8 threads, 15 GB RAM) with all data in memory. Every query was the same HTTP POST with a JSON body and a session token to each database, and every answer was the 50 full records as JSON. PostgreSQL, MongoDB and SQLite answered through a small HTTP server written for the test with the same libraries as DDBD, which reads the token, looks up the user's access level, runs the query and writes the answer. Each database had the indexes that serve the query, 300 queries per case after 30 warm-up queries, the four taking turns query by query, and the whole set ran 3 times; each number is the median of the runs' medians. All four returned the same records in the same order for every query.

Built for AI agents

A DDBD user makes an agent key for each agent or script, without asking an admin, and can revoke it at any time. A key can be limited to some collections, to a highest access level, to read-only, to private records or not, and to an expiry date:

// createAgentKey body { "name": "shop-bot", "collections": ["products"], "max_level": "05", "read_only": true, "expires": "30d" } 

An agent reads only what its user may read, within the key's limits. Every record it saves belongs to the user and names the key that saved it:

{ "record_id": "VZYu683bLzapjrgk", "user_id": "JFeCedoO", "agent": { "key_id": "mtxBMs", "name": "shop-bot" }, "data": { "title": "Desk Lamp" } } 

Two more things make it safe to let agents work on shared data:

• No network user is ever an admin. The only administrator is the operating system user who runs the server, through a local socket. A leaked token, at any level, never carries admin powers.
• Data even the admin cannot read. DDBD gives every user a salt and the settings to derive a key from their password with Argon2id. The app encrypts the data on the user's device before saving it, so the server, its backups and its admin only ever hold the encrypted result.

Try it on your machine

DDBD runs on Linux and macOS. On Linux, install it with one command:

curl -fsSL https://cdn.broadwayinc.com/get/ddbd/v1 | sh 

On macOS, install it with Homebrew:

brew install broadwayinc/ddbd/ddbd 

Then create a data folder and start the server:

ddbd init --data ~/ddbd-data ddbd start --data ~/ddbd-data 

In a second terminal, turn on the web admin by giving it a password, then open http://localhost:7070/ in your browser. From there you can add your first user, save records, run queries and make agent keys:

ddbd admin --data ~/ddbd-data web-password 

The server speaks plain HTTP by default, which is fine on your own machine. Anywhere else, run it behind HTTPS, or start it with --tls. Running the install command again later updates DDBD to the newest version; with Homebrew, brew upgrade ddbd does.

The documentation covers every request, with curl, JavaScript and Python examples: ddbd.skapi.com.

Thank you

Skapi has always been about letting you build without running a backend yourself, and that does not change: your Skapi projects keep running on the same design, managed by us. DDBD is for when you want that design on your own machine, next to your own agents and your own data, free.

Thank you for building with Skapi, and for every question, bug report and idea you send us. Try DDBD, break it, and tell us what you think at support@broadwayinc.com.