I'm building Arcyria, a multi-tenant data operations platform: data connections, ELT pipelines, SQL transformations, monitoring and an AI operator, in one place. This post covers the architecture decision that shaped the most: every pipeline run is its own short-lived container.

The problem with long-running workers

The obvious way to run pipelines is a pool of workers that pull jobs from a queue. It works, until it doesn't:

  • One tenant's heavy run starves everyone else.
  • State leaks between runs (temp files, caches, half-loaded tables).
  • Credentials end up living in long-lived processes.
  • Scaling means guessing how many workers you need.

The approach: one run, one container

When a user starts a run, the API does not execute anything. It publishes a small, credential-free event to a queue. A private control service consumes it and launches a one-shot task on AWS ECS Fargate.

Web app -> Go API -> SQS (run event, no secrets) | control service (consumer + scheduler) | ECS Fargate task (ELT job) | ephemeral DuckDB + dbt -> Parquet artifacts -> destination tables | idempotent callback to control

Inside the task, the run is staged through an ephemeral DuckDB database, transformed with dbt, written out as immutable Parquet artifacts, and then upserted into the destination tables. When the run finishes, the local DuckDB and dbt files are deleted. Nothing survives except the evidence the run published.

Why it works well

Isolation by default. A run can't see another run's files, and a crash only kills that run.

Credentials stay out of the pipeline path. The browser never talks to the ELT service. Credentials are encrypted by the API and never appear in queue messages, task overrides, logs or browser state.

Idempotent completion. The task reports back with a callback that is safe to retry, so a flaky network doesn't produce duplicate or lost results.

Cost follows usage. No idle worker fleet. You pay for the minutes a run actually takes.

Trade-offs

It isn't free. Cold-start latency is real, so this suits batch and scheduled ELT rather than sub-second jobs. You also need good run-level observability, since you can't just SSH into a worker. And destination tables must already exist with compatible columns before a run starts. We chose upsert-oriented delivery and strict contracts over magic schema creation.

What's next

I'm working toward a public beta and would love feedback from people who run ELT in production: what would you need before trusting a platform like this with your pipelines?

If you want to follow along, leave a comment or say hi.