Imagine this.
You have a distributed compute cluster running AI workloads. One of your workers suddenly becomes unreachable.
Your control plane stops receiving heartbeats. The execution lease expires. The scheduler assumes the worker is no longer available and reallocates the job.
But there's a problem.
The worker never actually stopped executing.
It might be partitioned from the control plane, experiencing network problems, or simply unable to communicate. Meanwhile, it's still consuming CPU, memory, and potentially expensive GPU resources.
Then it reconnects.
It reports that the job completed successfully. But another worker may already have executed the same job.
Whose result do you trust? Who owns the execution? And how do you prevent an old worker from modifying state it no longer owns?
These aren't theoretical questions. They're the kinds of problems that make distributed infrastructure difficult to get right.
I decided to build around these failure scenarios.
I'm the founder and primary engineer behind AetherGrid, an open-source distributed AI workload orchestration platform.
Rather than treating worker failures as simple offline events, I built the execution model around explicit ownership and recovery.
Some of the mechanisms include:
Time-bound execution leases: Workers renew ownership throughout job execution.
Lease fencing: Renewal, release, and outcome reporting are checked against lease identity, preventing stale owners from modifying execution state.
Multidimensional scheduling: Workloads are assigned based on CPU, memory, and VRAM requirements.
Reconciliation: Expired leases, dead workers, offline nodes, and abandoned jobs are handled through bounded recovery.
Tenant isolation: Isolation is enforced in both application logic and PostgreSQL using composite foreign keys.
Correctness matters more than the happy path.
One engineering decision that shaped AetherGrid was testing failure conditions against real PostgreSQL.
I don't want a test suite that only proves a worker can successfully execute a job. I want it to catch what happens when ownership changes, processes become unreachable, operations race, or a worker reports an outcome after losing its lease.
AetherGrid currently has more than 1,600 passing tests and over 100 architecture decision records.
It's built with Python, FastAPI, PostgreSQL, Go-based infrastructure experience, React, TypeScript, and Docker.
The project is live and open source.
Explore the implementation:
https://github.com/wycliffRotich-dev/aethergrid
Project website:
https://aethergrid.co.ke
Here's a question I'm genuinely interested in discussing:
When a worker loses its lease but continues executing, how do you handle the resources it is still consuming while preventing it from reporting a stale result?
Do you terminate immediately, wait for execution to finish, or use another mechanism?
I'm interested in hearing how other infrastructure teams approach this problem, and I'm open to technical collaborations, design partnerships, and contributors who want to challenge the implementation.
Because distributed systems don't become reliable just because everything works when the network behaves.

