Decentralized AI Inference Needs a Boring Job Queue

Whenever people talk about decentralized AI, the conversation usually revolves around GPUs, tokens, and governance.

But there's another problem that doesn't get nearly as much attention.

How do you get an AI request from an application to a remote GPU and back without things falling apart along the way?

Imagine sending a request to a GPU on a decentralized network. The GPU finishes the job, but the worker responsible for saving the result crashes.

Your backend gets a timeout.

Now you're left wondering: Did the GPU finish the job? Should I try again? And if I do, will I end up paying twice for the same work?

These might sound like small problems, but anyone who has worked with backend systems knows how messy things can get when different services depend on one another.

I'm a backend engineer, and I work with things like APIs, job queues, retries, and idempotency. So when I look at decentralized AI, I don't just think about how to access GPUs. I also think about what happens when things go wrong.

This article is my proposal for making decentralized GPU inference feel more like a regular backend service that developers can actually rely on.

It's an architecture I'm proposing, not a finished product.

A little about me

I'm a self-taught backend engineer from Nigeria. I work with TypeScript, Node.js, MongoDB, Redis, RabbitMQ, BullMQ, and Docker. I also build web and mobile products for Nigerian SMEs.

One thing that makes AI infrastructure interesting to me is the cost.

GPU compute is often priced in dollars, while many of the businesses around me earn in naira. As a product grows and starts making more AI requests, those costs can become harder to manage.

Depending entirely on one provider also comes with risks. Prices can change, services can become unavailable, and your application can become too dependent on a single company.

Decentralized GPU networks offer another option by connecting users with computing resources operated by different providers.

That's promising, but getting access to a GPU is only part of the story.

Developers still need a reliable way to send jobs, recover from failures, retrieve results, and keep track of what they're paying for.

That's the part I want to work on.

Having access to GPUs isn't enough

Let's say you're building an application that uses AI.

A user submits a request, and your backend sends it to a remote GPU to process.

• What happens if the request takes longer than expected?
• What if the GPU goes offline?
• What if the computation finishes but your backend never receives the result?

You also have to think about duplicate requests and billing. If your system retries a job after a timeout, how do you know the original job isn't still running?

These aren't problems unique to decentralized AI. They're challenges developers already face when building systems that depend on multiple services.

Thankfully, we already have tools that can help.

Job queues, databases, retries, and idempotency are all useful building blocks. They won't magically prevent every failure, but they can help us handle failures in a more controlled way.

And that's where I'd start.

How I'd build it

I'd build the system around four main components.

1. An API that accepts the request

The first component would be a simple API built with Express.js and TypeScript.

Its job would be to receive the request, check that it's valid, authenticate the user, and create a record for the job.

Instead of making the user wait for the entire AI computation to finish, the API would return a job ID immediately.

For example, a client could send a request to:

POST /v1/inference 

And receive something like this:

json

{ "jobId": "job_123", "status": "queued", "statusUrl": "/v1/jobs/job_123" } 

The user could then check the job's status or retrieve the result later.

The important thing is that the application wouldn't need to know which specific GPU provider is handling the request.

2. A queue that keeps track of the work

Next, I'd use BullMQ and Redis to manage background jobs, with PostgreSQL keeping the authoritative record of each job.

Think of the queue as a system that organizes work and makes sure it gets picked up for processing.

Each job would contain the information needed to execute it, including the model, input, deadline, retry limit, idempotency key, and billing details.

A job might move through these stages:

queued → dispatching → running → validating → succeeded 

If something goes wrong, it could move to failed or into a reconciliation state while the system figures out what happened.

That second option matters more than it might seem.

A timeout doesn't always mean the job failed. Sometimes the GPU has already finished the work, but your backend lost contact with it before saving the result.

If you immediately submit the job again, you could end up paying for another execution unnecessarily.

Redis and BullMQ would help coordinate the work, while PostgreSQL would preserve the job's history and status even if a worker or service restarts.

3. Connecting to a decentralized GPU provider

The third component would connect the backend to the actual GPU network.

I'd start with one provider, such as Nosana, rather than trying to support every decentralized compute network from day one.

A worker would take a job from the queue, prepare it for the provider, submit it, and track its progress.

I'd also create a small adapter that gives the rest of the application a consistent way to interact with the provider.

Conceptually, it would support three operations:

typescript

interface InferenceProvider { submit(job: InferenceJob): Promise<ProviderExecution>; getStatus(executionId: string): Promise<ProviderStatus>; getResult(executionId: string): Promise<InferenceResult>; } 

These are proposed interface methods. They don't mean that Nosana or any other provider necessarily exposes these exact operations.

The idea is simple: the application shouldn't have to deal with every provider's unique API and status codes.

The adapter would handle those differences and translate them into a format the backend understands.

Once the first integration works, I could look at adding other providers, self-hosted GPUs, or even centralized AI services.

But first, I'd make one integration work properly.

4. Checking and saving the result

Getting a response from a GPU doesn't automatically mean the job is successful.

The backend still needs to check the result before marking the job as complete.

For a text-generation request, that could mean checking:

• whether the execution finished
• whether the response has the expected structure
• whether the output is within the allowed size
• whether the model information matches the request

Large results could be saved in object storage, while PostgreSQL keeps the metadata, content hash, storage location, and execution details.

I could explore something like Arweave later if there's a real need for permanent, independently accessible storage.

But there's an important distinction here: a content hash can help show that a file hasn't changed. It doesn't prove that the GPU generated the correct answer.

For the first version, I'd focus on handling jobs and results correctly before adding more infrastructure.

What the whole process looks like

Here's how I'd expect a request to move through the system:

User submits a request | v API receives and validates it | v Job is saved in PostgreSQL | v BullMQ / Redis manages the job | v Worker sends it to the GPU provider | v GPU processes the request | v Backend retrieves and checks the result | v Result and final status are saved | v User retrieves the result 

Of course, things won't always go this smoothly.

A worker could crash, a provider could become unreachable, or a result could arrive when the backend isn't ready to save it.

That's why retries and recovery need to be part of the design from the beginning, rather than something added after the first failure.

What happens when things go wrong?

This is where I'd spend a lot of my attention.

Deadlines and recovery

Every job would have a deadline, and each provider execution would have its own ID.

If a worker loses contact with a provider, the system should first try to find out what happened.

Is the job still running? Did it finish? Did it fail?

If the provider offers a way to check execution status, the worker can use that information to decide what to do next.

If the result is still available, the backend could retrieve it instead of running the job again.

But what if the system can't determine the outcome?

In that case, it should record the uncertainty instead of blindly submitting another paid execution.

Sometimes the system won't have enough information to make a safe decision. I'd rather make that limitation visible than pretend every failure can be resolved automatically.

Preventing duplicate jobs and charges

Imagine a user submits an AI request, but their internet connection drops before the API responds.

They try again.

Without the right safeguards, the backend might create two separate jobs.

Idempotency helps prevent this.

The API would accept an idempotency key and use a database uniqueness constraint to ensure that repeated submissions with the same key return the existing logical job instead of creating another one.

I'd also give billing records their own duplicate protection and store each provider execution separately.

That way, the system can distinguish between one logical job and the different execution attempts associated with it.

However, there's a limitation worth being honest about.

Preventing duplicate jobs in my backend doesn't guarantee that the GPU provider will execute the work only once.

If a provider doesn't support safe repeated submissions or reliable status checks, the backend may not know whether an interrupted request was actually processed.

At that point, the system might need manual intervention or a decision about whether to risk another execution.

That's not a problem I'd hide behind a promise of "exactly-once execution." I'd document it and work within what the provider can actually guarantee.

Dealing with jobs that keep failing

Some jobs will fail repeatedly, even after retries.

Instead of retrying forever, I'd move them into a failure-handling workflow once they've exhausted their retry budget.

The system would preserve the error messages, execution history, and provider details so someone could investigate what happened.

An operator should be able to understand why a job failed and decide whether it's safe to try again.

Jobs shouldn't disappear silently, but retrying them endlessly isn't the answer either.

How do we know the result is correct?

This is another area that needs careful thought.

One possible approach is to run the same inference on two different nodes and compare their outputs.

That can be useful for deterministic workloads under controlled conditions.

But generative AI doesn't always return identical outputs for the same input. Sampling settings, execution environments, and implementation differences can all affect the result.

So if two nodes return different outputs, that alone doesn't prove that either one is dishonest.

I'd begin with basic validation and controlled test workloads where the expected behavior is easier to check.

Later, I could explore spot checks, redundant execution, and more advanced methods for verifying inference.

There's a trade-off, though. Running a job multiple times means paying for additional compute.

The extra verification should be worth the cost.

Keeping track of payments

I'd also keep job status and payment status separate.

A job might finish successfully while its payment is still pending. Likewise, a payment-related operation could fail even though the GPU completed the work.

The system should record estimated costs, provider-reported execution outcomes, and billing status independently.

If the provider supports an appropriate payment or escrow mechanism, I could integrate it later.

But I wouldn't promise escrow protection without confirming that the underlying provider can actually support it.

Why this matters for decentralized AI

I don't think decentralization is valuable simply because it's decentralized.

What interests me is the possibility of giving developers more flexibility and reducing their dependence on a single provider.

But that only becomes useful if integrating decentralized compute isn't a headache.

A developer should be able to submit an inference request, receive a job ID, check its status, and retrieve the result through a consistent interface.

They shouldn't have to rewrite their entire application every time they change GPU providers.

This architecture could also make it easier to choose between different providers based on cost, availability, latency, or privacy requirements.

For example, a request might go to a decentralized GPU network, a self-hosted GPU, or a centralized AI API.

Still, switching providers won't always be seamless. Different providers may support different models, input formats, performance levels, and privacy guarantees.

The adapter would make common operations easier, while clearly identifying the differences that matter.

That's the goal: reduce unnecessary complexity without pretending every provider is the same.

How I'd test the idea

This is still a proposal. I haven't benchmarked a working implementation, so I don't have results to claim yet.

I'd start with three experiments.

1. Send the same request multiple times

I'd submit the same inference request repeatedly using the same idempotency key.

Then I'd check whether the API returns the same logical job, whether duplicate queue messages are handled safely, and whether the billing records remain unique.

2. Interrupt a worker at the worst possible time

I'd simulate a worker crashing after the provider accepts a job but before the result is saved.

The question is whether the system can recover the existing execution, retrieve the result, or identify an uncertain outcome without automatically launching another paid job.

3. Measure the extra time introduced by the system

I'd compare sending a request directly to the provider with sending it through the gateway and queue.

I'd measure the extra latency, total completion time, success and failure rates, and how many provider executions were needed per logical job.

I'd also document the model, hardware or provider configuration, sample size, and testing conditions.

That way, anyone reading the results could understand what was tested and how meaningful the numbers are.

These are the experiments I'd run, not results I've already obtained.

What I'd build first

I'd keep the first version small.

With access to suitable Nosana credits, I'd start with:

• An Express.js API for submitting jobs and checking their status
• BullMQ and Redis for background processing
• PostgreSQL for job records and execution history
• One provider adapter connected to a real GPU workload
• Result storage and basic validation
• Tests for retries, recovery, and duplicate submissions

I'd leave multi-provider routing, permanent decentralized storage, on-chain escrow, and advanced inference verification for later.

There's no point building ten extra features before proving that the basic system works.

The first milestone is simple: send a request from an application to a remote GPU, get the result back, and show what happens when something goes wrong.

Final thoughts

Decentralized AI needs more than GPUs and a good token model.

It also needs the backend infrastructure that makes remote computation usable in real applications.

That means keeping track of jobs, handling retries carefully, recovering from failures, checking results, and making billing understandable.

Fortunately, we don't have to invent every building block from scratch. Many of these tools already exist.

The real challenge is putting them together in a way that accounts for the uncertainties of working with independently operated compute providers.

I'm approaching this as a backend engineer who's interested in solving a practical problem, not someone claiming to have already figured everything out.

My next step is to build a small working version, test the failure cases, and share what the measurements actually show.

If you're building in decentralized compute, I'd love to hear how you're handling execution recovery, result verification, and duplicate billing.

Those are the problems I want to understand better.