Your ERP just called an LLM synchronously. The model took 31 seconds. The ERP gave up at 30. Now 1,200 orders are stuck between "paid" and "shipped", and nobody knows which ones the AI already processed.
AI Middleware Architecture
The mistake is treating an LLM like a database call. It isn't one. It's slow, it's non-deterministic, and it fails in ways your ERP was never designed to handle. So you never connect them directly. You put a middleware layer in between, and that layer has one job: absorb the chaos.
Step one: the ERP fires a webhook and gets an immediate response. 200 milliseconds, done. The ERP is free.
Step two: the event lands in a queue. Not processed, queued. Now you control throughput, retries, and rate limits against the model provider.
Step three: stateless workers pull jobs and call the model. They scale horizontally: two workers on a quiet Tuesday, 40 on Black Friday.
Step four is the one most teams skip: validation. The model's answer is untrusted input. It gets parsed against a strict schema before it ever touches your ERP. If confidence is low, it goes to a human. Every decision gets written to an audit log. The ERP never waits, and the model never writes directly to your system of record.
Ingestion: ack fast, process later
The ingestion endpoint does three things before it answers.
First, it verifies the webhook signature. This endpoint can trigger financial operations, so unsigned requests die here.
Second, it parses the payload with Zod. Malformed events fail fast, before they cost you a single token.
Third, and this line matters, the job ID is the ERP's event ID. ERPs retry webhooks. If the same order arrives three times, the queue deduplicates it. That's your idempotency, in one parameter.
Then the endpoint returns 202 Accepted, not 200. It tells the ERP: received, not finished.
// webhook.ts app.post('/erp/events', async (req, res) => { if (!verifySignature(req)) return res.sendStatus(401); const event = ErpEvent.parse(req.body); await queue.add(event.type, event, { jobId: event.id }); res.sendStatus(202); }); The AI worker: schema, gate, audit
The worker handles a logistics routing case: given an order, pick the carrier and the warehouse. Notice the schema. The carrier isn't free text; it's an enum. The model cannot invent a carrier that doesn't exist in your contracts. The worker requests structured output from the model, and then parses it again anyway. Trust, but verify.
Then comes the confidence gate. Below 0.85, the decision goes to a human review queue instead of the ERP. You tune that threshold with real data, not intuition. Only then does the worker write back to the ERP and log the decision, including the model's reasoning.
// routing.worker.ts const RoutingDecision = z.object({ carrier: z.enum(['DHL', 'FEDEX', 'LOCAL_FLEET']), warehouseId: z.string(), confidence: z.number().min(0).max(1), reason: z.string(), }); new Worker('erp', async (job) => { const order = job.data; const decision = RoutingDecision.parse( await llm.generate({ schema: RoutingDecision, input: buildContext(order) }) ); if (decision.confidence < 0.85) { return reviewQueue.add('routing', { order, decision }); } await erp.updateShipment(order.id, decision); await audit.log(order.id, decision); }); What the business gets
- 0 ms AI latency added to the ERP
- 0 Duplicate orders processed
- 100% AI decisions auditable
What does this buy the business?
First, your ERP is isolated from AI latency. Model providers have bad days. Your order pipeline doesn't have to.
Second, idempotency. No order is routed twice. No invoice is flagged twice. In finance and logistics, that's the difference between an automation and an incident.
Third, auditability. When the CFO asks why an invoice was flagged, or why a shipment went to the wrong warehouse, you have the input, the output, the model version, and the reasoning for every decision.
The numbers are targets for an illustrative scenario, not measurements.
Same middleware, new use case
The pattern is also reusable. Swap the schema and the prompt, and the same middleware runs invoice analysis: duplicate charges, pricing mismatches against contracts, anomalies in payment terms. Same queue, same validation, same audit trail.
The model is the least important part of this architecture. The layer around it is what makes it enterprise-grade.
▶️ Watch the video: https://www.youtube.com/watch?v=JQ25ZvyS3Uk
💻 Full code: https://github.com/NodeNeuralAI/examples/tree/main/v1-ai-middleware
🔔 More enterprise AI architecture every week: https://www.youtube.com/@NodeNeuralAI?sub_confirmation=1
