A server that prints batch_limit: 4 in its own health endpoint has only told you what it was configured to accept. Four slots, four concurrent generations, and four requests that actually overlap on the GPU are three separate claims, and the field covers none of them. I found that out while trying to give one 24 GB card real batching, over two evenings of staged prototypes, and every number below came off probes I ran against my own engines.
The endpoints lied by omission
The installed inference server on the card answered /health with {ok: true, busy: ..., backend: exl3, context_length: 262144} and its sibling llama-swap member reported total_slots=4 next to n_ctx=262144. Both numbers looked like capacity. Neither was measured. Reading the server source over ssh is what settled it, serve_openai.py lines 45 and 315-318 hold a single gen_lock wrapping generation, so concurrency in that file is batch-1 asyncio, one generation at a time, extra requests waiting their turn.
The engine underneath had the same split. ExLlamaV3's own generator.py defaults max_batch_size=256, then caps it to cache.num_slots, and the hybrid cache initializes at -ambs 1. So even after I deleted the lock, the default left a second serializer in front of the one I removed. Removing gen_lock alone cannot produce multiple active sequences, the flag has to move too.
That is why I stopped trusting status fields at that point and wrote probes that count jobs while they run.
Counting jobs while they run
Staged one moved GPU steps off the aiohttp event loop with serialized enqueue and cancel handling, launched isolated on its own port, /health reporting batch_limit=4 against a shared aggregate cache, 22542 MiB resident on the card. verify_batch.py exited 0, peak_active_jobs came back 2 across 8 sampled health checks, both SSE streams generating at once and reaching finish_reason=length in 4.5-4.8 seconds.
Two overlapping generations, sampled eight times, was all I could honestly claim from that build. The probe proved overlap, stream termination, and that the health endpoint stayed readable mid-prefill. Answer quality, retrieval depth, production readiness, and four live jobs were all outside what it showed, so I wrote those four limits into the worklog entry beside the number, where they stay attached to it.
Staged two added an actual admission path. verify_batch4.py observed peak_active_jobs 4 across 31 samples, every stream ending at length. verify_function.py returned 4 alongside a correctly typed add(2, 3) tool call, so batching survived a tool round-trip. verify_cancel.py came back active=0, busy=false after a client disconnected, which is the one thing a naive batcher gets wrong most often, a cancelled request leaving a ghost job holding VRAM.
The same build ran verify_health.py while long prefill was in flight: 76,056 prompt tokens processed, 73 health checks served, zero errors, maximum health latency 72 ms. That is the argument for moving generator steps off the event loop, a health endpoint behind a blocking GPU step answers once per token batch, a health endpoint on a sliced loop answers in tens of milliseconds while the same prefill runs.
The proof that belonged to a different build
Then the long-context check. batch-needle.json records 240,118 real prompt tokens, both embedded facts retrieved, finish_reason=stop, 443.45 seconds wall time. Beautiful, that is the deepest retrieval I ever measured on that engine, and it happened on staged one. Staged two holds the four-job overlap. Neither artifact validates the other's configuration, and the queue entry I wrote at 6 pm says peak 4 active jobs with a separate line for the needle run, because reading them as one result would be a claim about a build that never had both properties tested together.
One habit fixes this class of mistake: every probe line gets written next to what it sampled, its sample count, and its termination reason. A number without those three is decoration.
Context math sets the ceiling the fields ignore
The model card says 262,144 native window, 64 layers with full attention every fourth layer, so 16 KV-bearing layers, 4 KV heads at head dimension 256, ideal raw KV at the full window 16 GiB in fp16, 8 GiB at 8-bit, 4 GiB at 4-bit, before scales, recurrent state from the linear layers, or server overhead. Four independent full windows at 4-bit need 16 GiB of KV plus weights, on a card with 24 GiB total. An aggregate pool shared across requests is a smaller promise than four private windows, and the health endpoint cannot tell those two apart. Neither can /props.
The same trap one level up
Worse than the engine's echo was my own runtime's. Both agent configs set single_session_lock and the router reports a local_backend_busy banner when it refuses a lane, yet grepping the shipped Hermes source found zero implementations reading either name. A config key nothing consumes is a comment wearing a boolean. Before I flipped any routing I needed the effective whole-turn gate reconciled against the batched design, which meant the flag had to be read somewhere, and it wasn't. Dead code beside a live-looking setting is how a self-heal loop ends up re-applying a patch that changes nothing.
What shipped
Production stayed the original single-generation server both nights. Every staged prototype ran isolated on its own port, cleanup stopped only its own process, restored the services and the watchdog timer, and cleared only markers whose prior mtime and size matched what we created. One morning both llama-swap units entered stop at 10:22:09 and timed out while stopping at 10:22:14 exactly when an empty maintenance marker of unknown ownership appeared on the box; no permission was inferred for clearing another owner's flag, and idle GPU occupancy was never treated as evidence that work had finished.
So the honest sequence for adding batching to your own local engine, in order of cost: read the serializer in the serving layer, check the runtime's own batch default behind it, sample active jobs during a prefill long enough to matter, then record what each probe did not prove next to its result. Configured capacity fields are the last thing you look at, first thing you doubt.
What does your health endpoint actually measure on a busy engine, milliseconds or honesty? If yours answers with a configured number, name the field, name the line of code that sets it, and I'll tell you whether it can ever disagree with reality.
