48-Hour Field Notes: I Raised Concurrency Until the Queue, Not the CPU, Told Me to Stop

I spent two nights tuning a nightly batch that talks to a model endpoint, and I picked the wrong instrument first: CPU. My laptop idled at single-digit load while the job's tail latency climbed past every budget I had set. So I did the obvious thing, doubled the client concurrency, and watched the whole run get slower. Here is what the next 48 hours taught me about measuring the right queue.

What I tested, and what this test cannot tell you

I ran a fixed set of 24 short prompts at four client concurrency levels on one laptop, over one home connection, across two evenings. The goal was never to rank a provider; it was to answer a question about my own code. Is my batch slow because of what I send, or because of how many things I send at once?

A sweep this small cannot tell you:

• provider-wide latency, because one laptop and 24 requests are not a benchmark;
• anything about a paid tier or region you have not actually tested;
• what happens past the concurrency levels you actually ran.

Free-tier limits, server sizing, and queue behavior change without notice, so check the current provider documentation before copying any number from this post. I deliberately publish no absolute latency claims, only the shape of the curve and the harness that produced it.

The first thing that broke: I set concurrency from a CPU chart

My original limit came from htop. If the box was not busy, I reasoned, the work must be waiting on the network, and more in-flight requests would fill the gap. That logic holds for local CPU work and fails badly for a call whose cost is paid somewhere else.

The failure was not subtle. At the higher concurrency level, p50 barely moved while p95 nearly doubled, which is the signature of contention rather than capacity. I had added workers to a queue I could not see, and the queue answered by making everyone wait longer.

The harness I would keep

The artifact I would repeat is small enough to paste into a repo and boring enough to trust. It has three parts: a fixed request set, a split between queue wait and service time, and a canary probe that runs outside the batch.

Step 1: a fixed request set

Vary the concurrency, never the payload. If the prompts change between runs, you are measuring two things at once and learning nothing.

# prompts.py PROMPTS = [ f'Summarize this log line in one sentence: ERROR {i} upstream timeout after 30s' for i in range(24) ] 

Step 2: separate queue wait from service time

The single most useful change I made was recording two timestamps around the semaphore, not one around the request. That split tells you whether your own limiter is the bottleneck or something downstream is.

# sweep.py import asyncio, json, time import httpx from prompts import PROMPTS def pct(values, p): if not values: return float('nan') s = sorted(values) k = max(0, min(len(s) - 1, round((p / 100) * (len(s) - 1)))) return s[k] async def one_call(client, sem, prompt, idx, rows): t_enqueue = time.perf_counter() async with sem: t_start = time.perf_counter() status = 'ok' try: r = await client.post('/v1/chat', json={'prompt': prompt}, timeout=60) r.raise_for_status() except Exception as exc: status = type(exc).__name__ t_end = time.perf_counter() rows.append({ 'idx': idx, 'status': status, 'queue_wait_s': round(t_start - t_enqueue, 3), 'service_s': round(t_end - t_start, 3), }) async def run(concurrency, base_url): sem = asyncio.Semaphore(concurrency) limits = httpx.Limits(max_connections=concurrency, max_keepalive_connections=concurrency) rows = [] async with httpx.AsyncClient(base_url=base_url, limits=limits, timeout=90) as client: await asyncio.gather(*[ one_call(client, sem, p, i, rows) for i, p in enumerate(PROMPTS) ]) ok = [r for r in rows if r['status'] == 'ok'] service = [r['service_s'] for r in ok] qwait = [r['queue_wait_s'] for r in ok] total = sum(service) + sum(qwait) return { 'concurrency': concurrency, 'ok': len(ok), 'failed': len(rows) - len(ok), 'queue_share': round(sum(qwait) / total, 3) if total else None, 'p50': round(pct(service, 50), 3), 'p95': round(pct(service, 95), 3), } if __name__ == '__main__': import sys base, conc = sys.argv[1], int(sys.argv[2]) print(json.dumps(asyncio.run(run(conc, base)))) 

Run it once per level and keep every row, failures included:

for c in 1 2 4 8; do python sweep.py "$MONKEYCODE_BASE_URL" "$c"; done 

The numbers below are illustrative output from a dry run against a stub server, not a provider measurement. What matters is the shape, not my values.

{"concurrency": 1, "ok": 24, "failed": 0, "queue_share": 0.002, "p50": 0.9, "p95": 1.1} {"concurrency": 4, "ok": 24, "failed": 0, "queue_share": 0.41, "p50": 1.2, "p95": 3.8} {"concurrency": 8, "ok": 21, "failed": 3, "queue_share": 0.67, "p50": 1.4, "p95": 7.9} 

Step 3: a canary probe that runs outside the batch

When the tail grows, you need to know whether the endpoint is backed up or your own loop is. A single request every ten seconds answers that, and it costs almost nothing:

while true; do curl -s -o /dev/null -w '%{time_total}\n' \ -X POST "$MONKEYCODE_BASE_URL/v1/chat" \ -H 'content-type: application/json' \ -d '{"prompt":"ping"}' >> canary.log sleep 10 done 

If the canary stays flat while the batch's p95 climbs, the pressure is yours. If the canary climbs too, stop adding workers, because you are now just queueing harder.

The second thing that broke: I trusted a percentile from 24 samples

A p95 over 24 rows is close to an anecdote. I quoted it in a commit message anyway, then spent an hour confused when a rerun disagreed.

Treat the percentile as a trend indicator and the queue share as the decision input. When two sweeps disagree at the same concurrency, you do not have enough samples, and the honest move is to repeat the run rather than pick the friendlier number.

The third thing that broke: my semaphore and my connection pool were different limits

I set a semaphore of eight and assumed eight requests in flight. The client had its own pool, and the effective concurrency was whatever survived both. Two limiters producing one apparent limit is a great way to tune fiction.

# match the pool to the semaphore, or measure the smaller one limits = httpx.Limits(max_connections=8, max_keepalive_connections=8) 

One more silent failure: exceptions recorded with a latency and no status will drag p50 down, because failures are often fast. Filter on status before you compute anything, then report the failure count beside the percentiles.

The decision table I now keep next to the harness

Where free model access and a free server actually fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A sweep is only useful if you can afford to repeat it, and repeated runs are exactly what a paid endpoint discourages during exploration. MonkeyCode's free model access and free server option are what let me run the same 24-prompt set at four concurrency levels twice, then throw the first evening away as warm-up.

That is the whole role it played. The harness does not care which endpoint answers, and the free tier tells you nothing reliable about a paid tier's latency. If you ship elsewhere, run the identical script there before you believe any of this.

What I would repeat, and what I would skip

• Repeat: the two-timestamp split, because it names the bottleneck instead of hinting at it.
• Repeat: the canary probe, because it separates your loop from everything else.
• Repeat: storing raw rows, so you can recompute after you realize a calculation was wrong.
• Skip: tuning from a CPU chart, which measures your machine and not the queue.
• Skip: quoting a percentile from one small sweep as if it were a property of the service.

Who should not do this

If you already have provider-side metrics, or a load-testing setup with real traffic shapes, this harness is a downgrade. If your batch is not latency-sensitive, or if a single retry costs more than the tail you are chasing, the tuning effort is not worth it either. And if you need statistically defensible numbers for a purchasing decision, hire the sample size instead of borrowing mine.

Two nights bought me one honest sentence for the runbook: measure queue wait before you add workers. If you want a first data point without a card, MonkeyCode's free models and free server are a reasonable place to run this harness once, and then re-run it wherever your traffic actually lives.