Workshop: Fair-Share Admission Control for One Shared Model Endpoint (100 Minutes)

A single shared model endpoint rarely fails because it is slow; it fails because the queue in front of it is unfair. One client fires forty requests, fills the in-flight window, and every other client waits behind work it did not create. The workshop below treats that as a measurable engineering problem rather than a provisioning problem.

The artifact is a 100-minute session: a measurement harness, a bounded admission gate, a deficit-round-robin scheduler, and a fairness test that students can rerun without spending a single token. Every number in this article is an illustrative example, not a measurement of a live service.

What you must verify before you book the room

Three facts decide whether these exercises work as written:

1. The current allowance and rate limits. Quotas move; read the provider's own documentation in the week you teach and write the observed value on the whiteboard.
2. Whether concurrency is capped per key or per account. If caps are per key, splitting a class across keys changes the entire measurement.
3. Whether classroom bursts are acceptable under the terms. A workshop is bursty by design, and bursts are exactly what limits exist to stop.

The substrate used in the outline is MonkeyCode's free model access, with its free server option hosting the proxy, both presented by the project as availability claims rather than capabilities verified here. Project promotion describes a 10,000,000-token allowance for free model access; treat that figure as operator-supplied, re-check it against current terms, and do not build a class budget on a number you have not confirmed today. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Separate the two failures before writing any code

'The endpoint is slow' covers two different problems that need different fixes:

• Overload: too many requests in flight, so p95 latency and error rate rise together.
• Unfairness: aggregate throughput is acceptable, but one client's median wait is many times another's.

Admission control addresses the first by bounding concurrency and rejecting early with a retry hint. Scheduling addresses the second by guaranteeing each client a floor on the permits that do exist. Students should be able to name which failure they observed before they touch a scheduler.

Timing for a 100-minute session

Exercise 1 (20 minutes): sweep concurrency and record the curve

Start with measurement, because a queue tuned against a guessed latency budget is tuned against nothing. The harness below is illustrative and was not executed for this article; it sweeps fixed concurrency levels and reports successes, errors, and latency percentiles.

# illustrative only - replace ENDPOINT and payload; not executed for this article import asyncio, statistics, time import httpx ENDPOINT = 'https://your-endpoint.example/v1/chat/completions' async def one_call(client, sem, payload): async with sem: t0 = time.perf_counter() status, ok = None, False try: r = await client.post(ENDPOINT, json=payload, timeout=60) status, ok = r.status_code, r.status_code < 400 except Exception as exc: status = type(exc).__name__ return {'ok': ok, 'status': status, 'ms': (time.perf_counter() - t0) * 1000} async def sweep(payload, levels=(1, 2, 4, 8, 16), n=24): report = {} async with httpx.AsyncClient() as client: for c in levels: sem = asyncio.Semaphore(c) rows = await asyncio.gather(*(one_call(client, sem, payload) for _ in range(n))) lat = sorted(r['ms'] for r in rows if r['ok']) report[c] = { 'ok': sum(r['ok'] for r in rows), 'errors': sum(not r['ok'] for r in rows), 'p50_ms': round(statistics.median(lat)) if lat else None, 'p95_ms': round(lat[int(0.95 * (len(lat) - 1))]) if lat else None, } return report if __name__ == '__main__': payload = {'messages': [{'role': 'user', 'content': 'Reply with the word ok.'}]} print(asyncio.run(sweep(payload))) 

Ask students to plot the sweep before interpreting it. An illustrative shape, useful only for teaching the reading, looks like this:

Two rules keep the reading honest. Record the timestamp and the advertised limits beside the table, because a curve without a date is folklore. Keep payloads identical across levels, since prompt length moves latency more than concurrency does.

Exercise 2 (15 minutes): a bounded admission gate

The gate needs three behaviours: pass when capacity is free, wait when the queue has room, and fail fast when it does not. A token bucket in front of a bounded waiter count gives all three without external dependencies.

# illustrative skeleton - single process, no persistence import asyncio, time class Admission: def __init__(self, rate_per_s, burst, max_waiters, max_wait_s): self.rate, self.burst = rate_per_s, burst self.tokens, self.stamp = burst, time.monotonic() self.max_waiters, self.max_wait_s = max_waiters, max_wait_s self.waiters = 0 def _refill(self): now = time.monotonic() self.tokens = min(self.burst, self.tokens + (now - self.stamp) * self.rate) self.stamp = now async def acquire(self): deadline = time.monotonic() + self.max_wait_s while True: self._refill() if self.tokens >= 1: self.tokens -= 1 return if self.waiters >= self.max_waiters: raise BufferError('queue_full') # map to 429 plus a retry hint  self.waiters += 1 try: need = (1 - self.tokens) / self.rate if time.monotonic() + need > deadline: raise TimeoutError('deadline_exceeded') await asyncio.sleep(min(need, 0.05)) finally: self.waiters -= 1 

Two details matter more than the code. Rejections must carry a retry hint, so a client backs off instead of hammering the gate. The waiter count must be bounded, because an unbounded queue converts an overload failure into a memory failure. Have students state the failure mode this gate catches and the one it does not.

Exercise 3 (20 minutes): give every client a floor with deficit round robin

Admission control still lets the loudest client take every permit it can reach. Deficit round robin fixes that by accumulating credit per client and spending it in rounds, so a client that is quiet now is not starved later.

from collections import deque class DeficitRoundRobin: def __init__(self, weights): self.weights = dict(weights) self.deficit = {c: 0 for c in self.weights} self.queues = {c: deque() for c in self.weights} def enqueue(self, client, item): self.queues[client].append(item) def next(self): for client in self.weights: # fixed iteration order  if not self.queues[client]: continue self.deficit[client] += self.weights[client] if self.deficit[client] < 1: continue self.deficit[client] -= 1 return client, self.queues[client].popleft() return None 

Have students argue about weight semantics out loud. A weight of four does not promise four times the throughput when request costs differ; it promises four times the turns, which is the same thing only for uniform work.

Exercise 4 (20 minutes): prove no starvation offline

The fairness check below is pure Python, so it runs offline and costs nothing. Queue a large backlog across three clients weighted 4:1:1 and assert the split lands within one item of the ideal share.

from collections import Counter def drain(sched, backlog, tolerance=1): for client, count in backlog.items(): for i in range(count): sched.enqueue(client, f'{client}-{i}') served, guard = Counter(), 0 while guard < 100_000: item = sched.next() if item is None: # fix: replenish a full round and retry instead of stopping  for c in sched.weights: sched.deficit[c] += sched.weights[c] if not any(sched.queues.values()): break continue served[item[0]] += 1 guard += 1 total = sum(served.values()) ideal = {c: total * w / sum(sched.weights.values()) for c, w in sched.weights.items()} return served, ideal weights = {'a': 4, 'b': 1, 'c': 1} sched = DeficitRoundRobin(weights) served, ideal = drain(sched, {'a': 40, 'b': 10, 'c': 10}) assert all(abs(served[c] - ideal[c]) <= 1 for c in weights), (served, ideal) 

Run it once without the replenish branch and the dispatcher will under-serve, which is the teaching moment. An illustrative passing run of that shape lands near 40, 10, and 10; a 60/0/0 result means the round order was reset after every dispatch, which is exactly what the deficit counter prevents.

Debrief (15 minutes): a decision table, not a slogan

Close the session by asking each team to fill the last two rows with the limits they personally recorded. A decision table built on unverified limits is a rumour with formatting.

Limitations and who should skip this

• Single process only. Fairness state dies with the process, and two replicas each believe they own the whole budget.
• Weights count turns, not tokens. Heterogeneous prompt sizes break the arithmetic; cost-weighted scheduling needs a different permit unit.
• No guarantee follows from a free tier. The gate bounds damage rather than promising service.
• Double queuing costs latency. If the provider already enforces fair per-key behaviour, a second local queue adds wait for nothing.
• Side effects need keyed retries. A gate that retries without idempotency keys can duplicate work.

Skip this design if you are a single developer on a single-user tool, or if a fast error is more useful to your users than a queued success. Skip it too if you need contractual latency, because free access is the wrong substrate for that promise.

What to change before production

Run Exercise 1 first and write the observed numbers on the board before quoting anyone's figures, including the illustrative ones above. Then convert weights from turns to estimated cost per request, and move the deferral state into shared storage so two replicas cannot overspend the same allowance.

If you want to teach this without provisioning, free model access to call the endpoint and a free server option to host the gate are enough for Exercises 1 and 2, with Exercises 3 and 4 running entirely offline.