Keeping DeepTutor's Local Inference Alive Past the Twelfth Turn
The page landed at 3:12 a.m. vLLM in front of DeepTutor had been OOM-killed on turn twelve of a calculus session, and the restart threw away the thread that lesson had just built. The weights were fine. The KV cache was not.
I integrate HKUDS/DeepTutor from the outside. I do not maintain the graph. The point of the stack is a lifelong loop: student state, skill traces, and a planner that still knows what the learner missed last week. The model call is a replaceable backend. The failure is inference quietly exhausting GPU memory because the KV cache was sized for a demo and left up across real lessons.
A 70B at Q4 or AWQ fits two 48 GB cards until you preallocate a 32k cache and let sessions pile up prefixes that never repeat. Weights sat under 40 GB. By turn twelve the process was dead: not out of weights, out of KV blocks for a context the lesson never used, with automatic placement so attention and cache fought over one device. If the live thread sits only in that chat session, the kill is data loss. DeepTutor's external memory helps only if the compact flushed first. Recovery is narrow: refuse new sessions until the server is back, replay the last persisted skill trace, and do not pretend the raw transcript survived on the GPU.
Treat context length, cache dtype, and concurrency as the tutor's memory budget, not as flags copied from a model card.
vllm serve meta-llama/Llama-3.1-70B-Instruct \ --quantization awq \ --max-model-len 8192 \ --gpu-memory-utilization 0.82 \ --kv-cache-dtype fp8 \ --enable-prefix-caching \ --max-num-seqs 4 \ --max-num-batched-tokens 8192 \ --port 8000 max-model-len 8192 is deliberate. DeepTutor already keeps structured history off the GPU. Eight thousand tokens covers the system prompt, the skill focus, and a few recent turns. A 32k raw transcript duplicates that store and fragments the cache. Prefix caching pays off on the stable system prompt and skill rubric, not on unique chat, so do not size for the chat. FP8 KV stopped the slow leak. Symbolic quality dipped slightly versus FP16, mostly on carried subscripts, not the final answer. Skip FP8 for a grader that must match a reference character for character. For a tutor checking the next step, the headroom was worth it. max-num-seqs 4 refuses a burst that allocates cache it cannot finish. The load generator looks worse. Hour three looks better. That is the number that matters.
If resident memory climbs while throughput stays flat, you are leaking cache, not compute. Cut context or batch size before you add a card. Utilization at 0.95 leaves fragmentation nowhere to go, and the page comes back later.
On Ollama the same contract belongs in the Modelfile, not in a serve flag forgotten on the next deploy:
FROM llama3.1:70b-instruct-q4_K_M PARAMETER num_ctx 8192 PARAMETER num_gpu 99 PARAMETER num_thread 8 num_gpu 99 still offloads what fits. The line that matters is num_ctx. The model default is how this OOM shows up under Ollama with a worse error string. A retry will not rebuild context you never persisted.
I measure with a small async client, not one curl: time to first token and full latency on the OpenAI-compatible route, same path for local vLLM or a remote baseline. To compare cache settings with a hosted 70B, I changed only the base URL. The same tutoring prompts through B-Lost's relay gateway (base_url = "https://b-lost.com/v1") kept the harness honest: same messages, same max tokens, different place the KV cache lives. Local wins on residency and repeat-prefix hits. The relay wins as a fallback provider without a client rewrite. One cold request tells you neither.
import asyncio, time, aiohttp URL = "http://127.0.0.1:8000/v1/chat/completions" # swap to https://b-lost.com/v1/chat/completions for the relay baseline MODEL = "meta-llama/Llama-3.1-70B-Instruct" async def one(session, i): payload = { "model": MODEL, "messages": [ {"role": "system", "content": "You are a calculus tutor. Track the student's last error."}, {"role": "user", "content": f"Session {i}: differentiate x^3 * sin(x). Show two steps only."}, ], "max_tokens": 128, "stream": True, } t0 = time.perf_counter() ttft = None async with session.post(URL, json=payload) as resp: async for raw in resp.content: if ttft is None and raw.strip(): ttft = time.perf_counter() - t0 total = time.perf_counter() - t0 return ttft, total async def main(n=8): async with aiohttp.ClientSession() as session: results = await asyncio.gather(*[one(session, i) for i in range(n)]) ttfts = [r[0] for r in results if r[0]] totals = [r[1] for r in results] print(f"n={n} ttft_p50={sorted(ttfts)[len(ttfts)//2]:.3f}s " f"lat_p50={sorted(totals)[len(totals)//2]:.3f}s") if __name__ == "__main__": asyncio.run(main()) Watch memory across a few hundred calls, not the median of eight. The leak did not appear in the first batch. It appeared after prefixes stopped matching and the allocator could not return blocks max-model-len had reserved.
The trade you accept is earlier summarization. A short window forces DeepTutor to compact history before the model sees it. Compact in the tutor, then let vLLM truncate with no summary, and you drop the personalization the stack exists for. FP8 and 8k will not make 70B cheap. They keep the process up long enough that the traces still matter.
The hard call is not the dtype. It is how much of a student's week you compress before the model sees it, knowing the uncompacted transcript will kill the process that was supposed to remember them.
Where do you draw that line on a box you get paged for: summarize hard and risk a bland tutor, or keep the raw thread and let turn twelve be when the GPU forgets the student?