Simon Willison shipped llm 0.37 on 2 November. The release adds gpt-6.1-sol and gpt-6.1-terra, a new llm openai endpoint add command for arbitrary OpenAI-compatible endpoints, and switches the default model to gpt-6-luna (half the price of the previous default). The more interesting line for people who actually ship tools is the embeddings change: the library now accepts different content-types for binary payloads, which is required for models like Gemini Embedding 2 that can embed text, audio, images, and video in one API.
That is a small API surface change. It is also the exact seam where most local real-time systems still fall over.
Embeddings are not free
An embedding call that accepts raw audio is useful for retrieval, speaker clustering, or “find the similar turn” later. It is almost never the thing you want on the hot path of a live call.
Latency budgets for a dual-track interview or call assistant look roughly like this:
- Capture buffer: 10–20 ms (WASAPI exclusive) or 40–100 ms (shared)
- Resample + VAD: 5–15 ms
- Streaming STT first partial: 150–400 ms depending on model and network
- Overlay render: <16 ms if you care about feeling responsive
A multimodal embedding model, even a fast one, is usually measured in hundreds of milliseconds once you count serialization, network (or GPU queue), and the fact that you rarely want to embed every 200 ms chunk. You embed the finalized turn, or you embed offline. The streaming path stays lean.
The practical pattern is the same one that shows up in any real agent loop once tool count or payload size leaves the toy regime: keep the expensive representation out of the critical path until you actually need it.
Dual-track capture is still the hard part on Windows
None of the new embedding APIs change the fact that a Windows call has two independent audio streams that must stay separate:
- Microphone (your voice)
- Loopback of the render endpoint (the other side)
Mixing them early destroys speaker labels and forces the recognizer to handle two languages or two accents in one beam. The correct architecture is two independent pipelines that only meet at the UI or the final transcript store.
On Windows the loopback path is still “open an input stream on a render device” via WASAPI. In Rust with cpal it looks like this:
// Simplified — real code also handles format negotiation and device changes use cpal::traits::{DeviceTrait, HostTrait, StreamTrait}; fn open_loopback(host: &cpal::Host, device_name: &str) -> Result<cpal::Stream, anyhow::Error> { let device = host .output_devices()? .find(|d| d.name().ok().as_deref() == Some(device_name)) .ok_or_else(|| anyhow::anyhow!("device not found"))?; let config = device.default_output_config()?; // cpal treats a render device opened as input as loopback let stream = device.build_input_stream( &config.into(), |data: &[f32], _: &cpal::InputCallbackInfo| { // push to ring buffer owned by the STT session // never block here }, |err| eprintln!("loopback error: {err}"), None, )?; stream.play()?; Ok(stream) } The callback must not allocate, must not lock a mutex that the UI or network thread might hold, and must tolerate the device disappearing mid-call. Those constraints are unchanged by any CLI or embedding update.
Shared mode lets other apps keep playing sound; exclusive mode drops latency but steals the device. Most interview helpers stay in shared mode and accept the extra 20–40 ms because the alternative is “your headphones go silent the moment the tool starts.”
Where the new llm features actually help
Once the turns are finalized you can do useful offline or near-line work with the same CLI that developers already use for everything else:
# embed a finalized wav snippet (content-type now respected) llm embed -m gemini-embedding-2 --binary --content-type audio/wav turn_incoming.wav # or just chat with a cheaper default while you triage logs llm -m gpt-6-luna "summarize the last 10 turns, mark code-switching" The llm openai endpoint add command is the BYOK escape hatch. Point it at a local llama.cpp server, an Ollama endpoint, or a company proxy and the rest of your scripts keep working. That is the same pattern you want inside a desktop app: the STT and translation models are configurable, the capture and overlay code is not.
AI-assisted PRs are mentioned in the release notes. Willison notes that most of the fixes in 0.37 came from contributors using models, and that triage with a stronger model kept the review load manageable. The same dynamic appears in any project that ships frequently: the volume of patches rises, the signal-to-noise depends on how good your local evaluation harness is. For a real-time audio tool the harness is “does the overlay stay under 200 ms end-to-end on a mid-range laptop while both tracks are live.”
Tradeoffs that do not go away
- Embedding the raw audio is tempting for speaker diarization. It is usually worse than running a proper streaming diarizer or simply trusting the two capture paths you already separated.
- Local models still lose to cloud STT on rare languages and heavy accents. The winning design keeps the key local and lets the user swap the endpoint.
- Overlay capture exclusion (so screen-share does not pick up the helper window) is a Windows-specific problem that no embedding API solves.
The models keep getting cheaper and more multimodal. The capture graph, the buffer discipline, and the decision of what stays on the hot path do not.
I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.
