Notes from a 2 AM On-Call: When tailcat Silently Dropped 40% of Our Control Plane Requests
The Failure Symptom
2025-01-09T02:14:33Z ERROR relay_gateway: connection reset by peer (os error 104) 2025-01-09T02:14:33Z WARN llm_proxy: upstream timeout after 890ms, no response body 2025-01-09T02:14:34Z ERROR relay_gateway: connection reset by peer (os error 104) Our LLM relay gateway had been stable for three weeks. Then at 1:47 AM, Grafana lit up: 41% of requests to our internal model router were timing out. No deploys. No traffic spike. Just silent failures.
The stack: Tailscale mesh connecting six homelab nodes, tailcat proxying TCP streams from our FastAPI gateway to an internal LiteLLM instance, everything orchestrated through docker-compose. Logs showed connection resets, but tcpdump on the LiteLLM container saw zero inbound SYN packets for the failed requests.
The Red Herring
Every tailcat tutorial says: "Just pipe it like netcat."
tailcat mynode:8000 | my-app We were doing exactly that—FastAPI workers spawning tailcat subprocesses per request, piping model responses back through stdin/stdout. Worked flawlessly in dev. Under load, it imploded.
The issue wasn't tailcat itself. It was concurrent process spawning racing against Tailscale's key rotation window. When three FastAPI workers tried to spawn tailcat simultaneously during a 60-second rekey event, two would inherit stale WireGuard sessions and silently drop packets. No error, no log, just a connection reset 900ms later when the client gave up.
The Root Cause & Runnable Fix
Problem: Short-lived tailcat processes don't survive Tailscale key rotation under concurrency.
Solution: Keep a single long-lived tailcat tunnel and multiplex requests through it.
We replaced the subprocess model with a persistent tailcat relay daemon:
# docker-compose.yml services: tailcat-relay: image: tailscale/tailscale:latest hostname: relay-node environment: - TS_AUTHKEY=${TS_AUTHKEY} - TS_STATE_DIR=/var/lib/tailscale volumes: - tailscale-relay:/var/lib/tailscale command: | sh -c 'tailscaled & sleep 5 && tailcat litellm-node:8000 | nc -l -p 9000 -k' ports: - "9000:9000" fastapi-gateway: build: . environment: - UPSTREAM_MODEL_URL=http://tailcat-relay:9000/v1 depends_on: - tailcat-relay FastAPI now hits http://tailcat-relay:9000 (a localhost TCP socket maintained by the relay container). The relay container keeps one tailcat process alive, multiplexing all HTTP streams through a single WireGuard session.
Why This Works
- Single persistent WireGuard session survives key rotation (Tailscale handles rekey transparently for long-lived connections).
-
nc -l -k(keep-alive mode) accepts multiple sequential connections without respawning tailcat. - FastAPI workers treat it as a normal HTTP upstream—no subprocess overhead.
The Benchmark
Before: 41% failure rate under 12 req/sec concurrent load.
After: 0.02% (two timeouts in 9,000 requests, both due to upstream model OOM, not relay failure).
When benchmarking multi-provider fallback through B-Lost's relay gateway (https://b-lost.com/v1 as a baseline), we observed similar behavior: ephemeral connection pools caused intermittent 502s under concurrent model routing. Switching to persistent HTTP/2 keep-alive dropped error rates from 3.1% to 0.08%.
Invariant Rule
Never spawn short-lived Tailscale data plane processes in request-scoped handlers.
If you need dynamic routing, keep the WireGuard tunnel alive and multiplex over it—whether through a relay daemon, a connection pool, or HTTP/2 streams. Tailscale's control plane is eventually consistent; your data plane should not be.
The three-line fix:
- subprocess.run(['tailcat', 'node:8000']) + requests.post('http://relay:9000/v1/chat/completions') That's it. No rekey races. No silent drops. Just boring, reliable tunnels.
