If you run LLMs in production—whether OpenAI, Anthropic, Gemini, or self-hosted Ollama/vLLM—you have probably been burned by silent model drift. An upstream provider tweaks quantization or safety filters overnight, refusal rates jump, or an update subtly breaks edge-case prompts without any announcement.

Most observability platforms solve this in two frustrating ways:

  1. They require shipping all your raw customer prompts to a third-party cloud.
  2. They rely on static threshold alerts (like alerting whenever response length changes), which spam false alarms whenever a model is naturally verbose.

Over the past few months (I’m 15, building on a basic setup with no GPU cluster or funding), I built Wallras—a single-container, self-hosted transparent proxy (base_url swap, ~3–5ms overhead) that catches silent drift, model swaps, and confident hallucinations while keeping 100% of your data on your own servers.

Here is a look under the hood at four engineering problems I had to solve while building it.

1. Replacing Naive Drift Thresholds With Permutation Math

A static rule like "alert if average response length shifts by 15%" breaks immediately on real traffic because some prompts naturally produce longer answers than others.

Instead, Wallras combines three statistical signals:

  • Population Stability Index (PSI) across response-length buckets
  • Jensen-Shannon Divergence (JSD) over the top 300 vocabulary tokens
  • Two-Proportion Z-Tests (|z| > 3) for multilingual refusal-rate spikes

To avoid crying wolf, it runs 150-iteration permutation shuffling on baseline vs. recent outputs to measure that specific model's own natural random variance. It only raises a drift flag when a shift exceeds both a practical floor and the model's 99th-percentile random noise ceiling (~1.7% false-alarm rate on noisy baselines).

You can read the exact formulas and thresholds on the measurement math page.

2. Catching "Confident Lies" via Logprob Entropy

LLMs frequently sound 95% confident in plain English even when they are guessing a date, API parameter, or legal citation.

When token logprobs are available (from OpenAI, OpenRouter, Ollama, or vLLM), Wallras computes Shannon entropy across the top alternative tokens alongside the geometric mean and minimum token probability:

internal_certainty = 0.5 * geometric_mean_prob + 0.5 * min_token_prob

Blending the average token probability with the single weakest token ensures that one guessed fact buried inside an otherwise fluent paragraph still triggers a mismatch flag when the model's text sounds overly certain.

3. Forensic Deletion: Wiping SQLite WAL Files & Free Pages

One of the trickiest security bugs I found during testing was with SQLite deletion requests.

When a user triggers a GDPR deletion—or turns on field-level AES-256-GCM encryption for existing records—running a standard SQL DELETE or UPDATE removes the row logically, but leaves the plaintext sitting inside SQLite's write-ahead log (.wal) and unallocated database file pages until reused.

To fix this properly:

  • Wallras overwrites removed content with zeros, flushes the write-ahead log after deletion, and compacts the database file once existing data is encrypted.
  • Every record is also linked in a tamper-evident hash chain—and during deletion, the random salt mixed into that record's fingerprint is wiped ("crypto-shredding") so an attacker cannot brute-force guesses against the ledger hash.
  • An automated attack test in CI inspects the raw binary SQLite file and .wal file after deletion to verify zero plaintext remains.

All 366 automated/attack tests, Semgrep/Bandit/Trivy scans, and live A+ Mozilla Observatory / SSL Labs checks are documented on the Wallras security page.

4. Making It Agent-Operated via Zero-Dependency MCP

Developers increasingly debug inside Claude Code or Cursor rather than clicking through separate dashboards.

Wallras ships a zero-dependency standard-library Python MCP server (wallras-mcp) and an llms.txt spec:

  • Read-only by default: Exposes 22 read-only inspection tools (wallras_drift, wallras_traces, wallras_blind_spots, wallras_prompt_diff) so your coding agent can diagnose production issues safely.
  • Gated prompt promotions: Write tools (--enable-writes) allow Cursor or Claude to promote a prompt update to production only if it passes your regression test suite first.

Looking for Honest Engineering Feedback

Payments aren't even turned on yet while I validate proof-of-concept—every plan up to Scale is 100% free to try with zero telemetry or prompt data leaving your machine.

Since I don't have a multi-GPU rig or heavy API budget to stress-test high concurrency, I would love feedback from anyone running local models or production LLM apps:

Feel free to roast the math, proxy overhead, or SQLite/Postgres architecture in the comments!