What 586 Real Traces Taught Us About LLM Agent Determinism

How temperature selectively degrades agent reasoning — and what that means for observability.

The Setup

On November 3, 2026, we ran 586 traces of a LangGraph folder-research agent against AgentPulse, our metadata-only observability platform. The agent used Ollama with qwen2.5:7b on an RTX 4060 (Legion), plus 6 traces from a Pixel 8 Pro running qwen3b via llama.cpp in Termux.

The experiment: same 10 questions, same corpus, four temperatures (0, 0.3, 0.7, 1.0). We measured trace shapes — span counts, durations, tool calls — to understand how sampling temperature affects agent behavior.

Why this matters: If you're building observability for AI agents, you need to know what "normal" looks like. Without a determinism baseline, every alert is a guess.

Finding 1: Temperature 0 Is Deterministic

At temp=0, qwen2.5:7b produced identical trace shapes across 20 runs per question. Nine out of ten questions had standard deviation 0.0 on span counts. The tenth (Q1) had sd 0.4 — still remarkably tight.

This is the baseline. When temp=0, trace-shape changes are genuine regression signal, not noise. Your alert thresholds can be tight.

Finding 2: Temperature Degrades Complex Reasoning First

Here's what surprised us. Temperature doesn't add uniform noise. It selectively degrades complex reasoning while simple lookups survive.

At temp=0.3:

• Q1 (complex): sd 1.7 — already breaking
• Q4 (complex): sd 2.0 — breaking
• Q3, Q7, Q9 (simple): sd 0.0 — perfect

At temp=1.0:

• Q7 (simple): sd 1.6 — finally breaks
• Q8 (simple): sd 0.0 — still perfect

The pattern: Multi-file synthesis questions degrade at 0.3. Simple lookups survive to 1.0. The breakdown is question-dependent, not just temperature-dependent.

Implication: Your eval thresholds can't just be "wider at higher temp." They need to be per-question-type. A simple lookup at temp=0.7 should still be deterministic. A complex reasoning task at temp=0.3 is already noisy.

Finding 3: Compare Distributions, Not Means

Our first eval prototype used mean comparison. It got 7/10 "correct" — but the 3 false positives were instructive.

At temp=0.7, traces become bimodal: sometimes the agent does full work (8 spans), sometimes it produces a degenerate 2-span trace. The mean shifts, triggering a "regression" alert. But the median and IQR tell the real story: high variance with overlapping IQR = noise, not regression.

The rule:

• Shifted median + similar IQR = regression (alert)
• Same median + wider IQR = noise (don't alert)
• Bimodal = noise (the agent is inconsistent, not systematically worse)

We validated this on 500 traces. Mean-based detection fails. Median/IQR works.

Finding 4: Three Devices, One Platform

We validated AgentPulse across three very different machines:

• Legion (RTX 4060): 580 traces, qwen2.5:7b, ~6s/trace
• Dell (i5-1135G7): 72h soak running, synthetic load
• Pixel 8 Pro (Tensor G3): 6 traces, qwen3b via llama.cpp, Termux

The phone was the hard one. We fought through:

• Rust ABI incompatibilities (uuid_utils on Python 3.13)
• Missing SSH client
• Model server configuration
• Registry label approval

But it works. The phone agent runs, generates traces, and uploads them to the same workspace. Metadata-only observability works on mobile.

What We Built

From these findings, we built the quality-warning evidence adapter — the statistical engine that connects real trace data to AgentPulse's alerting system.

Core logic:

1. Compare baseline vs candidate using median and IQR (not mean)
2. Infer question type from baseline shape (simple_lookup vs complex_reasoning)
3. Adjust thresholds by question type (complex gets 1.5x tolerance)
4. Classify: no_change, noise, regression, or insufficient_data
5. Bimodal distributions → noise, never regression

The adapter is live on GitHub (codex/quality-warning-evidence, commit 3df33f0). The API integration follows.

The Takeaway

If you're building agent observability:

1. Establish your temp=0 baseline first. Without it, you're guessing.
2. Thresholds must be per-question-type. Complex reasoning degrades faster.
3. Compare distributions. Means lie when data is bimodal.
4. Test on real hardware. The phone taught us more than the GPU did.

Temperature doesn't just make agents "more random." It selectively kills complex reasoning first. Your monitoring needs to know the difference.

586 traces. 3 devices. 1 night. The data is real, the findings are publishable, and the code is on GitHub.