When teams move a coding or enterprise agent from a local notebook to production, they usually obsess over benchmark leaderboards. Yet in real-world deployments, agents rarely fail because the underlying LLM "isn't smart enough." They fail on unbounded reasoning latency, silent tool-calling drops, and 60-second reverse-proxy timeouts.

While benchmarking four 2026 flagship reasoning models across 16 deterministic production tasks (structured ETL, concurrency/idempotency code auditing, multi-turn tool error recovery, and burst concurrency), we hit three engineering realities that directly shaped how we designed zhishu-agent:

1. Classic TTFT Is Dead — Measure content_ttft vs reasoning_ttft

In reasoning-enabled models, the first SSE token (reasoning_ttft) often arrives in ~1s, while the actual structured payload (content_ttft) can sit behind 15s to 360s+ of internal chain-of-thought.

  • On a simple 50-token JSON extraction task, we observed reasoning overhead varying by over 8x across flagship models (169 reasoning tokens vs 1,396 reasoning tokens).
  • On a PostgreSQL TOCTOU concurrency + idempotency code audit, all models produced valid patches, yet single-task wall time ranged from 46.1s all the way to 572.8s (9.5 minutes).

If your agent sits behind a standard Nginx/ALB 60s or 90s proxy_read_timeout, synchronous agent loops will trigger cascading 504 Gateway Timeouts and retry storms.

2. Multi-Turn Tool Error Recovery & Empty-Call Guards

In our two-turn tool-recovery probe, models handled explicit fallback hints (PRIMARY_RATE_LIMIT -> backup_query_order) reliably. However, on strict-typed financial write calls, a flagship model still occasionally returned zero tool_calls—replying in plain text instead of invoking the declared function.

In zhishu-agent, the deterministic harness treats every model turn as an untrusted state transition:

  • Async Job Queue + Heartbeat Streaming: Long-running audit/coding steps run asynchronously rather than holding synchronous HTTP connections hostage.
  • PreToolUse Deterministic Guardrails: Read-only tools (query_ledger) are separated from irreversible write actions (post_journal_entry), enforced in code—not just in system prompts.
  • Parameter Normalization Layer: Multi-model routing strips incompatible parameters per model family (e.g., models that throw HTTP 400 invalid_parameter_error when temperature is passed).

3. Route by Workload, Not Hype

  • High-frequency structured extraction / tool routing: Prioritize low-reasoning-overhead models (~1.6s–2.7s total latency).
  • Deep code audit / synchronous PR review: Prioritize concise reasoning chains (~37s–67s, half the total tokens).
  • Heavy offline batch reasoning: Push strictly into background worker queues with 300s+ timeouts.

Explore the full deterministic harness architecture and reproducible patterns in zhishu-agent on GitHub.