In Part 1 and Part 2, I used a 4B model (qwen3-4b-instruct-2507) inside a 4,096-token sliding window on a local Snapdragon NPU as an exploratory wind tunnel for a 3-tier smolagents hierarchy (L0 Supervisor $\rightarrow$ L1 Team Leads $\rightarrow$ L2 Specialists). As noted in Part 2, these runs are exploratory diagnostics rather than a controlled benchmark proving model parity.
SpaghettiLens-AI v0.6.0 (GitHub repository) adds two practical controls for this setup:
- Out-of-Window
CallLedger& Duplicate Static Call Guard (core/call_ledger.py,core/smol_hierarchy.py) - Same-Input Trace Diffing CLI (
core/trace_diff.py,cli/bench.py)
1. Catching Repeated Static Calls Under a Sliding Context Window
When multi-step ActionStep history rolls past a 4,096-token window, a compact model can re-emit a tool or sub-agent call it already ran earlier in the session. CallLedger targets a narrow case: repeated, static calls to read-only analysis tools that were already recorded as successful (OK).
- What counts as "static" & how the fingerprint works: Each generated Python step is parsed with
ast.parseto extract candidate tool/sub-agent calls. Only calls whose arguments are literal/constant expressions are treated as static; calls referencing loop variables, mutable runtime state, or non-literal expressions are treated as dynamic and left unblocked. (A loop body containing only identical literal arguments would still produce the same fingerprint unless excluded by the caller.) When the target signature is available, positional arguments are bound to parameter names and merged with keyword arguments into a sorted canonical map ((tool_name, canonical_args)):- Before:
extract_static_dependency_graph(path="fixtures/spaghettiCode.pl", include_sql=True)followed later byextract_static_dependency_graph(include_sql=True, path="fixtures/spaghettiCode.pl"). - After: Both normalize to
("extract_static_dependency_graph", {"include_sql": True, "path": "fixtures/spaghettiCode.pl"}). - Fallback: If a tool signature is unavailable, arguments are unpacked (
*args/**kwargs), or values cannot be normalized, the guard falls back conservatively and does not short-circuit.
- Before:
- Where the 420-character limit applies: The
<= 420-character cap applies only to the rendered[Executed Calls Ledger]digest, whichLedgerCodeAgentinserts directly beneath the system prompt (in framework-managed prompt context, ahead of the sliding step history). The underlying in-memoryCallLedgerretains the full session dictionary of recorded fingerprints, so compacting older entries out of the prompt digest does not cause the pre-execution guard to forget them. - Safety boundary & short-circuiting limits: A fingerprint match alone does not prove that skipping a call is behaviorally equivalent to running it again—even a static call can have side effects or depend on changing external state. In SpaghettiLens-AI, this guard is scoped to read-only, idempotent static-analysis tools operating on immutable legacy fixtures (e.g., schema graph extraction and static security audits), and should not be applied blindly to stateful or side-effecting tools. When every call in a step matches a prior
OKstatic entry,LedgerCodeAgentskipspython_executorand returns a synthetic text observation summarizing the priorOKresult (note: this assumes the agent reads the text observation rather than binding a live Python return object for downstream code). Calls that failed or raised exceptions are never short-circuited as cached successes, and each agent/task run uses its own isolated ledger instance.
(Note: I have not yet run a controlled timing/token benchmark with CallLedger enabled vs. disabled, so any latency or token savings remain qualitative until measured on a fixed fixture set.)
2. Comparing Execution Traces Alongside Deterministic Checks (cli/bench.py)
Comparing final generated code alone can miss cases where two runs take very different tool paths—or where code looks plausible on the surface while failing basic static or database checks.
python3 -m cli.bench \ --fixture fixtures/legacy_invoice_batch.pl \ --schema fixtures/schema.sql \ --baseline openai/gpt-4o \ --candidate npu/qwen3-4b-instruct-2507 \ --diff \ --out reports/invoice_trace_diff.md Here, "same-input" means the harness holds the fixture (fixtures/legacy_invoice_batch.pl), DDL schema (fixtures/schema.sql), tool definitions, prompt templates, and deterministic checks constant across both runs, while --baseline and --candidate select the model-specific provider adapter, weights, context ceiling, and inference settings (check cli/bench.py for exact provider/model identifier handling). Because LLM generation is nondeterministic, treat single-run diffs as diagnostic rather than statistical proof.
core/trace_diff.py evaluates two layers:
- Trace topology & call sequence: Compares step-level call groupings (
MATCH_EXACT,MATCH_TOPOLOGY_VARIANT,ROUTE_DIVERGENCE) alongside the flattened AST call sequence using Levenshtein edit distance. - Four deterministic checks (not proof of semantic correctness or full security): Runs four targeted checks—AST parsing (
ast.parse), basic SQL parameterization checks,decimal.Decimalusage checks on ledger-related code, and execution against an ephemeral SQLite schema. Passing these checks confirms those specific automated invariants; it does not establish general semantic equivalence or full security. - Oracle labels & baseline expectation: The baseline trace is a reference run, not a correctness oracle on its own—expected pass/fail status is determined by the four deterministic checks.
-
PARITY_PASS: Both baseline and candidate pass the four deterministic checks and match in call topology/sequence. (Matching a baseline that itself fails the deterministic checks does not count as a pass.) -
VALID_ALTERNATIVE_PATH: Candidate diverges in step grouping or AST call sequence from the baseline, yet still passes all four deterministic checks. -
CONFIRMED_SILENT_FAILURE: Candidate completes without an unhandled exception or visible crash, yet fails one or more of the four deterministic checks. (Visible runtime crashes are treated as explicit execution failures rather than silent failures.)
-
What's Next
The code, fixtures/legacy_invoice_batch.pl, and unit tests are in giuseppecamiolo-dev/spagnettilens. Next, I'll be applying this trace-logging and deterministic-check workflow to the NeurIPS 2026 × ML Intern Repro Hackathon (Nov 30–Dec 13, 2026; registration closes Nov 29)—reproducing a paper's main claim and exploring an RL evaluation environment around it.
If you've built loop guards or trace-diffing tools for small-model agents, I'd be interested in how you handle stateful tools and non-idempotent edge cases.


