What the app sits on
Ninaivu is an ML lab notebook agent. It trains real scikit-learn models on the Telco Customer Churn dataset from Kaggle, remembers each experiment, and uses that history to propose the next one. The code is public at github.com/divya240707/Ninaivu. Memory comes from the open-source agent memory system Hindsight; the Hindsight documentation describes the retain, recall and reflect operations the app uses. For the idea in general terms, see what agent memory is.
The app has three tabs: Results, Ask the memory, and Run history.
Bug 1: the page showed two different datasets
Our first Results tab displayed a table that said 360 runs and 64 fallbacks, while the chart underneath showed our newer, clean data. The cause was dull: the table read one results file and the chart was a saved image from another.
The fix was to compute every number and every chart from the same file, in the same script:
df = load_results()
df["bad_run"] = df["status"] != "ok"
trials = df.groupby("mode")["trial"].nunique()
Now the top-row metrics, the charts and the table can't disagree, because they all come from df. If your demo shows numbers, make them share a source.
Bug 2: a chart that said nothing
Our two agents scored within 0.001 F1 of each other, and the default chart put both lines on top of each other, in two nearly identical shades of blue. We moved to Altair with distinct colors, larger labels and thicker lines, and zoomed the y-axis so both lines were visible.
Zooming has a cost. It makes tiny differences look large. So we added a caption that states the real gap:
gap = (f1_wide["memory"] - f1_wide["baseline"]).abs().max()
st.caption(f"Largest gap between the two agents in any session: {gap:.3f} F1. The y-axis is zoomed in so both lines are visible.")
The bad-runs chart starts at zero, so its drop isn't exaggerated. [INSERT: screenshot of the two charts]
The answer page
The Ask the memory tab calls two Hindsight operations for each question: reflect to generate an answer, and recall to fetch the stored memories to show as evidence.
One detail that surprised us: Hindsight doesn't give your raw log lines back. It extracts facts from what you store. Recall returned sentences such as "Dropping the customerID column eliminated overfitting in the logistic regression model", not the JSON we had written. That makes evidence readable, but it means the evidence is a paraphrase, so the page has to put it on screen for people to check.
The first answer overclaimed
We asked, "Why should we drop the customerID column?" The answer quoted our numbers correctly (train F1 0.7543 against test 0.6047 with the column, then 0.6031 and 0.607 without it). But it also said dropping the column had made test scores "more stable and reliable". Our own results say the opposite: the test score did not change. Dropping the column removed the overfitting gap, not the score difference.
A language model asked a vague question will fill gaps with plausible detail. We changed the request and relabeled the output:
guarded = (
f"{question}\n\nAnswer in at most four sentences using only facts stored in memory. "
"Quote the numbers from the stored experiments. If memory does not cover something, say so."
)
answer = client.reflect(bank_id=BANK, query=guarded)
evidence = client.recall(bank_id=BANK, query=question)
The heading now reads "Answer (generated from memory, check it against the evidence below)". The new answer was shorter, and its numbers matched our runs exactly. It still added a general line about model families that we could only partly confirm from the evidence list, which is why we treat generated text as a draft to verify, not a result to quote.
One honest limit of the demo
The memory bank behind the app is seeded from one logged trial plus three manual checks, not from a live agent run. We say so in the README, because a demo that looks live when it isn't is its own kind of overclaim.
Lessons
1.Derive every displayed number from one data source. Two sources will eventually disagree on a slide.
2.If you zoom an axis, say how big the real difference is.
3.Put the sources next to generated text. Memory is only useful if you can see what it remembered.
4.Constrain the question. Asking for short answers that quote stored numbers cut the overclaiming sharply.
5.Read every generated sentence before you present it. One confident sentence we couldn't support was enough to change how we built the page.
The code is in the Ninaivu repository, and the Hindsight documentation is the best starting point if you want to add memory to your own app.
#AIAgents #AgentMemory #Hindsight #LLM
