I've been building agents with memory for a long time, and here's the honest truth: I still can't get an agent to build a good memory on its own. Every time I let one try, it's a disaster.

A few days ago I had my main agent set up a new agent for the work side of my life. It made a mess. I rebuilt the whole thing by hand.

And when I build one by hand? It's really good. That new agent is doing great.

That gap is the whole reason I'm hooked. Hand-built holds up. Agent-built falls apart. I want to know why.

Meet Odie

Odie is my agent. He's not my first. There were a lot of agents before him: trial and error, rebuild after rebuild. Odie is where it all finally came together, and now he's the one I use for basically everything.

His memory is a folder of plain markdown files. No database. The rules that keep it from rotting are the part I've sweated over:

  • One word opens one block. When I say "investing," he reads the whole investing block before he answers, and says "Pulled investing." If he grabbed the wrong thing, I see it right there.
  • The past leaves. When something changes, he rewrites the line on the spot. Git keeps the old version, where it never loads again. (I used to keep history files next to each block. They became a graveyard of old opinions that kept crawling back out.)
  • He sleeps. At the end of every chat he reads the actual transcript back, not his memory of it, and files everything before the next chat starts.
  • Receipts. A dumb little script does the loading and prints what actually loaded. Odie used to say "pulled!" when all he'd gotten was a preview of a much bigger file. He had no idea.

It's way better than where I started. It's also still tough. I spend real time looking after his memory with him, and some weeks it bites me.

→ How Odie remembers, the whole method, warts and all.

The shelves: my best shot

Everything I learned on Odie went into one thing: the shelves. It's my best shot at a memory an AI can actually use well from day one. Depth in the blocks, routing through one short index, trigger words that pull the right block at the right time, the sleep, the receipts. All the stuff I ground through, packed into a starter brain I hope works for anybody.

The newest piece is the one I'm proudest of: lenses.

Every block in the shelves opens with a lens: what the AI should think when it walks into that part of your life. Here's a real one. It's the lens that comes with the glass, which I'll get to in a second:

## My lens Their screen is their work, and they've let me onto it. I'm here so they never carry anything between their screen and me: they point, and I'm already there. - **Good here** is them barely having to say anything. My answer lands beside their mark while they're still looking at it. - **How I know, without asking them.** They keep sending from the glass instead of retyping in chat. Their silence means busy or tired, not that I got it right. - **What I want next.** To know their work well enough that "look at this," with nothing marked, lands on the right thing every time.

So it's not reading a list about me every time. It walks in knowing what matters there, and it rewrites the lens as it learns you. I tested that for a good while. It held up, so it's in the public one now.

You paste one line to your agent, it moves in, and from then on it remembers you.

→ The shelves

Then I got sidetracked

Here's my other problem. I get sidetracked building stuff. Constantly.

The big one is the glass. It came from a fight I've had with every AI I've ever used: showing it what I mean. I'd type ridiculous essays ("third paragraph down, the button on the left, no, the other left"). I'd screenshot things and circle them in Paint like a crime scene board. Then I'd ask the AI to teach me what I did wrong. And do it again. And again. And again.

Eventually I landed on the glass, and I've been grinding on it ever since. It's a see-through layer over my screen. I box anything, say what I want, and my agent answers right there in a little balloon. It goes the other way too: it circles the button I'm hunting for, lights up the paragraph that matters, and a card in the corner shows what it's doing, so I'm not stuck flipping back to the chat.

Here's me goofing around with it. I was sitting in the chat and told Odie: go find me a good source on how to plug in a microwave, and box the interesting stuff. He pulled up GE's own support page, boxed the paragraph that matters, flagged the fine print, and left me a note: "plug it into its own grounded outlet, no extension cord. So no, not the orange one from the garage."

The glass: my agent boxed the paragraph that matters and left notes, with the card in the corner saying Done

And here's where it ties back to memory. Remember that lens up top? That's this. A memory-based agent gets the glass as its own block, lens and skill, plug and play. It shows up already knowing how to work on my screen, and it keeps learning how I work there. No explaining myself every time I point at something.

→ The glass

The glass feeds the rest

The Den is where my agent and I actually work: a full-screen table with pictures, drafts and mock-ups side by side. Box a card with the glass and it knows exactly which one I mean. It pulls memory blocks straight onto the table too, so the plan, what I liked last time and what we already ruled out are already there.

The Den: a shared table with a picture, two versions to pick from, and this week's plan

UI Taste fixes the "folder of screenshots I like" problem. You box what you like, your agent guesses why ("a few big, easy targets across the top, so you never hunt or guess. Is that it?"), and you say Yes or Not quite. (I once told Odie "No, I mean with my eyeballs fella!") What it learns becomes a taste block, so the next thing it builds starts from what you love instead of its idea of "an app."

UI Taste: I boxed what I like on a site, said a few words, and Odie said back what he thinks I mean. Three agreed.

Memory first. Everything else leans on it.

The honest part

Everything here is pure experimentation. All of it.

I've started calling it a recipe: how I make an agent. Every time, I pick something to test, use it for real, and watch where it breaks. Then I change the recipe. Sometimes I find a better way. Sometimes I find out the thing I was proud of last month was quietly hurting me. (Hello again, history files.)

So what you just read is my best shot today. It'll be different soon, because I use this stuff every day and I can't leave it alone. The screenshots are already behind.

I'm not promising an answer. I work way too hard on this to pretend I have one. But a problem that never runs out is kind of the dream, and I love it.

The practical bits: it's all free and MIT. It runs on Claude Code right now, because that's what I use every day. The glass is Windows only for now. I build after my day job, so things break.

Everything: github.com/HunterBryson

If you're fighting your own agent's memory, I really want to hear how you're doing it. That's honestly why I'm posting.