What a Forgetful AI Taught Me About Memory
A movie arrives one frame at a time. The brain keeps eight of them close, and lets the rest go.
Watch the last ten minutes of any good film and try to notice the moment a scene stops being pixels and starts being meaning. A character walks back through a door. Nothing on screen tells you this matters. What tells you is everything that happened before — a promise made an hour ago, a look exchanged, a name you were supposed to remember. The image is new. The feeling is old.
For the past few months I have been trying to teach a machine to feel that difference, and the most interesting thing I learned is that it usually shouldn't.
Let me explain, because that sentence sounds like failure and it isn't.
The machines that watch us watch movies
There is a quiet, strange corner of neuroscience where researchers put people inside an fMRI scanner, play them episodes of Friends, and record which parts of the brain light up, second by second. Then they build a model — a "brain-encoding model" — that tries to predict the brain from the movie. Feed it a frame, a line of dialogue, a burst of laugh-track, and it guesses the pattern of activity across the cortex. When the guess is good, it means the model has captured something true about how a human head turns a television show into experience.
The best of these models is astonishing. It is called TRIBE v2, and it won first place out of 263 teams in a 2025 competition by fusing what the eyes see, what the ears hear, and what the words mean. But it has one very human-looking flaw that is actually deeply inhuman: it watches every moment as if it were the first. Each instant is processed alone, with no memory of the scene before it. It is the most sophisticated amnesiac ever built.
That bothered me. We know the brain doesn't work that way. Decades of work — I keep returning to Uri Hasson's — show the cortex is layered by time: the visual areas at the back live in the present tense, reacting to what is on screen right now, while the regions near the front hold on to context for seconds, even minutes, stitching a story together. So I asked a small, stubborn question:
If understanding a scene depends on what came before, shouldn't a model of the brain be allowed to remember?
Eight slots and a dial
The fix I built is almost embarrassingly modest, and I have come to think the modesty is the point.
I gave the model a memory of exactly eight past moments — a little rolling notebook that keeps the last several embeddings and throws the oldest away as new ones arrive. When the model predicts the current instant, it is allowed to glance back at those eight slots and ask whether any of them are relevant. Then comes the part I find genuinely charming: a single dial, which the model tunes itself during training, deciding how much of that remembered context to mix into the present.
I did not set the dial. I let the model find it. And across experiment after experiment, from different settings and different data, it kept landing in the same place — around twelve percent. Roughly seven-eighths of what it does is here and now, and about one-eighth is what I remember. It discovered, on its own, that memory should be a seasoning and not the meal.
Where remembering helps (warm) and where it only adds noise (cool). The map is the finding.
The average that almost lied to me
Here is the moment the project nearly ended, and the moment it actually began.
When I finished the fully polished version and looked at the single headline number — the average improvement across the whole brain — it was essentially zero. If I had trusted that one figure, I would have written "memory doesn't help," closed the laptop, and been wrong.
Because the average was hiding a fight. Underneath it, two opposite things were happening at once, and they were cancelling out.
When I broke the brain down by its functional networks, the story reversed itself completely. In the frontoparietal and salience networks — the front-of-brain systems that hold context, track goals, and decide what matters — memory helped, clearly and significantly. In the visual and somatomotor regions at the back, which just need to know what is on screen right now, memory didn't merely fail to help. It actively got in the way, like handing someone last week's newspaper to describe today's weather.
| Where in the brain | What it does | Effect of memory |
|---|---|---|
| Frontoparietal control | Holds context, tracks goals | Helps (+0.048) |
| Salience / attention | Decides what matters | Helps (+0.044) |
| Somatomotor | Immediate body/senses | Hurts (−0.025) |
| Visual | What's on screen now | Hurts (−0.038) |
That split — not the flat average — is the real result. The global number wasn't wrong so much as it was the wrong question. It asked "does memory help the brain?" when the honest answer is "which brain, doing what?"
Why this is bigger than one experiment
I want to be careful, because it is easy to oversell a small study, and this one is small: one subject in the main analysis, modest data, correlations that no one in this field gets to call large. I am not claiming I found the seat of memory. I built a compact model, and it behaved in a way that lined up — suspiciously well — with what neuroscientists have argued for years about how the cortex is organized by time.
But there is a lesson here that reaches past fMRI, and it is the reason I wanted to write this in plain language instead of leaving it in a paper.
Everyone building AI systems right now is obsessed with memory. Longer context windows. Retrieval systems that pull in documents. Agents that keep notes on themselves. The unspoken assumption is that more memory is more intelligence — that if we could just let the model hold everything, it would think better. My forgetful little brain model quietly disagrees. It found that memory is only valuable in the places whose job is to integrate across time, and that forcing it everywhere else makes things worse. The skill isn't remembering more. The skill is knowing where remembering belongs — and being willing to forget on purpose everywhere else.
There is something almost consoling in that, if you let it land. We tend to treat forgetting as a bug in ourselves, a failing of an overloaded mind. This tiny experiment suggests it might be closer to a design decision — that a system which keeps everything is not wiser, only louder. The brain appears to have made its peace with letting most of the movie go. It holds a few frames close, turns the dial to about twelve percent, and somehow that is enough to follow the whole story.
The full technical write-up — every figure, table, ablation, and the honest list of limitations — lives in the complete research article, and the code is open-source on GitHub if you'd like to take the dial for a spin yourself.
If this resonated, I'd genuinely love to hear how you think about memory — in brains, in models, or in yourself. The comment I keep coming back to is my own: maybe the intelligence was never in how much we hold on to. Maybe it was always in what we're willing to let go.
Related: Memory Embeddings Research Note · Ai Engineering · Software Engineering
