← RabbiWIKIMemory-Augmented Brain Encoding

Active research · Stockton University

Teaching an fMRI model to remember.

A compact memory module that lets a state-of-the-art brain-encoding model retrieve past context — and a clean answer to where in the cortex remembering actually helps.

Read the paper Full write-up
1,000 Schaefer parcels · 7 Yeo networks
+0.048peak ΔR — Frontoparietal Control
61.4%of 1,000 cortical parcels improved
88 : 12current-to-memory ratio the model learned
2.7Mparameters — just +42% over baseline

The Question / 01

A great film only makes sense because of what came before it. So does the brain that watches it. Current encoding models predict each moment in isolation — does giving one a memory of the recent past improve it, and where in the cortex does that memory help most?

The Architecture / 02

Memory-augmented encoding module schematic

A memory the size of eight moments.

The module keeps a circular buffer of the last eight stimulus embeddings. The current moment queries them through two-head cross-attention, and a single learned gate decides how much of the retrieved past to mix back in.

  • 8 slots — a deliberately tiny window; larger buffers only add noise.
  • 2 attention heads — enough to retrieve, cheap enough to run per timestep.
  • Learned gate σ(α) ≈ 0.12 — the model tunes its own reliance on memory.
  • Drop-in — adds just 42% parameters to any existing encoder.

The Finding / 03

Memory doesn't help everywhere.
That's the whole point.

The global average improvement is nearly zero — and that average lies. Split the cortex by network and two opposite effects appear: memory clearly helps the association networks that hold context, and clearly hurts the sensory networks that live in the present.

+0.048peak ΔR — Frontoparietal Control
61.4%of 1,000 cortical parcels improved
88 : 12current-to-memory ratio the model learned
2.7Mparameters — just +42% over baseline

Explore the Cortex / 04

Where remembering pays off.

Cortical map of the memory benefit
memory helps memory hurts
+0.048ΔR · p < 10⁻⁵ ***

Frontoparietal Control

Holds task context, goals, and plot structure across seconds to minutes — exactly the information a memory buffer supplies.

1 / 7

Honest Limits / 05

One subject

The main analysis rests on a single participant's fMRI — a direction, not a population result.

Small data

Just 478 TRs (~12 minutes) of training signal constrains how much the model can learn.

Modest absolutes

Predicting a brain from a movie is hard; nobody gets high raw correlations. The honest headline is ΔR, not accuracy.

A simple buffer

The circular memory is deliberately basic — hierarchical or consolidation-based memory may capture more.

Open Science / 06

Everything is public.

Paper (PDF)The full manuscript — methods, figures, ablations, limitations Code & notebooksImplementation, experiments, and every figure on GitHub Full write-upThe complete paper, rebuilt for the web with all figures Plain-language essayWhat a forgetful AI taught me about memory

MD Rabbi · Department of Computer Science, Stockton University · Manuscript prepared April 2026.