A Quick Note on Memory Embeddings and the Brain
Eight compact memory slots give the current moment a way to retrieve the past without attending to an entire movie.
Here is the short version of a research question that has kept me busy for the past few months:
If understanding a scene depends on what happened earlier, should a brain-encoding model have a memory of earlier scenes too?
Brain-encoding models learn a mapping from a stimulus—a movie frame, a sound, a line of dialogue— to the fMRI response it produces. Strong models can combine visual, audio, and language features, but they often make each prediction from a limited local window. That is a strange mismatch with how we experience a story. A character returning after ten minutes is not just a new image; the return means something because of what we remember.
The compact version of memory
I added a small memory module to the encoding pipeline. It keeps a circular buffer of eight past embeddings. The current embedding queries those slots through two-head cross-attention, then a learned gate decides how much retrieved context should be mixed back in.
The gate consistently settled near 0.12. In other words, the model relied mostly on the current stimulus and mixed in roughly twelve percent memory. That balance appeared across several configurations, which is more interesting to me than a model that simply learns to use as much history as possible.
The average almost hid the finding
In the final optimized experiment, the overall mean improvement was essentially zero. If I had stopped at one global number, the conclusion would have been that memory did not help.
The cortical-network breakdown told a different story:
| Network | Change in correlation, ΔR |
|---|---|
| Frontoparietal Control | +0.048 |
| Salience / Ventral Attention | +0.044 |
| Somatomotor | −0.025 |
| Visual | −0.038 |
Memory improved prediction in the higher-order networks involved in maintaining context and coordinating behavior. It added noise to regions that are better served by immediate sensory information. The global average flattened two opposite effects into one misleading summary.
That network-specific split is the main result.
What transferred—and what did not
On a separate Movie10 stimulus, a memory model trained directly on that data improved by ΔR = +0.020, with 57.1% of parcels improving. A model trained on Friends did not transfer zero-shot to Movie10. The usefulness of temporal context generalized; the exact memories and patterns did not.
That distinction matters. The experiment does not show that the model learned a universal memory mechanism. It shows that a compact memory pathway can help when it learns the temporal structure of the stimulus it is asked to encode.
Where I would take it next
The current model sends memory to every predicted parcel. The results argue for something more selective: memory for Frontoparietal and Salience networks, and a cleaner memoryless path for Visual and Somatomotor regions.
The study is still limited—one subject in the main analysis, a small number of training time points, and modest absolute fMRI correlations. I see the network result as a useful direction, not a final account of biological memory.
For the complete methods, figures, ablations, and limitations, read the full research article. The implementation and experiments are also available in the open-source repository.
