Nemori: Self-Organizing Agent Memory Inspired by Cognitive Science

📝 Paper Summary

Memory-Augmented Generation (MAG) Autonomous Agents

Nemori is a memory architecture that enables agents to autonomously segment conversational streams into coherent episodes and proactively distill semantic knowledge by analyzing the gap between predicted and actual events.

Core Problem

Existing agent memory systems rely on arbitrary chunking (losing context) and passive summarization (creating redundant or incomplete knowledge), preventing true long-term self-evolution.

Why it matters:

LLMs suffer from 'amnesia' in long interactions, resetting to a stranger state in new sessions without persistent memory
Standard RAG is designed for static knowledge bases and misaligns with the dynamic, stateful nature of agent conversational streams
Arbitrary segmentation (e.g., single messages or fixed windows) fractures semantic coherence, making complex reasoning across time impossible

Concrete Example: In current systems, a conversation is often split by fixed tokens or single messages. If a user changes intent from 'asking info' to 'making a decision' mid-stream, heuristic splitters miss the boundary, storing a fragmented memory that blends two distinct contexts. Nemori uses an intelligent detector to recognize this semantic shift and segment it into a distinct episode.

Key Novelty

Cognitively-Inspired Self-Organizing Memory (Nemori)

**Two-Step Alignment:** Instead of fixed-size chunks, the agent autonomously detects semantic boundaries (inspired by Event Segmentation Theory) and narrates the segment into a coherent episode.
**Predict-Calibrate Principle:** Instead of passively summarizing logs, the agent tries to predict an episode's content based on past memory; the 'surprise' (difference between prediction and reality) is distilled into new semantic knowledge.

Architecture

Overview of the Nemori architecture illustrating the interaction between its three core modules.

Evaluation Highlights

Significantly outperforms prior state-of-the-art systems on the LoCoMo benchmark (quantitative values not reported in provided text snippet)
Demonstrates pronounced advantages in longer contexts on the LongMemEval benchmark (quantitative values not reported in provided text snippet)

Breakthrough Assessment

8/10

Proposes a fundamental shift from passive storage to proactive, prediction-based learning (Free Energy Principle) for agents. While experimental numbers are missing in the snippet, the methodological contribution to MAG is significant.

⚙️ Technical Details

Problem Definition

Setting: Dynamic conversational memory management for autonomous agents

Inputs: Continuous stream of user-agent interaction messages M

Outputs: Structured episodic memory (e) and evolved semantic knowledge base (K)

Pipeline Flow

Input Processing: Message Buffer accumulates stream → Boundary Detector checks for semantic shifts
Episodic Generation: Episode Generator transforms segment into Title + Narrative
Semantic Learning: Unified Retrieval finds relevant context → Predictor guesses episode content → Distiller extracts knowledge from prediction error

System Modules

Boundary Detector

Determine if a meaningful semantic boundary has been crossed in the message stream

Model or implementation: LLM-based classifier (f_theta)

Episode Generator

Transform a raw segmented conversation into a structured narrative memory

Model or implementation: LLM-based generator (g_phi)

Unified Retrieval (Semantic Learning)

Retrieve relevant memories to support prediction

Model or implementation: Dense vector search with cosine similarity

Episode Predictor (Semantic Learning)

Forecast the content of an episode based solely on title and existing knowledge

Model or implementation: LLM-based predictor (h_psi)

Semantic Knowledge Distiller (Semantic Learning)

Identify the 'prediction gap' and extract new knowledge

Model or implementation: LLM-based distiller (r_omega)

Novel Architectural Elements

Asynchronous Predict-Calibrate pipeline: A loop that actively generates predictions and learns from the error, distinct from standard 'read-and-summarize' pipelines
Top-down intelligent Boundary Detector: An architectural decision to use a dedicated LLM module for segmentation decisions rather than heuristic token counting or fixed session windows

Modeling

Base Model: LLM-based (specific model architecture not reported in provided text)

Training Method: In-context learning / Database update (Non-parametric learning)

Adaptation: None (Inference-time architecture)

Key Hyperparameters:

boundary_confidence_threshold: sigma_boundary (configurable)
max_buffer_size: beta_max (predefined)

Compute: Not reported in the paper

Comparison to Prior Work

vs. MemGPT: Nemori uses semantically self-organized episodes rather than fixed context windows or simple message paging
vs. Mem0: Nemori preserves rich narrative episodes alongside facts, whereas Mem0 focuses on consolidated fact entries
vs. HEMA: Nemori uses a proactive 'Predict-Calibrate' cycle to learn, whereas HEMA uses passive summarization rules
+ 1 more
vs. EM-LLM: Nemori uses top-down semantic boundary detection rather than bottom-up token-level surprise [not cited in paper]

Limitations

Dependency on LLM performance for the boundary detector and predictor modules
Latencies introduced by the multi-step predict-calibrate cycle
Specific quantitative performance metrics and error analysis are not available in the provided text snippet

Reproducibility

Code: https://github.com/nemori-ai/nemori

Code is publicly available at https://github.com/nemori-ai/nemori. The provided text snippet does not list specific hyperparameters or model weights used for the experiments.

📊 Experiments & Results

Evaluation Setup

Evaluation of autonomous agent memory on long-context benchmarks

Benchmarks:

LoCoMo (Long Context Modeling)
LongMemEval (Long-term memory evaluation)

Metrics:

Not reported in the provided text
Statistical methodology: Not explicitly reported in the paper

Experiment Figures

Comparison of different input chunking strategies in Memory-Augmented Generation.

Main Takeaways

Nemori claims significant performance improvements over state-of-the-art systems on LoCoMo and LongMemEval benchmarks
The advantage of Nemori is reportedly more pronounced in longer context scenarios, validating the efficacy of the self-organizing memory structure
The Predict-Calibrate mechanism allows for 'genuine learning' by explicitly modeling the gap between expectation and reality, moving beyond heuristic knowledge extraction

📚 Prerequisite Knowledge

Prerequisites

Understanding of Retrieval-Augmented Generation (RAG) vs. Memory-Augmented Generation (MAG)
Basic principles of cognitive science (Episodic vs. Semantic memory)
Vector-based retrieval concepts

Key Terms

MAG: Memory-Augmented Generation—systems designed to organize and retrieve an agent's own dynamic, temporal experiences rather than static documents

Free-energy Principle: A cognitive science theory suggesting that intelligent systems learn by minimizing 'surprise' (the gap between their internal predictions and sensory reality)

Event Segmentation Theory: A psychological theory positing that humans naturally organize continuous experience into discrete, meaningful events or 'episodes'

Two-Step Alignment: Nemori's method of first detecting event boundaries (Boundary Alignment) and then generating narrative descriptions (Representation Alignment)

Episodic Memory: Memory of specific, temporally distinct events or experiences (the 'what, where, when')

Semantic Memory: Abstracted, general knowledge and facts distilled from accumulated experiences

Prediction Gap: The discrepancy between what the agent expected to happen based on past knowledge and what actually occurred in the conversation