AssoMem: Scalable Memory QA with Multi-Signal Associative Retrieval

📝 Paper Summary

Memory recall Long-term memory organization

AssoMem constructs an associative memory graph linking utterances to abstract clues to enable importance-aware retrieval, fusing relevance, importance, and temporal signals via mutual information for accurate memory recall.

Core Problem

As memory volume increases, retrieval performance deteriorates because memory pools accumulate highly similar items (e.g., repeated meeting topics), making it difficult to distinguish truly relevant information using only semantic distance.

Why it matters:

Standard RAG methods struggle with 'similarity-dense' scenarios where many memories are semantically close but distinct in importance or time.
Users ask open-ended recommendation questions (e.g., 'What do I usually complain about?') that require identifying important patterns rather than just specific fact retrieval.
Existing methods relying solely on semantic relevance fail to capture the associative and temporal nature of human memory organization.

Concrete Example: For the question 'What do I usually complain about at work?', a standard retriever might find random complaints. AssoMem identifies 'work complaints' as a high-importance clue node in the graph, retrieving the most frequent/connected grievances rather than isolated instances.

Key Novelty

Associative Memory Graph with Multi-Signal Fusion

Constructs a graph where raw memory utterances are linked to automatically extracted 'clues' (entities, events, topics), mimicking human associative memory.
Applies Personalized PageRank (PPR) on this graph to calculate an 'Importance' score, prioritizing memories central to the user's history.
Fuses three distinct signals—semantic relevance, graph-based importance, and temporal alignment—using an adaptive Mutual Information (MI) weighting strategy.

Architecture

The overall AssoMem framework, illustrating the two-step process: Memory Retrieval (Graph Construction + RITRanker) and Answer Generation.

Evaluation Highlights

Outperforms state-of-the-art baselines by 24.93% on average across benchmarks.
Achieves 57.3% accuracy on the newly introduced MeetingQA benchmark, surpassing the best baseline (Hybrid Retrieval) at 44.5%.
Demonstrates superior recall in similarity-dense environments compared to standard RAG and graph-based approaches.

Breakthrough Assessment

8/10

Significant advance in memory organization by moving beyond simple vector similarity to graph-based associative retrieval. Effectively integrates importance and temporal signals, addressing key failure modes in long-term memory assistants.

⚙️ Technical Details

Problem Definition

Setting: Memory recall QA over a memory bank M containing sessions of timestamped utterances.

Inputs: A memory question q and a memory bank M.

Outputs: A generated answer based on retrieved evidence E*.

Pipeline Flow

Graph Construction: Extract clues → Merge clues → Build Associative Graph
Retrieval: Query → Retrieve Clues → Candidate Utterances → RITRanker Scoring (Relevance + Importance + Temporal) → Top-K Evidence
Generation: Query + Top-K Evidence → Fine-tuned LLM → Answer

System Modules

Clue Extractor (Graph Construction)

Generate representative clues (aspects, entities, events) for each memory session.

Model or implementation: LLM agent (unspecified architecture)

Associative Memory Graph (Graph Construction)

Structure memories by linking utterances to clues and connecting similar items.

Model or implementation: Graph structure (Nodes: Clues/Utterances; Edges: Ownership/Similarity)

RITRanker

Score candidate utterances based on fused signals.

Model or implementation: Mathematical scoring function (Relevance: Cosine Sim; Importance: PPR; Temporal: TimeLlaMa Sim)

Answer Generator

Generate final answer and predict question type.

Model or implementation: Fine-tuned LLM (Multi-task)

Novel Architectural Elements

Associative Memory Graph linking raw utterances to abstract clues (unlike KGs which use only abstract entities).
Adaptive Mutual Information-driven fusion layer that dynamically weights retrieval signals (Relevance, Importance, Temporal) per query.

Modeling

Base Model: Llama-3-8B-Instruct (implied by context of recent memory papers, though exact base for generation not explicitly named in excerpt, extracted from typical setups in cited works)

Training Method: Multi-task Fine-tuning (Question Type Prediction + Answer Generation)

Training Data:

Denoising QA Dataset (QA+Mem) with mixed positive/negative contexts and negative-only contexts.

Compute: Not reported in the paper

Comparison to Prior Work

vs. Hybrid Retrieval: AssoMem adds 'Importance' (via graph) and explicit 'Temporal' alignment, not just text matching.
vs. Graph-based Retrieval: AssoMem links abstract clues to *exact* raw memories, whereas others often reason over abstract nodes only.
vs. Recency Decay: AssoMem uses explicit temporal embedding matching to handle specific time constraints, rather than just favoring recent items.

Limitations

Dependency on the quality of the LLM-generated clues; poor extraction degrades graph quality.
Computational cost of maintaining and updating the graph structure (PPR calculation) as memory grows.
Requires explicit timestamps for temporal alignment, which may not always be available in all data sources.

Reproducibility

Code: https://github.com/facebookresearch/AssoMem

Code is publicly available at https://github.com/facebookresearch/AssoMem. MeetingQA benchmark is introduced. Hyperparameters for graph construction (thresholds delta, gamma) are mentioned conceptually but exact values not in excerpt.

📊 Experiments & Results

Evaluation Setup

Memory recall QA over large-scale conversation history.

Benchmarks:

MeetingQA (QA over multi-turn meeting dialogues) [New]
Other memory benchmarks (Memory QA)

Metrics:

Accuracy (QA performance)
Retrieval Recall (implied)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
AssoMem significantly outperforms baselines on the newly introduced MeetingQA benchmark.

Main Takeaways

Consistent outperformace across three benchmarks and the new MeetingQA dataset (average +24.93%).
Importance scoring (PPR) is critical for recommendation-style questions where semantic relevance is low.
Temporal alignment significantly improves performance on time-constrained queries compared to simple recency decay.

📚 Prerequisite Knowledge

Prerequisites

Retrieval-Augmented Generation (RAG)
Graph theory (nodes, edges, adjacency matrix)
PageRank / Personalized PageRank
Mutual Information

Key Terms

clue: An automatically extracted abstract concept (entity, event, topic) associated with a set of memory utterances.

Personalized PageRank (PPR): A graph algorithm used here to score the 'importance' of memory nodes based on their connectivity to relevant clues.

Mutual Information (MI): A statistical measure used to adaptively weight the contribution of relevance, importance, and temporal scores based on the query type.

MeetingQA: A new benchmark dataset introduced in this paper simulating real-world meeting scenarios with multi-turn dialogues and QA pairs.

Recency decay: A common temporal retrieval method that prioritizes newer items, which the paper argues is insufficient for specific temporal constraints.

RITRanker: The paper's scoring module that combines Relevance, Importance, and Temporal alignment.

TimeLlaMa: A specific temporal embedding model used to compute similarity between time expressions in the query and memory timestamps.