KARMA: Augmenting Embodied AI Agents with Long-and-Short Term Memory Systems

📝 Paper Summary

Agentic RAG pipeline Layered memory Multi-task planning

KARMA equips embodied AI agents with a dual-memory system—a 3D scene graph for long-term environment maps and an adaptive short-term cache for object states—to improve long-sequence task planning.

Core Problem

Embodied AI agents executing long-sequence household tasks often exceed the in-context memory limits of Large Language Models (LLMs), causing them to forget critical object locations and states.

Why it matters:

Relying solely on LLM context windows leads to repetitive exploration and redundant actions, drastically reducing task success rates and efficiency in complex environments.
Prior memory mechanisms often save everything permanently (causing storage bloat) or use naive first-in-first-out eviction (discarding crucial context).

Concrete Example: When asked to 'wash an apple and place it in a bowl' and later 'bring an apple', an agent without short-term memory forgets the apple's location and state, forcing it to blindly re-explore the kitchen.

Key Novelty

Long-and-Short Term Memory System for Embodied Agents

Long-term memory maintains a non-volatile 3D map of the static environment, allowing the agent to understand spatial relationships without re-exploration.
Short-term memory acts as a dynamic cache for recently used objects, using an adaptive replacement policy (W-TinyLFU) to keep the most useful information readily available.

Evaluation Highlights

Improves Success Rate (SR) by 1.3x (43% vs 33%) and reduces execution time by 68.7% on Composite Tasks compared to CAPEAM.
Increases Success Rate (SR) by 2.3x (21% vs 9%) and reduces execution time by 69% on Complex Tasks compared to HELPER.
The W-TinyLFU memory replacement policy consistently achieves higher Memory Hit Rates (MHR) than standard FIFO across long task sequences.

Breakthrough Assessment

7/10

A practical and effective architectural design adapting classic computing memory hierarchy (LFU cache + persistent storage) to LLM-based embodied agents, showing strong empirical gains in simulation.

⚙️ Technical Details

Problem Definition

Setting: Planning and executing interconnected, long-sequence household tasks in indoor embodied environments.

Inputs: Natural language instruction I, decomposed into sub-tasks T, and multimodal sensor data (vision, location).

Outputs: Sequence of actionable skills or basic movements S (e.g., Python action code).

Pipeline Flow

Memory Construction & Update (Short-term & Long-term)
Memory Retrieval
LLM Planner

System Modules

Memory Construction & Update (Memory Management)

Analyzes captured images using a Vision-Language Model (VLM) to extract object states, combines with 3D coordinates, and embeds via text-embedding-3-large.

Model or implementation: VLM (not explicitly named) + text-embedding-3-large

Memory Retrieval (Memory Management)

Retrieves the top-K most similar short-term memories using cosine distance against the input instruction, and serializes the 3D Scene Graph (3DSG) long-term memory into text.

Model or implementation: text-embedding-3-large (for similarity matching)

LLM Planner

Generates executable action code based on the instruction, retrieved memories, and available skills APIs.

Model or implementation: Pre-trained LLM (not explicitly named, relies on LoTa-Bench framework)

Novel Architectural Elements

Dual-memory architecture splitting environmental context: 3DSG for static spatial data (long-term) and a bounded VLM-extracted cache for dynamic object states (short-term)
Integration of W-TinyLFU cache replacement policy into an LLM agent's short-term memory to optimize hit rates under capacity constraints

Modeling

Base Model: Pre-trained LLM (not explicitly named for planner; text-embedding-3-large for embeddings; LangSAM for real-world deployment)

Comparison to Prior Work

vs. CAPEAM: Uses a structured 3D scene graph and explicit W-TinyLFU short-term memory management rather than just semantic labels for object tracking
vs. HELPER: Implements an adaptive cache replacement policy (W-TinyLFU) instead of unbounded or naive memory accumulation
vs. MemGPT: MemGPT pages context in and out of the LLM window for generic text tasks, whereas KARMA is specifically tailored for embodied 3D spatial environments using scene graphs and multimodal state tracking [not cited in paper]

Limitations

Evaluations are primarily conducted in ideal, uninterrupted simulation environments without human or multi-agent interference
Memory operations and planning are open-loop (no feedback mechanism to evict or update incorrect memories)
Borrows biological terminology (long/short-term memory) but functions more like traditional computing caches without biological theoretical support

Reproducibility

Code: https://github.com/WZX-Swarm-Robotics/KARMA/tree/master

Available: Code repository (https://github.com/WZX-Swarm-Robotics/KARMA/tree/master) and experimental videos. Missing: The exact LLM used for the planner in the main experiments is not explicitly named in the text (though GPT-4o is mentioned as an example in the intro). Prompts are partially described but full templates require checking the codebase.

📊 Experiments & Results

Evaluation Setup

Simulated household tasks requiring long-sequence planning and execution in the AI2-THOR environment.

Benchmarks:

ALFRED-L (Long-sequence household tasks (Simple, Composite, Complex)) [New]
ALFWorld-R (Long-sequence tasks for memory replacement evaluation) [New]

Metrics:

Success Rate (SR)
Memory Retrieval Accuracy (MRA)
Memory Hit Rate (MHR)
Reduced Exploration (RE)
Reduced Time (RT)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
Main comparison results on the ALFRED-L dataset demonstrate KARMA's significant improvements in success rate and execution time over baseline embodied agents.
ALFRED-L (Composite Tasks)	Success Rate (SR)	0.33	0.43	+0.10
ALFRED-L (Composite Tasks)	Reduced Time (RT)	0.201	0.687	+0.486
ALFRED-L (Complex Tasks)	Success Rate (SR)	0.09	0.21	+0.12
ALFRED-L (Complex Tasks)	Reduced Time (RT)	0.011	0.690	+0.679
Ablation experiments reveal the distinct contributions of the short-term and long-term memory modules to the agent's overall performance.
ALFRED-L (Composite Tasks)	Success Rate (SR)	0.22	0.43	+0.21
ALFRED-L (Composite Tasks)	Reduced Time (RT)	0.210	0.687	+0.477

Experiment Figures

Memory Hit Rate (MHR) comparison between First-In-First-Out (FIFO) and Window Tiny Least Frequently Used (W-TinyLFU) policies over increasing task sequence lengths.

The relationship between Memory Hit Rate (MHR) and the proportion of Reduced Exploration (RE).

Main Takeaways

Short-term memory is critical for improving task success rates (SR drops significantly without it), while long-term memory primarily enhances task execution efficiency by reducing exploration (RT drops without it).
The W-TinyLFU memory replacement policy with a larger window segment size achieves the highest Memory Hit Rate (MHR) for long task sequences compared to standard First-In-First-Out (FIFO).
There is a linear correlation between Memory Hit Rate (MHR) and Reduced Exploration (RE), confirming that better memory retention directly translates to fewer redundant actions.
Memory Retrieval Accuracy (MRA) is lower for Complex Tasks than Composite Tasks because semantic matching struggles with ambiguous instructions (e.g., 'get me a high-calorie food').

📚 Prerequisite Knowledge

Prerequisites

Large Language Models (LLMs) for planning
3D Scene Graphs
Cache replacement policies

Key Terms

KARMA: The proposed memory-augmented system for embodied agents

LLM: Large Language Model—a powerful AI system trained on vast amounts of text, used here to plan the agent's actions

3DSG: 3D Scene Graph—a hierarchical map capturing spatial relationships and attributes of objects in an environment

W-TinyLFU: Window Tiny Least Frequently Used—an adaptive cache replacement policy that tracks usage frequency to keep highly reused items in memory

Hit rate: The percentage of times an agent successfully finds the required information in its short-term memory

VLM: Vision-Language Model—an AI model that analyzes images to extract text-based information, used here to determine object states

SLAM: Simultaneous Localization and Mapping—a method used by robots to map an environment and track their location within it

FIFO: First-In-First-Out—a simple memory replacement policy where the oldest item is discarded first

LRU: Least Recently Used—a memory replacement policy that discards the item that hasn't been accessed for the longest time

LFU: Least Frequently Used—a memory replacement policy that discards the item used least often

SR: Success Rate—the percentage of tasks fully completed by the agent

RT: Reduced Time—the proportion of time saved by reducing unnecessary actions during task execution

RE: Reduced Exploration—the proportion of unnecessary exploration attempts saved

MRA: Memory Retrieval Accuracy—whether related memory is successfully retrieved

MHR: Memory Hit Rate—the ratio of successful memory retrievals to total queries

ALFRED-L: A newly constructed dataset based on ALFRED, featuring long-sequence household tasks grouped into Simple, Composite, and Complex categories

AI2-THOR: An interactive 3D simulator used for evaluating embodied AI agents