Entropic Hetero-Associative Memory

📝 Paper Summary

Associative Memory Models Hetero-associative Memory

EHAM extends Entropic Associative Memory to hetero-association by storing pairs of objects in a 4D relation and retrieving them via constructive methods that handle memory indeterminacy.

Core Problem

Standard hetero-associative memories (like BAM) often struggle with capacity and indeterminacy, while the previous EAM model was limited to auto-associative tasks.

Why it matters:

Real-world memory involves associating objects of different domains (e.g., image to text), not just self-recall
Retrieving an associated object when the memory is indeterminate (overlapped content) is difficult because the specific cue for the target domain is missing
Existing neural associative models often suffer from catastrophic forgetting or limited capacity compared to table-based entropic approaches

Concrete Example: When a handwritten digit '3' (domain A) is used to retrieve a corresponding handwritten letter 'c' (domain B), the system identifies a 'memory plane' containing the 'c', but has no specific cue within domain B to extract it, unlike in auto-association where the input itself is the cue.

Key Novelty

Entropic Hetero-Associative Memory (EHAM)

Stores associations between two different object domains (e.g., digits and letters) as a 4D weighted relation, effectively creating a specific 'memory plane' for any given input cue
Introduces three specific retrieval strategies (Random, Sample and Test, Search and Test) to construct a valid object from the indeterminate, high-entropy memory plane returned by the cue

Architecture

Conceptual diagram of the Hetero-associative Memory Register and Retrieval operations. It illustrates two domains (A and B) and their codomains, mapping to a central association.

Evaluation Highlights

Achieves 100% precision and recall in recognizing registered pairs of MNIST digits and EMNIST letters
Retrieval using 'Search and Test' method achieves perfect recovery (0.0 mse) of stored objects given a cue from the associated domain
Demonstrates high capacity, successfully storing and retrieving 40,000 associations in a single memory instance with limited resources

Breakthrough Assessment

6/10

Offers a mathematically rigorous alternative to neural associative memories with perfect retrieval in tested scenarios, but relies on a specialized table-based structure rather than standard vector spaces.

⚙️ Technical Details

Problem Definition

Setting: Storage and retrieval of pairs of functions (objects) f_a: A -> V and f_b: B -> Z in a hetero-associative memory

Inputs: A cue object from one domain (e.g., f_a) to retrieve the associated object from the other (f_b)

Outputs: The constructed associated object f_b (or f_a)

Pipeline Flow

Register: Store pairs (f_a, f_b) into 4D table H
Recognition: Verify if a pair is stored
Retrieval: Input cue f_a -> Select sub-relation h_fa -> Solve missing cue -> Output f_b

System Modules

Stores the association between two objects by reinforcing cells in a 4D table corresponding to their attribute-value pairs

Model or implementation: Table-based weighted relation accumulation

Recognition (Eta)

Determines if a specific pair of objects is stored in the memory

Model or implementation: Logical implication check against filtered memory

Retrieval (Beta)

Recovers the associated object f_b given cue f_a using one of three construction methods

Model or implementation: Algorithmic reconstruction (Random, Sample-Test, or Search-Test)

Novel Architectural Elements

4D Memory Table structure representing the Cartesian product of two function domains (A x B -> V x Z)
Retrieval mechanism defined as an operation on 'memory planes' rather than vector-matrix multiplication
Explicit handling of 'missing cue' via entropy-guided search/sampling strategies

Modeling

Base Model: Table-based relational memory (non-neural)

Training Method: One-shot storage (Hebb-like learning)

Key Hyperparameters:

w: Max weight value (integer)
iota: Filtering threshold for noise reduction
kappa: Minimum recognition strength
+ 1 more
xi: Tolerance for missing info

Compute: Not reported in the paper

Comparison to Prior Work

vs. BAM: EHAM uses direct table storage/retrieval with arbitrary capacity limited only by table size, whereas BAM has capacity constraints related to neuron count
vs. Hopfield: EHAM is hetero-associative and constructive, whereas Hopfield is auto-associative and reconstructive
vs. SDM (Sparse Distributed Memory) [not cited in paper]: EHAM uses dense table overlaps and entropy, while SDM uses sparse address activation

Limitations

Computational resources scale with the size of the domains (4D table size)
Search and Test retrieval method is computationally expensive (exhaustive)
Random retrieval method has high error rates for precise recall
No comparison to modern vector database or transformer-based memory approaches

Reproducibility

No replication artifacts mentioned in the paper. The algorithms are described mathematically.

📊 Experiments & Results

Evaluation Setup

Association of MNIST digits (28x28) with EMNIST letters (28x28)

Benchmarks:

MNIST-EMNIST Composite (Hetero-associative retrieval (Digit <-> Letter)) [New]

Metrics:

Precision
Recall
F1 score
Mean Squared Error (mse)
Statistical methodology: Not explicitly reported in the paper

Key Results

Benchmark	Metric	Baseline	This Paper	Δ
MNIST-EMNIST Composite	Precision	1.0	1.0	0.0
MNIST-EMNIST Composite	Recall	1.0	1.0	0.0
MNIST-EMNIST Composite	mse	0.0	0.0	0.0
MNIST-EMNIST Composite	Successful Retrievals	0	40000	40000

Experiment Figures

Visual examples of retrieval using the three proposed methods: Random, Sample and Test, and Search and Test.

Main Takeaways

The memory system successfully extends EAM to hetero-associative tasks, preserving properties like indeterminacy and constructivity.
The 'Search and Test' retrieval method guarantees perfect retrieval but is slower; 'Random' is fast but noisy; 'Sample and Test' offers a balance.
The system handles large capacities (40k objects) without degradation in recognition performance, unlike many neural associative memories that saturate.

📚 Prerequisite Knowledge

Prerequisites

Set theory and relations
Associative memory concepts (auto vs. hetero-associativity)
Information theory (Shannon entropy)

Key Terms

EAM: Entropic Associative Memory—a memory system storing objects as functions in a 2D table where objects overlap, creating entropy

EHAM: Entropic Hetero-Associative Memory—the extension of EAM to store pairs of objects from potentially different domains

BAM: Bidirectional Associative Memory—a standard neural network model for hetero-association (e.g., Kosko's model)

memory plane: The 2D relation or table that holds the stored objects (or specific subset of associations) in the entropic memory

indeterminacy: The state where multiple objects overlap in the memory medium, making the content probabilistic rather than deterministic until retrieval

missing cue problem: The challenge in EHAM where the input cue identifies the correct memory plane for the associated object, but provides no information to extract the specific object from that plane