Memory Evaluation

​

Why Memory Evaluation Matters

​

Architecture Overview

​

Memory Extraction (Distillation)

1. Store New Memories — Conversation enters the pipeline asynchronously (after the agent responds)
2. Context Lookup — Find related existing memories to avoid duplicates
3. Distill Memories — Single-pass LLM extraction produces ADD-only facts from input + context
4. Deduplicate + Embed — Hash-based deduplication, then vectorize new memories
5. Entity Linking — Identify entities (proper nouns, quoted text, compound noun phrases) and link them across memories

​

Multi-Signal Retrieval

1. Semantic Search — Vector similarity scoring against memory embeddings
2. Keyword Search — Normalized term matching via BM25 with verb-form lemmatization
3. Entity Search — Entity graph matching boosts memories linked to query entities

​

Benchmarks

​

LoCoMo

​

LongMemEval

​

BEAM

​

Performance Summary

​

Running Evaluations

​

Setup

git clone https://github.com/mem0ai/memory-benchmarks.git
cd memory-benchmarks
pip install -r requirements.txt

# Set your API keys
export MEM0_API_KEY=m0-your-key
export OPENAI_API_KEY=sk-your-key

git clone https://github.com/mem0ai/memory-benchmarks.git
cd memory-benchmarks
pip install -r requirements.txt

# Copy and configure environment
cp .env.example .env
# Edit .env to add OPENAI_API_KEY

# Start local Mem0 server + Qdrant
docker compose up -d
# Mem0 server: http://localhost:8888
# Qdrant:      http://localhost:6333

​

Running a Benchmark

# ~300 questions across 10 conversations (fastest benchmark)
python -m benchmarks.locomo.run \
  --project-name my-eval \
  --backend cloud \
  --mem0-api-key $MEM0_API_KEY \
  --top-k 200

# Self-hosted
python -m benchmarks.locomo.run \
  --project-name my-eval \
  --top-k 200

​

Custom Model Configuration

# Available configs: openai.yaml, azure-openai.yaml, ollama.yaml
cp configs/azure-openai.yaml mem0-config.yaml
# Edit mem0-config.yaml with your model details

# Uncomment the volume mount in docker-compose.yml, then restart:
docker compose down && docker compose up -d

​

Viewing Results

npm install
npm run dev -- -p 3001
# Open http://localhost:3001

​

Result Format

{
  "id": "locomo_q_001",
  "group": "temporal",
  "question": "When did the user first mention moving?",
  "ground_truth": "During the March 3rd conversation",
  "retrieval": {
    "search_query": "when did user mention moving",
    "search_results": ["..."],
    "search_latency_ms": 123.4,
    "total_results": 42
  },
  "generation": {
    "generated_answer": "The user first mentioned moving on March 3rd",
    "model": "<answerer-model>",
    "prompt_tokens": 500,
    "completion_tokens": 100
  },
  "judgment": {
    "judgment": "CORRECT",
    "score": 0.85,
    "reason": "Answer correctly identifies the date",
    "model": "<judge-model>"
  },
  "cutoff_results": {
    "top_10": { "score": 0.75, "judgment": "CORRECT" },
    "top_50": { "score": 0.85, "judgment": "CORRECT" },
    "top_200": { "score": 0.90, "judgment": "CORRECT" }
  }
}

​

Interpreting Results

• Saturating a small benchmark is not the same as building a memory system that works at scale. Small benchmarks can be brute-forced with aggressive retrieval and frontier models.
• Token efficiency matters as much as accuracy. A system that scores 95% using 25K tokens per query isn’t comparable to one scoring 90% using 7K tokens. Report mean tokens per query alongside scores.
• Compare at equal constraints. Always compare systems using the same retrieval budget, the same model, and the same latency budget. A frontier model at maximum recall is not comparable to a smaller production-grade model at production-realistic retrieval depth.
• Watch for score ceiling effects. Categories like “single-session user” are already near-saturated (97%+). Improvements in these categories are less meaningful than gains in harder categories like temporal reasoning or multi-session.
• BEAM at 10M is the real test. Any system can look good at small scale. The 10M-token BEAM benchmark reveals whether the retrieval system actually scales.

​

FAQ

​

Resources