Building Memory Systems for AI Agents: A Developer's Guide

You've decided your AI agent needs memory. Great! Now comes the hard part: building it. This guide walks through the different approaches, their trade-offs, and when to use each one.

The Memory Architecture Spectrum

1. Simple Key-Value Storage

The simplest approach: store user preferences and facts in a key-value store (Redis, DynamoDB, etc.).

{
  "user_123": {
    "name": "Alice",
    "preferences": {
      "communication": "email",
      "timezone": "PST"
    }
  }
}

Pros: Simple, fast, easy to implement
Cons: No semantic search, limited structure, doesn't scale for complex queries

2. Conversation Buffer

Store recent conversation history and inject it into the context window.

messages = [
  {"role": "user", "content": "I love pizza"},
  {"role": "assistant", "content": "Great! What's your favorite topping?"},
  {"role": "user", "content": "Pepperoni"}
]

Pros: Maintains conversation flow, works with any LLM
Cons: Context window fills up fast, expensive at scale, no long-term memory

3. Summary-Based Memory

Use an LLM to periodically summarize conversations into key facts.

Summary after 20 messages:
- User prefers pepperoni pizza
- Lives in San Francisco
- Works in tech
- Interested in AI startups

Pros: Compresses context, captures key facts
Cons: Lossy compression, misses nuance, summarization costs add up

4. Vector Database + Semantic Search

Embed memories as vectors and retrieve semantically similar context.

# Store embeddings
embedding = embed("User loves pepperoni pizza")
vector_db.upsert(id="mem_123", vector=embedding, metadata={...})

# Query at inference
query_embedding = embed("What pizza does the user like?")
results = vector_db.search(query_embedding, top_k=5)

Pros: Semantic retrieval, scales to millions of memories, flexible
Cons: Complex to build, requires embedding management, retrieval quality varies

5. Hybrid: Structured + Vector

Combine structured metadata with vector embeddings for powerful, nuanced retrieval.

{
  "id": "mem_123",
  "text": "User loves pepperoni pizza",
  "type": "preference",
  "user_id": "user_123",
  "tags": ["food", "preferences"],
  "created_at": "2025-12-01T10:00:00Z",
  "vector": [0.123, 0.456, ...]
}

Pros: Best of both worlds, powerful filtering, high-quality retrieval
Cons: Most complex to implement, requires careful schema design

Key Design Decisions

Storage: What Goes In?

• Facts – "Lives in NYC", "Speaks Spanish"
• Preferences – "Prefers email", "Dark mode user"
• Events – "Purchased plan on 2025-11-15"
• Context – "Last discussed project X"

Retrieval: What Comes Out?

Not all memories are equally relevant. Your retrieval strategy should consider:

• Recency – Recent memories are often more relevant
• Semantic similarity – Match query intent
• Access patterns – Frequently accessed memories might be hot
• User context – Filter by user_id, session, etc.

Updates: How Do Memories Evolve?

Memory systems need to handle:

• Corrections – "Actually, I prefer sausage now"
• Deletions – User privacy, GDPR compliance
• Deduplication – Avoid storing "User loves pizza" 10 times
• Temporal decay – Old memories become less relevant

Common Pitfalls

1. Over-Retrieval

Pulling too many memories clutters the context and confuses the LLM. Aim for precision over recall.

2. Stale Context

A memory from 6 months ago might be outdated. Implement recency scoring or expiration policies.

3. Embedding Drift

If you change embedding models, old vectors become incompatible. Plan for migrations.

4. Privacy Nightmares

Storing sensitive data without proper encryption and access controls is a security disaster waiting to happen.

Which Approach Should You Use?

The Reality: Memory Is Hard

Building a robust memory system is not a weekend project. Between vector databases, embedding APIs, retrieval logic, deduplication, privacy, and performance optimization—it's a full engineering effort.

That's why we built photomem: to give developers the power of intelligent memory without the complexity of building it from scratch.