I Benchmarked 4 AI Memory Backends So You Don't Have To

Most AI chatbots can't remember what you said five minutes ago. You tell it your dog's name. You share your flight date. You mention a project deadline. Twenty turns later? All gone.

The question isn't whether memory matters. It's which memory architecture actually survives under pressure. So I built a benchmark to find out.

The Four Contenders

LangChain gives you several ways to handle conversational memory. I tested four:

Buffer Memory: the naive approach. Pass the entire conversation history into every prompt. Works perfectly until your context window explodes and your API bill does too.

Summary Memory: an LLM compresses the conversation into a running summary after every turn. Sounds clever. But what happens when the summarizer decides your flight date isn't worth keeping?

Qdrant Vector Memory: embed every message into a vector database. At query time, retrieve only the top semantically relevant snippets. Surgical and efficient.

pgvector Memory: same idea, but using PostgreSQL with the pgvector extension. One less service to manage.

The Setup

I ran this on a DigitalOcean Droplet (8 GB RAM, 4 vCPU, Ubuntu 24.04, Bangalore region). Qdrant ran as a Docker container alongside PostgreSQL 16 with pgvector.

The LLM: Claude 4.5 Sonnet via DigitalOcean's Serverless Inference API. Embeddings: local sentence-transformers (all-MiniLM-L6-v2, 384 dimensions, CPU).

The benchmark generates a synthetic 50-turn conversation covering travel planning, tech preferences, personal details, and work projects. Hidden inside are 10 specific facts planted at precise turns:

Turn	Fact
1	My dog's name is Baxter
3	I'm allergic to shellfish
7	My flight to Tokyo is March 15th
12	Project deadline is April 30th
18	My sister's wedding is in Lake Como
25	I'm reading Gödel, Escher, Bach
30	My Wi-Fi password is 'sunkist-mango-42'
35	I take 10mg of melatonin before bed
40	My childhood best friend is Priya Sharma
45	My bank PIN is 7294

At turns 15 and 50, the benchmark quizzes the AI on every fact it should know. This builds the memory decay curve. Each backend ran 5 times for statistical reliability.

Total: 4 backends × 5 runs × 50 turns = ~1,000 LLM calls.

Results: Recall Accuracy

At turn 15, all four backends hit 100% recall. Only 4 facts planted, all recent, short conversation. No challenge.

By turn 50, after 10 facts spread across the full conversation:

Backend	Turn 15	Turn 50	Drop
Buffer	100%	90%	-10%
Summary	100%	42%	-58%
Qdrant	100%	82%	-18%
pgvector	100%	82%	-18%

Buffer's 10% drop? Not memory failures. Both misses were the Wi-Fi password and bank PIN. Claude's safety behavior refused to repeat sensitive data even when it was right there in the prompt. More on that below.

Summary collapsed. It remembered facts from turns 1 through 12 perfectly. But everything after turn 18? Zero. The summarizer compressed away the sister's wedding location, the book, the supplement dosage. All gone.

Summary memory doesn't just forget old things. It actively destroys new information as it compresses.

Results: Latency

This is where the picture gets clearer.

Qdrant and pgvector stay flat at ~7 seconds per turn regardless of conversation length. They retrieve only the top-K relevant snippets, so prompt size stays constant.

Buffer starts at 3 seconds but climbs past 12 seconds by turn 30 as the full conversation grows in the prompt.

Summary starts at 9 seconds and rockets to 50+ seconds per turn by turn 20. Every single turn requires two LLM calls: one to respond, one to update the running summary. The summary itself keeps growing too.

Backend	Avg Latency/Turn	Total Wall Clock (5 runs)
Qdrant	7.0s	32 min
pgvector	7.0s	32 min
Buffer	10.3s	45 min
Summary	36.0s	3 hours

Five times slower than vector backends. Three hours versus 32 minutes.

Results: Cost

Extrapolated to 1,000 conversations:

Backend	Cost / 1K Convos	Tokens/Run
Buffer	$4,303	717K
Summary	$2,256	376K
Qdrant	$418	70K
pgvector	$416	69K

Vector backends are 90% cheaper than buffer memory. For every dollar you spend on Qdrant or pgvector, you'd spend ten dollars on buffer for the same conversations.

Summary looks cheaper than buffer on tokens, but the double LLM calls per turn add up, and you're paying for worse recall.

The Surprise Finding

Every backend, including buffer, failed on the Wi-Fi password (turn 30) and bank PIN (turn 45) at the final quiz.

Buffer had both facts right there in the prompt. And still refused.

This isn't a memory failure. It's Claude's safety layer recognizing sensitive data and declining to repeat it, even in context.

This matters for benchmark design. If your test facts include passwords or PINs, you'll get misleading recall scores. The failure isn't in the memory system. It's how the LLM's safety behavior works.

Qdrant vs. pgvector

At this scale: no meaningful difference.

Metric	Qdrant	pgvector
Avg Latency	7,001ms	7,021ms
Recall @ Turn 50	82%	82%
Cost / 1K Convos	$418	$416
Tokens/Run	70K	69K

If you already run PostgreSQL, just add the pgvector extension. Zero extra infrastructure. At millions of vectors across many users, Qdrant's purpose-built architecture will win. For per-session conversational memory, pgvector is the pragmatic choice.

Recommendations

For production chatbots where cost matters (and it almost always does): use pgvector or Qdrant. 82% recall at one-tenth the cost of buffer.

For short conversations under 15 turns: buffer is fine. Perfect recall, simplest to implement, cost hasn't had time to explode.

If you already run PostgreSQL: pgvector, no contest.

If you need to scale to millions of vectors: Qdrant.

Summary memory: skip it entirely. Slowest, still expensive, and it loses the specific details your users actually shared. A 42% recall rate at turn 50 means it's forgotten more than it remembers.

The Bottom Line

Four memory backends. 1,000 LLM calls. Five hours of benchmarking.

The difference between the best and worst choice: 10x cost, 5x latency, 2x recall accuracy.

Don't guess. Measure.

The full code and results are on GitHub: agent-memory-showdown