Seven Vector Databases Tied on Accuracy. Here's What Actually Separated Them

Most vector database comparisons are marketing copy dressed up as benchmarks. A controlled study using identical bge-m3 embeddings across seven open-source, self-hosted engines, including Milvus, Weaviate, pgvector, and Chroma, tested against real medical and technical query sets found something worth knowing before you pick one: retrieval accuracy barely varies between them. The nDCG@10 spread across all seven was just 0.014, and the gap to a theoretical oracle ranged only from 0.009 to 0.023. At matched recall, the "best" database by accuracy is statistically indistinguishable from the rest.

So what actually separates them

Speed, memory footprint, filtered-query throughput, and build cost, not accuracy. Milvus held the smallest memory footprint at a 2.25-million-vector scale, 17.0 GB on disk, and the fastest build time, 8 minutes. pgvector served dense retrieval-augmented-generation queries at 257 per second with full recall even under metadata filters. Different engines won on different operational dimensions, but none of them won meaningfully on the metric most comparison articles lead with.

The benchmark's authors also found something more useful than any single database's ranking: retrieval accuracy across every platform tested was capped by data and chunking quality, not database choice. A poorly chunked corpus performs poorly in any of these seven engines. A well-chunked one performs well in any of them. The database is not where the accuracy problem in most RAG systems actually lives.

What to actually optimize for

Stop evaluating vector databases on accuracy claims, since at matched recall they're within noise of each other. Pick based on your actual operational constraints: memory budget, need for hybrid search, expected query volume, and how fast you need to rebuild an index after a data change. Then spend the engineering effort you'd have spent shopping for "the most accurate" database on your chunking strategy instead, since that's the variable the data says actually determines whether your RAG system retrieves the right thing.