This Open-source solution eliminates the multi-billion-dollar gap between benchmark promises and production reality.

REDWOOD CITY, Calif. , July 31, 2025 /PRNewswire/ -- Zilliz, the company behind the popular open-source vector database Milvus, recently announced the release of VDBBench 1.0, an open-source benchmarking platform designed to test vector databases under realistic production conditions. Unlike traditional benchmarks that rely on static data and idealized scenarios, VDBBench evaluates systems during streaming data ingestion, metadata filtering, and concurrent workloads that mirror actual enterprise deployments.

"The vector database market has experienced explosive growth since 2023, but our benchmarking methodologies haven't kept pace," said James Luan, VP of Engineering at Zilliz. "Organizations are making multi-million dollar infrastructure decisions based on misleading benchmarks that don't reflect production realities. VDBBench 1.0 fundamentally changes this dynamic by establishing new standards for authentic, production-relevant performance evaluation."

• Advanced Filtering Analysis: Systematically tests metadata filtering across selectivity levels (50% to 99.9%), revealing the "hidden performance killer" that dramatically impacts query speeds and recall accuracy in production.
• Streaming Read/Write Testing: Simulates continuous data ingestion while serving queries, conditions where many databases fail despite impressive static benchmarks.
• Modern Datasets: Uses vectors from state-of-the-art embedding models, including OpenAI and Cohere, with dimensions ranging from 768 to 1,536, reflecting current AI workloads.
• Custom Dataset Support: Organizations can benchmark using their production data and embedding models for industry-specific testing requirements.
• Production-Focused Metrics: Prioritizes P95/P99 tail latency, sustainable throughput under load, and recall accuracy for realistic capacity planning.

VDBBench also features a redesigned dashboard with interactive visualizations that help engineers quickly identify performance gaps. Testing results for major platforms including Milvus, Zilliz Cloud, Pinecone, and Elasticsearch are available on the VDBBench Leaderboard.

VDBBench is available on GitHub with complete documentation and community support. Organizations ready to move beyond misleading benchmarks can access the platform immediately and customize it for specific testing requirements.

Download VDBBench 1.0 → | View Leaderboard → | Contact us for Support

Zilliz builds next-generation database technologies that help organizations unlock the value of unstructured data and rapidly develop AI applications. Headquartered in Redwood Shores, California, Zilliz is backed by leading investors, including Aramco's Prosperity7 Ventures and Temasek's Pavilion Capital. Learn more at www.zilliz.com.