Most teams building a retrieval-augmented generation (RAG) feature start by picking a large language model, then treat the vector database as an afterthought: whatever the tutorial used, or whatever has the flashiest landing page. That choice is harder to reverse than it looks. Swapping a vector database later means re-embedding your entire corpus, rebuilding indexes, and often rewriting your retrieval logic. This guide walks through what a vector database actually does, the real tradeoffs between the popular options, and a practical way to decide which one fits your project instead of whichever one trended last.
What a Vector Database Actually Does
A RAG system turns text, images, or other content into embeddings: numeric vectors that place similar meanings close together in a high-dimensional space. When a user asks a question, the system embeds the question the same way and searches for the nearest stored vectors. Those nearby chunks get pulled into the prompt as context before the language model generates an answer. We covered how that whole retrieval loop fits together in our overview of RAG for business AI; this guide goes one level deeper, into the storage and search layer underneath it.
A vector database is the storage and search layer that makes that lookup fast at scale. Comparing a query vector against every stored vector (exact nearest neighbor search) is accurate but slow once you have more than a few hundred thousand vectors. Vector databases solve this with approximate nearest neighbor (ANN) indexes that trade a small amount of accuracy for a large speed gain, plus the surrounding features real applications need: filtering by metadata, updating records, handling concurrent writes, and scaling across machines.
The Decision That Matters More Than the Vendor Name
Before comparing products, answer three questions. They narrow the field faster than any feature comparison.
- How much data, really? Tens of thousands of chunks behave very differently from tens of millions. Many teams overestimate their eventual scale and over-engineer for it.
- Is RAG even the right approach for this problem? If you have not already compared retrieval against fine-tuning or plain prompt engineering, our framework for choosing between them is worth reading before you lock in a vector database at all.
- Who operates the database? A managed service removes operational work but adds a recurring cost and a dependency on another vendor's uptime. Self-hosting gives you control and often lower cost at scale, but someone on your team now owns upgrades, backups, and capacity planning.
- Does the data already live somewhere? If your application data is already in Postgres, adding a vector extension to it is often simpler and cheaper than running a second database just for embeddings, at least until you outgrow it.
pgvector: When Your Existing Database Is Enough
pgvector is an open-source Postgres extension that adds vector similarity search directly to a database you likely already run. It supports standard, half-precision, binary, and sparse vector types, and several distance metrics including cosine, L2, and inner product.
The practical appeal is that vectors live next to your relational data, so you can join a similarity search against orders, users, or documents in a single query and get standard Postgres guarantees: transactions, replication, backups, and row-level security you already understand. The tradeoff is that pgvector scales with your Postgres instance, not independently of it. Teams in the low tens of millions of vectors on a well-tuned Postgres setup tend to do fine; teams well beyond that, or with very high query concurrency, usually end up evaluating a dedicated vector database instead.
Pinecone: Managed and Low-Ops, at a Price
Pinecone is a fully managed, serverless vector database. You do not provision servers, tune storage, or manage index rebuilds yourself, which is attractive for a small team that wants to ship a RAG feature without taking on new infrastructure to operate. The cost of that convenience is that you are paying for someone else's operations team, and managed vector services are usually priced per stored vector and per query, which can get expensive at high volume or high query rates compared with running your own instance. For a prototype or a feature with moderate, predictable traffic, that tradeoff is often worth it. For a high-volume product, it is worth modeling the cost at your expected scale before committing.
Weaviate and Qdrant: Self-Hosted Flexibility and Hybrid Search
Weaviate and Qdrant are both open-source vector databases that can be self-hosted or run as a managed cloud offering, which gives you a migration path if your needs change.
Both are known for strong hybrid search, combining vector similarity with traditional keyword search so that exact matches (names, part numbers, error codes, legal citations) are not lost the way they can be with pure vector similarity. Weaviate's hybrid search runs a vector query and a BM25F keyword query in parallel, then fuses the two result sets, with an alpha parameter to weight vector relevance against keyword relevance and configurable fusion methods for combining the scores.
Qdrant is often chosen for applications with heavy filtering needs: per-tenant data isolation, geographic constraints, date ranges, or permission checks that must run alongside the similarity search rather than after it. Its filtering supports exact matches, ranges, geo shapes, and nested boolean logic, and it is worth building payload indexes on the fields you filter on before you load data, since that is where a meaningful share of query performance comes from in practice.
Milvus: When You're Actually at Massive Scale
Milvus is built for the scale tier above what most applications ever reach: hundreds of millions to billions of vectors, often distributed across a cluster. It supports a wider range of index types and more granular tuning than most alternatives, which is exactly what very large deployments need and exactly what smaller teams do not. Running Milvus well generally assumes real experience operating distributed, stateful systems. Unless you have a concrete reason to expect that scale within a realistic timeframe, starting here is usually premature.
The Technical Tradeoffs That Decide the Outcome
Whichever product you choose, the index type underneath it drives most of the speed, memory, and accuracy you will experience.
HNSW vs. IVFFlat
HNSW (Hierarchical Navigable Small World) builds a multilayer graph and walks it from sparse upper layers down to denser lower layers to find close matches quickly. It generally offers a better speed-to-recall tradeoff and handles incremental inserts well, but it uses more memory and takes longer to build than simpler alternatives. IVFFlat clusters vectors with k-means and only searches the clusters nearest the query, which builds faster and uses less memory but needs careful tuning of how many clusters to search (commonly called nprobe or probes) to get acceptable recall. Get that parameter wrong and recall can be poor without any obvious error.
Filtering and search accuracy
A known weak point across most ANN indexes is that the index structure is built without knowledge of your filters. A naive implementation runs the similarity search first and applies filters afterward, which can return far fewer results than you asked for if the filter is selective. The fix, where supported, is to push filtering earlier in the query plan, or to use features like iterative index scans that keep searching until enough filtered results are found.
Hybrid search and reranking
Pure vector search is excellent at matching meaning and weak at matching exact terms: product SKUs, legal section numbers, patient identifiers. A hybrid approach that blends vector similarity with keyword search catches both. Many production pipelines also add a reranking step: retrieve a broader candidate set with the vector database, then use a separate, more precise model to reorder the top results before they reach the language model. This two-stage approach is usually cheaper than trying to make the first-pass retrieval perfect on its own.
Security and Data Handling Deserve a Place in the Decision
Embeddings are not as anonymous as they feel. Academic research on embedding inversion, including the vec2text method published at EMNLP 2023, has shown that text embeddings can be reconstructed with meaningful accuracy for shorter inputs, and that personal details can sometimes be recovered from embeddings of sensitive text such as clinical notes. In practice this means an embedding store containing sensitive content deserves the same access controls, encryption at rest and in transit, and audit logging you would apply to the source text, not a lighter standard because "it's just vectors."
If your RAG application touches regulated data such as protected health information, that affects more than storage: it affects who can query the index, how long chunks are retained, whether the embedding provider itself sees your raw text, and whether you can get a signed business associate agreement from every vendor in the pipeline, not just the model provider. Review these questions for each specific vendor and configuration rather than assuming any product is safe by default. If prompt injection and agent permissions are also part of your AI feature, our guide to LLM security risks covers the related attack surface in more depth.
A Practical Decision Framework
In rough order of what to check first:
- Under a few million vectors, already on Postgres: start with pgvector. It avoids a new piece of infrastructure and gets you to a working prototype fastest.
- Want to avoid operating infrastructure at all: a managed service such as Pinecone trades cost for operational simplicity. Model the cost at your real expected query volume before committing, not just at pilot scale.
- Need hybrid search or fine-grained filtering as a core requirement: Weaviate and Qdrant are both built around those needs and can run self-hosted or managed.
- Genuinely expect hundreds of millions of vectors or more: Milvus and similar distributed systems are designed for that tier, but budget for the operational expertise they require.
- Any regulated or sensitive data involved: narrow the shortlist by security and contractual terms before narrowing it by latency benchmarks.
Whatever you pick, benchmark it against a representative sample of your own data and your own query patterns before committing in production. Published benchmarks rarely match your actual chunk sizes, embedding dimensions, filter complexity, or traffic shape.
Common Mistakes We See
- Choosing the index before defining the query pattern. Filtering needs, update frequency, and latency requirements should drive the index choice, not the other way around.
- Retrofitting the metadata schema after ingestion. Deciding what you will need to filter on (tenant, permission level, document type, date, source) before you embed your first document saves a full re-ingestion later.
- Skipping hybrid search because "vector search is semantic enough." It is, until a user searches for an exact order number or product code and gets nothing relevant back.
- Re-embedding without re-evaluating. Changing chunk size, embedding model, or index parameters can quietly change retrieval quality in either direction. Measure it rather than assuming an improvement.
Frequently Asked Questions
Do I need a dedicated vector database if I'm already using Postgres?
Not necessarily at first. pgvector lets you add vector search to an existing Postgres database without introducing a new system. Most teams only move to a dedicated vector database once they outgrow what a well-tuned Postgres instance can comfortably handle, or once they need features like advanced hybrid search that aren't a strength of the extension.
Is a managed vector database always more expensive than self-hosting?
Usually at scale, but not always in the early stages. A managed service avoids the fixed cost of standing up and operating infrastructure, which can make it cheaper for a small or early-stage workload. Pricing models vary by vendor and typically scale with stored vectors and query volume, so the crossover point depends on your actual usage and is worth modeling rather than assuming.
Can I switch vector databases later without starting over?
You can, but it is real work, not a configuration change. You will generally need to re-embed your corpus if you change embedding models, rebuild indexes, and adjust any filtering or hybrid search logic built around the old product's API. Choosing deliberately the first time is cheaper than migrating twice.
Does encrypting the vector database make embeddings safe to store without other controls?
Encryption at rest protects against someone stealing the raw storage, but it does not address who can query the live system. Embedding inversion research shows embeddings themselves can leak information about the underlying text, so access control and query auditing matter as much as encryption for sensitive data.
Choosing a vector database is an architecture decision, not a vendor preference, and it is easiest to get right before your RAG feature is in production with real users depending on it. If you are scoping an AI feature and want a second opinion on the retrieval architecture before you commit to a stack, our team can walk through your data volume, security requirements, and query patterns on a strategy call, or you can see how this fits into a broader build at our SaaS development page.