Home→Insights→How to Pick a Vector Database (And When You Do Not Need One)
Artificial Intelligence
How to Pick a Vector Database (And When You Do Not Need One)
Sunil Sethi
Leader, AI & Workflow Specialist
· 20 min
Most mid-market AI use cases under 100k chunks do not need a vector database. The 3 architecture patterns (in-memory, embedded, dedicated), the 5 selection criteria when you do need one, and the architecture that makes vector storage decisions reversible.
Artificial Intelligence Solutions
Looking for a artificial intelligence partner?
We build domain-led systems tailored to your industry and workflow. 12 years. 2,100+ engagements.
Your team is about to spend thousands of dollars on a vector database for a use case that would run fine on a $200 server with the embeddings loaded in memory. The vector database vendors do not tell you this because they sell vector databases. Your AI lead does not tell you this because they read the announcement blog posts and the announcements emphasize features that matter at scale. Six months from now you are running an under-utilized vector database cluster, paying for capacity you do not need, and your team is debugging operational issues that only exist because you took on infrastructure you did not require. The honest framing on vector databases is this: most mid-market AI use cases do not need one. The ones that do need one have specific requirements that narrow the choice to 2 or 3 options. Picking the right one (or skipping it) is a 2-hour analysis, not a multi-quarter procurement.
The decision is structural. Vector databases earn their existence at scale, with high query volume, with filtered retrieval, or when you need operational features (multi-tenancy, replication, fine-grained access control) that in-memory cannot provide. Below a few hundred thousand chunks at moderate query volume, you have options that are simpler and cheaper. Above a few million chunks at high query volume, you need a real vector database and the choice gets nontrivial. The middle range is where teams either over-engineer or under-engineer; understanding the breakpoints prevents the wrong choice.
Below is the shape of the decision, the 3 architecture patterns that cover most cases, the 5 selection criteria when you do need a vector database, the 3 anti-patterns teams reach for when they pick a vector database for the wrong reasons, and the architecture that lets you skip the vector database when your scale does not require it.
100k
Chunks is the rough threshold below which in-memory retrieval beats a vector database on simplicity and cost.
3
Architecture patterns cover most cases: in-memory, embedded, dedicated vector database.
5
Selection criteria when you actually need a vector database: scale, filters, latency, ops, hybrid search.
50ms
Typical query budget for retrieval inside a RAG system; vector DBs hit it; in-memory beats it.
You will see when in-memory retrieval is the right answer, when embedded vector storage (pgvector, sqlite-vss) covers the next tier, when a dedicated vector database is genuinely required, and how the 5 selection criteria narrow the choice once you do need one. The work today is less about picking the most popular vector database and more about deciding whether you need one at all.
How Teams Over-Engineer Vector Storage for Use Cases That Do Not Need It
The default assumption from conference talks and vendor marketing is that you need a vector database to ship RAG. The assumption is wrong for most mid-market AI use cases. A typical mid-market RAG system has 10,000 to 100,000 chunks (the contents of your support docs, your product catalog, your knowledge base, your past customer interactions). At that scale, the entire embedding matrix fits in 100MB to 1GB of memory. Retrieval at that scale takes under 20ms with no special infrastructure. The vector database adds operational complexity, hosting cost, and a network hop without solving a problem you actually had. The diagram below shows the scale breakpoints.
Scale Breakpoints
When Each Approach Wins Based on Chunk Count
Tier 1: Under 100k Chunks
In-Memory Wins
Load embeddings into memory on app startup. Numpy or faiss does the similarity search. Query latency under 20ms. No vector database needed.
Tier 2: 100k to 5M Chunks
Embedded Vector Storage
pgvector extension on Postgres, sqlite-vss, or faiss persisted to disk. Same database you already use. Query latency under 50ms.
Tier 3: 5M+ Chunks
Dedicated Vector Database
Pinecone, Weaviate, Qdrant, Milvus, or similar. Real vector database earns its complexity through scale, filters, replication, and operations.
Shape, Not a Quote
Exact thresholds depend on query volume and embedding dimension. The shape is consistent. Tier 1 covers most mid-market RAG; Tier 3 is where teams over-engineer.
The shape above is the honest map of when each architecture pattern wins. Tier 1 (in-memory) handles most mid-market use cases including internal knowledge bases, product catalogs, support documentation, and policy libraries. Tier 2 (embedded) handles teams that need persistence and filtering but do not have the scale to justify a separate system. Tier 3 (dedicated vector database) handles teams with millions of chunks, high query volume, or operational requirements that the simpler tiers cannot meet.
The teams that ship the wrong tier usually do so because they read the documentation for Tier 3 systems and assume that is what RAG requires. The Tier 3 documentation describes features that solve Tier 3 problems; if you do not have those problems, you do not need those features. Your team should size the system for the use case rather than the architectural sophistication of the documentation.
3 Architecture Patterns That Cover Most Vector Storage Cases
01
In-Memory With faiss or Equivalent
Load your embeddings into memory at application startup. Use faiss for similarity search; it handles up to a few million vectors with reasonable latency on commodity hardware. Persist the embeddings to disk and reload on restart. No external dependency, no network hop, no operational burden beyond your application itself. This is the right architecture for chunks under 100,000 and for teams that value simplicity over horizontal scalability. Most internal knowledge bases and product RAG systems fit here comfortably.
02
Embedded Vector Storage in Your Existing Database
Add pgvector to your existing Postgres or use sqlite-vss for smaller deployments. Embeddings live alongside your other data, queryable in the same SQL connection. No new infrastructure; your existing backup, replication, and access control patterns extend to vector queries. This is the right architecture when you need persistence, filtering (where customer_id = X), or transactional consistency with your other data. Most production RAG systems at mid-market scale fit here.
03
Dedicated Vector Database When Scale or Features Demand It
Pinecone, Weaviate, Qdrant, or Milvus when you have millions of chunks, need replication and multi-tenancy, or require features (graph queries, sparse-dense hybrid retrieval at scale, sharded indexes) that the simpler tiers do not provide. These systems are real infrastructure with real operational requirements; pick one when the scale or feature set genuinely demands it. The 5 selection criteria below narrow the choice once you are in this tier.
The 3 patterns map to 3 different decision frames. In-memory is the right answer when simplicity matters most. Embedded is the right answer when you need database features (filters, transactions, joins) without adding a separate system. Dedicated is the right answer when scale or specialized features force the issue. Most teams should not start with the dedicated pattern; most teams should ship in-memory or embedded first and upgrade only when a measurable limit forces the change.
5 Selection Criteria When You Actually Need a Vector Database
The teams shipping production AI at scale converge on the same 5 selection criteria when picking a dedicated vector database. The right pair or triple decides which vendor fits your specific case. The diagram below lays out the 5.
5 Selection Criteria
How to Pick Among Pinecone, Weaviate, Qdrant, Milvus, and the Rest
Score each vendor against the 5 criteria. Most teams find 2 or 3 vendors meet their requirements; pick by operational fit.
Criterion 1
Scale Headroom
Total chunk count, dimensionality, and growth rate over 2 years. Some systems handle billions cleanly; others degrade past 100M.
Criterion 2
Filter Performance
If retrieval needs filtering (customer_id, date range, document type), filter quality varies enormously across vendors. Test with your actual filter shapes.
Criterion 3
Hybrid Search Support
Sparse plus dense retrieval native to the system, or bolt-on. Native support produces better results for production RAG.
Criterion 4
Operational Model
Managed SaaS, self-hosted, hybrid. Cost, control, and team capability decide which fits.
Criterion 5
Latency Under Load
P99 latency at your query volume matters more than median. Benchmark with your actual workload; vendor benchmarks rarely match production reality.
Shape, Not a Quote
Most teams that score on all 5 narrow to 2 or 3 vendors. The final decision usually comes down to operational model and team familiarity.
The 5 criteria share an underlying principle: vendor benchmarks lie under your actual workload. Each criterion needs evaluation against your data shape, your filter patterns, and your query distribution. Vendor selection without proof-of-concept benchmarking tends to produce surprises in production. The proof-of-concept itself usually takes 2 to 4 weeks and saves quarters of regret.
The operational model criterion is often the deciding factor for mid-market teams. Managed SaaS removes the operational burden but introduces cost predictability issues at scale and adds a vendor dependency. Self-hosted gives control but requires the team capability to operate it. Hybrid splits the difference but adds architectural complexity. The right answer depends on team capability and risk tolerance more than on technical fit.
3 Anti-Patterns When Teams Pick Vector Databases for the Wrong Reasons
01
Adopting a Vector Database for 10,000 Chunks
Your team has a 10,000-chunk knowledge base and signs a $24,000 annual contract with a vector database vendor. The system works fine but the chunk count would run in 50MB of memory with under 5ms latency. The cost and operational burden produce no measurable benefit. The fix is starting with in-memory and graduating only when scale or features demand it.
02
Picking the Vector Database by Popularity Instead of Fit
Your team picks the most-discussed vector database without scoring against the 5 criteria. The chosen system has weak filter support but your use case requires customer-scoped retrieval; the filter performance becomes the bottleneck. The fix is criterion-based selection rather than popularity-based selection. The popular vendor is sometimes the right choice; selection should still be deliberate.
03
Skipping the Proof-of-Concept Benchmark
Your team picks the vector database based on vendor benchmarks and ships to production. The vendor benchmark used 1M vectors with simple queries; your workload has 50M vectors with filtered hybrid search. P99 latency in production is 4 times worse than the benchmark suggested. The fix is benchmarking with your actual workload before commitment. Two weeks of benchmarking saves quarters of replacement effort.
The 3 anti-patterns share a common cause: the team treated vector database selection as a procurement exercise rather than an engineering decision. The right approach is scale-tier the use case, evaluate based on criteria that match your workload, and benchmark before committing.
5 Questions Before You Pick Your Vector Storage Approach
01
How many chunks do you have today and in 2 years?
Below 100k stays in-memory. 100k to 5M fits embedded vector storage. Above 5M needs a dedicated system. Your 2-year projection matters more than today.
02
Does your retrieval need filtering?
Customer-scoped, date-bounded, document-type-restricted retrieval requires filter support. In-memory handles it trivially; embedded handles it through SQL; dedicated vector databases vary widely in filter performance.
03
What is your latency budget?
Under 20ms requires in-memory. Under 100ms accommodates any of the 3 approaches. Above 200ms gives you maximum flexibility. Most user-facing RAG fits comfortably under 200ms total budget.
04
What operational capability does your team have?
In-memory is no-ops. Embedded extends your existing database operations. Dedicated requires operating a new system. Pick what your team can run reliably.
05
Will you need hybrid search?
Hybrid (sparse plus dense) usually outperforms pure vector in production. Plan for it now even if you start with pure vector. The architecture decision should support hybrid even if the first version skips it.
The 5 questions decide your starting tier. Most mid-market teams that answer honestly land in Tier 1 or Tier 2. The teams that genuinely need Tier 3 know it from the scale and feature requirements; if you cannot articulate why you need Tier 3 in one sentence, you probably do not.
How Vector Storage Fits in the Larger RAG Architecture
Architecture
Where Vector Storage Sits and How It Connects
Layer 1
Document Ingestion
Source documents flow in. Chunked, embedded, indexed. Updates trigger re-indexing for affected chunks.
→
Layer 2
Vector Storage
In-memory, embedded, or dedicated. Holds embeddings and metadata. The choice depends on scale and features.
→
Layer 3
Retrieval Logic
Hybrid search, re-ranking, filter application. Returns top relevant chunks to the model layer.
→
Layer 4
Application Boundary
The model and the application consume retrieved chunks. Vector storage is invisible to the upstream system.
Where the Architecture Decision Lives
Layer 2 is where the in-memory vs embedded vs dedicated decision happens. The other layers stay the same regardless. Migration between tiers is bounded because only Layer 2 changes.
The architecture above makes vector storage decisions reversible. Starting with in-memory and migrating to embedded later only changes Layer 2; the rest of the system stays the same. The same applies to migrating from embedded to dedicated. Teams that build clean abstraction at Layer 2 can change vendors or tiers without rewriting the application; teams that bake vendor-specific assumptions into the rest of the system pay a migration tax when the choice changes.
Frequently Asked Questions
Does pgvector really compete with dedicated vector databases?
Yes for most mid-market use cases. pgvector handles millions of vectors with reasonable latency on a decently-sized Postgres instance. Where it loses is at very high query volume with complex filters; dedicated systems handle that load better. For the 80 percent of teams whose query volume is modest, pgvector is the right choice and saves the operational cost of a separate system.
When should you actually pay for Pinecone or a managed vendor?
When you have over 5 million vectors, query volume above several hundred per second, or operational constraints that prevent self-hosting. The managed service is worth the cost when running it yourself would distract your team from the actual product work. For smaller deployments the operational burden of self-hosting is usually low and the cost savings substantial.
How long does the vector storage decision take to ship?
The decision: 1 to 2 weeks including the 5-criterion scoring and proof-of-concept benchmark. The implementation: 4 to 8 weeks for in-memory or embedded, 8 to 16 weeks for dedicated including operations setup. Most teams over-invest in the decision and under-invest in the production operations; the operational work is where the real time goes.
Can you migrate between tiers as you grow?
Yes when the architecture isolates the vector storage layer behind a clean abstraction. The application code does not change; only the storage layer swaps. Teams that build clean abstraction migrate in days to weeks; teams that bake vendor-specific patterns into the application pay much longer migration tax.
What about open source vector databases like Qdrant or Milvus?
Both are credible production systems. The choice between them and the commercial managed options usually comes down to team capability and cost preference. Open source self-hosted has lower direct cost but higher operational burden; managed has the inverse profile. Score against the 5 criteria; the open source options usually pass on technical merit and lose or win on operational model.
Do you need a vector database for graph-shaped data?
Sometimes. Weaviate and a few others support graph queries natively; if your retrieval needs to traverse relationships (find documents related to entity X mentioned in document Y), a graph-aware vector database earns its place. If your retrieval is mostly flat semantic search with metadata filters, you do not need graph features.
Can Entexis help pick and build the right vector storage approach?
Yes. We score the 5 criteria against your use case, run benchmarks with your actual data, and recommend the right tier. We build the resulting system whether in-memory, embedded, or dedicated, with the clean abstraction that lets you migrate later if scale changes. Most engagements skip the dedicated tier because the use case does not require it; some genuinely need it and we pick the vendor based on your data shape rather than what is popular.
The most important thing to take from this is that vector databases are infrastructure for use cases that genuinely need them. Most mid-market RAG does not. Picking the right tier for your scale saves cost, reduces operational burden, and lets your team focus on the use case rather than the infrastructure underneath it.
Want to Pick the Right Vector Storage Approach Without Over-Engineering?
At Entexis, we right-size vector storage to your use case. We score the 5 criteria against your data shape, benchmark with your actual workload, and recommend the tier that fits: in-memory for under 100k chunks, embedded for the middle range, dedicated when scale or features genuinely demand it. We build the system with clean abstraction so you can migrate tiers later without rewriting the application. Most engagements skip the dedicated tier; some need it and we pick the right vendor. Typical engagement is 4 to 8 weeks for in-memory or embedded, 8 to 16 weeks for dedicated. Start the conversation with Entexis.
Ready to Add AI to Your Business?
From intelligent chatbots to workflow automation, we build AI solutions that understand your domain, your data, and your users. Tell us what you need.
We'll get back within one business day.
Thank You!
We've received your message and will get back to you within one business day.
Try the AI workflows we build, for real, right now.
Same workflow patterns Entexis ships into client stacks. Try them in your browser, no signup. If one feels like it'd help your team, we build a private version tuned to your data.
