Vector Databases & RAG Best Practices

Ten practices for chunking, indexing, retrieving, and operating vector-backed agent memory so answers stay grounded, scoped, and measurable.

Use this list when launching a RAG agent or when fluency rises while factual accuracy slips after a corpus change.

How to Use This Checklist

Work top to bottom: data plane first, then retrieval tool design, then evals and ops.
Tick items only when they are true in production, not only in a notebook.
Re-run after embed model changes, major reindexes, or new agent tools.
Pair each unchecked item with an owner and a date.

A - Data and Indexing

1. Pin embed model, distance metric, and chunker config. Record model id, dimensions, splitter settings, and index build id next to the store so answers are reproducible.
2. Chunk for atomic facts with deliberate overlap. Prefer structure-aware splits for docs; keep constraints intact; version parent-child layouts when you use them. See Chunking Strategies That Affect Retrieval Quality.
3. Attach provenance metadata on every chunk. Require stable doc_id, source, tenant/ACL tags, and freshness fields; inherit them through the pipeline.

B - Retrieval and Tool Design

4. Enforce tenancy and ACL in the store query. Never trust the model to supply tenant ids; filter server-side before ranking.
5. Expose retrieval as a sharp tool (or a deliberate always-on packer). Write name, triggers, and non-goals; return ids + text + metadata in a structured envelope. See Building a Retrieval Tool an Agent Can Call.
6. Add hybrid search when exact tokens matter. Fuse vector and keyword/FTS (for example RRF) for SKUs, error codes, and legal cites. See Hybrid Search: Combining Vector Similarity with Keyword Filters.
7. Re-rank only after baseline metrics exist. Widen candidate_k, re-score, keep small top_n; drop re-rank if latency or cost wins over measured gain. See Re-Ranking Retrieved Results Before Passing Them to an Agent.

C - Quality and Safety

8. Gate releases on retrieval metrics and groundedness. Track hit-rate/MRR (or nDCG) plus answer faithfulness on a frozen question set before shipping chunker, embed, or prompt changes.
9. Treat retrieved text as untrusted. Docs can contain prompt-injection; bound result size; refuse or escalate when hits are empty or below policy thresholds.

D - Operations

10. Separate ingest from serve; log the full retrieval path. Batch/pipeline workers own upserts and deletes; request paths only query. Log query, filters, hit ids, scores, model ids, and latency per agent turn.

Applying the Habits in Order

Stage	Habits	Exit criterion
Design	1-3	Corpus layout, chunker, and versioning written
Implement	4-7	Scoped tool + retrieval stack matches workloads
Launch	8-9	Eval gates + injection-aware handling live
Operate	10	Ingest/serve split and traces in prod

Quick Anti-Patterns

Mixing embedding models inside one collection.
One mega-index for all tenants without filters.
Tool description: "search everything".
Raising top-k forever instead of fixing chunks.
Re-embedding the corpus on every API cold start.
Shipping prompt tweaks with no retrieval metrics.
Trusting chat fluency without inspecting source chunks.
Letting agents invent policy when retrieval returns nothing.

Minimal Reference Snippet

# Pattern: pin config + filtered search + structured tool result
INDEX = {
    "embed_model": "text-embedding-3-small",  # verify at build
    "chunker": "recursive-md@v3",
    "build_id": "2026-07-01.1",
}
 
def search_knowledge(query: str, top_k: int = 4) -> dict:
    # tenant from auth context - not from model args
    hits = hybrid_search(query, top_k=top_k, filters={"tenant": current_tenant()})
    return {
        "ok": True,
        "index_build_id": INDEX["build_id"],
        "results": [
            {"id": h.id, "text": h.text[:2000], "metadata": h.metadata}
            for h in hits
        ],
    }

FAQs

Which three habits are non-negotiable?

Pinned ingest versions (1), server-side tenancy filters (4), and retrieval eval gates (8). Without them, agents sound confident while drifting or leaking.

Should every app use hybrid search and re-ranking?

No. Start with solid chunks and filters. Add hybrid when exact tokens fail; add re-rank when evals show ranking, not recall, is the bottleneck.

How often should we reindex?

On corpus change, embed/chunker change, or scheduled freshness jobs for living documents. Measure before/after hit-rate.

Pinecone vs Chroma vs pgvector - which practice changes?

Ops and cost change; the ten habits stay. Pick stores with Choosing a Vector Store: Pinecone, Chroma, and pgvector Compared.

How do we stop hallucinations when retrieval is empty?

Detect empty or low-quality hits and force refuse/escalate tools instead of free-form generation.

What should tech leads review in PRs?

Embed/chunker pins, filter enforcement, tool descriptions, eval deltas, and that request paths cannot rebuild indexes inline.

Is always-on RAG a best practice?

It is fine for pure Q&A bots. Multi-tool agents usually want optional retrieval tools so not every turn pays search cost.

Where does long-term structured memory fit?

Keep atomic preferences and profile facts in structured stores; use vectors for prose. See the long-term memory section for extraction and conflicts.

Related: How Vector Search Gives Agents a Searchable Memory

Related: Vector Databases & RAG Basics

Related: Choosing a Vector Store: Pinecone, Chroma, and pgvector Compared

Related: Building a Retrieval Tool an Agent Can Call

Related: Chunking Strategies That Affect Retrieval Quality

Related: Hybrid Search: Combining Vector Similarity with Keyword Filters

Related: Re-Ranking Retrieved Results Before Passing Them to an Agent

Related: LlamaIndex Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Vector Databases & RAG Best Practices

Ten practices for chunking, indexing, retrieving, and operating vector-backed agent memory so answers stay grounded, scoped, and measurable.

Use this list when launching a RAG agent or when fluency rises while factual accuracy slips after a corpus change.

How to Use This Checklist

Work top to bottom: data plane first, then retrieval tool design, then evals and ops.
Tick items only when they are true in production, not only in a notebook.
Re-run after embed model changes, major reindexes, or new agent tools.
Pair each unchecked item with an owner and a date.

A - Data and Indexing

1. Pin embed model, distance metric, and chunker config. Record model id, dimensions, splitter settings, and index build id next to the store so answers are reproducible.
2. Chunk for atomic facts with deliberate overlap. Prefer structure-aware splits for docs; keep constraints intact; version parent-child layouts when you use them. See Chunking Strategies That Affect Retrieval Quality.
3. Attach provenance metadata on every chunk. Require stable doc_id, source, tenant/ACL tags, and freshness fields; inherit them through the pipeline.

B - Retrieval and Tool Design

4. Enforce tenancy and ACL in the store query. Never trust the model to supply tenant ids; filter server-side before ranking.
5. Expose retrieval as a sharp tool (or a deliberate always-on packer). Write name, triggers, and non-goals; return ids + text + metadata in a structured envelope. See Building a Retrieval Tool an Agent Can Call.
6. Add hybrid search when exact tokens matter. Fuse vector and keyword/FTS (for example RRF) for SKUs, error codes, and legal cites. See Hybrid Search: Combining Vector Similarity with Keyword Filters.
7. Re-rank only after baseline metrics exist. Widen candidate_k, re-score, keep small top_n; drop re-rank if latency or cost wins over measured gain. See Re-Ranking Retrieved Results Before Passing Them to an Agent.

C - Quality and Safety

8. Gate releases on retrieval metrics and groundedness. Track hit-rate/MRR (or nDCG) plus answer faithfulness on a frozen question set before shipping chunker, embed, or prompt changes.
9. Treat retrieved text as untrusted. Docs can contain prompt-injection; bound result size; refuse or escalate when hits are empty or below policy thresholds.

D - Operations

10. Separate ingest from serve; log the full retrieval path. Batch/pipeline workers own upserts and deletes; request paths only query. Log query, filters, hit ids, scores, model ids, and latency per agent turn.

Applying the Habits in Order

Stage	Habits	Exit criterion
Design	1-3	Corpus layout, chunker, and versioning written
Implement	4-7	Scoped tool + retrieval stack matches workloads
Launch	8-9	Eval gates + injection-aware handling live
Operate	10	Ingest/serve split and traces in prod

Quick Anti-Patterns

Mixing embedding models inside one collection.
One mega-index for all tenants without filters.
Tool description: "search everything".
Raising top-k forever instead of fixing chunks.
Re-embedding the corpus on every API cold start.
Shipping prompt tweaks with no retrieval metrics.
Trusting chat fluency without inspecting source chunks.
Letting agents invent policy when retrieval returns nothing.

Minimal Reference Snippet

# Pattern: pin config + filtered search + structured tool result
INDEX = {
    "embed_model": "text-embedding-3-small",  # verify at build
    "chunker": "recursive-md@v3",
    "build_id": "2026-07-01.1",
}
 
def search_knowledge(query: str, top_k: int = 4) -> dict:
    # tenant from auth context - not from model args
    hits = hybrid_search(query, top_k=top_k, filters={"tenant": current_tenant()})
    return {
        "ok": True,
        "index_build_id": INDEX["build_id"],
        "results": [
            {"id": h.id, "text": h.text[:2000], "metadata": h.metadata}
            for h in hits
        ],
    }

FAQs

Which three habits are non-negotiable?

Pinned ingest versions (1), server-side tenancy filters (4), and retrieval eval gates (8). Without them, agents sound confident while drifting or leaking.

Should every app use hybrid search and re-ranking?

No. Start with solid chunks and filters. Add hybrid when exact tokens fail; add re-rank when evals show ranking, not recall, is the bottleneck.

How often should we reindex?

On corpus change, embed/chunker change, or scheduled freshness jobs for living documents. Measure before/after hit-rate.

Pinecone vs Chroma vs pgvector - which practice changes?

Ops and cost change; the ten habits stay. Pick stores with Choosing a Vector Store: Pinecone, Chroma, and pgvector Compared.

How do we stop hallucinations when retrieval is empty?

Detect empty or low-quality hits and force refuse/escalate tools instead of free-form generation.

What should tech leads review in PRs?

Embed/chunker pins, filter enforcement, tool descriptions, eval deltas, and that request paths cannot rebuild indexes inline.

Is always-on RAG a best practice?

It is fine for pure Q&A bots. Multi-tool agents usually want optional retrieval tools so not every turn pays search cost.

Where does long-term structured memory fit?

Keep atomic preferences and profile facts in structured stores; use vectors for prose. See the long-term memory section for extraction and conflicts.

Related: How Vector Search Gives Agents a Searchable Memory

Related: Vector Databases & RAG Basics

Related: Choosing a Vector Store: Pinecone, Chroma, and pgvector Compared

Related: Building a Retrieval Tool an Agent Can Call

Related: Chunking Strategies That Affect Retrieval Quality

Related: Hybrid Search: Combining Vector Similarity with Keyword Filters

Related: Re-Ranking Retrieved Results Before Passing Them to an Agent

Related: LlamaIndex Best Practices

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Vector Databases & RAG Best Practices

Search Documentation

Vector Databases & RAG Best Practices