LlamaIndex Best Practices

Ten practices for building LlamaIndex knowledge bases and agents that stay grounded, measurable, and cheap enough to run.

Use this list when starting a RAG agent or when answers drift after a corpus update.

How to Use This Checklist

Work top to bottom: data plane first, then tools, then evals and ops.
Tick items only when they are true in production, not only in a notebook.
Re-run after embed model changes, major reindexes, or new agent tools.
Pair each unchecked item with an owner and a date.

A - Data and Ingestion

1. Pin chunking and embedding versions. Record splitter settings, embed model id, and index build id next to persisted storage so you can reproduce answers.
2. Put provenance on every Node. Require source, stable doc_id, and freshness fields; inherit metadata from Documents through the node parser.
3. Prefer specialized corpora over one mega-index. Split policies, product docs, and runbooks; join them with a router or multiple tools instead of one noisy collection.

B - Query and Tool Design

4. Tune similarity_top_k with metrics, not defaults. Plot hit-rate and faithfulness versus k; higher k is not free.
5. Write tool descriptions like API docs. QueryEngineTool name/description decide agent routing; include scope and exclusions.
6. Keep retrieval optional in multi-tool agents. Use query engines as tools so the agent can escalate or call APIs when docs are empty.

C - Quality and Safety

7. Gate releases on retrieval + faithfulness. Track hit-rate/MRR and groundedness on a frozen regression set before shipping chunker or prompt changes.
8. Enforce tenancy and secret hygiene. Filter by tenant metadata at retrieve time; never embed raw secrets or unrestricted PII dumps.

D - Operations

9. Persist indexes and separate ingest from serve. Batch ingestion/workers rebuild vectors; request paths only query.
10. Log routes, sources, and costs per answer. Store selector choices, node ids/scores, model ids, and token usage on every agent turn.

Applying the Habits in Order

Stage	Habits	Exit criterion
Design	1-3	Corpus layout + versioning story written
Implement	4-6	Query/tool config matches workloads
Launch	7-8	Eval gates + access controls live
Operate	9-10	Ingest/serve split and traces in prod

Quick Anti-Patterns

Re-embedding the whole tree on every API cold start.
One tool description: "search all company knowledge".
Shipping prompt tweaks with no retrieval metrics.
Mixing embed models inside a single vector collection.
Trusting chat fluency without source-node inspection.
Letting agents invent policy when the retriever returns nothing.

Minimal Reference Snippet

# Pattern: explicit chunking + query engine tool + sources logged
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.tools import QueryEngineTool
 
docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(
    docs,
    transformations=[SentenceSplitter(chunk_size=1024, chunk_overlap=128)],
)
qe = index.as_query_engine(similarity_top_k=4)
tool = QueryEngineTool.from_defaults(
    qe,
    name="policy_docs",
    description="Internal policies only; not for product how-to questions.",
)
# Agent wiring omitted - log response.source_nodes in your trace layer

FAQs

Which three habits are non-negotiable?

Versioned ingestion (1), tool/query clarity (5), and eval gates (7). Without them, agents sound confident while drifting.

Should every app use a router?

No. Routers help when partitions need different engines or prompts. A single clean corpus can stay one index.

How often should we reindex?

On corpus change, embed/chunker change, or scheduled freshness jobs for living documents. Measure before/after hit-rate.

Do these practices apply to LlamaIndex.TS?

Yes at the conceptual level: pin models, metadata, evals, and server-side secrets. APIs differ by language.

Where do rerankers fit?

After you have baseline hit-rate metrics. Rerankers can improve MRR but add latency and cost.

Is LlamaCloud required for production?

No. Managed parsing/indexing helps complex docs and ops scale; self-hosted pipelines remain valid.

How do we stop hallucinations when retrieval is empty?

Detect empty or low-score hits and force refuse/escalate tools instead of free-form generation.

What should tech leads review in PRs?

Chunker/embed pins, tool descriptions, eval deltas, and that request paths cannot rebuild indexes inline.

Related: LlamaIndex Basics

Related: The LlamaIndex Mental Model: Documents, Indexes, and Query Engines

Related: Ingesting and Chunking Documents for an Agent's Knowledge Base

Related: Building a Query Engine Agent Over Your Own Data

Related: Evaluating Retrieval Quality in a LlamaIndex Pipeline

Related: Combining Multiple Indexes with Router Query Engines

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

LlamaIndex Best Practices

Ten practices for building LlamaIndex knowledge bases and agents that stay grounded, measurable, and cheap enough to run.

Use this list when starting a RAG agent or when answers drift after a corpus update.

How to Use This Checklist

Work top to bottom: data plane first, then tools, then evals and ops.
Tick items only when they are true in production, not only in a notebook.
Re-run after embed model changes, major reindexes, or new agent tools.
Pair each unchecked item with an owner and a date.

A - Data and Ingestion

1. Pin chunking and embedding versions. Record splitter settings, embed model id, and index build id next to persisted storage so you can reproduce answers.
2. Put provenance on every Node. Require source, stable doc_id, and freshness fields; inherit metadata from Documents through the node parser.
3. Prefer specialized corpora over one mega-index. Split policies, product docs, and runbooks; join them with a router or multiple tools instead of one noisy collection.

B - Query and Tool Design

4. Tune similarity_top_k with metrics, not defaults. Plot hit-rate and faithfulness versus k; higher k is not free.
5. Write tool descriptions like API docs. QueryEngineTool name/description decide agent routing; include scope and exclusions.
6. Keep retrieval optional in multi-tool agents. Use query engines as tools so the agent can escalate or call APIs when docs are empty.

C - Quality and Safety

7. Gate releases on retrieval + faithfulness. Track hit-rate/MRR and groundedness on a frozen regression set before shipping chunker or prompt changes.
8. Enforce tenancy and secret hygiene. Filter by tenant metadata at retrieve time; never embed raw secrets or unrestricted PII dumps.

D - Operations

9. Persist indexes and separate ingest from serve. Batch ingestion/workers rebuild vectors; request paths only query.
10. Log routes, sources, and costs per answer. Store selector choices, node ids/scores, model ids, and token usage on every agent turn.

Applying the Habits in Order

Stage	Habits	Exit criterion
Design	1-3	Corpus layout + versioning story written
Implement	4-6	Query/tool config matches workloads
Launch	7-8	Eval gates + access controls live
Operate	9-10	Ingest/serve split and traces in prod

Quick Anti-Patterns

Re-embedding the whole tree on every API cold start.
One tool description: "search all company knowledge".
Shipping prompt tweaks with no retrieval metrics.
Mixing embed models inside a single vector collection.
Trusting chat fluency without source-node inspection.
Letting agents invent policy when the retriever returns nothing.

Minimal Reference Snippet

# Pattern: explicit chunking + query engine tool + sources logged
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.tools import QueryEngineTool
 
docs = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(
    docs,
    transformations=[SentenceSplitter(chunk_size=1024, chunk_overlap=128)],
)
qe = index.as_query_engine(similarity_top_k=4)
tool = QueryEngineTool.from_defaults(
    qe,
    name="policy_docs",
    description="Internal policies only; not for product how-to questions.",
)
# Agent wiring omitted - log response.source_nodes in your trace layer

FAQs

Which three habits are non-negotiable?

Versioned ingestion (1), tool/query clarity (5), and eval gates (7). Without them, agents sound confident while drifting.

Should every app use a router?

No. Routers help when partitions need different engines or prompts. A single clean corpus can stay one index.

How often should we reindex?

On corpus change, embed/chunker change, or scheduled freshness jobs for living documents. Measure before/after hit-rate.

Do these practices apply to LlamaIndex.TS?

Yes at the conceptual level: pin models, metadata, evals, and server-side secrets. APIs differ by language.

Where do rerankers fit?

After you have baseline hit-rate metrics. Rerankers can improve MRR but add latency and cost.

Is LlamaCloud required for production?

No. Managed parsing/indexing helps complex docs and ops scale; self-hosted pipelines remain valid.

How do we stop hallucinations when retrieval is empty?

Detect empty or low-score hits and force refuse/escalate tools instead of free-form generation.

What should tech leads review in PRs?

Chunker/embed pins, tool descriptions, eval deltas, and that request paths cannot rebuild indexes inline.

Related: LlamaIndex Basics

Related: The LlamaIndex Mental Model: Documents, Indexes, and Query Engines

Related: Ingesting and Chunking Documents for an Agent's Knowledge Base

Related: Building a Query Engine Agent Over Your Own Data

Related: Evaluating Retrieval Quality in a LlamaIndex Pipeline

Related: Combining Multiple Indexes with Router Query Engines

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

LlamaIndex Best Practices

Search Documentation

LlamaIndex Best Practices