The LlamaIndex Mental Model: Documents, Indexes, and Query Engines

Q: Can an agent use multiple knowledge bases?

Yes. Wrap each query engine as a QueryEngineTool with a clear description, or use a router that selects among engines.

LlamaIndex is a context-augmentation framework: it turns your files, APIs, and databases into structures an LLM can retrieve and reason over.

Agents built on LlamaIndex treat retrieval as a first-class tool, not as a one-off prompt hack.

Summary

Load data as Documents, split them into Nodes, store them in an Index, then answer questions through a Query Engine (or wrap that engine as an agent tool).
Insight: Pretrained models do not know your internal PDFs, tickets, or policies. LlamaIndex makes private context available at inference time without fine-tuning.
Key Concepts: Document, Node, Index, Retriever, Query Engine, Chat Engine, Agent / Workflow, Ingestion Pipeline.
When to Use: RAG chatbots, document Q&A agents, multi-source research tools, and any agent that must cite or ground answers in owned corpora.
Limitations/Trade-offs: Retrieval quality depends on chunking, embeddings, and metadata. A bare query engine is not a full multi-step agent; orchestration and evals are still your job.
Related Topics: ingestion and chunking, query-engine agents, router engines, evaluation, LlamaIndex.TS.

Foundations

Large language models ship with public knowledge only.

Your useful data usually lives in PDFs, wikis, SQL tables, tickets, or object storage.

Context augmentation injects the right slices of that data into the prompt at query time.

Retrieval-Augmented Generation (RAG) is the most common form: embed chunks, retrieve nearest neighbors, then generate an answer from those chunks.

LlamaIndex organizes that path into a small set of objects you can compose:

Object	Role
Document	Source unit with text plus metadata (path, author, source id).
Node	Chunk (or structured piece) derived from a Document; inherits metadata.
Index	Structure optimized for lookup (vector, summary/list, keyword, and others).
Retriever	Given a query, returns relevant Nodes.
Query Engine	Retrieve + synthesize a single-shot answer.
Chat Engine	Multi-turn conversational interface over the same data.
Agent / Workflow	LLM decides when to call tools (including query engines) across steps.

The high-level API hides most of this for a five-line demo.

The low-level API lets you replace any stage: readers, splitters, vector stores, retrievers, postprocessors, and synthesizers.

Python is the flagship surface.

LlamaIndex.TS brings the same document-index-engine story to Node, Deno, Bun, and related JS runtimes.

Managed pieces (LlamaParse, LlamaCloud indexing) sit on top when you need production parsing without owning every pipeline step.

Mechanics & Interactions

A typical read-path looks like this:

Load files or APIs into Document objects (SimpleDirectoryReader, LlamaHub readers, custom loaders).
Parse / chunk Documents into Nodes (SentenceSplitter, token splitters, semantic splitters, hierarchical parsers).
Index Nodes (most often VectorStoreIndex, which embeds text and stores vectors).
Query via index.as_query_engine(), which builds a retriever + response synthesizer.
Optionally agentize: wrap the query engine in a QueryEngineTool and hand it to a FunctionAgent (or similar).

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
 
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does the policy say about refunds?")

Each layer has a clear contract.

Documents are what you own.

Nodes are what you embed and retrieve.

Indexes are how you look Nodes up.

Engines are how users or agents ask questions.

Indexes are not all the same.

A vector index answers “find the paragraphs closest to this question.”

A summary or list-style index supports “synthesize across the whole collection.”

Keyword and hybrid approaches catch exact terms embeddings miss.

Router query engines sit above multiple engines and pick (or fan out) based on the query and tool descriptions.

Agents change the control flow.

A plain query engine always retrieves-then-answers once.

An agent can decide whether to query docs, call another tool, ask for clarification, or stop.

That is why LlamaIndex treats RAG pipelines as tools among tools, not as the entire product.

Ingestion is usually a pipeline, not a one-liner, once you leave the tutorial:

transformations (split, extract metadata, embed)
vector store as a sink
docstore / document store for originals
optional caching and incremental updates

Global Settings (LLM, embed model, chunker) configure defaults for many high-level constructors.

Prefer explicit models in production so agent code does not silently depend on ambient globals.

Advanced Considerations & Applications

For multi-agent systems, keep LlamaIndex focused on data plane work: ingest, index, retrieve, evaluate.

Use agents/workflows for control plane work: planning, tool choice, handoffs, and stopping conditions.

Patterns that fit the mental model well:

Query-engine-as-tool: one corpus per tool description; the agent chooses which knowledge base to hit.
Router over indexes: same user API, multiple backends (policies vs FAQs vs product specs).
Hybrid retrieval: vector + keyword/metadata filters before synthesis.
Eval loop: retrieval hit-rate/MRR and response faithfulness before shipping prompt or chunk changes.

Trade-offs versus neighboring stacks:

Approach	Strength	Weakness	Best fit
LlamaIndex-first RAG	Document/index abstractions, many connectors, eval helpers	Heavier than raw embeddings SDK	Knowledge-heavy agents
LangChain / LangGraph	Rich graph orchestration	You assemble more of the RAG stack yourself	Complex multi-step control flow
Direct vector DB + LLM SDK	Minimal dependencies	You own chunking, synthesis, and eval glue	Tiny single-index apps
LlamaCloud / LlamaParse managed path	Production parsing and pipelines	Cost and vendor coupling	Enterprise document quality

Version-sensitive surfaces (package names under llama-index-*, agent class names such as FunctionAgent vs older OpenAI agents) move quickly.

Treat import paths in examples as verify at build.

Common Misconceptions

"LlamaIndex is only a vector database." It is a framework around documents, indexes, engines, agents, and evals. The vector store is pluggable.
"Five lines of code equals production RAG." Demos hide chunking quality, metadata hygiene, persistence, and evaluation.
"One giant index always wins." Multiple specialized indexes plus routing often beat a single noisy collection.
"Query engines and agents are the same thing." A query engine answers once with retrieval. An agent may call many tools across turns.
"Embeddings fix bad documents." Garbage scans, missing structure, and wrong chunk boundaries still fail retrieval.

FAQs

What is the difference between a Document and a Node?

A Document is a source object (often one file or record). A Node is a chunk or structured piece derived from that Document, carrying inherited metadata for retrieval.

Do I need a vector index for every use case?

No. Vector indexes dominate Q&A, but summary/list indexes, keyword indexes, SQL engines, and routers cover other query shapes.

Where does the LLM actually run in a query engine?

Typically twice in spirit: embeddings (or other retrievers) select context, then a generative LLM synthesizes the final answer from the retrieved nodes (exact calls depend on configuration).

What is a retriever versus a query engine?

A retriever returns Nodes. A query engine retrieves and then synthesizes a natural-language response.

How do chat engines differ from query engines?

Chat engines maintain multi-turn conversation state over your data. Query engines are request/response Q&A interfaces.

Can an agent use multiple knowledge bases?

Yes. Wrap each query engine as a QueryEngineTool with a clear description, or use a router that selects among engines.

Is LlamaIndex only for Python?

Python is primary. LlamaIndex.TS covers TypeScript/JavaScript runtimes with the same context-engineering goals.

Where should I put metadata?

On Documents at load time (and ensure it propagates to Nodes). Filters and citations depend on clean source, tenant, and section metadata.

What is an IngestionPipeline?

A transform chain that runs readers' output through splitters, extractors, and embeddings before indexing, suitable for batch and incremental jobs.

How does this relate to MCP or A2A?

MCP and A2A are protocols for tools and agent-to-agent communication. LlamaIndex builds the retrieval and agent logic those protocols may expose.

When should I use LlamaParse?

When PDFs and complex layouts (tables, figures, nested structure) defeat plain text extractors and retrieval quality suffers.

Can I bring my own vector database?

Yes. Indexes commonly sit on top of Chroma, Pinecone, pgvector, Azure AI Search, and many others via integration packages (verify package names at build).

Does LlamaIndex replace my agent framework of choice?

Not necessarily. You can use LlamaIndex for RAG tools inside LangGraph, CrewAI, custom loops, or LlamaIndex's own agents and workflows.

What should I measure first?

Retrieval hit-rate or recall on a labeled question set, then faithfulness of answers to retrieved context. Generation quality without retrieval quality is noise.

LlamaIndex Basics - first query engine over a small folder
Ingesting and Chunking Documents for an Agent's Knowledge Base - load and split recipe
Building a Query Engine Agent Over Your Own Data - wrap RAG as a tool
Combining Multiple Indexes with Router Query Engines - multi-source routing
Vector Databases & RAG Basics - embedding and store fundamentals

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

Agents built on LlamaIndex treat retrieval as a first-class tool, not as a one-off prompt hack.

Summary

Load data as Documents, split them into Nodes, store them in an Index, then answer questions through a Query Engine (or wrap that engine as an agent tool).
Insight: Pretrained models do not know your internal PDFs, tickets, or policies. LlamaIndex makes private context available at inference time without fine-tuning.
Key Concepts: Document, Node, Index, Retriever, Query Engine, Chat Engine, Agent / Workflow, Ingestion Pipeline.
When to Use: RAG chatbots, document Q&A agents, multi-source research tools, and any agent that must cite or ground answers in owned corpora.
Limitations/Trade-offs: Retrieval quality depends on chunking, embeddings, and metadata. A bare query engine is not a full multi-step agent; orchestration and evals are still your job.
Related Topics: ingestion and chunking, query-engine agents, router engines, evaluation, LlamaIndex.TS.

Foundations

Large language models ship with public knowledge only.

Your useful data usually lives in PDFs, wikis, SQL tables, tickets, or object storage.

Context augmentation injects the right slices of that data into the prompt at query time.

Retrieval-Augmented Generation (RAG) is the most common form: embed chunks, retrieve nearest neighbors, then generate an answer from those chunks.

LlamaIndex organizes that path into a small set of objects you can compose:

Object	Role
Document	Source unit with text plus metadata (path, author, source id).
Node	Chunk (or structured piece) derived from a Document; inherits metadata.
Index	Structure optimized for lookup (vector, summary/list, keyword, and others).
Retriever	Given a query, returns relevant Nodes.
Query Engine	Retrieve + synthesize a single-shot answer.
Chat Engine	Multi-turn conversational interface over the same data.
Agent / Workflow	LLM decides when to call tools (including query engines) across steps.

The high-level API hides most of this for a five-line demo.

The low-level API lets you replace any stage: readers, splitters, vector stores, retrievers, postprocessors, and synthesizers.

Python is the flagship surface.

LlamaIndex.TS brings the same document-index-engine story to Node, Deno, Bun, and related JS runtimes.

Managed pieces (LlamaParse, LlamaCloud indexing) sit on top when you need production parsing without owning every pipeline step.

Mechanics & Interactions

A typical read-path looks like this:

Load files or APIs into Document objects (SimpleDirectoryReader, LlamaHub readers, custom loaders).
Parse / chunk Documents into Nodes (SentenceSplitter, token splitters, semantic splitters, hierarchical parsers).
Index Nodes (most often VectorStoreIndex, which embeds text and stores vectors).
Query via index.as_query_engine(), which builds a retriever + response synthesizer.
Optionally agentize: wrap the query engine in a QueryEngineTool and hand it to a FunctionAgent (or similar).

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
 
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does the policy say about refunds?")

Each layer has a clear contract.

Documents are what you own.

Nodes are what you embed and retrieve.

Indexes are how you look Nodes up.

Engines are how users or agents ask questions.

Indexes are not all the same.

A vector index answers “find the paragraphs closest to this question.”

A summary or list-style index supports “synthesize across the whole collection.”

Keyword and hybrid approaches catch exact terms embeddings miss.

Router query engines sit above multiple engines and pick (or fan out) based on the query and tool descriptions.

Agents change the control flow.

A plain query engine always retrieves-then-answers once.

An agent can decide whether to query docs, call another tool, ask for clarification, or stop.

That is why LlamaIndex treats RAG pipelines as tools among tools, not as the entire product.

Ingestion is usually a pipeline, not a one-liner, once you leave the tutorial:

transformations (split, extract metadata, embed)
vector store as a sink
docstore / document store for originals
optional caching and incremental updates

Global Settings (LLM, embed model, chunker) configure defaults for many high-level constructors.

Prefer explicit models in production so agent code does not silently depend on ambient globals.

Advanced Considerations & Applications

For multi-agent systems, keep LlamaIndex focused on data plane work: ingest, index, retrieve, evaluate.

Use agents/workflows for control plane work: planning, tool choice, handoffs, and stopping conditions.

Patterns that fit the mental model well:

Query-engine-as-tool: one corpus per tool description; the agent chooses which knowledge base to hit.
Router over indexes: same user API, multiple backends (policies vs FAQs vs product specs).
Hybrid retrieval: vector + keyword/metadata filters before synthesis.
Eval loop: retrieval hit-rate/MRR and response faithfulness before shipping prompt or chunk changes.

Trade-offs versus neighboring stacks:

Approach	Strength	Weakness	Best fit
LlamaIndex-first RAG	Document/index abstractions, many connectors, eval helpers	Heavier than raw embeddings SDK	Knowledge-heavy agents
LangChain / LangGraph	Rich graph orchestration	You assemble more of the RAG stack yourself	Complex multi-step control flow
Direct vector DB + LLM SDK	Minimal dependencies	You own chunking, synthesis, and eval glue	Tiny single-index apps
LlamaCloud / LlamaParse managed path	Production parsing and pipelines	Cost and vendor coupling	Enterprise document quality

Version-sensitive surfaces (package names under llama-index-*, agent class names such as FunctionAgent vs older OpenAI agents) move quickly.

Treat import paths in examples as verify at build.

Common Misconceptions

"LlamaIndex is only a vector database." It is a framework around documents, indexes, engines, agents, and evals. The vector store is pluggable.
"Five lines of code equals production RAG." Demos hide chunking quality, metadata hygiene, persistence, and evaluation.
"One giant index always wins." Multiple specialized indexes plus routing often beat a single noisy collection.
"Query engines and agents are the same thing." A query engine answers once with retrieval. An agent may call many tools across turns.
"Embeddings fix bad documents." Garbage scans, missing structure, and wrong chunk boundaries still fail retrieval.

FAQs

What is the difference between a Document and a Node?

A Document is a source object (often one file or record). A Node is a chunk or structured piece derived from that Document, carrying inherited metadata for retrieval.

Do I need a vector index for every use case?

No. Vector indexes dominate Q&A, but summary/list indexes, keyword indexes, SQL engines, and routers cover other query shapes.

Where does the LLM actually run in a query engine?

Typically twice in spirit: embeddings (or other retrievers) select context, then a generative LLM synthesizes the final answer from the retrieved nodes (exact calls depend on configuration).

What is a retriever versus a query engine?

A retriever returns Nodes. A query engine retrieves and then synthesizes a natural-language response.

How do chat engines differ from query engines?

Chat engines maintain multi-turn conversation state over your data. Query engines are request/response Q&A interfaces.

Can an agent use multiple knowledge bases?

Yes. Wrap each query engine as a QueryEngineTool with a clear description, or use a router that selects among engines.

Is LlamaIndex only for Python?

Python is primary. LlamaIndex.TS covers TypeScript/JavaScript runtimes with the same context-engineering goals.

Where should I put metadata?

On Documents at load time (and ensure it propagates to Nodes). Filters and citations depend on clean source, tenant, and section metadata.

What is an IngestionPipeline?

A transform chain that runs readers' output through splitters, extractors, and embeddings before indexing, suitable for batch and incremental jobs.

How does this relate to MCP or A2A?

MCP and A2A are protocols for tools and agent-to-agent communication. LlamaIndex builds the retrieval and agent logic those protocols may expose.

When should I use LlamaParse?

When PDFs and complex layouts (tables, figures, nested structure) defeat plain text extractors and retrieval quality suffers.

Can I bring my own vector database?

Yes. Indexes commonly sit on top of Chroma, Pinecone, pgvector, Azure AI Search, and many others via integration packages (verify package names at build).

Does LlamaIndex replace my agent framework of choice?

Not necessarily. You can use LlamaIndex for RAG tools inside LangGraph, CrewAI, custom loops, or LlamaIndex's own agents and workflows.

What should I measure first?

Retrieval hit-rate or recall on a labeled question set, then faithfulness of answers to retrieved context. Generation quality without retrieval quality is noise.

LlamaIndex Basics - first query engine over a small folder
Ingesting and Chunking Documents for an Agent's Knowledge Base - load and split recipe
Building a Query Engine Agent Over Your Own Data - wrap RAG as a tool
Combining Multiple Indexes with Router Query Engines - multi-source routing
Vector Databases & RAG Basics - embedding and store fundamentals

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.

The LlamaIndex Mental Model: Documents, Indexes, and Query Engines

Search Documentation

The LlamaIndex Mental Model: Documents, Indexes, and Query Engines