The LlamaIndex Mental Model: Documents, Indexes, and Query Engines
LlamaIndex is a context-augmentation framework: it turns your files, APIs, and databases into structures an LLM can retrieve and reason over.
Agents built on LlamaIndex treat retrieval as a first-class tool, not as a one-off prompt hack.
Summary
- Load data as Documents, split them into Nodes, store them in an Index, then answer questions through a Query Engine (or wrap that engine as an agent tool).
- Insight: Pretrained models do not know your internal PDFs, tickets, or policies. LlamaIndex makes private context available at inference time without fine-tuning.
- Key Concepts: Document, Node, Index, Retriever, Query Engine, Chat Engine, Agent / Workflow, Ingestion Pipeline.
- When to Use: RAG chatbots, document Q&A agents, multi-source research tools, and any agent that must cite or ground answers in owned corpora.
- Limitations/Trade-offs: Retrieval quality depends on chunking, embeddings, and metadata. A bare query engine is not a full multi-step agent; orchestration and evals are still your job.
- Related Topics: ingestion and chunking, query-engine agents, router engines, evaluation, LlamaIndex.TS.
Foundations
Large language models ship with public knowledge only.
Your useful data usually lives in PDFs, wikis, SQL tables, tickets, or object storage.
Context augmentation injects the right slices of that data into the prompt at query time.
Retrieval-Augmented Generation (RAG) is the most common form: embed chunks, retrieve nearest neighbors, then generate an answer from those chunks.
LlamaIndex organizes that path into a small set of objects you can compose:
| Object | Role |
|---|---|
| Document | Source unit with text plus metadata (path, author, source id). |
| Node | Chunk (or structured piece) derived from a Document; inherits metadata. |
| Index | Structure optimized for lookup (vector, summary/list, keyword, and others). |
| Retriever | Given a query, returns relevant Nodes. |
| Query Engine | Retrieve + synthesize a single-shot answer. |
| Chat Engine | Multi-turn conversational interface over the same data. |
| Agent / Workflow | LLM decides when to call tools (including query engines) across steps. |
The high-level API hides most of this for a five-line demo.
The low-level API lets you replace any stage: readers, splitters, vector stores, retrievers, postprocessors, and synthesizers.
Python is the flagship surface.
LlamaIndex.TS brings the same document-index-engine story to Node, Deno, Bun, and related JS runtimes.
Managed pieces (LlamaParse, LlamaCloud indexing) sit on top when you need production parsing without owning every pipeline step.
Mechanics & Interactions
A typical read-path looks like this:
- Load files or APIs into
Documentobjects (SimpleDirectoryReader, LlamaHub readers, custom loaders). - Parse / chunk Documents into
Nodes (SentenceSplitter, token splitters, semantic splitters, hierarchical parsers). - Index Nodes (most often
VectorStoreIndex, which embeds text and stores vectors). - Query via
index.as_query_engine(), which builds a retriever + response synthesizer. - Optionally agentize: wrap the query engine in a
QueryEngineTooland hand it to aFunctionAgent(or similar).
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What does the policy say about refunds?")Each layer has a clear contract.
Documents are what you own.
Nodes are what you embed and retrieve.
Indexes are how you look Nodes up.
Engines are how users or agents ask questions.
Indexes are not all the same.
A vector index answers “find the paragraphs closest to this question.”
A summary or list-style index supports “synthesize across the whole collection.”
Keyword and hybrid approaches catch exact terms embeddings miss.
Router query engines sit above multiple engines and pick (or fan out) based on the query and tool descriptions.
Agents change the control flow.
A plain query engine always retrieves-then-answers once.
An agent can decide whether to query docs, call another tool, ask for clarification, or stop.
That is why LlamaIndex treats RAG pipelines as tools among tools, not as the entire product.
Ingestion is usually a pipeline, not a one-liner, once you leave the tutorial:
- transformations (split, extract metadata, embed)
- vector store as a sink
- docstore / document store for originals
- optional caching and incremental updates
Global Settings (LLM, embed model, chunker) configure defaults for many high-level constructors.
Prefer explicit models in production so agent code does not silently depend on ambient globals.
Advanced Considerations & Applications
For multi-agent systems, keep LlamaIndex focused on data plane work: ingest, index, retrieve, evaluate.
Use agents/workflows for control plane work: planning, tool choice, handoffs, and stopping conditions.
Patterns that fit the mental model well:
- Query-engine-as-tool: one corpus per tool description; the agent chooses which knowledge base to hit.
- Router over indexes: same user API, multiple backends (policies vs FAQs vs product specs).
- Hybrid retrieval: vector + keyword/metadata filters before synthesis.
- Eval loop: retrieval hit-rate/MRR and response faithfulness before shipping prompt or chunk changes.
Trade-offs versus neighboring stacks:
| Approach | Strength | Weakness | Best fit |
|---|---|---|---|
| LlamaIndex-first RAG | Document/index abstractions, many connectors, eval helpers | Heavier than raw embeddings SDK | Knowledge-heavy agents |
| LangChain / LangGraph | Rich graph orchestration | You assemble more of the RAG stack yourself | Complex multi-step control flow |
| Direct vector DB + LLM SDK | Minimal dependencies | You own chunking, synthesis, and eval glue | Tiny single-index apps |
| LlamaCloud / LlamaParse managed path | Production parsing and pipelines | Cost and vendor coupling | Enterprise document quality |
Version-sensitive surfaces (package names under llama-index-*, agent class names such as FunctionAgent vs older OpenAI agents) move quickly.
Treat import paths in examples as verify at build.
Common Misconceptions
- "LlamaIndex is only a vector database." It is a framework around documents, indexes, engines, agents, and evals. The vector store is pluggable.
- "Five lines of code equals production RAG." Demos hide chunking quality, metadata hygiene, persistence, and evaluation.
- "One giant index always wins." Multiple specialized indexes plus routing often beat a single noisy collection.
- "Query engines and agents are the same thing." A query engine answers once with retrieval. An agent may call many tools across turns.
- "Embeddings fix bad documents." Garbage scans, missing structure, and wrong chunk boundaries still fail retrieval.
FAQs
What is the difference between a Document and a Node?
A Document is a source object (often one file or record). A Node is a chunk or structured piece derived from that Document, carrying inherited metadata for retrieval.
Do I need a vector index for every use case?
No. Vector indexes dominate Q&A, but summary/list indexes, keyword indexes, SQL engines, and routers cover other query shapes.
Where does the LLM actually run in a query engine?
Typically twice in spirit: embeddings (or other retrievers) select context, then a generative LLM synthesizes the final answer from the retrieved nodes (exact calls depend on configuration).
What is a retriever versus a query engine?
A retriever returns Nodes. A query engine retrieves and then synthesizes a natural-language response.
How do chat engines differ from query engines?
Chat engines maintain multi-turn conversation state over your data. Query engines are request/response Q&A interfaces.
Can an agent use multiple knowledge bases?
Yes. Wrap each query engine as a QueryEngineTool with a clear description, or use a router that selects among engines.
Is LlamaIndex only for Python?
Python is primary. LlamaIndex.TS covers TypeScript/JavaScript runtimes with the same context-engineering goals.
Where should I put metadata?
On Documents at load time (and ensure it propagates to Nodes). Filters and citations depend on clean source, tenant, and section metadata.
What is an IngestionPipeline?
A transform chain that runs readers' output through splitters, extractors, and embeddings before indexing, suitable for batch and incremental jobs.
How does this relate to MCP or A2A?
MCP and A2A are protocols for tools and agent-to-agent communication. LlamaIndex builds the retrieval and agent logic those protocols may expose.
When should I use LlamaParse?
When PDFs and complex layouts (tables, figures, nested structure) defeat plain text extractors and retrieval quality suffers.
Can I bring my own vector database?
Yes. Indexes commonly sit on top of Chroma, Pinecone, pgvector, Azure AI Search, and many others via integration packages (verify package names at build).
Does LlamaIndex replace my agent framework of choice?
Not necessarily. You can use LlamaIndex for RAG tools inside LangGraph, CrewAI, custom loops, or LlamaIndex's own agents and workflows.
What should I measure first?
Retrieval hit-rate or recall on a labeled question set, then faithfulness of answers to retrieved context. Generation quality without retrieval quality is noise.
Related
- LlamaIndex Basics - first query engine over a small folder
- Ingesting and Chunking Documents for an Agent's Knowledge Base - load and split recipe
- Building a Query Engine Agent Over Your Own Data - wrap RAG as a tool
- Combining Multiple Indexes with Router Query Engines - multi-source routing
- Vector Databases & RAG Basics - embedding and store fundamentals
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.