Ingesting and Chunking Documents for an Agent's Knowledge Base
Agents answer poorly when the knowledge base is a dump of raw files.
//
Search across all documentation pages
Agents answer poorly when the knowledge base is a dump of raw files.
Ingestion turns sources into clean Documents, chunked Nodes, and a queryable Index with metadata the agent can trust.
Load documents with readers, apply a deliberate splitter (or full IngestionPipeline), attach metadata, then index with an embedding model you pin for the life of the corpus.
source, tenant, doc_id, updated_at).SimpleDirectoryReader or a LlamaHub/custom reader; fail closed on empty or binary garbage.SentenceSplitter is a solid default) with explicit chunk_size and chunk_overlap.IngestionPipeline for multi-step transforms (split, extract, embed).VectorStoreIndex (or write Nodes into your vector store) with a pinned embed model.from llama_index.core import (
SimpleDirectoryReader,
VectorStoreIndex,
Settings,
StorageContext,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.ingestion import IngestionPipeline
from llama_index.embeddings.openai import OpenAIEmbedding # verify at build
from llama_index.llms.openai import OpenAI # verify at build
Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
documents = SimpleDirectoryReader(
input_dir="data/policies",
recursive=True,
required_exts=[".md", ".txt", ".pdf"],
).load_data()
for d in documents:
d.metadata.setdefault("corpus", "policies")
d.metadata.setdefault("source", d.metadata.get("file_path", "unknown"))
splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=128)
# Option A: transformations at index time
index = VectorStoreIndex.from_documents(
documents,
transformations=[splitter],
)
index.storage_context.persist(persist_dir="./storage/policies")
# Option B: explicit pipeline for batch jobs
pipeline = IngestionPipeline(transformations=[splitter])
nodes = pipeline.run(documents=documents)
print(f"nodes={len(nodes)} sample_meta={nodes[0].metadata if nodes else None}")Standalone splitter (tests and custom stores):
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter
parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = parser.get_nodes_from_documents(
[Document(text="...", metadata={"doc_id": "refunds-v3"})]
)Retrievers return Nodes, not whole PDFs.
If a policy rule is split across two oversized chunks without overlap, the agent never sees the full rule in one hit.
If chunks are tiny and context-free, the model retrieves fragments that look relevant but lack prerequisites.
| Signal | Prefer |
|---|---|
| Dense legal/policy prose | Medium-large chunks (e.g. 512–1024 tokens) with overlap |
| FAQ-style Q/A files | Smaller chunks; one Q/A pair per Node when possible |
| Code or configs | Language-aware splitters; keep functions intact |
| Tables / complex PDFs | Better parsing first (e.g. LlamaParse), then structure-aware splits |
Token vs character semantics depend on the splitter implementation.
Treat numbers as knobs you validate with retrieval evals, not universal constants.
updated_at for recency filtersMetadata on Documents should flow to child Nodes (LlamaIndex inherits attributes on split).
Use pipelines when you need repeatable batch jobs, caching, or multiple transforms before indexing.
Use from_documents(..., transformations=[...]) for simpler apps.
Global Settings.text_splitter can set defaults, but production agents should pin transformations in code or config for auditability.
When a source file changes, re-ingest that document id and upsert Nodes rather than rebuilding the world if your store supports it.
If you change embed model or chunker, schedule a full reindex and dual-run evals.
Local persist dirs are fine for demos.
Agents in production usually need a shared vector store (pgvector, Pinecone, Chroma, Azure AI Search, etc.) with backup and multi-process access.
Chunking rules stay the same; only the storage backend changes.
chunk_size, overlap, and embed model in deploy notes.| Approach | Pros | Cons |
|---|---|---|
SentenceSplitter defaults | Fast start, decent general text | Not optimal for code/tables |
| Token-based splitters | Closer to model token limits | May cut mid-sentence without care |
| Semantic / hierarchical splitters | Better topical boundaries | More cost and complexity |
| LlamaParse + structured nodes | Strong on complex docs | External dependency and cost |
| Manual hand-authored chunks | Highest precision for small FAQs | Does not scale |
Start with a sentence splitter around 512–1024 tokens and 10–20% overlap, then adjust using retrieval metrics on real questions.
No. VectorStoreIndex.from_documents with an explicit splitter is enough until you need batch transforms or incremental jobs.
Store tenant_id in metadata and enforce filters at retrieval time (and in tool wrappers) so the agent cannot cross tenants.
Only in separate collections for A/B tests. Never mix incompatible embeddings in one index space.
Offline or asynchronous ingestion. The agent should query a ready index, not re-chunk large corpora on every user turn.
Version the document id, delete or supersede old nodes for that id, re-ingest, and smoke-test questions that previously cited it.
You can store full docs for display, but retrieval still needs Node-sized units. Keep both if you need “open source doc” UX plus RAG.
Use multimodal or VLM-oriented parsing so figures become text or structured captions before embedding, or retrieve multimodally with a stack that supports it.
Good chunks plus metadata filters make keyword/vector hybrid search effective. Bad chunks make both channels noisy.
Retriever hit-rate/MRR on a fixed question set and a faithfulness sample on agent answers before and after the reindex.
Related: The LlamaIndex Mental Model: Documents, Indexes, and Query Engines
Related: LlamaIndex Basics
Related: Building a Query Engine Agent Over Your Own Data
Related: Evaluating Retrieval Quality in a LlamaIndex Pipeline
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.
Reviewed by Chris St. John·Last updated Jul 16, 2026