Ingesting and Chunking Documents for an Agent's Knowledge Base
Agents answer poorly when the knowledge base is a dump of raw files.
Ingestion turns sources into clean Documents, chunked Nodes, and a queryable Index with metadata the agent can trust.
Summary
Load documents with readers, apply a deliberate splitter (or full IngestionPipeline), attach metadata, then index with an embedding model you pin for the life of the corpus.
Recipe
- Inventory sources (folders, APIs, tickets) and define required metadata fields (
source,tenant,doc_id,updated_at). - Load with
SimpleDirectoryReaderor a LlamaHub/custom reader; fail closed on empty or binary garbage. - Choose a node parser (
SentenceSplitteris a solid default) with explicitchunk_sizeandchunk_overlap. - Optionally compose an
IngestionPipelinefor multi-step transforms (split, extract, embed). - Build
VectorStoreIndex(or write Nodes into your vector store) with a pinned embed model. - Persist storage / externalize the vector DB; record splitter + embed model versions next to the index.
- Smoke-test retrieval on 10–20 real agent questions before wiring tools.
Working Example
from llama_index.core import (
SimpleDirectoryReader,
VectorStoreIndex,
Settings,
StorageContext,
)
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.ingestion import IngestionPipeline
from llama_index.embeddings.openai import OpenAIEmbedding # verify at build
from llama_index.llms.openai import OpenAI # verify at build
Settings.llm = OpenAI(model="gpt-4o-mini")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
documents = SimpleDirectoryReader(
input_dir="data/policies",
recursive=True,
required_exts=[".md", ".txt", ".pdf"],
).load_data()
for d in documents:
d.metadata.setdefault("corpus", "policies")
d.metadata.setdefault("source", d.metadata.get("file_path", "unknown"))
splitter = SentenceSplitter(chunk_size=1024, chunk_overlap=128)
# Option A: transformations at index time
index = VectorStoreIndex.from_documents(
documents,
transformations=[splitter],
)
index.storage_context.persist(persist_dir="./storage/policies")
# Option B: explicit pipeline for batch jobs
pipeline = IngestionPipeline(transformations=[splitter])
nodes = pipeline.run(documents=documents)
print(f"nodes={len(nodes)} sample_meta={nodes[0].metadata if nodes else None}")Standalone splitter (tests and custom stores):
from llama_index.core import Document
from llama_index.core.node_parser import SentenceSplitter
parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)
nodes = parser.get_nodes_from_documents(
[Document(text="...", metadata={"doc_id": "refunds-v3"})]
)Deep Dive
Why chunking dominates agent quality
Retrievers return Nodes, not whole PDFs.
If a policy rule is split across two oversized chunks without overlap, the agent never sees the full rule in one hit.
If chunks are tiny and context-free, the model retrieves fragments that look relevant but lack prerequisites.
Choosing chunk size
| Signal | Prefer |
|---|---|
| Dense legal/policy prose | Medium-large chunks (e.g. 512–1024 tokens) with overlap |
| FAQ-style Q/A files | Smaller chunks; one Q/A pair per Node when possible |
| Code or configs | Language-aware splitters; keep functions intact |
| Tables / complex PDFs | Better parsing first (e.g. LlamaParse), then structure-aware splits |
Token vs character semantics depend on the splitter implementation.
Treat numbers as knobs you validate with retrieval evals, not universal constants.
Metadata that agents actually need
- Provenance: path, URL, ticket id
- Freshness:
updated_atfor recency filters - Tenancy: customer or workspace id for multi-tenant stores
- Access tier: so tools can refuse restricted corpora
Metadata on Documents should flow to child Nodes (LlamaIndex inherits attributes on split).
IngestionPipeline vs ad hoc transforms
Use pipelines when you need repeatable batch jobs, caching, or multiple transforms before indexing.
Use from_documents(..., transformations=[...]) for simpler apps.
Global Settings.text_splitter can set defaults, but production agents should pin transformations in code or config for auditability.
Incremental updates
When a source file changes, re-ingest that document id and upsert Nodes rather than rebuilding the world if your store supports it.
If you change embed model or chunker, schedule a full reindex and dual-run evals.
Relation to vector DB choice
Local persist dirs are fine for demos.
Agents in production usually need a shared vector store (pgvector, Pinecone, Chroma, Azure AI Search, etc.) with backup and multi-process access.
Chunking rules stay the same; only the storage backend changes.
Gotchas
- Default splitters hide configuration. Always log
chunk_size, overlap, and embed model in deploy notes. - PDFs without layout repair. Text extraction that shreds tables will poison embeddings no matter the splitter.
- Missing overlap on long dependencies. Adjacent sentences that complete a rule need overlap or hierarchical nodes.
- Metadata only on parents. If custom code strips metadata during transform, filters and citations break.
- Mixing embed models in one collection. Vectors from different models are not comparable.
- Indexing secrets. Scrub API keys and PII before embedding; vector DBs become long-lived copies.
- Agent tool over a dirty corpus. Tool-calling cannot fix retrieval that returns the wrong section every time.
Alternatives
| Approach | Pros | Cons |
|---|---|---|
SentenceSplitter defaults | Fast start, decent general text | Not optimal for code/tables |
| Token-based splitters | Closer to model token limits | May cut mid-sentence without care |
| Semantic / hierarchical splitters | Better topical boundaries | More cost and complexity |
| LlamaParse + structured nodes | Strong on complex docs | External dependency and cost |
| Manual hand-authored chunks | Highest precision for small FAQs | Does not scale |
FAQs
What chunk size should I start with?
Start with a sentence splitter around 512–1024 tokens and 10–20% overlap, then adjust using retrieval metrics on real questions.
Do I need an IngestionPipeline for a prototype?
No. VectorStoreIndex.from_documents with an explicit splitter is enough until you need batch transforms or incremental jobs.
How do I keep multi-tenant data isolated?
Store tenant_id in metadata and enforce filters at retrieval time (and in tool wrappers) so the agent cannot cross tenants.
Should I embed the same text twice with different models?
Only in separate collections for A/B tests. Never mix incompatible embeddings in one index space.
Where should chunking live relative to the agent loop?
Offline or asynchronous ingestion. The agent should query a ready index, not re-chunk large corpora on every user turn.
How do I handle updates to a single policy PDF?
Version the document id, delete or supersede old nodes for that id, re-ingest, and smoke-test questions that previously cited it.
Can I chunk after storing full documents only?
You can store full docs for display, but retrieval still needs Node-sized units. Keep both if you need “open source doc” UX plus RAG.
What about images and slides?
Use multimodal or VLM-oriented parsing so figures become text or structured captions before embedding, or retrieve multimodally with a stack that supports it.
How does this connect to hybrid search?
Good chunks plus metadata filters make keyword/vector hybrid search effective. Bad chunks make both channels noisy.
Which evals gate a reindex?
Retriever hit-rate/MRR on a fixed question set and a faithfulness sample on agent answers before and after the reindex.
Related
Related: The LlamaIndex Mental Model: Documents, Indexes, and Query Engines
Related: LlamaIndex Basics
Related: Building a Query Engine Agent Over Your Own Data
Related: Evaluating Retrieval Quality in a LlamaIndex Pipeline
Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana
@solana/web3.js+@solana/spl-token.