Building a Query Engine Agent Over Your Own Data

A query engine answers one RAG question well.

An agent decides whether and when to use that engine among other tools (calculators, tickets, CRM, web).

This page wires your index into a QueryEngineTool and a LlamaIndex agent loop.

Summary

Build or load a query engine, wrap it with a precise name and description via QueryEngineTool.from_defaults, then pass it to FunctionAgent (or ReActAgent when function calling is weak).

Recipe

1. Ingest and index your corpus (or load a persisted index).
2. Create query_engine = index.as_query_engine(...) with an explicit similarity_top_k.
3. Wrap it: QueryEngineTool.from_defaults(query_engine, name=..., description=...).
4. Choose agent type: FunctionAgent for tool-calling models; ReActAgent for broader model support.
5. Write a system prompt that states when to use the docs tool versus other tools.
6. Run async agent.run(...) and log tool calls + source nodes for traces.
7. Add stop conditions (max steps, timeouts) at the product layer even if the agent class has defaults.

Working Example

import asyncio
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.tools import QueryEngineTool, FunctionTool
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI  # verify imports at build
 
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=4)
 
docs_tool = QueryEngineTool.from_defaults(
    query_engine,
    name="company_docs",
    description=(
        "Search internal company policy and product documentation. "
        "Use for refunds, support hours, SLAs, and feature behavior. "
        "Input should be a full natural-language question."
    ),
)
 
def escalate(reason: str) -> str:
    """Open a human handoff ticket when docs are insufficient."""
    return f"ESCALATED: {reason}"
 
tools = [
    docs_tool,
    FunctionTool.from_defaults(escalate),
]
 
llm = OpenAI(model="gpt-4o-mini")  # verify at build
agent = FunctionAgent(
    tools=tools,
    llm=llm,
    system_prompt=(
        "You are a support agent. Prefer company_docs for policy facts. "
        "If documentation is missing or conflicting, call escalate."
    ),
)
 
async def main() -> None:
    answer = await agent.run(
        "A customer bought a plan 12 days ago. Are they still eligible for a refund?"
    )
    print(str(answer))
 
if __name__ == "__main__":
    asyncio.run(main())

Optional: return the tool result without an extra rewrite when the docs answer is the product response:

docs_tool = QueryEngineTool.from_defaults(
    query_engine,
    name="company_docs",
    description="...",
    return_direct=True,  # ends loop after tool; use carefully
)

Deep Dive

Why wrap a query engine instead of only chatting with the index?

index.as_chat_engine() is great for a pure RAG chatbot.

Agents need optional retrieval: some turns are pure tool calls, math, or handoffs with no corpus hit.

QueryEngineTool makes the RAG pipeline look like any other API from the model's point of view.

Tool description quality

The model selects tools from name + description + parameter schema.

Weak description: "docs tool".

Strong description: scope (which corpus), intended question types, and what not to use it for.

One tool per coherent corpus beats one mega-tool over unrelated folders.

FunctionAgent vs ReActAgent

Combining docs with other tools

Typical production mix:

• company_docs query engine tool
• ticket/CRM function tools
• escalation or approval tools
• calculators or pricing APIs

Keep the system prompt short and role-specific.

Put operational policy in tools and data, not a 2,000-token sermon.

Memory

Pass a ChatMemoryBuffer (or current memory API) when multi-turn sessions must remember prior tool results.

Bound memory so long RAG dumps do not blow the context window.

Observability

Log for every turn:

• user message
• selected tool name and arguments
• retrieved node ids / scores when available
• final answer and stop reason

Without this, “the agent ignored the docs” is guesswork.

Framework interop

You can expose the same query engine to LangGraph, CrewAI, or a custom runtime by wrapping query_engine.query as that framework’s tool type.

LlamaIndex does not have to own the outer loop if you only need its RAG stack.

Gotchas

1. Vague tool descriptions cause wrong tool picks. Rewrite descriptions using failed traces.
2. return_direct=True skips synthesis and multi-tool plans. Use only for pure Q&A paths.
3. Huge top-k fills the agent context. Prefer tighter retrieval and optional rerankers.
4. Blocking sync calls in async agents. Prefer async-capable paths where the stack supports them.
5. Stale indexes. Agent confidence stays high while docs rot; schedule reingestion.
6. No escalation tool. The model invents policy when retrieval is empty unless you force a handoff path.
7. Import drift. Agent class locations move across LlamaIndex versions - verify at build.

Alternatives

FAQs

Related

Related: LlamaIndex Basics

Related: Ingesting and Chunking Documents for an Agent's Knowledge Base

Related: Combining Multiple Indexes with Router Query Engines

Related: Building a Retrieval Tool an Agent Can Call

Related: Combining OpenRouter with LangChain, CrewAI, and LlamaIndex

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.