Combining Vision and Tool Use for Document-Processing Agents

A document-processing agent turns messy visual inputs (scans, photos, PDF page renders) into structured records, then uses tools to validate, enrich, and submit those records. Vision alone is not enough for production. You need schemas, validators, retries, and systems-of-record tools with clear authority boundaries.

Summary

Render or load page images, extract JSON with a multi-modal model against a strict schema, validate in code, repair with a second vision or text pass when needed, then call write tools only after checks pass.

Recipe

1. Define the output schema (Pydantic/JSON Schema) and acceptance rules (required fields, ranges, checksums).
2. Normalize inputs: PDF → page images, HEIC → JPEG, deskew/resize within model limits (verify at build).
3. Classify document type first if you handle multiple forms (invoice vs receipt vs ID).
4. Run a vision extract prompt that requests JSON only matching the schema for that type.
5. Validate with code; on failure, retry with validator errors + the same image (bounded retries).
6. Optionally cross-check totals, dates, or IDs with calculator/regex/DB lookup tools.
7. Gate any write/submit tool on validation_status=passed (and human approval for high risk).
8. Store source image references, model id, extract version, and confidence for audit.
9. Eval on a labeled fixture set; track field-level precision/recall, not only "looks good."
10. Redact or expire images per retention policy - documents are high-sensitivity data.

Working Example

Conceptual pipeline (provider-agnostic; verify vision APIs at build):

import base64
import json
from enum import Enum
from pathlib import Path
from pydantic import BaseModel, Field, ValidationError, field_validator
 
class DocType(str, Enum):
    invoice = "invoice"
    receipt = "receipt"
    unknown = "unknown"
 
class InvoiceExtract(BaseModel):
    vendor: str
    invoice_number: str
    invoice_date: str  # ISO date preferred
    currency: str = Field(min_length=3, max_length=3)
    subtotal: float
    tax: float
    total: float
    line_items: list[dict] = []
 
    @field_validator("total")
    @classmethod
    def total_nonneg(cls, v: float) -> float:
        if v < 0:
            raise ValueError("total must be >= 0")
        return v
 
def image_data_url(path: str) -> str:
    b64 = base64.b64encode(Path(path).read_bytes()).decode("ascii")
    return f"data:image/png;base64,{b64}"
 
def vision_json(client, model: str, prompt: str, data_url: str) -> str:
    resp = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": {"url": data_url}},
            ],
        }],
        max_tokens=1200,
    )
    return resp.choices[0].message.content or ""
 
def extract_invoice(client, model: str, path: str, max_repairs: int = 2) -> InvoiceExtract:
    data_url = image_data_url(path)
    prompt = (
        "Extract invoice fields as JSON with keys: vendor, invoice_number, "
        "invoice_date, currency, subtotal, tax, total, line_items. "
        "Use ISO dates. Numbers as JSON numbers. No markdown."
    )
    last_err = None
    text = vision_json(client, model, prompt, data_url)
    for _ in range(max_repairs + 1):
        try:
            data = json.loads(text)
            inv = InvoiceExtract.model_validate(data)
            # Business rule example
            if abs((inv.subtotal + inv.tax) - inv.total) > 0.05:
                raise ValueError("subtotal + tax != total")
            return inv
        except (json.JSONDecodeError, ValidationError, ValueError) as e:
            last_err = e
            text = vision_json(
                client,
                model,
                prompt + f"\nPrevious output failed validation: {e}\nReturn corrected JSON only.",
                data_url,
            )
    raise RuntimeError(f"extract failed: {last_err}")
 
# Tools the agent may call after validation
def tool_lookup_vendor(vendor: str) -> dict:
    return {"ok": True, "vendor_id": "V-104", "normalized_name": vendor.strip().title()}
 
def tool_submit_invoice(payload: dict) -> dict:
    # Only call when host sets validated=True
    return {"ok": True, "erp_id": "INV-9001"}

Agent policy sketch:

def run_doc_agent(client, model: str, image_path: str):
    inv = extract_invoice(client, model, image_path)
    vendor = tool_lookup_vendor(inv.vendor)
    record = {**inv.model_dump(), **vendor, "validated": True}
    # Human gate for amounts over threshold would sit here
    return tool_submit_invoice(record)

For multi-page PDFs, loop pages, extract per page or per doc, and merge with explicit rules (header on page 1, line items on following pages).

Deep Dive

Why combine vision with tools

OCR prepass vs pure vision

Many production systems OCR for search indexing and still use vision (or layout models) for field extraction on the same file.

Multi-document and multi-page control

• Classify before extract so prompts stay small.
• Keep a page index in state: {doc_id, page, status}.
• Prefer plan-and-execute: classify → extract all pages → validate → submit.
• Cap pages and megapixels per run to control cost.

Confidence and humans

Models may not emit calibrated confidence. Proxy signals: validator pass, arithmetic checks, vendor lookup hit, second-model agreement. Send low-trust cases to human review queues rather than auto-posting to ERP.

Gotchas

1. Free-text extracts in production - unparseable prose skips automation.
2. No arithmetic checks - plausible invoices with impossible totals.
3. Submitting before validation - garbage in the system of record.
4. Whole-PDF as one giant image - resolution and token blowups; paginate.
5. Logging full document images forever - privacy and compliance risk.
6. Unbounded repair loops - spend caps needed on vision retries.
7. Trusting letterhead as vendor identity - spoof risk; bind to known vendor lists when paying.
8. Mixing read tools and pay/transfer tools in one allowlist - least privilege.

Alternatives

FAQs

Related

• How Vision Models Extend an Agent Beyond Text - multi-modal foundations
• Multi-Modal & Browser Agents Basics - first image loop
• Building a Browser-Controlling Agent with Playwright - portal downloads
• Multi-Modal & Browser Agents Best Practices - reliability checklist
• Tool Use and Function Calling Basics - tool loop
• Human in the Loop - escalation for payments

Stack versions: Pins from the category manifest (verify at build): OpenRouter (~315+ models, July 2026 pricing/fees); LangGraph 1.0+; CrewAI 1.14+; Microsoft Agent Framework 1.0; Vercel AI SDK 6; Pydantic AI (latest); LlamaIndex (latest); OpenAI Agents SDK (latest + MCP); MCP (Linux Foundation governance); A2A (HTTP+SSE+JSON-RPC 2.0); Solana @solana/web3.js + @solana/spl-token.