Intelligent Document Processing (IDP) - Multi-Agent System
Enterprise-level Multi-Agent Orchestration Framework using LangGraph and Gemini 2.5 Flash to automate data extraction from complex Mutual Fund forms. Features self-correcting agents and confidence-based automation.
- Role: AI Engineer
- Timeline: Jan 2025 - Present
- Team: Core AI Team
- Technologies: LangGraph, Gemini 2.5, Vertex AI, Python, Pydantic, Docker
- Link: https://github.com/Laxman824/
Problem
Processing complex financial documents (such as Mutual Fund redemption requests and multi-page TIFFs) was a manual bottleneck. The lack of standardized layouts, presence of handwritten text, and strict compliance requirements made traditional OCR insufficient, resulting in slow turnaround times and high operational costs for manual verification.
Solution
Engineered a Multi-Agent Orchestration Framework using LangGraph to handle the full document lifecycle. The system utilizes a 'Router-Extractor-Auditor' architecture where a Semantic Router classifies documents, a Vision Extractor (Gemini 2.5) pulls structured data, and a Self-Reflective Auditor enforces business logic. If validation fails, the Auditor triggers a feedback loop, prompting the Extractor to self-correct before final output.
Impact
- Achieved Straight-Through Processing (STP) for high-confidence documents
- Automated extraction of Folios, PANs, and Signatures from unstructured inputs
- Reduced manual verification effort by utilizing confidence-based gating
- Implemented self-healing workflows where agents correct their own errors
- Standardized output formats using Pydantic schema validation
Key features
- Multi-Agent Orchestration using LangGraph state machines
- Multimodal extraction (Text + Vision) for scanned TIFFs/PDFs
- Semantic Routing to dynamically select extraction schemas
- Self-Reflective Auditor agent for automated quality control
- Feedback loops for Hallucination reduction
- Confidence-Based Gating for hybrid AI-Human workflows
Tech stack
- Backend: Python, LangGraph, LangChain
- Ai: Google Vertex AI, Gemini 2.5 Flash, Vision APIs
- Data: Pydantic, Pandas, JSON Structured Output
- Infrastructure: GCP, Docker, Cloud Run
System architecture
Trust boundary — Runs inside CAMS GCP — Cloud Run. Documents carry PAN numbers, folio numbers and customer signatures, so no production output can be shown publicly. The graph below is the system.
Document intake — The inputs traditional OCR could not handle
- Mutual Fund forms — Redemption requests, multi-page TIFFs. Non-standardised layouts, scanned quality, and handwritten fields — the combination that makes template-based OCR fail and forces a vision-model approach.
- Ingestion & page handling — Normalisation before the graph runs. Multi-page documents are split and normalised so each run enters the state machine with a predictable payload.
↓ Multi-page TIFF / PDF + document metadata
LangGraph state machine — StateGraph(AgentState) — a cyclic graph, not a pipeline
- StateGraph orchestrator — Compiled LangGraph app. A pipeline runs each stage once. This is a graph with a cycle in it, so the extractor can be re-entered when the auditor rejects its output — which is what makes the system self-correcting rather than merely sequential.
- AgentState — document · extracted_data · errors · retries. The typed state every node reads and writes. Carrying `errors` and `retries` in shared state is what lets the auditor's findings reach the extractor on the next pass, and what makes the retry cap enforceable. (TypedDict, Shared state)
↓ AgentState initialised
Agent 01 · Semantic Router — semantic_router_agent — which AMC, and therefore which schema
- AMC classification — SBI · HDFC · Jio · others. Every asset management company uses its own form layout, so the first decision is which AMC issued this document. It classifies semantically rather than by template match, which is why an unseen variant of an SBI or HDFC form still routes correctly.
- AMC-specific extraction schema — Per-AMC field set and rules. Each AMC maps to its own field set — the fields that exist, where they sit, and what a valid value looks like differ per issuer. Selecting the schema up front keeps the extractor's prompt narrow and makes its output checkable against something specific rather than generic.
↓ AMC identity + AMC-specific field schema
Agent 02 · Vision Extractor — gemini_vision_agent — multimodal, re-entrant
- Gemini 2.5 Flash — Vision + text in one pass. Reads the page as an image rather than as OCR text, which is what allows handwritten fields and irregular layouts to be extracted. Flash was chosen deliberately for the cost/accuracy balance at this volume. (Vertex AI, Multimodal)
- Target field extraction — Folio · PAN · signatures. Pulls the compliance-relevant fields into the schema chosen by the router — the values downstream systems actually consume.
- Re-entry with error context — Retry pass, not a blind repeat. On a retry the extractor receives the auditor's specific findings, so the second attempt is corrective rather than a re-roll of the same prompt.
↓ Structured JSON candidate + per-field confidence
↺ Validation failed → re-extract with error context
Agent 03 · Self-Reflective Auditor — business_logic_validator — the reflection step
- Business-logic validation — Domain rules, not just shape. Checks that extracted values make sense against mutual-fund business rules — a well-formed but wrong PAN is still a failure. This is the check that catches hallucinated values.
- Pydantic schema enforcement — Strict typed output. Every output is validated against a Pydantic model, so malformed JSON never reaches a downstream system. Schema violations become structured errors the extractor can act on.
- AMC cross-verification — Webhook lookup against the issuer. The strongest check in the system: extracted identifiers such as the account or folio number are sent by webhook to the corresponding AMC, which searches its own records and returns the matching data. The extraction is then reconciled against the issuer's system of record — so correctness is confirmed against ground truth rather than inferred from model confidence. (Webhook, Account no. match, Source of record)
- Confidence scoring — Feeds the automation gate. Combines model confidence with the outcome of the AMC match to produce the score the conditional edge uses to choose between straight-through processing and human review.
↓ errors[] · confidence · retries
Conditional edge · check_quality — add_conditional_edges("auditor", …) — three outcomes
- No errors → END — Straight-through processing. A clean, high-confidence document finishes with no human involvement at all — the STP path the whole design exists to maximise.
- Errors → extractor — Self-correction cycle. Validation failures send the document back to the extractor with the error list attached. This is the cycle that gives the system its self-healing behaviour.
- retries > 3 → human_review — Bounded autonomy. A hard retry cap. Without it a genuinely unreadable document would loop forever burning tokens; with it, difficult cases degrade gracefully into a human queue instead of failing loudly or silently.
↓ Routed to its terminal state
Terminal states — Automated or escalated — never dropped
- Validated structured output — Schema-conformant JSON. Typed, validated data released straight to downstream processing for high-confidence documents.
- Human-in-the-loop dashboard — manual_queue node. Only sub-threshold cases surface here, each pre-filled with the extraction attempt, the auditor's findings and the AMC match result. A reviewer corrects a draft instead of keying a form from scratch, which is where the bulk of the manual-effort reduction comes from — the default flips from 'a human checks everything' to 'a human sees only what the system flagged'.
What this replaced
- Template-based OCR — Insufficient for these documents. Traditional OCR assumes a standardised layout and printed text. Against non-standard mutual-fund forms with handwriting, it produced results that still required full manual verification — which is what made a multi-agent vision approach necessary.
- Full manual verification — The original bottleneck. Every document previously passed through a human reviewer. The confidence gate inverts that default: humans now see only the cases the system flags.
Control loops
- Reflection loop — The auditor returns structured findings and the extractor re-runs against them — the pattern that reduces hallucinated field values instead of just detecting them.
- Retry ceiling — Capped at 3 passes. Bounded autonomy: the graph cannot spin on a document it will never parse.
- Confidence gating — The automation rate is a tunable threshold, not fixed behaviour — the gate can be tightened per AMC or for compliance-sensitive document types.
- Ground-truth reconciliation — Extracted identifiers are matched back against the issuing AMC's own records by webhook, so a confidently-wrong extraction is caught by data rather than by a score.
What K Laxman learned
- Orchestrating stateful multi-agent systems with cyclic graphs
- Prompt engineering for multimodal LLMs (Vision + Text)
- Implementing 'Reflection' patterns to reduce AI hallucinations
- Balancing cost vs. accuracy with Gemini Flash models
- Enforcing strict JSON schemas using Pydantic in production
Explore more
- Home — overview, skills and a built-in AI assistant
- Experience — roles at Think360 AI (CAMS), CAMS Mutual Funds and IIT Delhi
- Projects — GenAI, LLM, RAG and full-stack builds
- Education — IIT Delhi, M.Tech & B.Tech Computer Science
- GitHub Activity — open-source contributions
- Contact / Hire me