OpsPilot — Agentic Request Triage & Remediation System

Developed an end-to-end Agentic AI platform that automates customer support operations by understanding incoming emails, planning intelligent remediation workflows, validating decisions through policy guardrails, and providing transparent execution with human oversight and complete auditability.

• Role: AI Engineer & Solo Developer
• Timeline: 2026
• Team: Solo Developer (proof-of-concept / take-home style build, ~3 days)
• Technologies: Gemini 2.5, Python, Streamlit, SQLite, Pandas, Altair
• Link: https://opspilot-autonomous-agent.streamlit.app/

Problem

Built OpsPilot, an AI-powered Agentic Operations platform that transforms incoming customer emails into intelligent, policy-aware workflows. The system leverages LLMs to classify requests, generate explainable multi-step remediation plans, execute automated actions with human-in-the-loop safeguards, and maintain a complete audit trail—demonstrating production-ready AI orchestration, governance, and operational transparency.

Solution

Built a custom, framework-free ~140-line agent loop (no LangChain/AutoGPT) with 7 stages: perceive (30-day sender-history recall), plan (one Gemini 2.5 call producing both classification and an ordered, justified tool-call plan), guard (a deterministic policy layer validates the plan against a per-request-type playbook — inserting missing mandatory steps, stripping forbidden ones, capping length), gate (confidence-based routing to a human review queue), act (executes the final plan through a 12-tool registry), reflect (a critic LLM call reviews and rewrites any customer-facing draft against a compliance checklist), and record (every classification, both plan versions, every guardrail repair, and every tool result persisted to SQLite for per-case replay).

Impact

• "Agent proposes, policy disposes": guardrails.py encodes each request type's mandatory/optional/forbidden actions as data, so a deterministic function — not the LLM — has final say over what actually executes
• 3-tier resilience chain (primary model → fallback model → offline keyword classifier) means the app degrades gracefully instead of crashing on API quota errors or a missing key
• Fused classify+plan into a single structured-output call, roughly halving LLM request volume versus separate calls and keeping the plan's rationale coherent with the classification's
• Live dashboard computes real stats (avg confidence, % plans repaired, % human-overridden, status mix) directly from the audit database rather than hardcoded numbers

Key features

• JSON-schema-constrained Gemini call returns classification (type, urgency, confidence, sentiment, rationale, department) and an ordered, justified remediation plan in one pass
• Policy-guardrail layer (guardrails.py) repairs agent-proposed plans against a per-type playbook and logs every repair for audit
• Confidence-gated human review queue: reclassify, keep/drop individual plan steps, approve — case re-enters the same pipeline tagged as a human override
• Critic/reflection pass rewrites any customer-facing draft that fails a 5-point compliance checklist, showing original vs. revised text side-by-side
• Sender-memory recall: looks up a sender's 30-day case history and folds it into the planning prompt, so repeat complainants get urgency bumps and supervisor alerts
• 12-tool action catalog (draft reply, KB lookup, structured extraction, routing, escalation, supervisor alert, SLA timer, human hold, simulated send, mark resolved, etc.) with stable, swappable-for-real-integrations contracts
• Full SQLite audit trail (cases / plans / audit_log) with a per-case drill-down UI that replays the entire reasoning trace on demand
• Batch CSV tab processes a queue of requests sequentially with a progress bar, throttled to the Gemini free-tier rate limit

Tech stack

• Ai: Gemini 2.5 Flash, Gemini 2.5 Flash-Lite, google-genai (structured output)
• Backend: Python 3.12, Streamlit
• Data: SQLite, pandas
• Viz: Altair

System architecture

Trust boundary — Policy-constrained autonomous mail reply Agent. The agent composes its own remediation plan and justifies every step; a deterministic policy layer has final say over what actually executes. Nothing runs below the confidence threshold.

01 · Perceive — Intake and sender history

• Request intake — Form · simulated inbox · batch CSV. Customer requests arrive through a form, a simulated inbox, or a batch CSV queue processed sequentially and throttled to the model's rate limit.
• Sender memory — 30-day case-history recall. Before planning, the sender's prior cases are recalled and folded into the prompt — so a repeat complainant gets an urgency bump and a supervisor alert the agent would not otherwise have proposed. This is the difference between a stateless classifier and an agent with context.

↓ Normalised request + 30-day sender context

02 · Agent 01 · Triage-Planner — One fused, schema-constrained Gemini call

• Classification — type · urgency · confidence · sentiment. Type and urgency are judged independently, each with a written rationale. Emitting confidence here is what makes the downstream gate possible.
• Self-composed plan — Ordered tool calls + per-step justification. The agent writes its own remediation plan rather than triggering a hardcoded branch, and every optional step needs a concrete, request-specific justification visible on the plan row. This is the line between an agent and an if/else tree with a classifier in front of it.
• Fused single call — Classify + plan in one pass. Classification and planning share one structured-output call — roughly halving request volume against the free-tier rate limit, and keeping the plan coherent with the rationale that produced it. (JSON schema, google-genai)
• 3-tier resilience chain — backoff → Flash-Lite → offline. Backoff retries, then a fallback to Gemini 2.5 Flash-Lite, then a fully offline keyword classifier with confidence capped at 0.5 — which by construction routes every case to human review. A quota error degrades the system instead of crashing it.

↓ Classification + ordered, justified tool plan

03 · Guard — guardrails.py — the agent proposes, policy disposes

• Playbook as data — 4 request types. Complaint, General Enquiry, Service Request and Escalation/Urgent each declare mandatory, optional and forbidden steps plus a terminal status. Policy lives as data, so changing it does not mean changing the agent. (ESCALATED, RESOLVED, ROUTED, HELD_FOR_HUMAN)
• validate_and_repair_plan() — Mandatory inserted · forbidden stripped. A deterministic function — not the model — has the last word on what executes. Missing mandatory steps are inserted, forbidden actions removed, length capped, and every repair surfaced in the UI rather than applied silently.
• Proposed vs. final plan — Both versions persisted. Storing what the agent wanted alongside what policy allowed gives a diff view per case — and a live metric for the share of plans the guardrail had to repair.

↓ Repaired plan + flagged repairs

04 · Gate — Confidence threshold, default 0.70

• confidence ≥ 0.70 — Straight to execution. Above the threshold the plan executes unattended. The threshold is adjustable at runtime, so the automation rate is an operator decision rather than a fixed property of the system.
• Below threshold → Human Review — Nothing executes. An ambiguous request stops dead — no side effects at all. An operator reclassifies, keeps or drops individual plan steps, and approves; the case then re-enters the same pipeline badged as a human override.
• Autonomy modes — Auto-run or propose-and-confirm. The whole system can be switched to require approval for every plan regardless of confidence — the same agent, a different trust setting.

↓ Approved plan cleared for execution

05 · Act — Tool executor over a 12-tool catalog

• Tool executor — Runs the final plan step by step. Each step is wrapped so that one tool failure marks the remaining steps SKIPPED and flags the case, rather than crashing the run mid-way and leaving it in an unknown state.
• 12-tool catalog — Draft · KB lookup · route · escalate · SLA. Draft reply, knowledge-base lookup, structured extraction, department routing, escalation, supervisor alert, SLA timer, human hold, simulated send, mark resolved and more — the vocabulary the planner composes from.
• Structured artifacts — Simulated side effects. Routing records, alerts and reminders are persisted as structured artifacts rather than sent for real. The tool contracts are stable, so swapping in Gmail, Slack or a ticketing system is integration work, not a rewrite.

↓ Drafts, routing records, alerts, SLA timers

↺ Draft rejected → revised against the checklist

06 · Agent 02 · Critic — Compliance review of anything customer-facing

• 5-point compliance checklist — No fault admission · no promises · no invented facts. Every customer-facing draft is reviewed against a fixed checklist before it can leave. In a regulated support context, the risk is not a clumsy sentence — it is an AI admitting liability or promising compensation on the company's behalf.
• Revision pass — Original vs. revised, side by side. A rejected draft is rewritten rather than just blocked, and both versions are shown together — so the reviewer can see what the critic objected to and judge whether it was right.

↓ Approved or revised draft

07 · Record — Glass-box audit trail

• SQLite audit trail — cases · plans · audit_log. Every classification, both plan versions, every guardrail repair and every tool result is persisted — enough to replay a case's entire reasoning trace on demand.
• Agent Trace view — Per-case replay. The reasoning is inspectable on screen rather than buried in logs. Being able to show an operator exactly why the agent did what it did is what makes the autonomy defensible.
• Live dashboard — Computed from the audit DB. Volumes, status mix, average confidence, share of plans the guardrail repaired and share human-overridden — all computed from the audit database rather than hardcoded. (Altair, pandas)

Deliberately out of scope

• Real channel integrations — Email · Slack · ticketing. Side effects are simulated on purpose in a proof-of-concept. Because the tool contracts are stable, connecting real channels is roadmap rather than rework.
• Agent frameworks — No LangChain / AutoGPT. The orchestration is a ~140-line loop written from scratch. In a 3-day build the orchestration was the thing being assessed, so every line is explainable and there is no framework dependency to defend.

Control loops

• Reflection loop — The critic reviews and rewrites customer-facing drafts before they leave, with both versions retained for inspection.
• Human override loop — An operator's corrections re-enter the same pipeline tagged as an override, so the audit trail records the disagreement rather than hiding it.
• Graceful degradation — Model unavailable? Confidence is capped at 0.5, which routes every case to a human by construction — the failure mode is more oversight, never silent wrong action.

Architecture highlights

• 7-stage agent loop (perceive → plan → guard → gate → act → reflect → record) implemented as a single ~140-line module, no agent framework dependency
• Guardrail playbooks per request type (Complaint, General Enquiry, Service Request, Escalation/Urgent) each define mandatory, optional, and forbidden steps plus a terminal status
• SQLite schema stores both the agent's originally proposed plan and the guardrail-repaired final plan per case, enabling a diff view of what the agent wanted vs. what policy allowed
• Try/except-wrapped tool execution means a single tool failure marks remaining steps SKIPPED and flags the case, rather than crashing the run

What K Laxman learned

• Encoding business policy as plain data structures that a deterministic function validates against is more reliable than trusting an LLM to self-enforce rules
• Fusing classification and planning into one structured-output call cuts API usage without sacrificing plan-rationale coherence
• A confidence-capped offline fallback (keyword-based, no API) is a simple way to guarantee an app never silently fails when a model or key is unavailable
• Designing tool contracts (`(req, cls, params, ctx) -> ToolResult`) up front makes 'simulate now, wire in real integrations later' a contained change instead of a rewrite

← All 27 projects by K Laxman

Explore more

• Home — overview, skills and a built-in AI assistant
• Experience — roles at Think360 AI (CAMS), CAMS Mutual Funds and IIT Delhi
• Projects — GenAI, LLM, RAG and full-stack builds
• Education — IIT Delhi, M.Tech & B.Tech Computer Science
• GitHub Activity — open-source contributions
• Contact / Hire me

GitHub · LinkedIn · Email