SEBI Debarred Entities Extraction — Regulatory Document Intelligence
Cloud-native event-driven RAG backend monitoring SEBI regulatory orders, routing multimodal PDFs via Camelot or Gemini Vision, scoring page relevance with FAISS, and extracting structured debarred PAN records using schema-constrained Gemini 2.5 Flash.
- Role: Backend Developer & AI Engineer
- Timeline: August 2024 - October 2024
- Team: Solo + Compliance Team Review
- Technologies: Gemini 2.5, FAISS, Python, Google Cloud, Pub/Sub, Firestore, Docker
- Link: https://github.com/Laxman824/
Problem
SEBI regularly publishes legal PDF orders debarring entities from securities markets. Compliance teams manually reviewed hundreds of pages per PDF to identify debarred individuals and extract PAN details—an expensive, error-prone process critical for KYC compliance.
Solution
Built an event-driven RAG pipeline on GCP (Pub/Sub, Cloud Run, GKE, Firestore) that automatically ingests regulatory PDFs. The system routes documents based on type (Camelot for text-native vs. Gemini Vision for scanned OCR), scores page relevance using FAISS vector search on SEBI-specific document patterns, and passes only highly relevant context into Gemini 2.5 Flash with schema enforcement for 100% structured PAN extraction.
Impact
- 50% reduction in manual compliance review time across regulatory teams
- 6,000+ SEBI orders automatically evaluated and indexed in Firestore
- 10 concurrent PDF extractions handled simultaneously via Pub/Sub queue
- 100% schema-constrained structured outputs (PAN, entity name, debarment period) extracted with zero hallucination
Key features
- Automated SEBI regulatory order ingestion and deduplication
- Multimodal document router (Camelot for native text vs. Gemini Vision for scanned OCR)
- FAISS vector similarity scoring to isolate relevant debarment pages and prune legal boilerplate
- Schema-constrained Gemini 2.5 Flash extraction for PAN numbers, entity names, and durations
- Event-driven GCP architecture using Pub/Sub, Cloud Run, and GKE with autoscaling
- Searchable Firestore database for instant downstream KYC watchlist matching
Tech stack
- Backend: Python, FastAPI, Camelot, PyPDF2
- Ai: Gemini 2.5 Flash, Gemini Vision, FAISS, Document AI
- Cloud: Cloud Run, Pub/Sub, Cloud Scheduler, Firestore, GKE
- Tools: Docker, Cloud Build, Secret Manager
System architecture
Trust boundary — Event-driven SEBI Regulatory Document Intelligence Pipeline. Event-driven pipeline routes each PDF by type, scores page relevance with FAISS, and returns schema-constrained extractions.
01 · Ingestion & Queue — Pub/Sub event pipeline
- SEBI order PDF — Pub/Sub · 10 concurrent. Automated scrapers ingest regulatory PDF orders from SEBI into GCP Pub/Sub, processing up to 10 concurrent PDF streams simultaneously across Cloud Run and GKE worker nodes.
↓ SEBI order PDF → Concurrent Pub/Sub Queue
02 · Document Router — Multimodal parsing decision tree
- Document router — text vs scanned. Inspects incoming PDF stream character density to determine document structure: native digital text PDFs are dispatched to Camelot, while scanned image PDFs route to Gemini Vision OCR.
- Camelot — text extraction. Extracts text and table boundaries directly from native digital PDFs fast without network LLM overhead.
- Gemini Vision — scanned OCR. Multimodal vision OCR fallback for image-only or low-quality scanned legal orders to convert PDF bitmap pages into clean markdown text.
↓ PDF content check → Camelot OR Gemini Vision
03 · Context Scoring (RAG) — FAISS page relevance search
- FAISS scoring — page relevance. Embeds document pages and scores vector similarity against SEBI legal directive patterns. Filters out 90%+ of irrelevant legal boilerplate pages before LLM inference.
↓ Extracted pages → Relevant context pages
04 · Structured Extraction — Gemini 2.5 Flash schema enforcement
- Gemini 2.5 Flash — schema-constrained. Evaluates the relevance-filtered pages using strict JSON schema output constraints to extract 100% structured PAN numbers, entity names, and debarment terms without hallucination.
↓ Relevant context → Validated JSON entities
05 · Persistence — Firestore compliance store
- Debarred PANs — Firestore. Persists structured debarment entity profiles into Firestore with deduplication, enabling instant KYC watchlist search and compliance checks.
↓ PAN JSON → Firestore Database
Deliberately out of scope / Future Work
- Real-time KYC Webhooks — Core banking alerts. Automated webhook dispatches to core banking systems upon discovering newly debarred entities are planned for future integration.
Control loops
- Concurrency Autoscaling — Cloud Run and GKE scale worker instances dynamically up to 10 concurrent PDF extractions based on Pub/Sub queue depth.
- Multimodal Fallback — If native text parsing via Camelot yields low character density, the document automatically re-routes to Gemini Vision OCR.
Architecture highlights
- 5-stage event-driven pipeline: Ingest (Pub/Sub) → Route (Decision) → Parse (Camelot / Gemini Vision) → Score (FAISS) → Extract & Save (Gemini 2.5 Flash → Firestore)
- FAISS vector store isolates debarment-relevant pages, pruning irrelevant legal boilerplates before LLM invocation
- Fail-safe fallback: Scanned or damaged PDFs automatically redirect to Gemini Vision OCR when native text extraction returns low confidence
- Schema enforcement guarantees structured entity records formatted specifically for instant KYC matching
What K Laxman learned
- Filtering multi-page PDFs using FAISS vector page scoring before passing context to LLMs drastically reduces token consumption and response latency
- Routing documents early (Camelot for text vs. Gemini Vision for scanned OCR) preserves extraction accuracy while keeping costs optimal
- Enforcing strict JSON schema constraints on Gemini calls prevents formatting hallucinations in downstream compliance databases
- Decoupling PDF intake and parsing via GCP Pub/Sub ensures high concurrency without worker memory crashes
Explore more
- Home — overview, skills and a built-in AI assistant
- Experience — roles at Think360 AI (CAMS), CAMS Mutual Funds and IIT Delhi
- Projects — GenAI, LLM, RAG and full-stack builds
- Education — IIT Delhi, M.Tech & B.Tech Computer Science
- GitHub Activity — open-source contributions
- Contact / Hire me