Document classification & triage engine for a European claims desk
Engineered a calibrated classification model behind an active-learning triage queue, reducing intake processing time from 4.2 hours to under 18 minutes.
Key Production Metric & Outcome
99.1% F1-precision on critical settlement tiers; 6.4x operator throughput increase with automated disagree-and-retrain loop.
A European claims and insurance desk received over 35,000 mixed document packs per month across 14 document categories. Prior to our intervention, intake specialists opened every multipage PDF manually to assign workflow urgency and routing.
Problem & Baseline Constraints
A naïve multi-class classifier was accurate enough in aggregate (~88%) but dangerous on edge cases: false negatives on urgent hospital claims cost thousands in SLA penalties. There was no mechanism for desk reviewers to disagree with model confidence scores or feed corrections back into model retraining.
Architecture & Engineering Approach
We structured the intake engine around calibrated uncertainty and human-in-the-loop triage:
- Feature Pipeline: Hybrid text extraction (PDF raster features + layout-aware token embeddings) processed within an isolated sandboxed worker pool.
- Calibrated Inference Service: FastAPI microservice backed by ONNX Runtime with int8 quantization, delivering P95 inference latency < 42ms.
- Active-Learning Feedback Store: Operator corrections are committed to an append-only label store in PostgreSQL with confidence stratification.
- Frozen Eval Harness: Automated CI eval suite testing across 2,400 frozen edge-case document packs before any production model promotion.
Shadow Mode Deployment & Continuous Calibration Engine
Every candidate model checkpoint is deployed in shadow mode alongside production traffic for 14 days, evaluating live prediction divergence against frozen ground-truth distributions. Automated conformal prediction boundaries guarantee a maximum error rate bound of < 0.1% on catastrophic liability classes before traffic promotion.