Contract Risk Classification Model
Research system for automated contract risk classification: 41 CUAD clause labels plus a contract-level risk rating (Low/Medium/High) with evidence-backed, citation-traceable outputs.
Key metrics
0.866
Final Macro-F1
Hybrid Phase 3 reasoner
0.759
Legal-BERT Macro-F1
Phase 2 clause detection
0.719
Classical baseline
Lexical signal only
0.903
Oracle ceiling
RF on ground-truth clauses
0.159
Failed BN ablation
Documented, not hidden
Research question
Legal risk assessment on long contracts is either fully manual (slow, inconsistent across reviewers) or handled by black-box AI tools that give a risk score with no traceable reasoning. Neither scales, and neither survives an auditor asking why.
Method
A three-phase research pipeline, each phase built to be falsifiable, not just functional. Phase 1 established a classical ML baseline to know what's recoverable from lexical signal alone. Phase 2 fine-tuned Legal-BERT with LoRA adapters for clause detection across long, sometimes-scanned contracts.
Phase 3 converts clause-level signal into a contract-level risk decision. A Bayesian Network collapsed under class imbalance; we diagnosed why and replaced it with a calibrated Random Forest reasoner instead of patching the failure.
Architecture
- 01
Phase 1, Classical ML baseline to measure what lexical signal alone can recover
- 02
Phase 2, Legal-BERT with LoRA adapters for 41-label CUAD clause detection on long and scanned contracts
- 03
Phase 3, Clause-frequency features into a calibrated Random Forest reasoner after a Bayesian Network collapse under class imbalance
- 04
Delivery, Upload UI returns risk class with citation-traceable evidence
Findings
- Macro-F1 progression: 0.719 (classical baseline) → 0.759 (Legal-BERT) → 0.866 (final hybrid Phase 3 reasoner)
- Failed Bayesian Network (Macro-F1 0.159) documented and replaced, kept as ablation evidence
- Oracle ceiling test (RF fed ground-truth clauses) hits 0.903, remaining headroom is in upstream clause extraction, not the reasoning layer
Method details
- Task
- Clause detection (41 CUAD categories) + contract-level risk classification
- Backbone
- nlpaueb/legal-bert-base-uncased, LoRA fine-tuned
- Reasoner
- Calibrated Random Forest (SMOTE-balanced) on clause-frequency features
- Ingestion
- pdfplumber, with EasyOCR fallback for scanned contracts
- Delivery
- Web UI (upload → risk class + evidence)
Stack
References
- CUAD v1, Contract Understanding Atticus Dataset
- nlpaueb/legal-bert-base-uncased (LoRA fine-tuned)
- pdfplumber + EasyOCR for scanned-contract ingestion
Applying this method in production?
Talk research → product