How I Built a Multi-Agent AI Contract Analyzer

I started building BeforeJD in early 2026 to solve a problem I kept running into personally: I was signing contracts I did not fully understand, and professional legal review was too expensive for what I needed. I am a solo developer who built this as a side project. Here is what I built and what I learned.

This post is a technical deep dive into the architecture, the decisions that shaped it, and the hardest problems I had to solve along the way. If you have ever wondered what goes on inside an AI system that reads legal documents, this is the honest account.

Why Multiple Agents Instead of One

The Single-Model Trap

My first prototype was a single large language model reading a contract and producing a risk report. It worked surprisingly well for simple documents. But the moment I fed it a 40-page commercial lease or a dense SaaS agreement with nested indemnification clauses, the quality collapsed. The model would hallucinate clause text, miss entire sections, and produce confident-sounding analysis that was factually wrong.

The core problem is that contract analysis requires multiple cognitive modes. Extracting structured data from raw text is a different task than interpreting what that data means legally. And both of those differ from adversarial stress-testing, where you try to break the analysis by looking for what was missed. Asking one model to do all of this in a single pass is like asking one person to simultaneously be the court reporter, the judge, and the opposing counsel.

The Multi-Agent Pipeline

The architecture I settled on runs separate AI models in sequence, each from a different provider. One agent acts as the Extractor. Its job is purely mechanical: read the raw document text and produce structured JSON that identifies every clause, obligation, party, date, and dollar amount. It does not interpret or evaluate. It just extracts.

Downstream, the Interpreter receives the structured extraction and performs the actual legal risk analysis. It evaluates each clause against known risk patterns, scores severity, identifies missing provisions, and generates plain-language explanations of what each clause means for the person signing. This model was chosen specifically for its strength in nuanced reasoning and calibrated confidence, which is exactly what legal interpretation demands.

Adversarial agents round out the council. The Adversary receives everything the upstream agents produced and tries to find flaws. Did the Extractor miss a clause? Did the Interpreter under-rate a risk? Is there an interaction between two clauses that no single agent caught individually? The Adversary exists to catch the mistakes of the earlier stages. If you want to understand why a council of agents catches more than one model, I wrote a separate post diving deeper into the multi-agent philosophy.

Provider Heterogeneity as a Design Principle

Why No Single Vendor Sees the Full Pipeline

One of the most deliberate decisions in the architecture is that the agents run on different AI providers. One provider handles extraction. Another handles interpretation. Yet another handles adversarial review. No single provider sees the complete pipeline or has access to the full analysis chain.

This matters for two reasons. First, it is a privacy architecture. When you upload a sensitive contract, no single AI provider can reconstruct the full analysis. The Extractor sees raw text but produces only structured data. The Interpreter sees structured data but never the original document. The Adversary sees the analysis but not the extraction logic. Each provider holds one piece of the puzzle.

Second, provider diversity creates resilience against model-specific blind spots. Every large language model has failure modes shaped by its training data. A clause pattern that one model consistently misreads may be handled correctly by a model from a different provider with different training data. By distributing the work across providers, the system inherits the strengths of each while limiting the blast radius of any single model’s weaknesses.

The Patent-Pending Architecture

This cross-provider sequential pipeline, where each agent’s output becomes the next agent’s input and no single provider sees the full chain, is patent pending. I believe it represents a genuinely novel approach to building trustworthy AI systems for sensitive document analysis. The idea is simple in hindsight, but the engineering required to make it work reliably was anything but.

The Quote Verification Challenge

Hallucinated Clause Text Is Dangerous

The most dangerous failure mode in AI contract analysis is not missing a risk. It is fabricating one. When an AI tool quotes contract language that does not actually exist in the document, it creates a false basis for decision-making. A user might refuse to sign a contract, demand a renegotiation, or hire a lawyer, all based on a clause the AI invented.

Early in development, I discovered that every model I tested would occasionally generate plausible-sounding contract language that was not in the source document. The hallucinated text was always syntactically correct, stylistically consistent with the rest of the contract, and topically relevant. It just was not real. In a medical or financial context, this is called a confabulation. In contract analysis, it is a liability.

How the System Catches Fabricated Quotes

The solution is a post-processing verification step that runs after the agents complete their work. Every quoted passage in the final report is checked against the original document text using fuzzy string matching. If a quote cannot be matched to the source document within a confidence threshold, it is either corrected to the nearest matching passage or removed entirely.

This sounds straightforward, but the implementation is surprisingly complex. Contracts contain repeated boilerplate. A liability limitation clause might appear in nearly identical language in three different sections. The fuzzy matcher needs to find the right instance, not just any instance. It also needs to handle OCR artifacts from scanned documents, inconsistent whitespace, and Unicode variants of common characters like curly quotes and em dashes.

The verification step catches approximately 3 to 5 percent of all quoted passages in a typical analysis. That number may sound small, but in a report with 30 to 50 quoted clauses, it means one or two fabricated quotes per document. Without verification, those fabricated quotes would reach the user presented with the same confidence as legitimate findings. As I explored in my post on AI contract review versus hiring a lawyer, trust in the output is the foundation that everything else depends on.

Benchmarking Against CUAD

What CUAD Measures

CUAD, the Contract Understanding Attainment Dataset, is an academic benchmark created by legal AI researchers. It contains over 500 real-world contracts annotated by law students and attorneys across 41 clause categories. It is the closest thing the field has to a standardized test for contract analysis systems.

I built an evaluation framework that runs BeforeJD’s full multi-agent pipeline against the CUAD test set and compares the results to the human annotations. The system measures precision (did the system flag things that are actually risks?), recall (did the system catch all the risks that humans identified?), and F1 score (the harmonic mean of precision and recall).

The Benchmark Result

On the risk identification task, BeforeJD identifies nine in ten material contract risks on the CUAD v1 benchmark, a locked set of more than 13,000 clauses marked up by legal experts across 41 categories. That means the system identifies the large majority of risks that trained legal annotators identify, with a practical balance of precision and recall.

I want to be precise about what this does and does not mean. It means the system is good at finding risks that exist. It does not mean the system provides legal advice. It does not mean the system replaces an attorney for high-stakes transactions. And it does not mean the system is perfect. But for the use case of giving individuals and small businesses a fast, affordable first pass at understanding what they are signing, this represents a meaningful level of reliability.

The CUAD benchmark also revealed where the system struggles. Highly context-dependent clauses, where the risk depends on facts outside the document, produce the most disagreement between the system and human annotators. This is expected. An AI reading a contract cannot know the signer’s financial situation, risk tolerance, or negotiating leverage. Those are human inputs that the system cannot access.

The Hardest Engineering Problem

Schema Compliance Across Multiple Providers

If you asked me at the start of this project what the hardest part would be, I would have guessed the AI prompting, the legal reasoning quality, or the document parsing. I would have been wrong. The hardest engineering challenge was getting multiple AI providers to consistently produce valid, schema-compliant JSON outputs that could be reliably passed from one agent to the next.

Each agent in the pipeline produces structured JSON that conforms to a strict TypeScript schema. The Extractor outputs a ContractExtraction schema. The Interpreter outputs a RiskInterpretation schema. The Adversary outputs an AdversaryReview schema. These schemas define exactly what fields must be present, what types they must be, and what values are valid. If any agent produces malformed output, the entire pipeline fails.

The Ways Models Break Schema

Each provider breaks schema compliance in different ways. One model wraps its JSON in markdown code fences. Another omits required fields when it is not confident. Yet another invents new fields that are not in the schema. One provider occasionally returns a conversational explanation instead of JSON entirely, as if it forgot it was supposed to be producing structured data.

The solution is a multi-layer validation and repair pipeline. First, the raw output is cleaned: code fences are stripped, common formatting artifacts are removed, and truncated JSON is detected. Second, the cleaned output is parsed and validated against the schema. Third, if validation fails, a repair pass attempts to fix the output by filling in defaults for missing required fields, removing unexpected fields, and coercing types where possible. Fourth, if repair fails, the agent is re-invoked with an error message describing exactly what went wrong and asking for corrected output.

This retry-and-repair loop accounts for a significant portion of the codebase. The main pipeline route is approximately 2,000 lines of code, and a substantial fraction of that is error handling, validation, and recovery logic. It is not glamorous work, but it is the difference between a demo that works 80% of the time and a product that works 99% of the time.

Lessons Learned

Building this system taught me that the hard problems in AI applications are rarely the AI itself. The models are impressive. The difficulty is in the plumbing: parsing documents reliably across 20+ file formats, keeping multiple providers in sync, verifying every quoted passage, handling edge cases in schema validation, and building confidence scoring that honestly communicates uncertainty.

It also reinforced something I already believed: a solo developer can build systems of genuine complexity if they are willing to iterate relentlessly. The first version of BeforeJD was rough. The current version is the result of dozens of sprints, each one fixing the problems the previous sprint revealed. There is no shortcut for that process. You build, you test against real contracts, you find the failure modes, and you fix them.