Which model should answer
a legal question?
To build a good legal AI tool, we need to know which models work best on legal matters. A benchmark is only as valid as the models it tested, and new models arrive every 11 to 30 days, so read each result as a dated snapshot.
Summary
The wider landscape
Which AI should run each part of your legal work?
Benchmarks tell you which model is good. The AI maps say which one should run which activity, from cloud frontier to local. One map each for law firms, in-house teams and government.
See the AI maps
Is AI worth owning? The deck
The slides behind the film above, from our LTF Online talk on 13 October 2026. Strategic autonomy for law firms and legal departments: what AI costs when it is metered, rented or owned, which parts to own, and who can reach your data. The talk is the first 19 slides; the rest is backup with the sources.
Six things our run found
We ran DELTA 1.1.0 ourselves: 15 practitioner assignments across nine Dutch practice areas, 273 lawyer-written criteria, five models answering every task 2 times with no retrieval and no tools under DELTA’s own fixed system prompt, graded blind by two judges from different model families. We are not publishing a ranking. DELTA publishes one, over a far larger held-back set, and theirs is the one to read. These six are what our run can say that theirs does not.
Finding 1: Price and quality come apart, and DELTA has no cost axis at all
This is the finding that actually answers “which model should I use”, and it exists nowhere upstream: DELTA grades quality and never prices it. Across our five models the spread is 54× in cost per task against 28.4 points of substance. There is no order to read down the chart below, only a frontier to read across.
Answering cost only, at list prices on 2026-09-13; judging is excluded because it is a property of our harness, not of the model. The price axis is logarithmic — on a linear one, four of five models sit on the origin. Monocle NL is absent because a product endpoint has no per-call token accounting to place it with.
The ordering holds. The exchange rate does not.
On criteria met, price and quality track in rank order exactly — we searched every pair and found no inversion. What collapses is what the money buys. The last step up the ladder costs 20× per point of what the first step costs.
| Step up | Extra per task | Points bought | Cost per point |
|---|---|---|---|
| DeepSeek V4 Pro toGPT-6 Astra | +$0.071 | +12.8 | $0.006 |
| GPT-6 Astra toOpus 5 | +$0.36 | +13 | $0.027 |
| Opus 5 toFable 5.1 | +$0.31 | +2.6 | $0.12 |
On the strict measure, it does invert.
“Tasks fully passed” requires every criterion on a task, not most of them — the difference between mostly right and right. Measured that way, paying more buys less, twice:
- Opus 5
- $0.4426.7%
- Fable 5.1
- $0.7523.3%
Fable 5.1 costs 1.7× more than Opus 5 and scores below it.
- Grok 4.6
- $0.0146.7%
- GPT-6 Astra
- $0.0853.3%
GPT-6 Astra costs 6.1× more than Grok 4.6 and scores below it.
And one of these is just a rate card. Opus 5 emitted 527,993 output tokens against Fable 5.1’s 447,912 — more work, not less — and still cost 41% less, because its published rate is half. No quality measurement produced that gap and none would predict it.
Finding 2: The benchmark cannot see a fabricated citation
Every citation criterion in DELTA is a presence test, never an accuracy test. Does this article number appear, does this ECLI appear. An answer can cite the wrong article for a rule, or the wrong ECLI for a case, and collect the criterion anyway. 7 of our 9 independent reviewers found this on their own, without seeing each other’s work.
What that blindness is worth, in score: DeepSeek V4 Pro scores 51.4% of substance criteria while inventing Dutch law outright — not misremembering it, inventing it.
art. 29a MwCompetition lawNo art. 29a Mw exists. The lowered healthcare thresholds come from an AMvB under the art. 29 delegation.
art. 3:246 lid 5 BWInsolvencyThere is no derogation clause there. Lid 5 is the uitstel-van-betaling power, and the answer reaches the opposite of the law.
art. 3:97 lid 2 BWProperty lawQuoted statutory text that does not exist, about “een overeenkomst tot het verrichten van arbeid”.
The consequence binds every number on this page, and DELTA’s too. No ranking built on this benchmark can be read as a quality ordering at the bottom, because the rubric cannot tell recalling a case from making one up. Differences between systems are more trustworthy than the levels. It is also why the second benchmark on this page exists: it is the test that can see a fabricated citation, and it is further down.
Finding 3: Which task shape breaks which model
The spread within a model is wider than the spread between models — 76.3 points against 28.4. So the useful question is not which model, it is which kind of question. DELTA tags every citation criterion with the authority it points at, so we could split the tasks by whether that authority is an article or a decision without labelling anything by hand.
Blue is the mean score across all five models; amber is the mean gap between the best and worst model on a task of that shape. Task counts are in the labels — these are small groups, so read the direction, not the decimal. Shape is derived from DELTA’s own cited_authority fields in tasks.jsonl.
Where the answer is an article, every model scores well and the choice barely matters. Mean 81.8% across all five. This is the cheap-model case.
Where the answer is a decision, the mean falls to 53.6% and the gap between best and worst model widens to 38.2 points. This is where paying more is defensible.
The sharpest separator in the whole set is a pure case-law question. Tort law, duty to warn: which Hoge Raad decision leads on when a warning is an adequate measure. Fable 5.1 takes 100% of its criteria; DeepSeek V4 Pro takes 20.8%. That is 79.2 points on one question.
| Which case controls task | Fable 5.1 | Opus 5 | Grok 4.6 | GPT-6 Astra | DeepSeek V4 Pro |
|---|---|---|---|---|---|
Director liability, selective payments Insolvency | 54.2% | 54.2% | 58.3% | 66.7% | 45.8% |
Interpretation of notarial deeds Real estate | 57.7% | 57.7% | 34.6% | 34.6% | 26.9% |
Pledgee settlement authority Insolvency | 58.8% | 32.4% | 35.3% | 41.2% | 20.6% |
Animal liability and exoneration Tort law | 66.7% | 66.7% | 53.3% | 53.3% | 46.7% |
Inappropriate conduct at work Employment law | 77.5% | 77.5% | 57.5% | 55% | 37.5% |
Duty to warn (Jetblast) Tort law | 100% | 100% | 33.3% | 83.3% | 20.8% |
The six case-anchored tasks, hardest first. Substance criteria met, Anthropic judge, both runs averaged.
Finding 4: The public 15 tasks are easier than the held-back set
A warning about our own numbers before anyone borrows them. Fable 5.1 passed 30% of tasks for us, against DELTA’s published 15% on the full set — same protocol, same fixed prompt. The 15 public tasks are the ones DELTA chose to publish; 200+ are held back.
Read this as directional, not exact. Our judge pair is not DELTA's, so part of the gap is the grading and part is the task set. The point is the warning, and it applies to us first: any number taken off a public task set and read as a leaderboard number overstates by roughly double. Ours included.
Finding 5: Judge reliability varies by practice area
Two judges from different model families graded every answer blind. Where they disagreed we put the call to independent review: 122 criterion-level disagreements, split 75/47 in the Anthropic judge’s favour overall. But the split inverts in 2 of the nine areas.
Net calls per practice area. Bars to the right are disagreements the reviewers gave to the Anthropic judge, bars to the left to the xAI judge. The reviewers are not lawyers, and none of this was carried back into any score on this page.
Cross-family judging is necessary, not sufficient. Picking one judge because it looks more faithful overall would have been wrong in property law and family law. So every figure on this page is the both-agree number: a criterion counts as met only where both judges passed it. That is the conservative floor, not an adjudicated score.
Finding 6: Running twice catches retrieval variance, not model variance
We ran every system through the whole set 2 times, expecting to average out model noise. There was none to average: the raw models gave the same answer in both passes. What moved was the system that retrieves.
KelderluikJetblastMonocle NL returned a different leading authority on the duty-to-warn question in each pass. No raw model moved between passes at all, so a second pass buys nothing against model noise and everything against a retrieval layer landing somewhere else.
And one thing we did not learn: whether retrieval helps
Monocle NL appears in the full grid below and nowhere else on this page. Its row is a product measurement, not a controlled comparison: it is the only system that could use retrieval and the only one that could not be given DELTA’s fixed system prompt, so nothing separates the effect of retrieval from the effect of a different prompt. Its shape is suggestive — strong on statute-anchored questions, weak on case-law ones — and that is as far as it goes. It is labelled everywhere and never ranked against the models.
Two criterion defects, filed upstream
DELTA invites criterion disputes as contributions, so we checked each one against the statute and filed the two that survived:
S-005Property lawCredits art. 3:97 lid 2 BW with enabling advance delivery. That is lid 1; lid 2 ranks competing advance deliveries.
Models that state the law correctly are failed for it; an answer that got it wrong would pass. 3 of the area's 19 judge disagreements are this one criterion.
delta#1S-009Family lawConditional on a “dynamische uitleg” argument that no candidate makes — zero occurrences across all five models.
The condition never fires, so the criterion has no determinate value. One judge failed all five answers, the other passed three.
delta#2
A third one we withdrew, because it was wrong
Our reviewers reported that art. 1:250 BW is undivided, so the criterion’s demand for “art. 1:250 lid 1 BW” pointed at a lid that does not exist. It does exist. Art. 1:250 BW has leden, and lid 1 is the bijzondere-curator provision on a conflict of interest, which is exactly what the criterion asks for. There is also a lid 2 on the raad voor de kinderbescherming (art. 242a).
Nine independent reviewers agreeing is not verification. This one was caught by reading the statute before filing, which is the check that should have run first. We are leaving this on the page rather than quietly dropping it: finding 2 argues that a benchmark should be able to tell a real citation from an invented one, and a page making that argument does not get to hide the moment its own reviewers failed the same test.
Source: Burgerlijk Wetboek Boek 1, BWBR0002656, text in force at the task’s law_as_of of 2026-08-30.
Substance criteria met, Anthropic judge, both runs averaged; totals are both-agree. This grid is not a ranking — 15 public tasks cannot order five models, and finding 2 above says the bottom of any such order is not a quality order anyway. For a general ranking, DELTA publishes one over 200+ held-back tasks. Monocle’s zeroes are not wrong answers: on those tasks it asked a clarifying question instead of answering, which scores zero under a strict one-shot protocol. DELTA 1.1.0 is licensed CC BY 4.0. DELTA is curated by Legal Benchmarks with Zeno.Law as Founding Research Partner. Every figure above is generated from the run artifacts; none is typed by hand.
The gap DELTA cannot score
Finding 2 above is that DELTA’s citation criteria are presence tests: they ask whether an ECLI appears, never whether it is real. That is the single most important signal for a practising litigator, and no score on this page can see it. So we built the test that can, and pointed it at the one thing DELTA’s rubric leaves uncovered. Half the items are true bibliographic statements about real legal documents; the other half carry exactly one controlled mutation. The model judges true or false from memory alone, across 227 jurisdictions rather than one.
This measures a model’s unaided memory, not the Legalcomplex product. The number below comes from asking Grok 4.1 Fast (non-reasoning) true/false questions about citations with no retrieval, no search, and no source documents — the exact condition Monocle is built to never operate in. Monocle looks up official sources, pins verified provisions, and judges its own citations before answering. The two numbers are not comparable, and a skim that concludes “the product scores 5.8%” is reading this page wrong.
Grok 4.1 Fast (non-reasoning) correctly confirmed a real legal citation exists, from memory, 5.8% of the time — against a 50% chance baseline.
Grok 4.1 Fast (non-reasoning) · xai · run 2026-06-20 · full sweep (227/227 packs)
Source: labs/benchmarks/s3-framework/SWEEP-REPORT.md @ 53cba04
Methodology note: 16.3% of the true items (221 of 1,358) carry a year-precision source date rendered as the full January 1, so a model that doubts a suspicious Jan-1 datestamp is graded wrong even where that skepticism is defensible. Excluding those items entirely, Grok 4.1 Fast (non-reasoning) reads 6.9% on real citations. The headline keeps the full set.
Skepticism without knowledge
The model is nearly perfect at rejecting a fabricated citation (98.1%) and nearly incapable of confirming a real one (5.8%). That gap is the finding: it doubts everything, including documents that genuinely exist, rather than actually knowing which ones do.
How the test works
Every jurisdiction pack has 12 bibliographic statements about real, recently collected official legal documents. Half are verbatim true. Half carry exactly one controlled mutation.
The model gets one true/false call per item, temperature 0, no search and no source documents. Grading is exact-match on the first true/false token in the response. Chance is 50%.
2,716 items across every jurisdiction in the world-law catalog, source-grounded and generated in-house.
Accuracy by mutation class
These are FALSE items only, how often the mutation was caught. A wrong-source pairing (a real citation attached to the wrong document) is detectable on internal consistency, without knowing the source.
Accuracy by statement shape
Every jurisdiction, not a curated few
A handful of packs score well above chance. Inspection shows that’s mostly FALSE mutations caught on internal-consistency grounds, not genuine knowledge of the documents — most of the world sits inside the chance band.
Number of jurisdiction packs (12 items each, one exception at 10 or 6) landing in each accuracy band. Chance is 50%.
| Jurisdiction | Items | Accuracy |
|---|---|---|
| ArubaAW | 12 | 75% |
| BoliviaBO | 12 | 75% |
| IndiaIN | 12 | 75% |
| NorwayNO | 12 | 75% |
| TuvaluTV | 12 | 75% |
| Central African RepublicCF | 12 | 67% |
| Costa RicaCR | 12 | 67% |
| CyprusCY | 12 | 67% |
| EgyptEG | 12 | 67% |
| GabonGA | 12 | 67% |
| Hong KongHK | 12 | 67% |
| LatviaLV | 12 | 67% |
| PolandPL | 12 | 67% |
| SloveniaSI | 12 | 67% |
| SpainES | 12 | 67% |
| AlgeriaDZ | 12 | 58% |
| BarbadosBB | 12 | 58% |
| BelgiumBE | 12 | 58% |
| Burkina FasoBF | 12 | 58% |
| BurundiBI | 12 | 58% |
| ChadTD | 12 | 58% |
| Cook IslandsCK | 12 | 58% |
| CzechiaCZ | 12 | 58% |
| Democratic Republic of the CongoCD | 12 | 58% |
| Dominican RepublicDO | 12 | 58% |
| European UnionEU | 12 | 58% |
| KiribatiKI | 12 | 58% |
| KosovoXK | 12 | 58% |
| MadagascarMG | 12 | 58% |
| MoldovaMD | 12 | 58% |
| NamibiaNA | 12 | 58% |
| NicaraguaNI | 12 | 58% |
| NigerNE | 12 | 58% |
| NigeriaNG | 12 | 58% |
| ParaguayPY | 12 | 58% |
| PeruPE | 12 | 58% |
| Republic of the CongoCG | 12 | 58% |
| SingaporeSG | 12 | 58% |
| South AfricaZA | 12 | 58% |
| Sri LankaLK | 12 | 58% |
| TaiwanTW | 12 | 58% |
| TogoTG | 12 | 58% |
| United StatesUS | 12 | 58% |
| UruguayUY | 12 | 58% |
| Åland IslandsAX | 12 | 50% |
| AlbaniaAL | 12 | 50% |
| American SamoaAS | 6 | 50% |
| AndorraAD | 12 | 50% |
| AngolaAO | 12 | 50% |
| AnguillaAI | 12 | 50% |
| Antigua and BarbudaAG | 12 | 50% |
| ArmeniaAM | 12 | 50% |
| AustraliaAU | 12 | 50% |
| AustriaAT | 12 | 50% |
| AzerbaijanAZ | 12 | 50% |
| BahamasBS | 12 | 50% |
| BahrainBH | 12 | 50% |
| BangladeshBD | 12 | 50% |
| BelizeBZ | 12 | 50% |
| BermudaBM | 12 | 50% |
| BhutanBT | 12 | 50% |
| BotswanaBW | 12 | 50% |
| BrazilBR | 12 | 50% |
| British Virgin IslandsVG | 12 | 50% |
| BruneiBN | 12 | 50% |
| BulgariaBG | 12 | 50% |
| CambodiaKH | 12 | 50% |
| CameroonCM | 12 | 50% |
| CanadaCA | 12 | 50% |
| Cape VerdeCV | 12 | 50% |
| Caribbean NetherlandsBQ | 12 | 50% |
| Cayman IslandsKY | 12 | 50% |
| ChileCL | 12 | 50% |
| ChinaCN | 12 | 50% |
| ColombiaCO | 12 | 50% |
| ComorosKM | 12 | 50% |
| Côte d'IvoireCI | 12 | 50% |
| Council of EuropeCOE | 12 | 50% |
| CroatiaHR | 12 | 50% |
| CubaCU | 12 | 50% |
| CuraçaoCW | 12 | 50% |
| DenmarkDK | 12 | 50% |
| DjiboutiDJ | 12 | 50% |
| DominicaDM | 12 | 50% |
| EcuadorEC | 12 | 50% |
| El SalvadorSV | 12 | 50% |
| Equatorial GuineaGQ | 12 | 50% |
| EritreaER | 12 | 50% |
| EstoniaEE | 12 | 50% |
| EswatiniSZ | 12 | 50% |
| EthiopiaET | 12 | 50% |
| Falkland IslandsFK | 12 | 50% |
| Faroe IslandsFO | 12 | 50% |
| FijiFJ | 12 | 50% |
| FinlandFI | 12 | 50% |
| French PolynesiaPF | 12 | 50% |
| GambiaGM | 12 | 50% |
| GeorgiaGE | 12 | 50% |
| GermanyDE | 12 | 50% |
| GhanaGH | 12 | 50% |
| GibraltarGI | 12 | 50% |
| GreeceGR | 12 | 50% |
| GreenlandGL | 12 | 50% |
| GrenadaGD | 12 | 50% |
| GuamGU | 12 | 50% |
| GuatemalaGT | 12 | 50% |
| GuernseyGG | 12 | 50% |
| GuineaGN | 12 | 50% |
| Guinea-BissauGW | 12 | 50% |
| GuyanaGY | 12 | 50% |
| HaitiHT | 12 | 50% |
| HondurasHN | 12 | 50% |
| HungaryHU | 12 | 50% |
| IcelandIS | 12 | 50% |
| IndonesiaID | 12 | 50% |
| InternationalINTL | 12 | 50% |
| IranIR | 12 | 50% |
| IraqIQ | 12 | 50% |
| IrelandIE | 12 | 50% |
| Isle of ManIM | 12 | 50% |
| IsraelIL | 12 | 50% |
| ItalyIT | 12 | 50% |
| JamaicaJM | 12 | 50% |
| JerseyJE | 12 | 50% |
| JordanJO | 12 | 50% |
| KazakhstanKZ | 12 | 50% |
| KenyaKE | 12 | 50% |
| KyrgyzstanKG | 12 | 50% |
| LaosLA | 12 | 50% |
| LebanonLB | 12 | 50% |
| LesothoLS | 12 | 50% |
| LiberiaLR | 12 | 50% |
| LibyaLY | 12 | 50% |
| LithuaniaLT | 12 | 50% |
| LuxembourgLU | 12 | 50% |
| MacaoMO | 12 | 50% |
| MalawiMW | 12 | 50% |
| MaldivesMV | 12 | 50% |
| MaliML | 12 | 50% |
| MaltaMT | 12 | 50% |
| Marshall IslandsMH | 12 | 50% |
| MauritaniaMR | 12 | 50% |
| MicronesiaFM | 10 | 50% |
| MonacoMC | 12 | 50% |
| MongoliaMN | 12 | 50% |
| MontenegroME | 12 | 50% |
| MontserratMS | 12 | 50% |
| MoroccoMA | 12 | 50% |
| MozambiqueMZ | 12 | 50% |
| MyanmarMM | 12 | 50% |
| NauruNR | 12 | 50% |
| NetherlandsNL | 12 | 50% |
| New ZealandNZ | 12 | 50% |
| NiueNU | 12 | 50% |
| North MacedoniaMK | 12 | 50% |
| OmanOM | 12 | 50% |
| PakistanPK | 12 | 50% |
| PalauPW | 12 | 50% |
| PalestinePS | 12 | 50% |
| PanamaPA | 12 | 50% |
| Papua New GuineaPG | 12 | 50% |
| PhilippinesPH | 12 | 50% |
| Pitcairn IslandsPN | 12 | 50% |
| PortugalPT | 12 | 50% |
| Puerto RicoPR | 12 | 50% |
| QatarQA | 12 | 50% |
| RomaniaRO | 12 | 50% |
| RussiaRU | 12 | 50% |
| RwandaRW | 12 | 50% |
| Saint BarthélemyBL | 12 | 50% |
| Saint HelenaSH | 12 | 50% |
| Saint Kitts and NevisKN | 12 | 50% |
| Saint LuciaLC | 12 | 50% |
| Saint MartinMF | 12 | 50% |
| Saint Pierre and MiquelonPM | 12 | 50% |
| Saint Vincent and the GrenadinesVC | 12 | 50% |
| SamoaWS | 12 | 50% |
| San MarinoSM | 12 | 50% |
| Saudi ArabiaSA | 12 | 50% |
| SenegalSN | 12 | 50% |
| SerbiaRS | 12 | 50% |
| SeychellesSC | 12 | 50% |
| Sierra LeoneSL | 12 | 50% |
| Sint MaartenSX | 12 | 50% |
| SlovakiaSK | 12 | 50% |
| Solomon IslandsSB | 12 | 50% |
| SomaliaSO | 12 | 50% |
| South KoreaKR | 12 | 50% |
| South SudanSS | 12 | 50% |
| SudanSD | 12 | 50% |
| SurinameSR | 12 | 50% |
| SwedenSE | 12 | 50% |
| SwitzerlandCH | 12 | 50% |
| SyriaSY | 12 | 50% |
| TajikistanTJ | 12 | 50% |
| TanzaniaTZ | 12 | 50% |
| ThailandTH | 12 | 50% |
| Timor-LesteTL | 12 | 50% |
| TongaTO | 12 | 50% |
| Trinidad and TobagoTT | 12 | 50% |
| TunisiaTN | 12 | 50% |
| TurkeyTR | 12 | 50% |
| TurkmenistanTM | 12 | 50% |
| Turks and Caicos IslandsTC | 12 | 50% |
| U.S. Virgin IslandsVI | 12 | 50% |
| UgandaUG | 12 | 50% |
| United Arab EmiratesAE | 12 | 50% |
| United KingdomUK | 12 | 50% |
| United NationsUN | 12 | 50% |
| UzbekistanUZ | 12 | 50% |
| VanuatuVU | 12 | 50% |
| Vatican CityVA | 12 | 50% |
| VietnamVN | 12 | 50% |
| Wallis and FutunaWF | 12 | 50% |
| YemenYE | 12 | 50% |
| ZambiaZM | 12 | 50% |
| ZimbabweZW | 12 | 50% |
| BeninBJ | 12 | 42% |
| Bosnia & HerzegovinaBA | 12 | 42% |
| FranceFR | 12 | 42% |
| JapanJP | 12 | 42% |
| LiechtensteinLI | 12 | 42% |
| MauritiusMU | 12 | 42% |
| MexicoMX | 12 | 42% |
| UkraineUA | 12 | 42% |
| VenezuelaVE | 12 | 42% |
| ArgentinaAR | 12 | 33% |
Across models
| Model | Run date | Items | Aggregate | TRUE items | FALSE items | Est. cost |
|---|---|---|---|---|---|---|
| Claude Opus 4.8 | 2026-07-19 | 2,716 | 52.6% | 7.4% | 97.6% | $5.23 |
| Claude Haiku 4.5 | 2026-07-19 | 2,716 | 52.3% | 6.6% | 98% | $0.90 |
| Grok 4.1 Fast (non-reasoning) | 2026-06-20 | 2,716 | 51.9% | 5.8% | 98.1% | $0.09 |
| Claude Sonnet 5 | 2026-07-19 | 2,716 | 50.5% | 1.5% | 99.5% | $2.26 |
Est. cost is what the full run cost to answer, at list prices on the run date. Every run above covers the same 2,716 items, so the column compares directly. Read it next to accuracy, never on its own: the cheapest run here is within a point of the most expensive.
How Monocle answers instead
Retrieves from official sources — government legislation portals, court registries, and gazette systems, not model memory.
Pins verified provisions for the statute-level questions that matter most, each citation deep-linked to the official text, checked against contradiction across repeated judge runs before it ships.
Judges its own citations with a calibrated NLI model before showing them, and only marks a citation verified when the judge finds full support with no flags.
Abstains honestly when the facts don’t support a ranking — a tie stays a tie, a gap stays a gap, and a citation the judge can’t support is never presented as confirmed.
Other receipts
Support-accuracy of the judge that verifies citations inside Monocle answers (a different, retrieval-grounded task from the S3 memory test above). Single calibration run at this n; treat deltas as noise until re-run.
gemini-flash-lite · n=24 · 2026-07-02 · labs/benchmarks/citation-nli/
What this does and doesn’t show
- This is a memory-only condition: no retrieval, no web search, no source documents. It is not a measurement of any retrieval-augmented system, ours or anyone else’s.
- Packs are generated in-house from source-grounded facts. The pack set itself is not published (it would stop being a clean test the moment a model could train on it) — the sweep report and this page are the receipts instead.
- Grading is exact-match true/false on a single response. It doesn’t credit partial reasoning or hedged answers, by design — the question is binary, so the grade is too.
- Where a pack scores well above chance, that is usually a mutation caught on internal-consistency grounds, not genuine knowledge of the underlying document — see the per-jurisdiction section above.
- Every number on this page traces to a dated run artifact. None are hand-typed estimates.
Which model sends a question to the right sources
Before any answer, a small model decides whether a question needs official law, our market data or the open web. The costly mistake is a legal question answered without the law. 98 rows written fresh (30 market, 30 legal, 30 general, 6 hybrid, 2 controls), 3 runs per arm, worst run is the unit. Latency measured from one Mac, compare accuracy across arms and latency only within an arm.
Three-way routing, six arms
| Arm | Model | Right (worst run) | Law missed | Wrong legal lane | p50 ms | p95 ms | Per 1,000 |
|---|---|---|---|---|---|---|---|
| Today's word lists alone | none (deterministic) | 41/96 | 23 | 2 | 0 | 1 | none |
| Router prompt v3 | Grok fast | 83/96 | 1 | 1 | 2658 | 6452 | $0.32 |
| Three-way prompt | Grok fast | 92/96 | 1 | 1 | 830 | 3593 | $0.25 |
| Three-way prompt | Mistral Large 4, reasoning off | 96/96 | 0 | 0 | 756 | 1465 | $0.28 |
| Three-way prompt | Haiku 5.5, reasoning low | 91/96 | 3 | 0 | 1102 | 3210 | $0.07 |
| Three-way prompt | Haiku 5.5, reasoning off | 94/96 | 0 | 1 | 1081 | 1486 | $0.06 |
Run 2026-10-09, worst of 3 runs. 98 rows with the author's labels (two general rows are arguable), written the same day, one Mac, arms run in sequence. The three-way prompt returns places, not lanes, so it is a classifier beside the lane router. Not a decision on its own.
Merged production router prompt (v4), same corpus, run 2026-10-09T04:41Z
| Model | Right (worst run) | Law missed | Wrong legal lane | p50 ms | p95 ms | Per 1,000 |
|---|---|---|---|---|---|---|
| Grok fast, reasoning off | 93/96 | 1 | 2 | 1461 | 3659 | $0.32 |
| Haiku 5.5 via OpenRouter, reasoning off | 93/96 | 0 | 2 | 1224 | 1650 | $0.14 |
| Mistral Large 4 via OpenRouter, reasoning off | 95/96 | 0 | 1 | 1316 | 2371 | $0.32 |
Jev against Grok as the jurisdiction router
| Model | Hard | Wrong lane | Missed routes | p50 ms | p95 ms | Per 1,000 | Cost basis |
|---|---|---|---|---|---|---|---|
| Grok 4.20 | 0 | 0 | 2 | 762 | 2221 | $0.59 | derived from a rate table |
| Jev 1.13 | 0 | 2 | 0 | 314 | 472 | $0.038 | measured |
Run 2026-09-20. 151 rows (65 fire a lane), 5 runs per arm, worst case across runs. Both clear the gate and fail in opposite directions: Grok errs toward silence, Jev toward action (place confusion, e.g. a Dutch dismissal question routed to the EU). Re-measured 2026-10-06 and again as a fallback on 2026-10-09, where Jev failed (a brand question to a US lane, a Dutch question with no jurisdiction). Not shipped.
Draft or review: what a reader wants done with attached files
When a reader attaches files, the request decides whether Monocle should review them or draft something new. We compared Monocle's current word-list patterns with Jev 1.13, a typed decision model, in nine languages. 216 synthetic cases: 24 requests, each written in 9 languages (English, Dutch, German, Spanish, French, Italian, Portuguese, Polish, Greek), every case with 2 files attached. 9 requests want a draft, 9 want a review or an answer about the files, 6 are ambiguous. Written and labelled by us; the translations were not read by native speakers. Jev 1.13 ran 5 times per case (1,080 calls, 0 failed), Monocle's word-list patterns once (they are deterministic).
| The request | Outcome | Word-list patterns (per case) | Jev, no threshold (per run) | Jev, threshold 0.8 (per run) |
|---|---|---|---|---|
| Wants a draft (n = 81 / 405) | Drafted | 11 (14%) | 405 (100%) | 362 (89%) |
| Wants a draft (n = 81 / 405) | Asked | 63 (78%) | 0 (0%) | 43 (11%) |
| Wants a draft (n = 81 / 405) | Missed (read as a review) | 7 (9%) | 0 (0%) | 0 (0%) |
| Wants a review (n = 81 / 405) | Reviewed | 56 (69%) | 400 (99%) | 341 (84%) |
| Wants a review (n = 81 / 405) | Asked | 25 (31%) | 5 (1%) | 64 (16%) |
| Wants a review (n = 81 / 405) | Wrong draft | 0 (0%) | 0 (0%) | 0 (0%) |
| Ambiguous (n = 54 / 270) | Asked | 43 (80%) | 88 (33%) | 116 (43%) |
| Ambiguous (n = 54 / 270) | Acted on a guess | 11 (20%) | 182 (67%) | 154 (57%) |
Run 2026-10-24. n is cases for the patterns and case-runs (5 per case) for Jev. A missed draft is recoverable, since the review offers "Draft from these"; a wrong draft, a review request answered with a new document, is the harmful direction. Jev at a threshold asks whenever it is less confident than the threshold.
- Cost and speed: $0.02 for all 1,080 Jev calls, median 181 ms, 95th percentile 232 ms, slowest 1,646 ms.
- Stability: 1 of 216 cases did not give the same action in all 5 runs with no threshold, 5 at 0.8 and 8 at 0.99.
- Jev's weak spot is vague requests. On the 6 ambiguous ones it acted on a guess more often than it asked, for example by drafting on "Fix this contract".
- The word lists miss most drafts outside English and Dutch, and one request ending in a question mark is read as a question about the files in 7 languages.
What this does not show: how real readers phrase requests (24 requests, translated in parallel, labelled by us); a calibrated threshold (0.8 is a direction); requests with no file attached; whether native speakers would write the same sentences. Jev is not part of Monocle today. Patterns are counted per case, Jev per case-run, so compare shares, not raw counts. A threshold of 0.8 is a direction, not a calibrated value: it was tuned on 12 intents, and the threshold our scoring rule picked (0.99) asks on 30 to 35% of clear requests.
Every benchmark we track, and what we have actually run
The three benchmarks above are the ones we ran and have results for. The rest of the legal AI field is in our inventory too: we read each one and wrote down what it teaches. A row below says plainly whether we ran it. Plan: run every benchmark in the inventory against Monocle. Scheduling lives in the roadmap.
| Benchmark | Origin | Year | Scope | Scale | Our status | Next | What we took from it |
|---|---|---|---|---|---|---|---|
| Legora BAR | Legora | 2026 | 30 practice areas | 17 models, real-world cases | Studied, not run | Not scheduled | Agentic reasoning benchmark that tracks Legora's own harness across frontier models month by month. We have read the October update (chart below); we have not run it, and its figures are Legora's own. |
| Cost against accuracy for legal work (George Hannah, from Artificial Analysis data) | George Hannah, Artificial Analysis | 2026 | Legal work, cost per task | About 30 models, data as of 14 October 2026 | Studied, not run | Not scheduled | A published chart of all-pass rate against cost per task. We have read it (chart below) and not run it; the figures are the author's and Artificial Analysis's, read off the graphic by eye. |
| Ivo x micro1 ContractBench | Ivo, micro1 | 2026 | Contract review, US and UK law | 350 tasks, 5,862 checklist items, 7 models | Studied, not run | Not scheduled | Models mark up Word contracts with tracked changes as one side's lawyer, scored against attorney-written checklists. We have read the page (section below) and not run it; the figures are Ivo's own. |
| BigLaw Bench | Harvey AI | 2024 | US | ~100 tasks | Studied, not run | To schedule | Open-ended tasks scored against a rubric by an LLM judge and humans. Notes and a worked sample task are in the folder; no model has been run through it. |
| Harvey Legal Agent Benchmark (LAB) | Harvey AI | 2026 | US BigLaw, 24 practice areas | 1,251 tasks, 75k rubric criteria | Studied, not run | To schedule | Agent harness in a sandbox, all-pass scoring. We compared it with our own benches in a gap table; running one model through one task is the open first step. |
| LegalBench | Stanford / HazyResearch | 2023 | US | 162 tasks | Studied, not run | To schedule | Broad task coverage; public long enough that absolute scores are contamination-prone, so tier gaps matter more than headline accuracy. |
| CaseHOLD | Stanford RegLab | 2021 | US case law | 53,137 multiple-choice items | Studied, not run | To schedule | Holding-selection multiple choice over case law. |
| LeCaRD v2 | THUIR (Tsinghua) | 2024 | China (criminal) | 800 queries, 55,192-document corpus | Studied, not run | To schedule | Case retrieval benchmark; useful as a design reference for retrieval ground truth. |
| COLIEE | Annual competition | 2018 | Canada and Japan | ~400 queries a year | Studied, not run | To schedule | Deep dive corrects the claimed 0.35 ceiling for Task 1 with year-by-year data. |
| Isaacus Legal RAG Bench | Isaacus | 2026 | Australia (Victorian criminal) | 100 triplets | Studied, not run | To schedule | First benchmark with explicit error decomposition (retrieval versus generation). |
| RegLab Hallucination-Free | Stanford RegLab | 2024 | US | 202 vendor queries | Studied, not run | To schedule | The vendor confrontation: legal RAG tools still hallucinate. |
| RegLab Reasoning-Focused Retrieval | Stanford RegLab | 2025 | US | ~10k questions | Studied, not run | To schedule | Retrieval that needs reasoning rather than keyword match. |
| CrowS-Pairs | NYU | 2020 | Bias, not jurisdiction-specific | 1,508 minimal pairs | Studied, not run | To schedule | Minimal-pair design and a parity metric; points at a missing legal fairness benchmark. |
| prinzbench | Private, single expert | 2026 | Mixed | Private | Studied, not run | To schedule | Contamination-proof by construction; the trade-off is a one-person oracle. Deep dive carries a frontier leaderboard. |
| LawBench | Fei et al. / OpenCompass | 2024 | China | 20 tasks, 3 Bloom levels | Studied, not run | Run as a control, or port the tiering | The Bloom tiering is the takeaway, not the Chinese-law content. |
| Contract Drafting Study | Legal Benchmarking (Singapore) | 2025 | Commercial contracting | 30 tasks, 450 outputs, 13 tools plus a human baseline | Studied, not run | Replicate the harness | Reliability, usefulness and workflow scored separately, with a real human baseline. |
| AILQA | Nigam et al. | 2026 | India | AIBE-anchored question answering | Studied, not run | Read; data and code unconfirmed | Paper only so far. |
| Indian Law 100 | Rohas Nagpal | 2026 | India | 100 micro-cases, 13 seeded traps | Studied, not run | Run | Seeded wrong answers give a trap subscore. Self-published, so we take the mechanism, not the leaderboard. |
| ROHAS Legal AI Benchmark | Rohas Nagpal | 2026 | General legal screening | 10 questions, 5 competencies, 2 disqualifiers | Studied, not run | Replicate the harness | The point is a disqualifier tier in our grader. |
| One Word AI Benchmark | Rohas Nagpal | 2026 | Cross-domain bias | Up to 10 models, 3 questions a run | Studied, not run | Run | Cheap, browser-local forced-choice test. |
| CaseLaw-Bench | Legalcomplex | 2026 | US | 4 tracks: retrieval, open, synthesis, validity | Designed, not run | Build the corpus, then run | Designed; the corpus build needs an API token. |
| Practice-Bench | Legalcomplex | 2026 | US | 11 tasks with embedded source documents | Smoke test only | Run the core set | Core tasks written; only a two-file smoke test has run. |
| L3-Framework (S3) | Legalcomplex | 2026 | 227 jurisdictions | 2,716 items in the full sweep | Run by us | Re-run per new model | Benchmark 2 above: citation verification from memory, four models swept. |
| Citation-NLI | Legalcomplex | 2026 | Grounding, not jurisdiction-specific | 24-pair calibration set | Run by us | Calibrate any judge before use | Four calibration runs of the citation judge; its receipt appears in the citation section. |
| DELTA | Legal Benchmarks and Zeno.Law | 2026 | Netherlands | 15 tasks, 273 criteria, 9 practice areas | Run by us | Held-back set is not public | Benchmark 1 above: five models and the Monocle product, graded blind by two judges. |
Legora BAR, October 2026 update: quality against cost per case
Legora publishes its own benchmark, built on real-world cases, to track how its harness performs across frontier models over time. The October update covers 17 models and 30 practice areas. This is the chart most relevant to our routing and cost questions: output quality against average cost per case, both as a percentage difference from the 17-model average.
- Highest quality in the October update: Opus 5.5 at about +6% against the average, at roughly two and a half times the average cost per case.
- Above-average quality at below-average cost: Muse Wonder, Gemini 3.8 Flash, Kimi K3, GLM 5.3, Grok 4.7 and Muse Spark 1.3.
- Mistral Large 4 sits at about +3% quality for about 30% above the average cost. Legora notes it ran the public-preview version.
- The cheapest models are not the best value: GLM 5.3 Flash, DeepSeek V4 Pro and GPT-6 Luna are 91 to 95% cheaper than average and 6 to 12 points below it.
| Model | Maker | Quality vs average (approx.) | Cost per case vs average (approx.) |
|---|---|---|---|
| Opus 5.5 | Anthropic | +6% | +146% |
| Muse Wonder | Meta | +5% | -51% |
| Gemini 3.8 Flash | +4% | -79% | |
| Fable 5.1 | Anthropic | +4% | +158% |
| Kimi K3 | Moonshot | +3% | -50% |
| Mistral Large 4 | Mistral | +3% | +29% |
| Sonnet 5.5 | Anthropic | +3% | +47% |
| GLM 5.3 | Z.ai | +2% | -49% |
| Grok 4.7 | xAI | +2% | -22% |
| Muse Spark 1.3 | Meta | +1% | -82% |
| DeepSeek V4.1 Flash | DeepSeek | 0% | -95% |
| Gemini 4 Argon | -1% | +28% | |
| GPT-6.1 Sol | OpenAI | -2% | -23% |
| GPT-6 Astra | OpenAI | -4% | +323% |
| GLM 5.3 Flash | Z.ai | -6% | -95% |
| DeepSeek V4 Pro | DeepSeek | -6% | -91% |
| GPT-6 Luna | OpenAI | -12% | -93% |
Legora, October 2026 update. Legora's own benchmark, run by Legora on its own harness; we did not run it and cannot verify it. The values below are read off Legora's published chart by eye, so treat each as approximate (about 1 point of quality and 5 points of cost). Costs use public list prices. The line-up and baseline changed from September, so figures are not comparable month to month, and Mistral Large 4 was a preview version. Differences are relative to the 17-model average, not absolute scores. None of this is a Legalcomplex or Monocle result.
George Hannah's cost against accuracy chart for legal work, October 2026
A LinkedIn graphic by George Hannah that plots AI models for legal work by cost per task in US dollars against their all-pass rate, with data credited to Artificial Analysis. We show only the models at or above roughly a 3% pass rate; the rest of the field sits below that, mostly at under $2 per task. Pass rates are low across the board, so read the ranking, not the absolute level.
- Highest pass rate: Wonder Muse (max) at about 10%, for about $15 per task.
- Best value in the graphic's own low-cost, high-pass quadrant: Grok 4.7 (high) at about 9.4% for about $10, and Muse Spark 1.3 (max) at about 8.9% for under $5.
- The Claude models in the graphic run with a fallback and are the most expensive: Fable 5.1 at about 6.4% for about $21 and Opus 5.6 at about 4.2% for about $18.
- Mistral Large 4 (preview) is at about 3.1% for about $3.50, among the cheaper models above 3%.
| Model | All-pass rate (approx.) | Cost per task, USD (approx.) |
|---|---|---|
| Wonder Muse (max) | 10.0% | $14.8 |
| Grok 4.7 (high) | 9.4% | $10.0 |
| Muse Spark 1.3 (max) | 8.9% | $4.6 |
| Astra GPT-6 (max) | 8.6% | $14.5 |
| GPT-6.1 Sol (max) | 6.9% | $2.2 |
| Claude 5.1 Fable w/ fallback | 6.4% | $21.5 |
| Kimi K3 (max) | 5.3% | $6.4 |
| Claude Opus 5.6 w/ fallback | 4.2% | $18.3 |
| GPT-6 Sol (max) | 3.8% | $4.5 |
| Luna GPT-6 (max) | 3.3% | $0.4 |
| Mistral Large 4 Preview | 3.1% | $3.5 |
George Hannah, citing Artificial Analysis, data as of 14 October 2026. A third-party chart we did not reproduce or verify: we do not know what task set sits behind it or exactly how an all-pass is scored, and the graphic does not say. Values are read off the graphic by eye, so treat each as approximate (about $0.50 of cost and 0.3 points of pass rate). Models below roughly a 3% pass rate are left out. Dashed lines mark the graphic's own quadrant boundaries, about $13 and 7%. None of this is a Legalcomplex or Monocle result.
Ivo x micro1 ContractBench: marking up contracts as one side's lawyer, October 2026
Ivo, which sells contract-review software, published this benchmark with micro1 on 13 October 2026. A model gets a Word contract, a company playbook and the deal context, acts as the lawyer for one side, and edits the file with tracked changes, comments and escalation flags. The result is scored against pass or fail checklists that attorneys wrote after doing the same redline. Seven general-purpose models were scored. Grok 4.7 came first at 58.3%, a hair ahead of Opus 5.5 at 58.1%. The page names no individual authors.
Models act as the lawyer for one side on a Word contract with a company playbook and deal context, editing with tracked changes: a first redline of a clean agreement (Turn 1) or a reply to a counterparty's markup (Turn 2). There are 35 scenarios across five agreement types (30 under US law, 5 under UK law), 350 tasks, and 5,862 pass or fail checklist items written by attorneys, with 630 runs per model. Three AI judges from different providers grade each item and the majority decides; the page describes no human grading of the models' output.
Results as the source states them
- The top three are 1.2 points apart (Grok 4.7, Opus 5.5, Kimi K3). The page gives no margin of error for these scores, so we would read them as a tie.
- Opus 5.5 finished only 80.8% of its runs: 121 of its 630 hit the output limit before writing an edit. Its score covers the runs that finished, so it is not like for like with models that finished nearly all of theirs.
- Cost and speed separate the leaders more than quality does. Kimi K3 scored 1.2 points below Grok 4.7 at 38% less cost per finished run and 13% faster. Gemini 3.8 Flash scored 3.3 points below at 59% less cost and 52% faster.
- Models are good at leaving text alone (every model passed more than 96% of the Turn 2 items where the right move is to change nothing) and weak at active work: new edits passed between 15.1% and 36.5%.
| Model | Weighted pass rate | First redline (Turn 1) | Reply to markup (Turn 2) | Runs finished | Cost per finished run, USD | Minutes per finished run |
|---|---|---|---|---|---|---|
| Grok 4.7 | 58.3% | 62.7% | 53.9% | 99.5% | $3.63 | 24.5 |
| Opus 5.5 | 58.1% | 60.9% | 55.2% | 80.8% | $5.84 | 33.3 |
| Kimi K3 | 57.1% | 57.8% | 56.4% | 100.0% | $2.25 | 21.4 |
| Gemini 3.8 Flash | 55.0% | 58.5% | 51.6% | 99.8% | $1.50 | 11.8 |
| GLM-5.3 | 55.0% | 56.1% | 53.8% | 95.2% | $1.87 | 28.7 |
| DeepSeek V4 Pro | 48.4% | 52.5% | 44.2% | 99.5% | $0.56 | 20.0 |
| GPT-6 Astra | 45.6% | 48.7% | 42.6% | 100.0% | $5.81 | 19.7 |
For our own benchmarks: No overlap with the draft-or-review test above: that decides whether a request means review or draft, and ContractBench tests the review itself. Running it would need the dataset (not opened by us, terms unchecked), a harness that returns an edited .docx with tracked changes, and three AI judges. Not scheduled.
Ivo AI, Inc. (dataset hosted under micro1), published 13 October 2026. Ivo sells contract-review software and published this with micro1; no Ivo product was scored, no individual authors are named and there is no paper or code. The judges are AI, pass rates leave out failed runs (19.2% of Opus 5.5's), and Gemini 3.8 Flash and GLM-5.3 show 55.0% in the page's chart where its text says roughly 55%. Ivo's own benchmark, run by Ivo; we did not run or verify it. Every number here is printed in the source's text or tables. None of this is a Legalcomplex or Monocle result. Read the original · Dataset
These receipts are one line on a longer plan. The roadmap shows what shipped, what is building and what is still a question.
Model: Grok 4.1 Fast (non-reasoning) (xai) · Run date: 2026-06-20 · Items: 2,716 (full sweep)
536,107 tokens in / 82,961 tokens out · est. $0.09 · 10 min wall time