Can AI solve high-stakes legal questions? Let's test: is buying Autonomy for $11.1 billion by HP a good deal?

Can AI solve high-stakes legal questions? Let's test: is buying Autonomy for $11.1 billion by HP a good deal?

40 contracts said yes. 28,000 documents said no. HP 'lost' $8.8 Billion and no AI can solve this, unless...

From the top:

40 contracts at ~25 pages = 643,068 tokens, which fits nicely in 1 session with a 1 million context window

But KPMG's reviewer wrote in 2011 that he was not sure the 40 "captured everything"

During trial 28,000 documents were pulled from a database of many millions

28,000 documents = 360,117,994 tokens = 360 sessions at 1 million

What's the problem? 🤷🏽‍♂️

You need to reason and remember across all 360 sessions to answer legal questions

Questions like "how much did Autonomy sell, and does it match the revenue reported?"

That answer lives in no single document. It only exists once you have added up all 28,000

Caching is bounded by 1M context window per session RAG returns slices so the picture is incomplete TAR returns text matches, it can't calculate numbers Map-reduce offers you one shot at the right question LLM fine-tuning gives you parametric memory, which is lossy

Models cannot answer high-value legal questions But models can build a machine that can 😇

Enjoy The Weeknd 🎧