LegalForecastBench
This benchmark gives models motion to dismiss briefs from real federal cases and asks the model to predict how the judge will resolve each issue in the motion. We released the first results on October 13, 2026.
Why this benchmark matters
Testing models’ ability to predict motion outcomes is valuable because it is an objectively verifiable, high-level judgment task. In real-world litigation, predicting how a judge is likely to react to a contemplated litigation strategy or a potential motion is usually performed by the most senior and highly paid lawyers on a litigation team. Unlike many legal work-product tasks, moreover, the outcome is an externally observable judicial disposition.
I also wanted to reduce the risk that models would already know the answers from training. The judges issued these decisions between June 30 and August 7, 2026. Fifteen of the eighteen configurations report knowledge or training cutoffs before June 30. I’ve built a pipeline that ingests new cases, so that as models’ training cutoffs advance, I can create a new version that uses the most recent decisions at that point in time.
First release: results
The beta includes 91 cases and 387 scored claim–defendant units. In the released results, GPT-6 Sol had the lowest micro-Brier loss (0.118) and the highest accuracy (84.5%) of eighteen model configurations tested. Seventeen of the eighteen did better than predicting dismissal every time.
The paper and results explain the findings in more detail; the methods describe how we specified and scored the task.
Next publication: Harvey’s Legal Agent Benchmark
We are also planning to publish a separate audit of the litigation tasks in Harvey’s Legal Agent Benchmark later this month (October 2026).
Current research: LegalForecastGym
I am also building LegalForecastGym, a reinforcement learning environment for federal motion-to-dismiss forecasting. I am post-training open-weight models to assess whether training on expert-labeled litigation prediction tasks improves model performance and prediction calibration in a way that generalizes. I am particularly interested in exploring whether the expert-labeled data provides meaningful training signal that could generalize beyond prediction tasks.
Discuss the research
I’d welcome feedback on the task design, calibration results, and next evaluation cycle. Get in touch.