Legalarxiv.

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré et al.

All authors

Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, Zehua Li

Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track

Read the paperPublication source

What it adds

Connects task evaluation to six legally recognisable reasoning categories.

Read with this limitation

English-only, US-skewed, short-document tasks omit ambiguous and integrated legal reasoning; scores cannot establish deployment suitability.

Jurisdiction tags
United States · Canada · European Union
Language
English

Follow the source

Selected statements located in the linked primary sources. These results have not been independently reproduced here.

  1. The benchmark contains 162 tasks from 36 data sources.

    1 Introduction

  2. GPT-4 scored 82.2 for rule-application correctness and 79.7 for analysis under the paper’s manual evaluation.

    5.1 Setup; 5.2 Performance trends
Entry status, source edition & rights
Index entry
Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
Archive admission
This bibliographic entry does not confer admission to the versioned archive.
Edition inspected
10.52202/075280-1915
Reading coverage
Targeted main-text methods, results and limitations (sections 1-5); appendices B, C, E.1-E.2 and G.4.3; relevant reference entries. The 157-page appendix was not exhaustively audited.
Rights
External references only. Papers and research artifacts remain with their original hosts; their own terms apply.
Entry updated
2026-10-15

Source-linked summary · independent assessment pending.

Research updates

A monthly digest of selected legal AI papers, datasets and benchmarks, plus published research checks.