The reading frame
Ask what the result establishes.
For each paper, identify the question, the material tested and the standard used to judge success. Then ask which parts of a real legal workflow remain outside that test.
| Question | Evidence to inspect |
|---|---|
| Did retrieval find the right material? | Relevance labels, source spans, ranking and retrieval metrics |
| Does the material support the answer? | Exact propositions, cited passages and the grounding rubric |
| Was the legal task solved? | Task definition, jurisdiction, labels and scoring method |
| Can the finding be obtained again? | Pinned artifacts, model version, commands and observed results |
Our first executed dataset check illustrates why version identity matters: an available LegalBench artifact and a historical table agree on the total but differ on two class counts.
01 / Reading path
Does the answer hold up?
These studies examine factual correctness and source support. Read their query design and study dates alongside the findings; the results concern historical versions and test periods.
2024
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Separates externally checkable answers from contradiction-based hallucination estimates.
Read with this limit. Contradiction-based estimates are lower bounds; federal-case sampling and historical models do not represent all legal practice.
2025
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
A preregistered, manually labelled evaluation of proprietary legal RAG products.
Read with this limit. Small, curated sample; changing proprietary systems prevent conclusions about current versions or other workflows.
02 / Reading path
Can it find the evidence?
Retrieval, extraction and inference require different evaluations. A relevant clause, a correctly extracted span and a supported conclusion are separate achievements.
2021
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Expert-supervised clause annotations across multiple commercial contract types.
Read with this limit. EDGAR contracts skew toward public-company transactions; clause extraction does not assess legal advice or risk judgments.
2021
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts
Pairs document-level entailment judgments with supporting evidence spans.
Read with this limit. Fixed hypotheses across train/test and NDA-only coverage limit claims about unseen contract questions; label imbalance hides weak contradiction detection.
2024
LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
Transforms existing legal annotations into retrieval queries with character-span targets.
Read with this limit. Table 1 totals 6,858; Table 2 totals 6,889. ContractNLI counts differ (946/977).
2025
ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting
Graded relevance judgments for drafting-oriented clause retrieval.
Read with this limit. Many one-star negatives inherit relevance from other clause categories; describing every pair as individually expert-rated would overstate the annotation process.
03 / Reading path
What does the benchmark measure?
Compare tasks, splits, prompts and scoring rules before comparing model scores. Benchmark coverage sets the boundary of the claim.
2022
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Unifies seven English datasets with accessible evaluation code and comparable metrics.
Read with this limit. English-only and simplified tasks; independent expert re-annotation and human-performance estimates are missing.
2023
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Connects task evaluation to six legally recognisable reasoning categories.
Read with this limit. English-only, US-skewed, short-document tasks omit ambiguous and integrated legal reasoning; scores cannot establish deployment suitability.
2024
LawBench: Benchmarking Legal Knowledge of Large Language Models
Groups legal tasks into memorisation, understanding and application.
Read with this limit. Possible training contamination, brittle answer extraction and ROUGE-based generative scoring weaken practical-competence conclusions.
04 / Reading path
Whose law and language?
Language, jurisdiction and data provenance shape which claims can transfer. A multilingual corpus widens the material available for research; practical competence still needs evaluation.
2024
MultiLegalPile: A 689GB Multilingual Legal Corpus
Combines four corpora and evaluates adapted multilingual and monolingual encoders.
Read with this limit. No deduplication; legal-mC4 may overlap other subsets. No large generative model was pretrained.
05 / Reference trail
Research builds on research.
Follow a checked connection, then inspect the passage that establishes it.
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Cited by
Compares extraction with inference; tests CUAD-fine-tuned models.
Inspect the citation · 1; 4.3, printed 1907 and 1912Four source-checked citation links. A citation does not establish support, agreement or reproduction. This map is deliberately partial.
06 / Scope
A starting collection, with visible gaps.
This is an editorial reading guide prepared with agent assistance. The original papers’ methods, findings, limitations and selected references were inspected to the extent recorded in the private intake dossiers. Our dossiers and guide have not received qualified human review; no LegalArxiv admission decision has been made.
The selection emphasises empirical and technical research. It does not yet cover doctrinal scholarship, many jurisdictions, or a source-assessed field experiment on human legal workflows. It is not an exhaustive review or a current model ranking.
References link to their original hosts. The guide adds orientation and a small citation map; it does not republish the papers or resolve their remaining licensing and artifact questions.