Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys
ACM Conference on Computer Science and Law (CSLAW 2026), pages 101–125
What it adds
Evaluates statutory-survey systems against state unemployment-insurance law, tracing apparent mistakes to retrieval, legal reasoning and omissions in the human reference compilation.
Read with this limitation
One legal domain and Boolean labels limit generalization. Comprehensive answer-key correction focused on STARA, so its corrected score cannot be fairly compared with uncorrected commercial scores. Different query limits and opaque commercial retrieval also constrain comparison.
- Jurisdiction tags
- United States
- Language
- English
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
The evaluation uses 1,647 questions concerning unemployment-insurance provisions across US states.
3.1 Experimental SetupSource verification found valid statutory provisions absent from the human compilation, while the comprehensive corrections focused on STARA's apparent errors.
3.3 Validation of System Outputs and DOL Report Accuracy; limits in 5.2 on PDF page 10
Connected research
Related question
Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
Both evaluate legal research tools against source evidence. Benchmarking Legal RAG concentrates on statutory surveys and the reliability of benchmark answers. This is a thematic connection.
Related question
Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task
Editorial connection: both studies examine what a human reference can establish in a legal-AI evaluation. Here the reference is ordinary-speaker interpretation; the Stanford study checks statutory survey answers against source law. This is not a citation, shared-dataset or head-to-head comparison claim.
Entry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Edition inspected
- arXiv:2603.03300v1
- Reading coverage
- Exact arXiv 2603.03300v1 PDF inspected: methods, results, limitations and selected appendices. The arXiv record identifies acceptance at CSLAW 2026; Stanford's publication record confirms proceedings publication. Publisher-deposited metadata gives online publication on 22 May 2026. Proceedings status is distinct from the pinned reviewed copy. No systems rerun.
- Rights
- External source links only. No full text or research data is mirrored; original-host terms apply. Publication and source inspection do not establish reproduction.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.