Can GPT-3 Perform Statutory Reasoning?
ICAIL 2023; inspected arXiv v2
What it adds
Separates knowledge of statutory text from applying rules to facts. Alongside SARA tax-law tasks, newly generated synthetic statutes test reasoning without relying on memorised statutory language.
Read with this limitation
Tests the historical text-davinci-003 model on constrained tax-law and synthetic tasks. Simplified, unambiguous rules omit much interpretive work; the findings should not be presented as current model performance or demonstrated competence in tax practice.
- Jurisdiction tags
- United States
- Language
- English
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
Table 4 reports approximately 78% accuracy on its simplest synthetic statutes, with lower scores as structures widen or deepen.
5.2. Zero-Shot Experimental Setup; Table 4
Connected research
Related question
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Both examine statutory reasoning using SARA-derived tasks; the statutory-reasoning study also probes synthetic statutes.
Entry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Edition inspected
- 2302.06100v2
- Reading coverage
- Exact arXiv v2 full PDF inspected, especially sections 3–5 and Tables 1, 3, 4. Conference DOI confirmed; PDF retained privately with SHA-256.
- Rights
- External links only; no full text mirrored on the Site. No permission to republish the paper is asserted.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.