Re-evaluating GPT-4’s bar exam performance
Artificial Intelligence and Law 33, 581–604; published 30 March 2024, September 2025 issue
What it adds
Audits the comparison populations and grading behind reported bar-exam performance, distinguishing a model's score from claims about its standing relative to lawyers.
Read with this limitation
The analysis concerns launch-era GPT-4 and historical exam results, not current models or practical legal competence. Some percentile estimates depend on statistical assumptions. The paper does not establish that GPT-4 failed the bar exam.
- Jurisdiction tags
- United States
- Language
- English
Research artifacts
Resources linked by the authors.
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
The paper shows that the estimated percentile changes materially with the comparison population, including first-time candidates and successful examinees.
Abstract: reanalysis of comparison populationsThe author replicates the multiple-choice score while identifying methodological problems in the reported grading of the written components.
Abstract: verification of scaled UBE score
Connected research
Related question
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Both concern the validity of legal-model evaluation, using different tasks and comparison methods. This is a thematic connection, not a citation claim.
Entry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Edition inspected
- 10.1007/s10506-024-09396-9
- Reading coverage
- Final publisher HTML inspected: abstract, methodological reanalysis, reported results, notes, author affiliation and code statement. Version of record published 30 March 2024; September 2025 is the issue date. The publisher's exact OSF link is retained, but code access and execution were not verified.
- Rights
- External source and artifact links only. No full text or research data is mirrored; original-host terms apply. An artifact link does not establish access or reproduction.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.