{"schema_version":"legalarxiv.paper-index.v1","updated":"2026-10-15","collection":"Bibliographic paper index; distinct from versioned archive publications","assessment":"Agent-assisted source summaries; independent scholarly assessment pending; no admission or reproduction implied","total_indexed":33,"total_matching":33,"offset":0,"limit":20,"next_offset":20,"items":[{"id":"ai-powered-lawyering-2026","title":"AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice","year":2026,"authors":["Daniel Schwarcz","Sam Manning","J.J. Prescott","Patrick Barry","David R. Cleveland","Beverly Rich"],"venue":"Journal of Law & Empirical Analysis 3(1), 220–250","source_url":"https://doi.org/10.1177/2755323X261427048","content_url":"https://journals.sagepub.com/doi/10.1177/2755323X261427048","identifier":"10.1177/2755323X261427048","group":"Practice","jurisdictions":["United States"],"languages":["English"],"contribution":"A randomized study tests whether upper-level law students produce better work with Vincent AI or o1-preview. It separates speed, accuracy and other quality dimensions across six realistic assignments.","limitation":"Participants were students at two selective US law schools, not practising lawyers handling client matters. Effects varied by task; the tested tools date to October 2024. Subjective grading and fixed task order also limit generalization.","source_coverage":"Publisher full HTML inspected: methods, results, limitations and references. Journal edition dated April 2026; earlier 2025 drafts are not treated as identical.","claims":[{"text":"137 students completed at least one task; 125 completed all six.","url":"https://journals.sagepub.com/doi/10.1177/2755323X261427048","section":"3. Methodology; Table 1"},{"text":"The study found no statistically significant speed or quality benefit on its NDA drafting task.","url":"https://journals.sagepub.com/doi/10.1177/2755323X261427048","section":"4. Results; 5. Limitations"}],"related":[{"id":"legal-aid-2025","relation":"cites","context":"Cites the legal-aid field study as prior evidence on AI use in practice.","evidence_url":"https://journals.sagepub.com/doi/10.1177/2755323X261427048"},{"id":"legalbench-2023","relation":"cites","context":"Distinguishes model benchmarking from measuring the performance of people using AI.","evidence_url":"https://journals.sagepub.com/doi/10.1177/2755323X261427048"}],"rights_note":"External links only; no full text mirrored on the Site. Publisher labels the article CC BY-NC 4.0.","coverage_level":"full_text","index_summary":"Tests how AI assistance changes law students’ speed and work quality across six realistic legal assignments.","url":"https://legalarxiv.org/papers/ai-powered-lawyering-2026"},{"id":"benchmarking-legal-rag-2026","title":"Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys","year":2026,"authors":["Mohamed Afane","Emaan Hariri","Derek Ouyang","Daniel E. Ho"],"affiliations":["Stanford University"],"venue":"ACM Conference on Computer Science and Law (CSLAW 2026), pages 101–125","publication_status":"peer_reviewed","source_url":"https://reglab.stanford.edu/publications/benchmarking-legal-rag-the-promise-and-limits-of-ai-statutory-surveys/","content_url":"https://arxiv.org/pdf/2603.03300v1","identifier":"10.1145/3788646.3789533","group":"Reliability","jurisdictions":["United States"],"languages":["English"],"index_summary":"A statutory-survey evaluation shows how model errors, retrieval failures and incomplete human answer keys interact.","contribution":"Evaluates statutory-survey systems against state unemployment-insurance law, tracing apparent mistakes to retrieval, legal reasoning and omissions in the human reference compilation.","limitation":"One legal domain and Boolean labels limit generalization. Comprehensive answer-key correction focused on STARA, so its corrected score cannot be fairly compared with uncorrected commercial scores. Different query limits and opaque commercial retrieval also constrain comparison.","source_coverage":"Exact arXiv 2603.03300v1 PDF inspected: methods, results, limitations and selected appendices. The arXiv record identifies acceptance at CSLAW 2026; Stanford's publication record confirms proceedings publication. Publisher-deposited metadata gives online publication on 22 May 2026. Proceedings status is distinct from the pinned reviewed copy. No systems rerun.","coverage_level":"full_text","claims":[{"text":"The evaluation uses 1,647 questions concerning unemployment-insurance provisions across US states.","url":"https://arxiv.org/pdf/2603.03300v1#page=4","section":"3.1 Experimental Setup"},{"text":"Source verification found valid statutory provisions absent from the human compilation, while the comprehensive corrections focused on STARA's apparent errors.","url":"https://arxiv.org/pdf/2603.03300v1#page=4","section":"3.3 Validation of System Outputs and DOL Report Accuracy; limits in 5.2 on PDF page 10"}],"related":[{"id":"hallucination-free-2025","relation":"shared_question","context":"Both evaluate legal research tools against source evidence. Benchmarking Legal RAG concentrates on statutory surveys and the reliability of benchmark answers. This is a thematic connection."}],"artifacts":[],"rights_note":"External source links only. No full text or research data is mirrored; original-host terms apply. Publication and source inspection do not establish reproduction.","inspected_identifier":"arXiv:2603.03300v1","url":"https://legalarxiv.org/papers/benchmarking-legal-rag-2026"},{"id":"patent-expertise-2026","title":"Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting","year":2026,"authors":["David Autor","Tanya Rodchenko","Josh Martin","Zanna Iscenko","Scott Strand","David Pearl","Melissa Ferere"],"affiliations":["Massachusetts Institute of Technology and NBER (David Autor)","Google (Tanya Rodchenko, Josh Martin, Zanna Iscenko, Scott Strand, David Pearl, Melissa Ferere)"],"venue":"NBER Working Paper 35720, September 2026","publication_status":"working_paper","source_url":"https://www.nber.org/papers/w35720","content_url":"https://shapingwork.mit.edu/wp-content/uploads/2026/09/Autor-et-al-Sept-2026.pdf","identifier":"10.3386/w35720","group":"Practice","jurisdictions":["United States"],"languages":["English"],"index_summary":"A three-month patent-lawyer experiment separates better AI-assisted work from retained professional skill.","contribution":"Tests assisted drafting and later unassisted patent review in a preregistered experiment with practicing lawyers, using blinded expert grading to distinguish work quality from retained expertise.","limitation":"Only 91 of 133 participants completed the protocol. The selected firms had Google relationships; Google funded the study. No baseline skills assessment was possible, and unassisted-task compliance was imperfectly observable. Benchmark-task results and a three-month horizon do not establish long-term client outcomes.","source_coverage":"September 2026 NBER working-paper PDF: author information, introduction, study design, results and discussion inspected, including PDF pages 2–4 and 38–40. DOI and month verified against publisher-deposited metadata. The paper expressly states that it has not been peer reviewed. No experiments or artifacts executed.","coverage_level":"full_text","claims":[{"text":"The experiment randomized 133 patent lawyers from eleven US firms; 91 completed the full protocol.","url":"https://shapingwork.mit.edu/wp-content/uploads/2026/09/Autor-et-al-Sept-2026.pdf#page=4","section":"1 Introduction"},{"text":"After three months, the reported gains on unassisted patent redlining were concentrated among senior lawyers; juniors showed no average gain.","url":"https://shapingwork.mit.edu/wp-content/uploads/2026/09/Autor-et-al-Sept-2026.pdf#page=38","section":"6 Discussion"}],"related":[{"id":"ai-powered-lawyering-2026","relation":"shared_question","context":"Both examine AI-assisted legal work. The patent-drafting experiment adds sustained exposure and an unassisted assessment of practicing lawyers. The connection is thematic, not a citation claim."}],"artifacts":[{"label":"AEA trial registration cited by the paper; registry contents not independently inspected","url":"https://www.socialscienceregistry.org/trials/15823","kind":"registration"}],"rights_note":"External source and artifact links only. No full text or research data is mirrored; original-host terms apply. An artifact link does not establish access or reproduction.","url":"https://legalarxiv.org/papers/patent-expertise-2026"},{"id":"sense-and-sensitivity-2026","title":"Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task","year":2026,"authors":["Dawson Petersen","Abhishek Purushothama","Nathan Schneider"],"affiliations":["Georgetown University"],"venue":"CoNLL 2026","publication_status":"peer_reviewed","source_url":"https://aclanthology.org/2026.conll-main.4/","content_url":"https://aclanthology.org/2026.conll-main.4.pdf","identifier":"10.18653/v1/2026.conll-main.4","inspected_identifier":"ACL Anthology 2026.conll-main.4, proceedings edition (July 2026), pp. 45–56","group":"Reliability","jurisdictions":["United States"],"languages":["English"],"index_summary":"A direct human–model comparison separates resistance to prompt changes from agreement with ordinary speakers in insurance interpretation.","contribution":"Compares US participants and five language-model configurations on matched insurance-coverage scenarios, independently varying substantive facts, statement framing and answer order. The design tests both prompt sensitivity and agreement with ordinary speakers.","limitation":"One English-language insurance-interpretation task with 288 retained US participants and five model configurations. Each participant answered six items, with a median of three human observations per conditioned prompt; model–human correlations are not legal-accuracy scores. Absence of a statistically detected framing effect is not proof of complete invariance. The released OpenAI responses date to February 2026, and results do not establish the performance of current products or general judicial decision-making.","source_coverage":"Full-text source check on 15 October 2026 of the final CoNLL proceedings PDF: title-page affiliations, sections 3–5, Table 2, limitations and appendices A–D. Proceedings date: 3–4 July 2026. First public manuscript date was not verified; code history begins in February and the repository was created in May, so July is not asserted as first release. A separate structural audit of four released OpenAI request/response file pairs found 552 unique, matching, identically ordered IDs per pair; response timestamps are 17 February 2026. This checks artifact alignment only. The statistical analysis and model calls have not been independently reproduced. The authors warn that the stripped-down release was not fully tested; the analysis-data landing page responds, but its contents were not inspected.","rights_note":"External references only. ACL states that post-2016 Anthology papers are licensed CC BY 4.0. No explicit repository licence was found in the inspected tree; do not infer reuse permission for code or data from public availability.","coverage_level":"full_text","claims":[{"text":"The study constructs 552 prompt variants from 46 insurance scenarios, each with three substantive versions, two statement framings and two answer orders. It retains 288 of 300 recruited US participants; each participant answers six items.","url":"https://aclanthology.org/2026.conll-main.4.pdf#page=2","section":"Section 3, pp. 46–47; Appendix A, p. 55"},{"text":"The human sample and GPT-5 Mini and GPT-5.2 with medium reasoning show no statistically significant frame or answer-order effects in the reported likelihood-ratio tests. All three non-reasoning configurations show a significant frame effect. This is a result of these tests, not a guarantee of invariance.","url":"https://aclanthology.org/2026.conll-main.4.pdf#page=12","section":"Sections 4.1–4.3; Appendix D, Tables 4–9, p. 56"},{"text":"GPT-5.2 without reasoning has a reported framing odds ratio of 0.545 (95% CI 0.387–0.769) for covered responses under positive versus negative framing, despite the same substantive scenario.","url":"https://aclanthology.org/2026.conll-main.4.pdf#page=5","section":"Section 4.2, p. 49"},{"text":"Point-biserial correlations between model decisions and averaged human coverage judgments are 0.47 for GPT-5.2 without reasoning, 0.52 for GPT-5.2 with medium reasoning, and 0.57 for GPT-5 Mini with medium reasoning. These correlations are not percentages of legally correct answers.","url":"https://aclanthology.org/2026.conll-main.4.pdf#page=7","section":"Section 4.5 and Table 2, pp. 50–51"},{"text":"The authors expressly limit generalisation beyond English insurance contracts and the tested models. The analysis concerns approximation of ordinary speakers’ interpretations, not a comparison with judicial rulings or a test of professional legal advice.","url":"https://aclanthology.org/2026.conll-main.4.pdf#page=8","section":"Discussion, conclusion and limitations, p. 52; motivation in section 2, p. 46"}],"related":[{"id":"benchmarking-legal-rag-2026","relation":"shared_question","context":"Editorial connection: both studies examine what a human reference can establish in a legal-AI evaluation. Here the reference is ordinary-speaker interpretation; the Stanford study checks statutory survey answers against source law. This is not a citation, shared-dataset or head-to-head comparison claim.","evidence_url":"https://aclanthology.org/2026.conll-main.4.pdf#page=7"}],"artifacts":[{"label":"Author code and archived model responses","url":"https://github.com/Dawson-Petersen/Sense-and-Sensitivity/tree/4a30dbf7437137cd8697b004b06382c3badc4508","kind":"code"},{"label":"Author-linked human data and statistical analysis (contents not inspected)","url":"https://georgetown.box.com/v/sense-and-sensitivity-analysis","kind":"data"},{"label":"July note on batch-ID alignment and released-code testing","url":"https://github.com/Dawson-Petersen/Sense-and-Sensitivity/blob/4a30dbf7437137cd8697b004b06382c3badc4508/README.md","kind":"supplement"}],"url":"https://legalarxiv.org/papers/sense-and-sensitivity-2026"},{"id":"sg-legalcite-2026","title":"SG-LegalCite: A Principle-Augmented Benchmark for Legal Citation Retrieval in Singapore Law","year":2026,"authors":["Shannon Lee Yueh Ern","Kaidong Feng","Yingpeng Du","Chloe Lee En Jia","Zhu Sun"],"affiliations":["Singapore University of Technology and Design","Nanyang Technological University","Singapore Management University, Yong Pung How School of Law"],"venue":"arXiv preprint","publication_status":"preprint","source_url":"https://arxiv.org/abs/2605.21057","content_url":"https://arxiv.org/pdf/2605.21057v1","identifier":"arXiv:2605.21057","inspected_identifier":"arXiv:2605.21057v1 (20 May 2026)","group":"Retrieval","jurisdictions":["Singapore"],"languages":["English"],"index_summary":"Singapore case-citation retrieval tests the value of stating the legal principle for which authority is sought.","contribution":"Introduces a benchmark linking LLM-generated factual summaries, extracted legal principles and cited authorities from Singapore Supreme Court judgments, and compares fact-only with principle-augmented retrieval.","limitation":"Main results use principles extracted from completed judgments, an oracle setting that differs from starting with client facts. Evaluation uses one gold citation and 999 sampled negatives, not the full case collection. LLM-generated fields receive partial expert validation; each model is evaluated in one fixed-seed run. The later author-repository cold-start update uses a different sample and must be kept separate from the inspected May preprint.","source_coverage":"Full-text review of arXiv:2605.21057v1, including the title-page author affiliations, sections 3–5, Tables 4–7 and stated limitations. First arXiv release: 20 May 2026. Authors and their lab report acceptance at AACL 2026 Main; a final proceedings PDF was not located. The separately labelled September claim below cites the authors’ 15 September 2026 repository update, which expands the cold-start sample to 1,000 test pools. Its final conference PDF was not inspected. The other claims cite the May preprint. Code and data links were checked for availability; no training, retrieval evaluation or independent reproduction was performed.","rights_note":"External references only. Authors release the dataset under CC BY 4.0 and code under MIT; source judgments retain their original terms. A source check does not establish unrestricted downstream reuse or independent reproduction.","coverage_level":"full_text","claims":[{"text":"The benchmark contains 100,890 case–principle pairs extracted from 8,523 Singapore Supreme Court judgments spanning 2000–2025.","url":"https://arxiv.org/pdf/2605.21057v1#page=5","section":"4.5 The Dataset and Expert Validation; Table 4"},{"text":"The retrieval evaluation compares 11 baselines using LLM-generated fact summaries, with or without an extracted principle, on pools containing one gold cited case and 999 randomly sampled negatives. Metrics are mean reciprocal rank and Recall@k.","url":"https://arxiv.org/pdf/2605.21057v1#page=6","section":"5.1 Experiment Setup"},{"text":"In the main table, SaulLM-7B achieves mean reciprocal rank of 13.0 with facts alone and 38.2 with facts plus an extracted legal principle, on the paper’s 0–100 reporting scale. The principle comes from the completed citing judgment.","url":"https://arxiv.org/pdf/2605.21057v1#page=7","section":"5.2 Main Results, Table 5; oracle qualification in 5.3, page 8"},{"text":"The May preprint’s separate 50-case experiment reports MRR of 17.1 for facts alone, 21.9 with a principle predicted from facts, and 49.5 with the extracted oracle principle. These figures describe that original small-sample experiment, not the later repository update.","url":"https://arxiv.org/pdf/2605.21057v1#page=8","section":"5.3 Gold Principle vs. Predicted Principle; Table 7"},{"text":"In their 15 September 2026 repository update, the authors report a cold-start evaluation on 1,000 sampled test pools. With fine-tuned SaulLM-7B, the cited case ranks first in 7.5% of fact-summary-only searches, 8.6% when DeepSeek-V3 predicts a principle from those summaries, and 20.8% when the extracted oracle principle is supplied. These are later author-reported repository results, distinct from the May preprint’s 50-case experiment; the final conference PDF was not inspected and no uncertainty intervals are reported.","url":"https://github.com/ShannonLeeYuehErn/SG-LegalCite/commit/3a9627c5d1bce07ae1c7c1c21c3950df98619e2b","section":"15 September 2026 README update; 6. Cold-Start Evaluation (Section 5.3), R@1"},{"text":"Three legal experts rated 131 case–principle pairs from 15 sampled judgments; extracted principles received a mean rating of 3.22 out of 5, with reported inter-rater agreement ICC(2,k) of 0.54.","url":"https://arxiv.org/pdf/2605.21057v1#page=5","section":"4.5 The Dataset and Expert Validation"}],"related":[{"id":"lecardv2-2024","relation":"cites_as_prior_benchmark","context":"Cites LeCaRDv2 as a fact-based similar-case retrieval benchmark when motivating a facts-plus-principle query. This is a comparison of task design, not reuse of LeCaRDv2 data or a head-to-head performance claim.","evidence_url":"https://arxiv.org/pdf/2605.21057v1#page=2"},{"id":"pile-of-law-2022","relation":"evaluates_model_from","context":"Includes Pile-of-Law BERT as a retrieval baseline. This connects model provenance to the new evaluation; it does not reproduce the Pile of Law study.","evidence_url":"https://arxiv.org/pdf/2605.21057v1#page=6"}],"artifacts":[{"label":"Author code and evaluation documentation","url":"https://github.com/ShannonLeeYuehErn/SG-LegalCite","kind":"code"},{"label":"Author-released SG-LegalCite dataset","url":"https://huggingface.co/datasets/ShannonLeeYuehErn/SG-LegalCite","kind":"data"},{"label":"15 September repository evaluation update","url":"https://github.com/ShannonLeeYuehErn/SG-LegalCite/commit/3a9627c5d1bce07ae1c7c1c21c3950df98619e2b","kind":"supplement"}],"url":"https://legalarxiv.org/papers/sg-legalcite-2026"},{"id":"acord-2025","title":"ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting","year":2025,"authors":["Steven H. Wang","Maksim Zubkov","Kexin Fan","Sarah Harrell","Yuyang Sun","Wei Chen","Andreas Plesner","Roger Wattenhofer"],"venue":"ACL 2025, 24739–24762","source_url":"https://aclanthology.org/2025.acl-long.1206/","content_url":"https://aclanthology.org/2025.acl-long.1206.pdf","identifier":"10.18653/v1/2025.acl-long.1206","group":"Contracts","jurisdictions":["United States"],"languages":["English"],"contribution":"Graded relevance judgments for drafting-oriented clause retrieval.","limitation":"Many one-star negatives inherit relevance from other clause categories; describing every pair as individually expert-rated would overstate the annotation process.","source_coverage":"Published sections 2–7, Table 3, Appendices A–C and selected implementation details; project/repository documentation.","claims":[{"text":"The benchmark has 114 queries and more than 126,000 query-clause pairs.","url":"https://aclanthology.org/2025.acl-long.1206.pdf#page=5","section":"3 and Table 2, printed 24743–24744"},{"text":"BM25 plus GPT-4o reranking reports NDCG@10 79.7 versus 54.0 for BM25 alone.","url":"https://aclanthology.org/2025.acl-long.1206.pdf#page=7","section":"5, printed 24745"}],"related":[],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","coverage_level":"full_text","url":"https://legalarxiv.org/papers/acord-2025"},{"id":"legal-aid-2025","title":"Generative AI and Legal Aid: Results from a Field Study and 100 Use Cases to Bridge the Access to Justice Gap","year":2025,"authors":["Colleen V. Chien","Miriam Kim"],"venue":"Loyola of Los Angeles Law Review 57(4), 903–988","source_url":"https://digitalcommons.lmu.edu/llr/vol57/iss4/2/","content_url":"https://digitalcommons.lmu.edu/cgi/viewcontent.cgi?article=3210&context=llr","identifier":"https://digitalcommons.lmu.edu/llr/vol57/iss4/2/","group":"Practice","jurisdictions":["United States"],"languages":["English"],"contribution":"A field pilot examines AI adoption in legal aid, including practical use cases and randomized access to extra support. It brings practitioners' experiences into a collection otherwise dominated by benchmarks and simulated tasks.","limitation":"Reported productivity gains are participants' assessments, not independently measured client outcomes. Randomization concerns additional support, not AI access versus no AI. Our current inspection covers the journal abstract, so detailed methods still require full-text assessment.","source_coverage":"Primary journal abstract and bibliographic record inspected. Full journal PDF retrieval returned 403; no claim of complete full-text review. Publisher citation uses 2025 despite printed issue-year ambiguity.","claims":[{"text":"The pilot gave 91 people up to two months of paid-tool access; a randomly selected subset received additional support.","url":"https://digitalcommons.lmu.edu/llr/vol57/iss4/2/","section":"Abstract"}],"related":[{"id":"ai-powered-lawyering-2026","relation":"shared_question","context":"Both examine how access and support affect people's use of legal AI, with different study designs.","evidence_url":"https://digitalcommons.lmu.edu/llr/vol57/iss4/2/"}],"rights_note":"External links only; no full text mirrored on the Site. No permission to republish the paper is asserted.","coverage_level":"abstract","index_summary":"A field study of AI adoption and support in legal aid, drawing on practitioners’ reported experiences.","url":"https://legalarxiv.org/papers/legal-aid-2025"},{"id":"hallucination-free-2025","title":"Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools","year":2025,"authors":["Varun Magesh","Faiz Surani","Matthew Dahl","Mirac Suzgun","Christopher D. Manning","Daniel E. Ho"],"venue":"Journal of Empirical Legal Studies 22(2), 216–242","source_url":"https://onlinelibrary.wiley.com/doi/10.1111/jels.12413","content_url":"https://nlp.stanford.edu/~manning/papers/Magesh-Hallucination%E2%80%90Free-2025.pdf","identifier":"10.1111/jels.12413","group":"Reliability","jurisdictions":["United States"],"languages":["English"],"contribution":"A preregistered, manually labelled evaluation of proprietary legal RAG products.","limitation":"Small, curated sample; changing proprietary systems prevent conclusions about current versions or other workflows.","source_coverage":"Journal sections 4–7, Figure 4, data availability and cited references; earlier arXiv and Stanford drafts compared selectively. Appendices not exhaustively audited.","claims":[{"text":"The evaluation uses 202 queries across four legal-query families.","url":"https://nlp.stanford.edu/~manning/papers/Magesh-Hallucination%E2%80%90Free-2025.pdf#page=7","section":"5.2; 7, printed 222–224 and 231"},{"text":"The historical tested products hallucinated in 17–33% of responses under the study rubric.","url":"https://nlp.stanford.edu/~manning/papers/Magesh-Hallucination%E2%80%90Free-2025.pdf#page=1","section":"Abstract; 6.1 and Figure 4, printed 216 and 224–225"}],"related":[{"id":"legalbench-2023","relation":"cites","context":"Uses 20 Rule QA questions.","evidence_url":"https://nlp.stanford.edu/~manning/papers/Magesh-Hallucination%E2%80%90Free-2025.pdf#page=8"}],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","coverage_level":"full_text","url":"https://legalarxiv.org/papers/hallucination-free-2025"},{"id":"judges-shenzhen-2025","title":"How do judges use large language models? Evidence from Shenzhen","year":2025,"authors":["John Zhuang Liu","Xueyao Li"],"affiliations":["University of Hong Kong, Faculty of Law (John Zhuang Liu)","Shanghai Jiao Tong University, Koguan School of Law (Xueyao Li)"],"venue":"Journal of Legal Analysis 16(1), 235–262; 2024 issue, published online 2 January 2025","publication_status":"peer_reviewed","source_url":"https://academic.oup.com/jla/article/16/1/235/7941565","content_url":"https://academic.oup.com/jla/article/16/1/235/7941565","identifier":"10.1093/jla/laae009","group":"Courts","jurisdictions":["China"],"languages":["Chinese"],"index_summary":"Observed judicial AI use and a judge experiment examine the risks of generating reasons after choosing a decision.","contribution":"Connects observation of a Shenzhen court's AI workflow with a vignette experiment involving experienced judges, examining how reasons generated after an initial decision may affect deliberation.","limitation":"The experiment was exploratory and unregistered, using fixed GPT-3.5 outputs in a simulated case. Its judges were not from Shenzhen. Evidence that AI amplifies bias is mixed and statistically inconclusive; this is not a causal evaluation of the deployed court system.","source_coverage":"Final publisher HTML inspected for author affiliations, publication date, descriptive court study, experimental design, results and replication link. Year follows online publication; the volume carries 2024. Chinese denotes the experimental materials; the article is English. Linked Dataverse files were not retrieved or executed.","coverage_level":"full_text","claims":[{"text":"The observed workflow begins with the judge's decision, followed by AI-generated supporting reasons and judicial revision.","url":"https://academic.oup.com/jla/article/16/1/235/7941565","section":"Abstract and introduction: observed judge–AI interaction"},{"text":"The analysis includes 139 judges and finds at most mixed evidence that AI assistance aggravates bias.","url":"https://academic.oup.com/jla/article/16/1/235/7941565","section":"5. Experiment Results"}],"related":[{"id":"fairlex-2022","relation":"shared_question","context":"Both examine bias in legal AI. The Shenzhen study examines interaction with judges, whereas FairLex evaluates disparities in model performance. No citation is implied."}],"artifacts":[{"label":"Author-linked replication materials; files not inspected","url":"https://doi.org/10.7910/DVN/73WTKD","kind":"data"}],"rights_note":"External source and artifact links only. No full text or research data is mirrored; original-host terms apply. An artifact link does not establish access or reproduction.","url":"https://legalarxiv.org/papers/judges-shenzhen-2025"},{"id":"lawflow-2025","title":"LawFlow: Collecting and Simulating Lawyers’ Thought Processes on Business Formation Case Studies","year":2025,"authors":["Debarati Das","Khanh Chi Le","Ritik Sachin Parkar","Karin De Langis","Brendan Madson","Chad M. Berryman","Robin M. Willis","Daniel H. Moses","Brett McDonnell","Daniel Schwarcz","Dongyeop Kang"],"venue":"COLM 2025; inspected arXiv v2","source_url":"https://arxiv.org/abs/2504.18942v2","content_url":"https://arxiv.org/html/2504.18942v2","identifier":"2504.18942v2","group":"Practice","jurisdictions":["United States"],"languages":["English"],"contribution":"Records how trained law students plan, revise and carry out simulated business-formation work. Comparing these traces with model-generated workflows exposes differences that final-answer scores can miss.","limitation":"Only ten finalized scenarios, with students role-playing lawyers and clients. Structured task plans and prompting can shape the observed differences. Logged actions reveal external behaviour rather than the models' internal reasoning.","source_coverage":"Exact arXiv v2 full HTML inspected and retained privately; sections 3, 4, 6, 7 and references. Conference acceptance independently listed by COLM.","claims":[{"text":"The final dataset contains ten scenarios, ten memos and eight agreements.","url":"https://arxiv.org/html/2504.18942v2#S3","section":"3. Dataset of Legal Workflows"},{"text":"The authors caution that more linear model workflows may partly reflect the prompting setup.","url":"https://arxiv.org/html/2504.18942v2#S6","section":"6. Conclusion"}],"related":[{"id":"legalbench-2023","relation":"cites","context":"Uses LegalBench when motivating the gap between isolated tasks and workflow evaluation.","evidence_url":"https://arxiv.org/html/2504.18942v2#S2"}],"rights_note":"External links only; no full text mirrored on the Site. arXiv labels this version CC BY 4.0.","coverage_level":"full_text","index_summary":"Compares human and model workflows through simulated business-formation cases, extending evaluation beyond final answers.","url":"https://legalarxiv.org/papers/lawflow-2025"},{"id":"arablegaleval-2024","title":"ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models","year":2024,"authors":["Faris Hijazi","Somayah AlHarbi","Abdulaziz AlHussein","Harethah Abu Shairah","Reem AlZahrani","Hebah AlShamlan","Omar Knio","George Turkiyyah"],"affiliations":["THIQAH","King Abdullah University of Science and Technology (KAUST)"],"venue":"Second Arabic Natural Language Processing Conference (ArabicNLP 2024), pages 225–249","source_url":"https://aclanthology.org/2024.arabicnlp-1.20/","content_url":"https://aclanthology.org/2024.arabicnlp-1.20.pdf","identifier":"10.18653/v1/2024.arabicnlp-1.20","group":"Multilingual","jurisdictions":["Saudi Arabia"],"languages":["Arabic"],"contribution":"Combines Arabic questions grounded in Saudi legal sources with translated LegalBench tasks to evaluate legal knowledge and text understanding.","limitation":"Saudi sources and translated tasks do not represent every Arabic-speaking jurisdiction. Expert-validation depth and task metadata are limited. External retrieval is outside scope; benchmark performance does not establish operational legal competence.","index_summary":"Arabic legal evaluation using Saudi sources and translated English-language benchmark tasks.","source_coverage":"Published ArabicNLP 2024 PDF: sections 1, 3–5 and section 6 limitations inspected. Arabic is the evaluation language. Translated LegalBench tasks were selected for limited localisation requirements; no additional national-law facet is inferred. Publication affiliations and official artifact links checked. No code execution.","coverage_level":"full_text","publication_status":"peer_reviewed","claims":[{"text":"The benchmark combines Arabic multiple-choice questions, question answering with relevant Saudi regulations supplied, and Arabic translations of LegalBench tasks.","url":"https://aclanthology.org/2024.arabicnlp-1.20.pdf#page=2","section":"1 Introduction; task composition"},{"text":"The study does not evaluate retrieving relevant laws or facts from external knowledge bases.","url":"https://aclanthology.org/2024.arabicnlp-1.20.pdf#page=2","section":"1 Introduction; evaluation scope"}],"related":[{"id":"legalbench-2023","relation":"shared_question","context":"Both evaluate structured legal tasks; ArabLegalEval examines Arabic performance with Saudi-source questions and translated tasks."},{"id":"lawbench-2024","relation":"shared_question","context":"Both examine legal model capabilities beyond English, with different languages, legal systems and task designs."}],"artifacts":[{"label":"Official evaluation code and data","url":"https://github.com/Thiqah/ArabLegalEval","kind":"code"}],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","url":"https://legalarxiv.org/papers/arablegaleval-2024"},{"id":"better-call-gpt-2024","title":"Better Call GPT, Comparing Large Language Models Against Lawyers","year":2024,"authors":["Lauren Martin","Nick Whitehouse","Stephanie Yiu","Lizzie Catterson","Rivindu Perera"],"venue":"arXiv preprint; Onit AI Center of Excellence","source_url":"https://arxiv.org/abs/2401.16212v1","content_url":"https://arxiv.org/pdf/2401.16212v1","identifier":"2401.16212v1","group":"Practice","jurisdictions":["United States","New Zealand"],"languages":["English"],"contribution":"Compares models, junior lawyers and legal-process outsourcers on identifying and locating contractual issues. Separate measures reveal that recognising an issue and pinpointing its supporting clause can produce different rankings.","limitation":"A company-authored preprint using only ten anonymised procurement contracts and senior-lawyer consensus labels. Historical model results and estimated processing costs do not establish equivalent performance across negotiations, supervision or complete client engagements.","source_coverage":"Exact arXiv v1 PDF and HTML inspected; methods, Tables 2–5, discussion and prompts. PDF retained privately with SHA-256.","claims":[{"text":"Ten procurement contracts cover US and New Zealand settings.","url":"https://arxiv.org/pdf/2401.16212v1#page=4","section":"3.3.1. Dataset Preparation"},{"text":"On issue location, outsourcers achieved an F-score of 0.770, above the best model’s 0.740.","url":"https://arxiv.org/pdf/2401.16212v1#page=8","section":"Table 3"}],"related":[{"id":"cuad-2021","relation":"shared_question","context":"Both investigate contract analysis; their annotation targets and evaluation settings differ.","evidence_url":"https://arxiv.org/pdf/2401.16212v1#page=4"}],"rights_note":"External links only; no full text mirrored on the Site. arXiv labels this version CC BY 4.0.","coverage_level":"full_text","index_summary":"Compares models, junior lawyers and legal-process outsourcers on finding and locating contractual issues.","url":"https://legalarxiv.org/papers/better-call-gpt-2024"},{"id":"building-better-lawyer-2024","title":"Building a better lawyer: Experimental evidence that artificial intelligence can increase legal work efficiency","year":2024,"authors":["Aileen Nielsen","Stavroula Skylaki","Milda Norkute","Alexander Stremitzer"],"affiliations":["Harvard Law School (Aileen Nielsen)","Thomson Reuters Labs (Stavroula Skylaki, Milda Norkute)","ETH Zurich, Center for Law and Economics (Alexander Stremitzer)"],"venue":"Journal of Empirical Legal Studies 21(4), 979–1022; first published 17 November 2024","publication_status":"peer_reviewed","source_url":"https://onlinelibrary.wiley.com/doi/10.1111/jels.12396","content_url":"https://onlinelibrary.wiley.com/doi/10.1111/jels.12396","identifier":"10.1111/jels.12396","group":"Practice","jurisdictions":["United States"],"languages":["English"],"index_summary":"A law-student experiment compares AI highlighting and summaries, showing why interface choices matter for legal work.","contribution":"Separates the effects of AI-generated summaries and text highlighting on legal complaint-reading tasks, connecting interface design to completion time and measured quality.","limitation":"Student complaint-reading tasks do not establish experienced lawyers' performance on client matters or different tasks. The journal's full final text was not accessible in this review. Replication data are available on request rather than through a verified public package.","source_coverage":"Journal metadata, abstract and data-availability statement inspected; displayed claims refer to this final journal abstract. The separate October 2024 Harvard Discussion Paper 1111 was inspected for methodological context, including PDF pages 11–19 and 34–35, and is explicitly linked as an earlier manuscript. No final-journal full-text review or experiment rerun is asserted.","coverage_level":"abstract","claims":[{"text":"The study compares AI-generated summaries and highlighting in an experiment with 206 law students.","url":"https://onlinelibrary.wiley.com/doi/10.1111/jels.12396","section":"Abstract"},{"text":"The abstract reports faster completion with highlighting and no reduction in measured quality indicators; summaries alone produced no change in performance metrics.","url":"https://onlinelibrary.wiley.com/doi/10.1111/jels.12396","section":"Abstract"}],"related":[{"id":"ai-powered-lawyering-2026","relation":"shared_question","context":"Both study AI-assisted student legal work. Building a better lawyer isolates the presentation of assistance. The relationship is thematic, not a citation claim."}],"artifacts":[{"label":"Earlier full manuscript: Harvard Discussion Paper 1111, October 2024; not the final journal edition","url":"https://laweconcenter.law.harvard.edu/wp-content/uploads/2025/07/1111_Nielsen.pdf","kind":"supplement"}],"rights_note":"External source and manuscript links only. No full text or research data is mirrored; original-host terms apply. Linking an earlier manuscript does not identify it as the final journal edition.","url":"https://legalarxiv.org/papers/building-better-lawyer-2024"},{"id":"il-tur-2024","title":"IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning","year":2024,"authors":["Abhinav Joshi","Shounak Paul","Akshat Sharma","Pawan Goyal","Saptarshi Ghosh","Ashutosh Modi"],"affiliations":["IIT Kanpur","IIT Kharagpur"],"venue":"ACL 2024, Volume 1: Long Papers","source_url":"https://aclanthology.org/2024.acl-long.618/","content_url":"https://aclanthology.org/2024.acl-long.618v2.pdf","identifier":"10.18653/v1/2024.acl-long.618","group":"Multilingual","jurisdictions":["India"],"languages":["English","Bengali","Hindi","Gujarati","Malayalam","Marathi","Telugu","Tamil","Punjabi","Oriya"],"contribution":"Brings eight Indian legal tasks into one benchmark, spanning document understanding, precedent retrieval, summarisation, prediction and translation.","limitation":"Multilingual coverage applies to translation; bail prediction is Hindi and other tasks are English. Legal subdomains remain incomplete. The authors endorse research use only; predicting historical outcomes does not establish legally valid reasoning.","index_summary":"Indian legal understanding across eight tasks, with translation between English and nine Indian languages.","source_coverage":"ACL publisher v2 PDF: main-text task design and limitations, Table 2, and Appendix A.8 language inventory inspected. Publication affiliations recorded; author order preserved. The full appendix was not exhaustively audited. Linked task artifacts were inspected, not executed.","coverage_level":"full_text","publication_status":"peer_reviewed","claims":[{"text":"The benchmark combines eight tasks, including legal entity recognition, rhetorical roles, precedent retrieval, summarisation and legal translation.","url":"https://aclanthology.org/2024.acl-long.618v2.pdf#page=4","section":"3 IL-TUR: Legal-NLP Benchmark; Table 2"},{"text":"The translation task covers English and nine Indian languages; the limitations identify bail prediction as Hindi and the remaining tasks as English.","url":"https://aclanthology.org/2024.acl-long.618v2.pdf#page=10","section":"Limitations; language inventory in Appendix A.8"}],"related":[{"id":"ildc-2021","relation":"shared_question","context":"Both examine Indian court outcome prediction and explanations; IL-TUR also evaluates other legal document tasks."},{"id":"lextreme-2023","relation":"shared_question","context":"Both organise legal NLP evaluation across languages, with different jurisdictions and task coverage."}],"artifacts":[{"label":"Official task code and evaluation links","url":"https://github.com/Exploration-Lab/IL-TUR","kind":"code"},{"label":"IL-TUR datasets","url":"https://huggingface.co/datasets/Exploration-Lab/IL-TUR","kind":"data"}],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","url":"https://legalarxiv.org/papers/il-tur-2024"},{"id":"large-legal-fictions-2024","title":"Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models","year":2024,"authors":["Matthew Dahl","Varun Magesh","Mirac Suzgun","Daniel E. Ho"],"venue":"Journal of Legal Analysis 16(1), 64–93","source_url":"https://doi.org/10.1093/jla/laae003","content_url":"https://arxiv.org/pdf/2401.01301v2","identifier":"2401.01301v2","group":"Reliability","jurisdictions":["United States"],"languages":["English"],"contribution":"Separates externally checkable answers from contradiction-based hallucination estimates.","limitation":"Contradiction-based estimates are lower bounds; federal-case sampling and historical models do not represent all legal practice.","source_coverage":"Exact arXiv v2 main methods/results, Figures 1–2, sections 2–6 and artifact description; 80-page document including appendices not exhaustively audited.","claims":[{"text":"For sampled verifiable case questions, reported hallucination rates range from 58% for GPT-4 to 88% for Llama 2.","url":"https://arxiv.org/pdf/2401.01301v2#page=6","section":"1 Introduction"},{"text":"Sampling draws 5,000 cases at each of three federal-court levels.","url":"https://arxiv.org/pdf/2401.01301v2#page=17","section":"4.1 Data"}],"related":[],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","coverage_level":"full_text","url":"https://legalarxiv.org/papers/large-legal-fictions-2024"},{"id":"lawbench-2024","title":"LawBench: Benchmarking Legal Knowledge of Large Language Models","year":2024,"authors":["Zhiwei Fei","Xiaoyu Shen","Dawei Zhu","Fengzhe Zhou","Zhuo Han","Alan Huang","Songyang Zhang","Kai Chen","Zhixin Yin","Zongwen Shen","Jidong Ge","Vincent Ng"],"venue":"EMNLP 2024, 7933–7962","source_url":"https://aclanthology.org/2024.emnlp-main.452/","content_url":"https://aclanthology.org/2024.emnlp-main.452.pdf","identifier":"10.18653/v1/2024.emnlp-main.452","group":"Reasoning","jurisdictions":["China"],"languages":["Chinese"],"contribution":"Groups legal tasks into memorisation, understanding and application.","limitation":"Possible training contamination, brittle answer extraction and ROUGE-based generative scoring weaken practical-competence conclusions.","source_coverage":"Published sections 2–4, Tables 1–6, limitations, Appendix E and relevant references; author repository README/file tree. Appendix task prompts and all model tables not exhaustively audited.","claims":[{"text":"LawBench comprises 20 tasks with 500 test instances per task.","url":"https://aclanthology.org/2024.emnlp-main.452.pdf#page=3","section":"3, printed 7935–7936"},{"text":"The published evaluation tests 21 LLMs in zero-shot and one-shot settings.","url":"https://aclanthology.org/2024.emnlp-main.452.pdf#page=4","section":"4.1, printed 7936"}],"related":[{"id":"lexglue-2022","relation":"cites","context":"Positions Chinese tasks against English legal NLP benchmarks.","evidence_url":"https://aclanthology.org/2024.emnlp-main.452.pdf#page=2"}],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","coverage_level":"full_text","url":"https://legalarxiv.org/papers/lawbench-2024"},{"id":"lecardv2-2024","title":"LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset","year":2024,"authors":["Haitao Li","Yunqiu Shao","Yueyue Wu","Qingyao Ai","Yixiao Ma","Yiqun Liu"],"affiliations":["Tsinghua University","Quan Cheng Laboratory"],"venue":"SIGIR 2024, pages 2251–2260","source_url":"https://doi.org/10.1145/3626772.3657887","content_url":"https://arxiv.org/pdf/2310.17609v1","identifier":"10.1145/3626772.3657887","group":"Retrieval","jurisdictions":["China"],"languages":["Chinese"],"contribution":"Defines case relevance through legal characterisation, penalty and procedure, with expert annotations and two-stage candidate pooling.","limitation":"Coverage is Chinese criminal cases with pooled relevance judgments. Retrieval scores do not establish exhaustive coverage of all relevant cases or transfer to other jurisdictions. Full-text review used the 2023 preprint, not a complete comparison with the published edition.","index_summary":"Chinese criminal case retrieval with expert judgments of substantive and procedural relevance.","source_coverage":"Full-text methods inspected in arXiv:2310.17609v1 (26 October 2023), especially sections 1 and 3–4. SIGIR 2024 title, author order, affiliations and DOI checked against publisher metadata. Published PDF retrieval was blocked; claims cite the reviewed preprint. No reproduction performed.","coverage_level":"full_text","publication_status":"peer_reviewed","claims":[{"text":"The dataset contains 800 query cases and 55,192 candidate cases selected from 4.3 million criminal case documents.","url":"https://arxiv.org/pdf/2310.17609v1#page=1","section":"Abstract; 1 Introduction"},{"text":"The relevance criteria distinguish characterisation, penalty and procedure; the dataset uses two-stage candidate pooling and legal-expert annotation.","url":"https://arxiv.org/pdf/2310.17609v1#page=1","section":"Abstract; detailed construction in section 3"}],"related":[{"id":"legalbench-rag-2024","relation":"shared_question","context":"Both evaluate retrieval for legal applications, using different retrieval units and relevance criteria."},{"id":"cail2018-2018","relation":"shared_question","context":"Both use Chinese criminal judgments; LeCaRDv2 studies case retrieval while CAIL2018 studies judgment prediction."}],"artifacts":[{"label":"Official data and baseline references","url":"https://github.com/THUIR/LeCaRDv2","kind":"data"}],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","inspected_identifier":"arXiv:2310.17609v1 (26 October 2023)","url":"https://legalarxiv.org/papers/lecardv2-2024"},{"id":"legalbench-rag-2024","title":"LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain","year":2024,"authors":["Nicholas Pipitone","Ghita Houir Alami"],"venue":"arXiv preprint; peer-reviewed venue not established","source_url":"https://arxiv.org/abs/2408.10343v1","content_url":"https://arxiv.org/pdf/2408.10343v1","identifier":"2408.10343v1","group":"Retrieval","jurisdictions":["United States"],"languages":["English"],"contribution":"Transforms existing legal annotations into retrieval queries with character-span targets.","limitation":"Table 1 totals 6,858; Table 2 totals 6,889. ContractNLI counts differ (946/977).","source_coverage":"Main sections 2–5 and Tables 1–7; tables 4–5 inspected as rendered pixels; author repository README and file tree.","claims":[{"text":"The corpus draws from ContractNLI, CUAD, MAUD and PrivacyQA.","url":"https://arxiv.org/pdf/2408.10343v1#page=3","section":"3.1 and Table 1"},{"text":"Experiments use the 776-query miniature benchmark, not the full corpus.","url":"https://arxiv.org/pdf/2408.10343v1#page=5","section":"3.4; 4; Table 3"}],"related":[{"id":"cuad-2021","relation":"cites","context":"Converts clause annotations to retrieval targets.","evidence_url":"https://arxiv.org/pdf/2408.10343v1#page=3"}],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","coverage_level":"full_text","url":"https://legalarxiv.org/papers/legalbench-rag-2024"},{"id":"multilegalpile-2024","title":"MultiLegalPile: A 689GB Multilingual Legal Corpus","year":2024,"authors":["Joel Niklaus","Veton Matoshi","Matthias Stürmer","Ilias Chalkidis","Daniel E. Ho"],"venue":"ACL 2024, Volume 1: Long Papers","source_url":"https://aclanthology.org/2024.acl-long.805/","content_url":"https://aclanthology.org/2024.acl-long.805.pdf","identifier":"10.18653/v1/2024.acl-long.805","group":"Multilingual","jurisdictions":["United States","Canada","European Union","Switzerland","United Kingdom","Germany","France","Brazil"],"languages":["Bulgarian","Czech","Danish","German","Greek","English","Spanish","Estonian","Finnish","French","Irish","Croatian","Hungarian","Italian","Lithuanian","Latvian","Maltese","Dutch","Polish","Portuguese","Romanian","Slovak","Slovenian","Swedish"],"contribution":"Combines four corpora and evaluates adapted multilingual and monolingual encoders.","limitation":"No deduplication; legal-mC4 may overlap other subsets. No large generative model was pretrained.","source_coverage":"Canonical PDF sections 1-5, limitations, Appendix A disclosure, Appendix D/Table 8, Appendix E/Table 9; relevant references. Corresponding arXiv v3 HTML inspected selectively.","claims":[{"text":"The corpus totals 689GB across 24 languages and 17 jurisdictions.","url":"https://aclanthology.org/2024.acl-long.805.pdf","section":"1 Contributions; 3.2 Description"},{"text":"Final LEXTREME aggregate: Legal-XLM-R-large 59.5 versus XLM-R-large 59.3; improvement is small.","url":"https://aclanthology.org/2024.acl-long.805.pdf","section":"5.3 Evaluation on LEXTREME"}],"related":[],"rights_note":"External references only. Papers and research artifacts remain with their original hosts; their own terms apply.","coverage_level":"full_text","url":"https://legalarxiv.org/papers/multilegalpile-2024"},{"id":"reevaluating-bar-exam-2024","title":"Re-evaluating GPT-4’s bar exam performance","year":2024,"authors":["Eric Martínez"],"affiliations":["Massachusetts Institute of Technology, Department of Brain and Cognitive Sciences"],"venue":"Artificial Intelligence and Law 33, 581–604; published 30 March 2024, September 2025 issue","publication_status":"peer_reviewed","source_url":"https://link.springer.com/article/10.1007/s10506-024-09396-9","content_url":"https://link.springer.com/article/10.1007/s10506-024-09396-9","identifier":"10.1007/s10506-024-09396-9","group":"Reliability","jurisdictions":["United States"],"languages":["English"],"index_summary":"Re-examines GPT-4's bar-exam headline through comparison populations, score construction and evaluation transparency.","contribution":"Audits the comparison populations and grading behind reported bar-exam performance, distinguishing a model's score from claims about its standing relative to lawyers.","limitation":"The analysis concerns launch-era GPT-4 and historical exam results, not current models or practical legal competence. Some percentile estimates depend on statistical assumptions. The paper does not establish that GPT-4 failed the bar exam.","source_coverage":"Final publisher HTML inspected: abstract, methodological reanalysis, reported results, notes, author affiliation and code statement. Version of record published 30 March 2024; September 2025 is the issue date. The publisher's exact OSF link is retained, but code access and execution were not verified.","coverage_level":"full_text","claims":[{"text":"The paper shows that the estimated percentile changes materially with the comparison population, including first-time candidates and successful examinees.","url":"https://link.springer.com/article/10.1007/s10506-024-09396-9","section":"Abstract: reanalysis of comparison populations"},{"text":"The author replicates the multiple-choice score while identifying methodological problems in the reported grading of the written components.","url":"https://link.springer.com/article/10.1007/s10506-024-09396-9","section":"Abstract: verification of scaled UBE score"}],"related":[{"id":"legalbench-2023","relation":"shared_question","context":"Both concern the validity of legal-model evaluation, using different tasks and comparison methods. This is a thematic connection, not a citation claim."}],"artifacts":[{"label":"Author-linked analysis code; access and execution unverified","url":"https://osf.io/c8ygu/?view_only=dcc617accc464491922b77414867a066","kind":"code"}],"rights_note":"External source and artifact links only. No full text or research data is mirrored; original-host terms apply. An artifact link does not establish access or reproduction.","url":"https://legalarxiv.org/papers/reevaluating-bar-exam-2024"}]}