Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task
CoNLL 2026
What it adds
Compares US participants and five language-model configurations on matched insurance-coverage scenarios, independently varying substantive facts, statement framing and answer order. The design tests both prompt sensitivity and agreement with ordinary speakers.
Read with this limitation
One English-language insurance-interpretation task with 288 retained US participants and five model configurations. Each participant answered six items, with a median of three human observations per conditioned prompt; model–human correlations are not legal-accuracy scores. Absence of a statistically detected framing effect is not proof of complete invariance. The released OpenAI responses date to February 2026, and results do not establish the performance of current products or general judicial decision-making.
- Jurisdiction tags
- United States
- Language
- English
Research artifacts
Resources linked by the authors.
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
The study constructs 552 prompt variants from 46 insurance scenarios, each with three substantive versions, two statement framings and two answer orders. It retains 288 of 300 recruited US participants; each participant answers six items.
Section 3, pp. 46–47; Appendix A, p. 55The human sample and GPT-5 Mini and GPT-5.2 with medium reasoning show no statistically significant frame or answer-order effects in the reported likelihood-ratio tests. All three non-reasoning configurations show a significant frame effect. This is a result of these tests, not a guarantee of invariance.
Sections 4.1–4.3; Appendix D, Tables 4–9, p. 56GPT-5.2 without reasoning has a reported framing odds ratio of 0.545 (95% CI 0.387–0.769) for covered responses under positive versus negative framing, despite the same substantive scenario.
Section 4.2, p. 49Point-biserial correlations between model decisions and averaged human coverage judgments are 0.47 for GPT-5.2 without reasoning, 0.52 for GPT-5.2 with medium reasoning, and 0.57 for GPT-5 Mini with medium reasoning. These correlations are not percentages of legally correct answers.
Section 4.5 and Table 2, pp. 50–51The authors expressly limit generalisation beyond English insurance contracts and the tested models. The analysis concerns approximation of ordinary speakers’ interpretations, not a comparison with judicial rulings or a test of professional legal advice.
Discussion, conclusion and limitations, p. 52; motivation in section 2, p. 46
Connected research
Related question
Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys
Editorial connection: both studies examine what a human reference can establish in a legal-AI evaluation. Here the reference is ordinary-speaker interpretation; the Stanford study checks statutory survey answers against source law. This is not a citation, shared-dataset or head-to-head comparison claim.
Entry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Reading coverage
- Full-text source check on 15 October 2026 of the final CoNLL proceedings PDF: title-page affiliations, sections 3–5, Table 2, limitations and appendices A–D. Proceedings date: 3–4 July 2026. First public manuscript date was not verified; code history begins in February and the repository was created in May, so July is not asserted as first release. A separate structural audit of four released OpenAI request/response file pairs found 552 unique, matching, identically ordered IDs per pair; response timestamps are 17 February 2026. This checks artifact alignment only. The statistical analysis and model calls have not been independently reproduced. The authors warn that the stripped-down release was not fully tested; the analysis-data landing page responds, but its contents were not inspected.
- Rights
- External references only. ACL states that post-2016 Anthology papers are licensed CC BY 4.0. No explicit repository licence was found in the inspected tree; do not infer reuse permission for code or data from public availability.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.