Better Call GPT, Comparing Large Language Models Against Lawyers
arXiv preprint; Onit AI Center of Excellence
What it adds
Compares models, junior lawyers and legal-process outsourcers on identifying and locating contractual issues. Separate measures reveal that recognising an issue and pinpointing its supporting clause can produce different rankings.
Read with this limitation
A company-authored preprint using only ten anonymised procurement contracts and senior-lawyer consensus labels. Historical model results and estimated processing costs do not establish equivalent performance across negotiations, supervision or complete client engagements.
- Jurisdiction tags
- United States · New Zealand
- Language
- English
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
Ten procurement contracts cover US and New Zealand settings.
3.3.1. Dataset PreparationOn issue location, outsourcers achieved an F-score of 0.770, above the best model’s 0.740.
Table 3
Connected research
Related question
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Both investigate contract analysis; their annotation targets and evaluation settings differ.
Entry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Edition inspected
- 2401.16212v1
- Reading coverage
- Exact arXiv v1 PDF and HTML inspected; methods, Tables 2–5, discussion and prompts. PDF retained privately with SHA-256.
- Rights
- External links only; no full text mirrored on the Site. arXiv labels this version CC BY 4.0.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.