Does the dataset match
the paper’s counts?

01 / The comparison

One declared target. Exact counts.

We extracted the five category counts from Table 7 of LegalBench v1 and compared them with a commit-pinned copy of the task’s test file. Counts had to match exactly. We interpret the table’s n values as evaluation-slice counts; the paper supplies no linked checksum identifying that historical split.

The training file contains five additional examples. The pinned task README states 99 samples in one place, while its construction description implies 100. The files contain 100.

Exact text comparisons and comparisons ignoring case and repeated whitespace found no duplicate texts within either split or overlapping texts between train and test. These checks do not establish the absence of semantic duplicates or contamination.

02 / Interpretation

A discrepancy worth preserving.

The available artifact and the paper’s historical description do not fully agree. A changed dataset, a documentation error or another explanation could account for this; the check does not determine which.

No language model was run. No reported accuracy was reproduced or disproved. We did not evaluate the legal correctness of the labels. This is a check of dataset composition, with an unresolved historical-version question.

The work was prepared and executed by a Codex research assistant using deterministic counting code. It has not received qualified human review or a formal archive admission assessment.

03 / Execution

Recorded conditions, visible limits.

Environment

Python 3.14.7 · macOS arm64 · standard library only

Execution

Local process, 10-second CPU limit and 30-second wall timeout. No upstream code executed.

Memory limit

The requested limit was rejected by macOS and was not enforced.

Observed runtime

Approximately 0.034 seconds for the audit; 0.128 seconds for the wrapped process.

Replay

A second offline run produced the same checks and observations on identical input hashes.

Additional paid services

$0; no model API calls. Hardware energy and assistant orchestration costs were not measured.

This local experiment does not constitute a hosted execution service or a sandbox for submitted code.

04 / Materials

Inspect it. Repeat it.

• Machine-readable reportExpected and observed values, environment and limitations
• Execution logOriginal events with the local workstation path removed
• Source and input manifestPinned URLs, retrieval times and SHA-256 hashes
• Download the repeatable checkPython scripts, protocol, reports and instructions; source files are fetched separately

Sources: pinned task documentation and pinned dataset files. Task attribution: Neel Guha and LegalBench contributors. Dataset licence: CC BY 4.0.

Read the legal-AI reliability guide · Read the checking protocol