Research checks

01 / The process

One claim. A traceable check.

A successful run shows that a particular result can be obtained under recorded conditions. It does not establish the paper’s overall validity, legal correctness or suitability for practice.

1. Specify the claim

   Identify one table, metric or finding in an exact paper version. State the expected result and acceptable tolerance before running the check.

2. Pin the materials

   Record the code commit, data snapshot, environment, model version, prompts, evaluation split and exact command. Establish the right to use each artifact.

3. Run within limits

   An agent prepares the plan; an isolated runner executes it within fixed time, compute and cost budgets. Preserve failed attempts and keep any agent repairs separate from the original run.

4. Compare and explain

   Compare observed and expected values using the declared method. Publish logs, differences, coverage and limitations. Interpretation and independence remain separately assessed.

02 / Reading a result

Say exactly what happened.

Materials, execution and independent review answer different questions. Each would be recorded separately.

Materials

Not assessed · incomplete · ready to run · restricted access

Execution

Queued · running · completed · blocked · cancelled

Result

Matches stated tolerance · partially matches · does not match · inconclusive

Review

Automated check · human-reviewed check · independent reproduction · independent replication

Unavailable data, a retired model or an exhausted budget means a check is blocked. A result can be reported only for the work actually performed. Independence must identify who carried out the check and their relationship to the authors.

Checks remain attached to the version they tested. A revised paper needs a new assessment; an earlier successful run cannot silently validate it.

03 / Legal AI checks

The benchmark needs context.

Record jurisdiction, language and the date of the applicable law alongside the benchmark version, task, split, metrics and model settings. Check for duplicate cases and temporal leakage across training and evaluation data.

A citation that exists may not support the proposition. A reproducible score may still rely on disputed labels. Separate source resolution, source support, annotation quality and substantive legal judgment.

Doctrinal and normative work needs source and argument assessment, not a forced code-execution test. Comparisons should stay within compatible tasks and conditions.

04 / Prepare the materials

Start with a small, public result.

The first pilot checks dataset composition using public artifacts and an inspectable Python script. A model-performance rerun would additionally require the exact model, prompts and scoring conditions.

• Exact paper version and target claim, table or metric.
• Pinned code, dependencies, data and their licences.
• Execution command, expected output, tolerance and resource estimate.
• Model and API versions, prompts, retrieval settings and random seeds.
• Known limitations, restricted materials and an accountable contact.

Keep credentials and confidential client or case material out of public artifacts. Current submission intake supports a PDF and structured metadata; code and data intake is not available yet.