SG-LegalCite: A Principle-Augmented Benchmark for Legal Citation Retrieval in Singapore Law
arXiv preprint
Preprint edition
What it adds
Introduces a benchmark linking LLM-generated factual summaries, extracted legal principles and cited authorities from Singapore Supreme Court judgments, and compares fact-only with principle-augmented retrieval.
Read with this limitation
Main results use principles extracted from completed judgments, an oracle setting that differs from starting with client facts. Evaluation uses one gold citation and 999 sampled negatives, not the full case collection. LLM-generated fields receive partial expert validation; each model is evaluated in one fixed-seed run. The later author-repository cold-start update uses a different sample and must be kept separate from the inspected May preprint.
- Jurisdiction tags
- Singapore
- Language
- English
Research artifacts
Resources linked by the authors.
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
The benchmark contains 100,890 case–principle pairs extracted from 8,523 Singapore Supreme Court judgments spanning 2000–2025.
4.5 The Dataset and Expert Validation; Table 4The retrieval evaluation compares 11 baselines using LLM-generated fact summaries, with or without an extracted principle, on pools containing one gold cited case and 999 randomly sampled negatives. Metrics are mean reciprocal rank and Recall@k.
5.1 Experiment SetupIn the main table, SaulLM-7B achieves mean reciprocal rank of 13.0 with facts alone and 38.2 with facts plus an extracted legal principle, on the paper’s 0–100 reporting scale. The principle comes from the completed citing judgment.
5.2 Main Results, Table 5; oracle qualification in 5.3, page 8The May preprint’s separate 50-case experiment reports MRR of 17.1 for facts alone, 21.9 with a principle predicted from facts, and 49.5 with the extracted oracle principle. These figures describe that original small-sample experiment, not the later repository update.
5.3 Gold Principle vs. Predicted Principle; Table 7In their 15 September 2026 repository update, the authors report a cold-start evaluation on 1,000 sampled test pools. With fine-tuned SaulLM-7B, the cited case ranks first in 7.5% of fact-summary-only searches, 8.6% when DeepSeek-V3 predicts a principle from those summaries, and 20.8% when the extracted oracle principle is supplied. These are later author-reported repository results, distinct from the May preprint’s 50-case experiment; the final conference PDF was not inspected and no uncertainty intervals are reported.
15 September 2026 README update; 6. Cold-Start Evaluation (Section 5.3), R@1Three legal experts rated 131 case–principle pairs from 15 sampled judgments; extracted principles received a mean rating of 3.22 out of 5, with reported inter-rater agreement ICC(2,k) of 0.54.
4.5 The Dataset and Expert Validation
Connected research
Cites as an earlier benchmark
LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset
Cites LeCaRDv2 as a fact-based similar-case retrieval benchmark when motivating a facts-plus-principle query. This is a comparison of task design, not reuse of LeCaRDv2 data or a head-to-head performance claim.
Inspect the sourceEvaluates a model from
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Includes Pile-of-Law BERT as a retrieval baseline. This connects model provenance to the new evaluation; it does not reproduce the Pile of Law study.
Inspect the sourceEntry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Edition inspected
- arXiv:2605.21057v1 (20 May 2026)
- Reading coverage
- Full-text review of arXiv:2605.21057v1, including the title-page author affiliations, sections 3–5, Tables 4–7 and stated limitations. First arXiv release: 20 May 2026. Authors and their lab report acceptance at AACL 2026 Main; a final proceedings PDF was not located. The separately labelled September claim below cites the authors’ 15 September 2026 repository update, which expands the cold-start sample to 1,000 test pools. Its final conference PDF was not inspected. The other claims cite the May preprint. Code and data links were checked for availability; no training, retrieval evaluation or independent reproduction was performed.
- Rights
- External references only. Authors release the dataset under CC BY 4.0 and code under MIT; source judgments retain their original terms. A source check does not establish unrestricted downstream reuse or independent reproduction.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.