LawBench: Benchmarking Legal Knowledge of Large Language Models

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou et al.

All authors

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, Vincent Ng

2024EMNLP 2024, 7933–7962

What it adds

Groups legal tasks into memorisation, understanding and application.

Possible training contamination, brittle answer extraction and ROUGE-based generative scoring weaken practical-competence conclusions.

Selected statements located in the linked primary sources. These results have not been independently reproduced here.

LawBench comprises 20 tasks with 500 test instances per task.
3, printed 7935–7936
The published evaluation tests 21 LLMs in zero-shot and one-shot settings.
4.1, printed 7936

Cites

Positions Chinese tasks against English legal NLP benchmarks.

Both examine Chinese legal-model evaluation, with different tasks and generations of models. This is a thematic connection, not a citation claim.

Both examine legal model capabilities beyond English, with different languages, legal systems and task designs.

Entry status, source edition & rights

Index entry: Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
Archive admission: This bibliographic entry does not confer admission to the versioned archive.
Edition inspected: 10.18653/v1/2024.emnlp-main.452
Reading coverage: Published sections 2–4, Tables 1–6, limitations, Appendix E and relevant references; author repository README/file tree. Appendix task prompts and all model tables not exhaustively audited.
Rights: External references only. Papers and research artifacts remain with their original hosts; their own terms apply.
Entry updated: 2026-10-15

Source-linked summary · independent assessment pending.