LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain

Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi et al.

All authors

Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, Ilias Chalkidis

2023Findings of EMNLP 2023

What it adds

Brings eleven legal datasets into a common evaluation framework, reporting both dataset and language aggregates across 24 languages. It enables more systematic comparisons of classification and entity-recognition models.

Read with this limitation

The tasks do not establish general legal understanding. Coverage concentrates on European languages and Brazilian Portuguese; generation is outside this edition. Several outcome tasks use court-written accounts, which differ from documents available before a decision.

Jurisdiction tags: Brazil · European Union · Switzerland · United Kingdom · Other European jurisdictions
Language: 24 languages
Bulgarian · Czech · Danish · German · Greek · English · Spanish · Estonian · Finnish · French · Irish · Croatian · Hungarian · Italian · Lithuanian · Latvian · Maltese · Dutch · Polish · Portuguese · Romanian · Slovak · Slovenian · Swedish

Follow the source

Selected statements located in the linked primary sources. These results have not been independently reproduced here.

The benchmark selects eleven datasets and reports aggregate results for 24 languages.
Introduction and contributions; Table 6 on PDF page 8
Its authors explicitly caution that benchmark results cannot demonstrate general natural legal language understanding.
Limitations

Cites

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Positions multilingual evaluation alongside LexGLUE.

Inspect the source

Cites

FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing

Discusses FairLex as prior multilingual fairness research.

Inspect the source

IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning

Both organise legal NLP evaluation across languages, with different jurisdictions and task coverage.

Entry status, source edition & rights

Index entry: Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
Archive admission: This bibliographic entry does not confer admission to the versioned archive.
Edition inspected: 10.18653/v1/2023.findings-emnlp.200
Reading coverage: Publisher PDF inspected for benchmark scope, task descriptions, evaluation tables, limitations and cited references (PDF pages 1–10). No experiments rerun. Languages listed follow Table 6; source inconsistencies are retained in the accompanying notes.
Rights: External source links only. No paper full text or underlying dataset is mirrored by the site; downstream reuse requires checking the source licence.
Entry updated: 2026-10-15

Source-linked summary · independent assessment pending.