FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing
ACL 2022
What it adds
Tests disparities between groups across four legal datasets and five languages. Comparing robustness methods makes visible the differences that a single overall accuracy score can conceal.
Read with this limitation
Some sensitive attributes are automatically extracted or simplified, including binary gender. Performance parity is the selected operational definition of fairness; historical judgments are jurisdiction-dependent labels, not proof of legally or socially just outcomes.
- Jurisdiction tags
- Council of Europe · United States · Switzerland · China
- Language
- English · German · French · Italian · Chinese
Follow the source
Selected statements located in the linked primary sources. These results have not been independently reproduced here.
The tested robustness methods show mixed results; no single method consistently performs best across datasets and groups.
Table 4 and ConclusionsThe benchmark acknowledges that historical judgment labels reflect judges’ interpretations within particular legal frameworks.
Limitations
Connected research
Cites
CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction
Identifies CAIL2018 as the original Chinese judgment-prediction resource; its fairness experiments use a derived setting.
Inspect the sourceCited by
LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Discusses FairLex as prior multilingual fairness research.
Inspect the sourceRelated question
A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction
Both support examination of group-related bias in legal prediction, using different jurisdictions and evaluation designs.
Related question
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Pile of Law examines data filtering; FairLex examines fairness evaluation. Both study how legal datasets shape model behaviour.
Related question
How do judges use large language models? Evidence from Shenzhen
Both examine bias in legal AI. The Shenzhen study examines interaction with judges, whereas FairLex evaluates disparities in model performance. No citation is implied.
Entry status, source edition & rights
- Index entry
- Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
- Archive admission
- This bibliographic entry does not confer admission to the versioned archive.
- Edition inspected
- 10.18653/v1/2022.acl-long.301
- Reading coverage
- Publisher PDF inspected for dataset construction, group definitions, Table 4 results and stated limitations (PDF pages 1–4 and 9); selected bibliography entries inspected. No models rerun or attributes independently validated.
- Rights
- External source links only. No paper full text or underlying dataset is mirrored by the site; downstream reuse requires checking the source licence.
- Entry updated
- 2026-10-15
Source-linked summary · independent assessment pending.