ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah et al.

Authors & affiliations

Faris Hijazi, Somayah AlHarbi, Abdulaziz AlHussein, Harethah Abu Shairah, Reem AlZahrani, Hebah AlShamlan, Omar Knio, George Turkiyyah

Affiliations reported at publication:

THIQAH
King Abdullah University of Science and Technology (KAUST)

2024Second Arabic Natural Language Processing Conference (ArabicNLP 2024), pages 225–249

Read the paper Publication source

What it adds

Combines Arabic questions grounded in Saudi legal sources with translated LegalBench tasks to evaluate legal knowledge and text understanding.

Read with this limitation

Saudi sources and translated tasks do not represent every Arabic-speaking jurisdiction. Expert-validation depth and task metadata are limited. External retrieval is outside scope; benchmark performance does not establish operational legal competence.

Jurisdiction tags: Saudi Arabia
Language: Arabic

Research artifacts

Resources linked by the authors.

Official evaluation code and data

Follow the source

Selected statements located in the linked primary sources. These results have not been independently reproduced here.

The benchmark combines Arabic multiple-choice questions, question answering with relevant Saudi regulations supplied, and Arabic translations of LegalBench tasks.
1 Introduction; task composition
The study does not evaluate retrieving relevant laws or facts from external knowledge bases.
1 Introduction; evaluation scope

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Both evaluate structured legal tasks; ArabLegalEval examines Arabic performance with Saudi-source questions and translated tasks.

LawBench: Benchmarking Legal Knowledge of Large Language Models

Both examine legal model capabilities beyond English, with different languages, legal systems and task designs.

Entry status, source edition & rights

Index entry: Source-linked summary, prepared with agent assistance. Independent scholarly review is pending.
Archive admission: This bibliographic entry does not confer admission to the versioned archive.
Edition inspected: 10.18653/v1/2024.arabicnlp-1.20
Reading coverage: Published ArabicNLP 2024 PDF: sections 1, 3–5 and section 6 limitations inspected. Arabic is the evaluation language. Translated LegalBench tasks were selected for limited localisation requirements; no additional national-law facet is inferred. Publication affiliations and official artifact links checked. No code execution.
Rights: External references only. Papers and research artifacts remain with their original hosts; their own terms apply.
Entry updated: 2026-10-15

Source-linked summary · independent assessment pending.