TerjamaBench: A Cultural Benchmark for English-Darija Machine Translation

Introduction

We introduce TerjamaBench, an evaluation benchmark for English-Darija machine translation. Darija, the Moroccan Arabic dialect, presents unique challenges for machine translation due to its informal nature, regional variations, and scarcity of digital resources. TerjamaBench features meticulously curated parallel texts in English, Arabic-script Darija, and Latin-script Darija (Arabizi), representing a wide range of cultural contexts and regional differences. We assess multiple state-of-the-art models, including proprietary LLMs and open-source models, utilizing various evaluation methods. We also show the limitations of widely used metrics in Machine Translation (MT) tasks in evaluating Darija translations by analysing their correlation with human judgment. Our results demonstrate significant gaps in current translation capabilities and provide insights for improving Darija-English translation systems.

Examples from TerjamaBench dataset. The dataset is available in atlasia/TerjamaBench.

Benchmark Design

TerjamaBench was built through a careful process, addressing the unique challenges of Darija, which exhibits significant regional and linguistic diversity. The benchmark’s development involved curating varied data, extracting valuable insights, and acknowledging the dataset’s inherent limitations.

Data Curation Process

The dataset was curated manually by 16 annotators and 14 reviewers, all native Moroccans. Each annotator brought regional expertise, ensuring a broad representation of Darija’s variations across Morocco. The goal was to capture both formal and informal expressions, with an emphasis on the spoken nature of the language. We followed a structured approach:

• Clear guidelines for annotation.
• Validation steps to ensure linguistic and cultural authenticity.
• Documentation of regional variations and their frequency.
• Multiple review rounds by native speakers to ensure accuracy.

Key Insights and Statistics

The dataset contains 850 entries, structured into six columns:

• Topic: Broad category of the sentence.
• Subtopic: More specific classification within the topic.
• Arabizi: Latin-script written Darija.
• English: English translation of the Darija text.
• Darija (in Arabic letters): Arabic-script written Darija.
• Annotator Dialect (City): The regional variation spoken by the annotator.

The dataset includes both standard phrases and idiomatic expressions, with a focus on minimizing bias, and capturing the frequent code-switching seen in Darija, where speakers blend Arabic, Tamazight, French, and sometimes English.

The topics span a wide range of categories:

Limitations

Despite the thorough curation process, the dataset still has some limitations. First, there’s a regional bias; even though we tried to represent a diverse range of dialects, certain regions remain overrepresented. Another challenge is the orthographic variations in written Darija. Since Darija lacks a standardized writing system, inconsistencies in spelling and grammar are common, complicating machine translation models. The use of Arabizi, an informal and phonetically driven script without formal rules, adds further complexity, making normalization difficult for machine translation models.

Experimental Setup

Dataset

The initial dataset contained 850 entries. After deduplication and removing the "dialect_variation" topic due to its complexity, our final experimental subset contained 788 samples. For human evaluation, we selected a stratified random sample of 237 entries (30% of 788), ensuring proportional representation across all topics.

Models

We evaluated a diverse set of both proprietary and open-source models to benchmark translation performance in English-to-Darija:

• gemini-exp-1206, claude-3-5-sonnet-20241022, gpt-4o-2024-08-06: These proprietary models were selected based on top-tier human judgment on English2Darija translation.
• atlasia/Terjman-Large-v1.2, atlasia/Terjman-Nano: AtlasIA’s MT models, fine-tuned specifically for English2Darija translation.
• MBZUAI-Paris/Atlas-Chat-9B: An open-source Darija LLM.
• facebook/nllb-200-3.3B: Used as a baseline.

Evaluation Approaches

Metric-Based Evaluation

To evaluate model performance, we employed three standard and widely used metrics in MT: BiLingual Evaluation Understudy (BLEU), CHaRacter-level F-score (chrF), and Translation Error Rate (TER). BLEU: Measures n-gram overlap between the model’s output and reference translations. chrF: Focuses on character-level n-grams, providing finer-grained insights into similarity, particularly for morphologically rich languages like Darija. TER: Computes the number of edits required to transform the model’s output into the reference translation. However, we acknowledge their limitations, particularly for a language with high orthographic and linguistic variability like Darija. The next sections highlight why these metrics may fall short in fully capturing translation quality in the context of Moroccan Darija.

LLM-as-a-Judge Evaluation

To complement traditional metrics and provide a more context-sensitive evaluation, we leveraged Claude 3.5 Sonnet (2024-10-22) as an evaluation judge. Using the prompt in APPENDIX 1, we assessed translations by feeding the model both the reference and the generated output, scoring each sample on a nuanced 4-point scale: -1: Translation contains repetitive tokens or clear bugs. 0: Translation is incorrect, nonsensical, or lacks any Darija words. 1: Translation is mostly correct but includes Modern Standard Arabic elements (at least one Darija word) or has minor typos. 2: Translation is fully correct and entirely in Darija.

Human Evaluation

To validate the reliability of our evaluation approaches, we conducted a human evaluation using a random subsample of 30% from each topic, resulting in 241 samples. We used the same 4-point scale as the LLM-as-a-judge. The primary goal was to assess whether metrics-based approaches (BLEU, chrF, TER) and LLM-as-a-judge evaluations are correlated with human judgments.

Results and Analysis

Model Performance Comparison

Metric-based

Table 1: BLEU, chrF, and TER scores for each model. Higher BLEU and chrF scores indicate better alignment with reference translations, while lower TER scores indicate fewer edits needed.

Proprietary models consistently outperform open-source models, with gemini-exp-1206 and claude-3-5-sonnet-20241022 leading across both metrics-based and topic-specific evaluations (cf Table 2 in Appendix). Among open-source models, atlasia/Terjman-Large performs the best, though it significantly lags behind proprietary counterparts. Topics like "religion" and "single_words" are easier, as evidenced by higher scores across models, while "idioms" and "long_sentences" are notably challenging, particularly for open-source models, highlighting their struggle with context-sensitive and structurally complex translations.

LLM-as-a-Judge

The LLM-as-judge evaluation (Appendix - Table 3) maintains the same hierarchy as metrics-based evaluation for proprietary models, but reveals a more nuanced picture for open-source ones. While proprietary models still lead (63.58% high-quality translations for gemini), Atlas-Chat-9B (39.47%) performs competitively with Terjman-Large (32.87%) despite having lower metric scores. The topic breakdown (Appendix - Table 4) shows interesting patterns - models maintain high performance on "common_phrases" and "named_entities" but struggle more with "idioms" and "long_sentences", particularly open-source ones.

Human evaluation

Human judgments (Appendix - Table 5) corroborate the superiority of proprietary models, with gemini-exp-1206 achieving the highest ratings. However, the topic-level analysis (Appendix - Table 6) shows even proprietary models struggle with culturally-loaded topics like "humor" and "idioms", while excelling at more straightforward topics like "religion" and "common_phrases". Overall, the consistently strong performance of proprietary models across all evaluation approaches, particularly on challenging topics, highlights the current limitations of open-source alternatives in handling Darija's linguistic complexity.

Also, human evaluators rated translations more favorably overall compared to the LLM judge for most models as shown below.

Table 7: % of 2-scored samples using llm-as-a-judge and human evaluation on the evaluation subset.

This suggests current automated evaluation approaches do not align with human assessment. This progressive analysis through different evaluation lenses reveals that while metric-based approaches capture broad performance trends, they may underestimate both the absolute quality of translations and the true difficulty gap between simple and complex topics.

Correlation between human evaluation and other approaches

To validate the reliability of our automated evaluation methods, we conducted a comprehensive correlation analysis between human evaluation scores and other evaluation approaches on the same subset described in section Human evaluation. Table 7 presents the Spearman correlation coefficients.

Table 8: Correlation between human evaluation and other evaluation approaches. Statistical significance is less than 10−4

• Metric Reliability: chrF shows the strongest correlation with human judgment.

• Error Metrics: TER shows a moderate negative correlation with human evaluation, indicating that while it captures some aspects of translation quality, it may not fully align with human perception of dialectal translation adequacy.

The LLM-as-a-judge approach shows a comparable and higher correlation to chrF, indicating its potential as a robust evaluation metric.

All correlations are statistically significant (p < 0.001), indicating the reliability of these relationships. However, the moderate strength of these correlations suggests that no single automated metric can fully replace human evaluation for assessing Darija translation quality. This finding underscores the importance of using multiple evaluation approaches, as we have done in this study, to get a comprehensive understanding of translation quality. These results also highlight the need for developing more sophisticated evaluation metrics specifically designed for dialectal Arabic translation, potentially incorporating features that better align with human judgment of translation quality in dialectal contexts.

Qualitative analysis of proprietary LLMs performance

• Gemini-exp-1206 demonstrates a solid capability in handling Darija, although it sometimes produces unnatural or awkward phrases. Its most frequent mistakes are literal translations from English and occasional use of Standard Arabic. Some of the issues include the following:
  • Awkward constructions
    • Example: "مال هاد الضحك غير على سبة" (What’s with this silly laughter)
  • Verb-subject agreement
    • Example: "ألف درهم راه" (should be "راها")
  • Collocation errors
    • Example: "ديال أيام القراية" (“from school days” sounds unnatural)
  • Missing articles
    • Example: "شكلاط سخون عفاك" (should include an article like "شي" or "واحد ال")
  • Standard Arabic vocabulary
    • Examples: "مضحك", "ما كياخد حتى حاجة محمل الجد"
  • Literal translations from English
    • Examples: "كيفاش كتقدر" (how dare you), "السلام الصاحب" (hello friend)
• GPT-4o is also competent in Darija but struggles with consistent word choices and literal translations that reduce the naturalness of the output. Some of these issues include the following:
  • Incorrect translation/word usage
    • Examples: "شحال فعامك؟" (How old are you), "واش عطيتي لماما العصير" (Have you given your mother juice)
  • Literal translations
    • Examples: "ضعت لي مفاتيح الدار" (I lost my house keys), "غادي نطيح من الضحك" (I’ll faint from laughter)
  • Inconsistent handling of “آ”
    • Examples: "سير دي ولادك لغابة المعمورة صاحبي", "شنو قلت ليك يا هاجر؟"
  • Missing suffixes
    • Example: "شفت الماتش ديال Barça البارح؟" (should be "شفتي")
  • Collocation issues
    • Example: "شنو النهار اليوم؟" (should be "شمن نهار")
  • Occasional Standard Arabic usage
    • Examples: "ماخصش الواحد يزعل بسهولة", "مضحك"
• Claude 3.5 sonnet handles Moroccan Darija with notable difficulties, especially the use of literal translations and vocabulary or expressions more common in Standard Arabic. Below are some of the issues that were observed:
  • Verb-object agreement
    • Examples: وصلو رسالة, بلي وقع شي مخالفة (should be وصلاتو)
  • Collocation issues
    • Examples: "تحكم عليه فالمحكمة الفيدرالية ف 2008 على رشوة", "للمعاش ديالو عند التقاعد" (should be بالرشوة)
  • Word order issues
    • Example: Nothing has changed -> والو ما تبدل (should be ما تبدل والو)
  • Excessive definiteness
    • Example: “human persons have a right to life” -> “البنادم عندو الحق فالحياة” (should be بنادم)
  • Use of Standard Arabic (Fus7a) vocabulary/expressions
    • Examples: المقبلة, بسماحهم, القسط, بنايات, على طول, لرفيقتو
  • Literal translations
    • Example: “cease to be blind” -> “توقفو تكونو عميان”
  • Issues with certain question forms
    • Example: “Which of the following observations about revolutions and gender is best supported by the first passage?” -> “شنو من هاد الملاحظات على الثورات والجنس هي اللي كتدعمها أحسن الفقرة الأولى؟”

Summary and Future Research Directions

TerjamaBench makes several significant contributions to the advancement of English-Darija machine translation:

1. A rich, diverse, and culturally specific benchmark dataset designed to reflect authentic Moroccan Darija usage, spanning categories like everyday expressions, technical vocabulary, and regional dialects.
2. Comparative study of diverse evaluation approaches, highlighting their consistency with human evaluations.
3. A detailed quantitative and qualitative evaluation of various models across different linguistic challenges, such as syntax, semantics, and dialectal variations

Our findings reveal that while proprietary models show promising results, significant challenges remain in:

• Handling regional dialectal variations
• Translating idiomatic expressions
• Maintaining consistency in long-form translations
• Processing mixed-language content

Future work should focus on:

• Expanding the benchmark to include more regional variations
• Developing Darija-specific evaluation metrics
• Improving open-source models' performance on cultural expressions

The significant gap between proprietary and open-source models highlights the need for more investment in open-source Darija translation capabilities to improve accessibility of these technologies.

Acknowledgments

Special recognition goes to the contributors: Aissam Outchakoucht, Chaymae Rami, Mahmoud Bidry, Zaid Chiech, Imane Momayiz, Abdelaziz Bounhar, Abir Arsalane, Abdeljalil ElMajjodi, Aymane ElFirdoussi, Nouamane Tazi, Salah-Eddine Iguiliz, Hamza Essamaali, Ihssane Nedjaoui, Anas Amchaar, Yousef Khoubrane, Khaoula Alaoui, Salah-Eddine Alabouch, Adnan Anouzla, Bilal El Hammouchi, Taha Boukhari, Mustapha Ajeghrir, Ikhlas Elhamly, Fouad Aurag, Omar Choukrani, Ali Nirheche, Yanis Bardes, Abdelmonaim Bounite.

Citation

@article{atlasia2024terjamabench,
  title={TerjamaBench: A Culturally Specific Dataset for Evaluating Translation Models for Moroccan Darija},
  author={Imane Momayiz and Aissam Outchakoucht and Omar Choukrani and Ali Nirheche},
  year={2024},
  url={https://huggingface.co/datasets/atlasia/TerjamaBench/}
  institution={AtlasIA}
}

Appendix

Prompt used to generate translations

You are a Moroccan Arabic (Darija) translator. Your task is to translate text from English to Moroccan Arabic using Arabic script, following these guidelines:

1. Maintain any JSON formatting in the original text
2. For words without common Arabic equivalents, use their French translations as Moroccans would do
3. Preserve all code and technical terms in French/English
4. Adapt any culturally sensitive content to be appropriate for Moroccan audiences
5. For idioms, literature, examples, and questions, provide natural Moroccan Arabic translations
6. Use Moroccan Arabic instead of Modern Standard Arabic whenever possible (VERY IMPORTANT)

Format your response as:
[
{"original": "I love going to the beach", "translation": "كنبغي نمشي للبحر"},
{"original": "The weather is nice today", "translation": "الجو زوين اليوم"}
]

Please translate the following JSON list of texts:

Prompt used for LLM as a judge

You are a native Moroccan Arabic (Darija) speaker and expert linguist. You will evaluate machine translations into Moroccan Arabic.

For each example, you will be given:
1. The original English text
2. The ground truth Moroccan Arabic in Arabic script
3. A machine-generated translation in Arabic script

Please evaluate the machine translation by:
1. Comparing it to the ground truth version (Arabic script). But keep in mind that the ground truth can be in a different dialect.
2. Checking for:
   - Accuracy of meaning
   - Natural Moroccan dialect usage
   - Appropriate colloquial expressions
   - Correct grammar and word choice
3. Give a score where:
   -1 = Contains repetitive tokens or bugs
   0 = Translation is incorrect, makes no sense, or contains no Darija words
   1 = Translation is correct but mixed with Modern Standard Arabic (contains at least one Darija word) or has minor typos
   2 = Translation is correct and fully in Darija

Format your response in JSON format, with each line containing a JSON object with these fields in this order:
- analysis: Brief explanation of score, highlighting strengths/weaknesses
- score: Integer score (-1, 0, 1, or 2)


# Example evaluation
Input:
{
   "English": "the speech was addressed to all the people who were present",
   "Darija": "الهدرة توجهات لكاع الناس اللي كانو حاضرين",
   "machine": "الخطاب توجه لجميع الناس اللي كانو حاضرين"
}


Output:
{"analysis": "استعملو 'جميع' عوض 'كاع'، وهادي أقرب للعربية الفصحى", "score": 1}
{"analysis": "مسا الخير صحيحة بالدارجة غير هي بلهجة مختلفة", "score": 2}



Here are 15 samples to evaluate:

Table 2: BLEU score per model per topic. (Higher is better)

Table 3: Percentage of translations rated as high quality (score = 2) by LLM-as-a-judge for each model.

Table 4: Percentage of translations rated as high quality (score = 2) by LLM-as-a-judge for each model per topic.

Table 5: Percentage of high-quality translations (score = 2) rated by human evaluators for each model.

Table 6: Percentage of translations rated as high quality (score = 2) by human evaluators for each model per topic.