posted by user: lefterav || 347 views || tracked by 1 users: [display]

WMT-metrics 2024 : WMT24 Metrics Task: Call for Participation


When Nov 12, 2024 - Nov 13, 2024
Where Miami, Florida, USA
Submission Deadline Jul 23, 2024
Categories    NLP   machine translation   evaluation

Call For Papers

The 9th Conference on Machine Translation (WMT24), collocated with EMNLP 2024, will be featuring this year the Shared Task on evaluation of Automatic Metrics. We are looking for both reference-based metrics and reference-free metrics to evaluate the quality of MT systems. We’ll be using expert-based MQM annotations on English-German, English-Spanish and Japanese⇾Chinese as the primary gold standard for evaluating metrics. Details are at

We’ll be continuing the challenge sets subtask [1] this year: we invite anyone to submit a new test suite and/or an analysis paper on metric behaviour for specific perturbations/phenomena (you’re welcome to resubmit last year’s challenge set!)

New this year:
- New language pairs: English-Spanish and Japanese⇾Chinese
- Additional 13 typologically diverse African languages in a challenge set
- We will be using the Codabench platform to improve the metric submission experience

Important dates:
- Challenge sets submission deadline: 11th July
- Metrics inputs ready to download: 23rd July
- Metric submission deadline: 30th July
- Metric scores for challenge sets distributed: 6th August
- Paper submission deadline to WMT: 20th August

Please register your metric submissions here [2] and challenge set submissions [3] here so we can keep track of participants.

Looking forward to your submissions,
WMT24 Metrics team


Related Resources

Ei/Scopus-AACIP 2024   2024 2nd Asia Conference on Algorithms, Computing and Image Processing (AACIP 2024)-EI Compendex
WMT-Testsuites 2024   'Help us break LLMs' - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)
MEN 2024   7th International Conference on Mechanical Engineering
ARDUOUS 2024   Annotation of useR Data for UbiquitOUs Systems
IEEE Big Data - MMAI 2024   IEEE Big Data 2024 Workshop on Multimodal AI (Hybrid)
IJFLS 2024   International Journal of Fuzzy Logic Systems
NLAI 2024   5th International Conference on NLP & Artificial Intelligence Techniques
Call For Papers Special Issue 2024   Smart Cities, innovating in the Transformation of Urban Environments
Migrating Minds (3) 2025   Migrating Minds. Journal of Cultural Cosmopolitanism-- Call for submissions for Vol.3, Issue 2 (Fall 2025)
NLPTT 2024   5th International Conference on NLP Trends & Technologies