posted by user: lefterav || 1034 views || tracked by 4 users: [display]

WMT-metrics 2024 : WMT24 Metrics Task: Call for Participation

FacebookTwitterLinkedInGoogle

Link: http://www2.statmt.org/wmt24/metrics-task.html
 
When Nov 12, 2024 - Nov 13, 2024
Where Miami, Florida, USA
Submission Deadline Jul 23, 2024
Categories    NLP   machine translation   evaluation
 

Call For Papers

The 9th Conference on Machine Translation (WMT24), collocated with EMNLP 2024, will be featuring this year the Shared Task on evaluation of Automatic Metrics. We are looking for both reference-based metrics and reference-free metrics to evaluate the quality of MT systems. We’ll be using expert-based MQM annotations on English-German, English-Spanish and Japanese⇾Chinese as the primary gold standard for evaluating metrics. Details are at http://www2.statmt.org/wmt24/metrics-task.html.

We’ll be continuing the challenge sets subtask [1] this year: we invite anyone to submit a new test suite and/or an analysis paper on metric behaviour for specific perturbations/phenomena (you’re welcome to resubmit last year’s challenge set!)

New this year:
- New language pairs: English-Spanish and Japanese⇾Chinese
- Additional 13 typologically diverse African languages in a challenge set
- We will be using the Codabench platform to improve the metric submission experience

Important dates:
- Challenge sets submission deadline: 11th July
- Metrics inputs ready to download: 23rd July
- Metric submission deadline: 30th July
- Metric scores for challenge sets distributed: 6th August
- Paper submission deadline to WMT: 20th August

Please register your metric submissions here [2] and challenge set submissions [3] here so we can keep track of participants.

Looking forward to your submissions,
WMT24 Metrics team

[1] http://www2.statmt.org/wmt24/metrics-task.html#_challenge_set_subtask
[2] https://docs.google.com/forms/d/e/1FAIpQLSenoN9svmqkfJshM8bHd8p3Ofyepdvcg9Mtns0VdEbsFunvEA/viewform?usp=pp_url
[3] https://forms.office.com/pages/responsepage.aspx?id=vfGpYaB-aECyMbtKa_y3APGD0hnjQCxCom5Ag3yDFjJUQVZHV1FSUlJRSTBYRU9JV1oyRjNRWk9LRi4u

Related Resources

SOTICS 2025   The Fifteenth International Conference on Social Media Technologies, Communication, and Informatics
IEEE-Ei/Scopus-ITCC 2025   2025 5th International Conference on Information Technology and Cloud Computing (ITCC 2025)-EI Compendex
IJIBM 2025   IJIBM 2025 : Call For Papers - International Journal of Information, Business and Management
ACL 2025   The 63rd Annual Meeting of the Association for Computational Linguistics
IEEE WHC 2025   IEEE World Haptics Conference 2025: First Call for Participation
CMIT 2025   13th International Conference of Managing Information Technology
Migrating Minds 4 (1), 2026   Migrating Minds. Journal of Cultural Cosmopolitanism -- Call for papers for Vol. 4, Issue 1, Spring 2026
CoNLL 2025   29th Conference on Computational Natural Language Learning
OP 2025   Call for Papers - Kant's Concept of Spontaneity and Its Legacy in Later Theories of Subjectivity
Carpathian culture and heritage 2025   Call for Papers: The cultures and heritage of the communities along the Carpathian Arc