posted by user: lefterav || 730 views || tracked by 4 users: [display]

WMT-metrics 2024 : WMT24 Metrics Task: Call for Participation

FacebookTwitterLinkedInGoogle

Link: http://www2.statmt.org/wmt24/metrics-task.html
 
When Nov 12, 2024 - Nov 13, 2024
Where Miami, Florida, USA
Submission Deadline Jul 23, 2024
Categories    NLP   machine translation   evaluation
 

Call For Papers

The 9th Conference on Machine Translation (WMT24), collocated with EMNLP 2024, will be featuring this year the Shared Task on evaluation of Automatic Metrics. We are looking for both reference-based metrics and reference-free metrics to evaluate the quality of MT systems. We’ll be using expert-based MQM annotations on English-German, English-Spanish and Japanese⇾Chinese as the primary gold standard for evaluating metrics. Details are at http://www2.statmt.org/wmt24/metrics-task.html.

We’ll be continuing the challenge sets subtask [1] this year: we invite anyone to submit a new test suite and/or an analysis paper on metric behaviour for specific perturbations/phenomena (you’re welcome to resubmit last year’s challenge set!)

New this year:
- New language pairs: English-Spanish and Japanese⇾Chinese
- Additional 13 typologically diverse African languages in a challenge set
- We will be using the Codabench platform to improve the metric submission experience

Important dates:
- Challenge sets submission deadline: 11th July
- Metrics inputs ready to download: 23rd July
- Metric submission deadline: 30th July
- Metric scores for challenge sets distributed: 6th August
- Paper submission deadline to WMT: 20th August

Please register your metric submissions here [2] and challenge set submissions [3] here so we can keep track of participants.

Looking forward to your submissions,
WMT24 Metrics team

[1] http://www2.statmt.org/wmt24/metrics-task.html#_challenge_set_subtask
[2] https://docs.google.com/forms/d/e/1FAIpQLSenoN9svmqkfJshM8bHd8p3Ofyepdvcg9Mtns0VdEbsFunvEA/viewform?usp=pp_url
[3] https://forms.office.com/pages/responsepage.aspx?id=vfGpYaB-aECyMbtKa_y3APGD0hnjQCxCom5Ag3yDFjJUQVZHV1FSUlJRSTBYRU9JV1oyRjNRWk9LRi4u

Related Resources

VCOI 2024   International Conference on Vision and Computational Intelligence
WMT-Testsuites 2024   'Help us break LLMs' - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)
IEEE Big Data - MMAI 2024   IEEE Big Data 2024 Workshop on Multimodal AI
QuASoQ 2024   12th International Workshop on Quantitative Approaches to Software Quality
RTEE 2024   10th International Conference on Recent Trends in Electrical Engineering
IJRAP 2024   International Journal of Recent advances in Physics
MLDS 2024   5th International Conference on Machine Learning Techniques and Data Science
ARDUOUS 2024   Annotation of useR Data for UbiquitOUs Systems
BIBC 2024   5th International Conference on Big Data, IOT and Blockchain
Call For Papers Special Issue 2024   Smart Cities, innovating in the Transformation of Urban Environments