posted by organizer: lefterav || 595 views || tracked by 2 users: [display]

WMT-Testsuites 2023 : Call for Participation: Test suite sub-task of the Eighth Conference on Machine Translation (WMT23)


When Dec 6, 2023 - Dec 7, 2023
Where Singapore
Submission Deadline Jun 19, 2023
Notification Due Jul 27, 2023
Final Version Due Sep 1, 2023
Categories    machine translation

Call For Papers

the “Test suites” sub-task will be included for the sixth time in the General MT Shared Task of the Conference on Machine Translation (WMT23).


Test suites are custom extensions to the test sets of the General MT Shared Task, constructed so that they can focus on particular aspects of the MT output. They cοnsist of a source-side test-set and a customized evaluation service. As opposed to the standard evaluation process which produces generic quality scores, test suites often produce separate fine-grained results for each phenomenon.

Given the massive improvement of MT and the emergence of LLMs recently, test suites can be useful at revealing serious flaws of otherwise highly scoring systems.


19th June: Test suite source texts must reach us

27th July: Translated test suites shipped back to test suites authors:

TBC - September: Test suite description and analysis paper

Potential participants are kindly requested to fill in this form


The sub-task is co-ordinated by Ondrej Bojar and Eleftherios Avramidis.

Further information can be found in the dedicated page of the WMT website

Related Resources

WMT-Testsuites 2024   'Help us break LLMs' - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)
NLPTT 2024   5th International Conference on NLP Trends & Technologies
Call For Papers Special Issue 2024   Smart Cities, innovating in the Transformation of Urban Environments
BIOEJ 2024   Bioscience & Engineering: An International Journal
Call for Chapters - CRC Press 2024   Internet of Things: Challenges and Advances
NLCA 2024   5th International Conference on Natural Language Computing Advances
PJA Call for Guest Editors 2025   The Polish Journal of Aesthetics: Call for Special Issue Proposals and Guest Editor Recruitment
NLPA 2024   5th International Conference on Natural Language Processing and Applications
Dialogical Approaches to the Sphere ‘in 2024   Call For Papers - Dialogical Approaches to the Sphere ‘in-between’ Self and Other: The Methodological Meaning of Listening
Philosophical Approaches to Games and Ga 2024   Call For Papers - Philosophical Approaches to Games and Gamification: Ethical, Aesthetic, Technological and Political Perspectives