The “test suites” sub-task will be included for the sixth time in the General MT Shared Task of the Conference on Machine Translation (WMT24).

*OVERVIEW*

Test suites are custom extensions to the test sets of the General MT Shared Task, constructed so that they can focus on concrete aspects of the MT output. They consist of a source-side test-set and a customized evaluation service. As opposed to the standard evaluation process which produces generic quality scores, test suites often produce separate fine-grained results for each phenomenon.

Since the usage of LLMs for translation is getting more popular, and we are expecting more LLMs participations in WMT this year, the theme of this year’s test suite sub-task is "Help us break LLMs", i.e. to reveal weaknesses and serious flaws of LLMs when translating, hidden within the overall high-quality generation.

*IMPORTANT DATES*

11th April: Test suite source texts may be submitted for a pre-run on SoTA MT systems
12th June: Test suite source texts must reach us
11th July: Translated test suites shipped back to test suites authors:
TBC - August: Test suite description and analysis paper
12th-13th November: Conference

Potential participants are kindly requested to fill in this form
https://forms.office.com/e/e4JuMTSWFF

Further information can be found in the dedicated page of the WMT website
http://www2.statmt.org/wmt24/testsuite-subtask.html