posted by user: lefterav || 446 views || tracked by 1 users: [display]

WMT-Testsuites 2024 : 'Help us break LLMs' - Test suite sub-task of the Ninth Conference on Machine Translation (WMT24)


When Nov 12, 2024 - Nov 13, 2024
Where Miami, Florida
Submission Deadline Jun 12, 2024
Categories    machine translation   NLP

Call For Papers

The “test suites” sub-task will be included for the sixth time in the General MT Shared Task of the Conference on Machine Translation (WMT24).


Test suites are custom extensions to the test sets of the General MT Shared Task, constructed so that they can focus on concrete aspects of the MT output. They consist of a source-side test-set and a customized evaluation service. As opposed to the standard evaluation process which produces generic quality scores, test suites often produce separate fine-grained results for each phenomenon.

Since the usage of LLMs for translation is getting more popular, and we are expecting more LLMs participations in WMT this year, the theme of this year’s test suite sub-task is "Help us break LLMs", i.e. to reveal weaknesses and serious flaws of LLMs when translating, hidden within the overall high-quality generation.


11th April: Test suite source texts may be submitted for a pre-run on SoTA MT systems
12th June: Test suite source texts must reach us
11th July: Translated test suites shipped back to test suites authors:
TBC - August: Test suite description and analysis paper
12th-13th November: Conference

Potential participants are kindly requested to fill in this form

Further information can be found in the dedicated page of the WMT website

Related Resources

ACM-Ei/Scopus-CCISS 2024   2024 International Conference on Computing, Information Science and System (CCISS 2024)
SDM 2024   SIAM International Conference on Data Mining
NLPTT 2024   5th International Conference on NLP Trends & Technologies
NTIRE 2024   CVPR 2024 New Trends in Image Restoration and Enhancement workshop and challenges
COMIT 2024   8th International Conference on Computer Science and Information Technology
CSET 2024   Cyber Security Experimentation and Test (CSET)
SIPRO 2024   10th International Conference on Signal and Image Processing
IMW 2024   CVPR 2024 Workshop on Image Matching: Local Features and Beyond
JEDT 2024   International Journal of Electronic Design and Test
AISC 2024   12th International Conference on Artificial Intelligence, Soft Computing