posted by user: kwchang || 705 views || tracked by 1 users: [display]

MML-Shared Task 2022 : Multilingual Multimodal Learning 2022 Shared Task


When May 27, 2022 - May 27, 2022
Where ACL 2022
Submission Deadline Apr 30, 2022
Notification Due May 7, 2022
Final Version Due May 14, 2022
Categories    multimodal   multilingual   shared task

Call For Papers

The multilingual multimodal learning (MML) workshop, co-located at ACL 2022, is hosting a shared task on multilingual visually grounded reasoning. The task will be centred around the MaRVL dataset, introduced by Liu et al. (EMNLP 2021). This dataset extends the NLVR2 task (Suhr et al., ACL 2019) to multicultural and multilingual (Indonesian, Mandarin, Swahili, Tamil, Turkish) inputs: Given two images and a textual description, a system needs to predict whether the description applies to both images (True/False).

The standard setup consists of fine-tuning a multilingual vision-and-language model in the English NLVR2 dataset and then evaluating on MaRVL. We consider two subtasks, as detailed below: zero-shot transfer and few-shot transfer. Both setups have been shown to be challenging (Bugliarello et al., 2022), and we look forward to seeing your approaches to the tasks!

Participants will be invited to describe their system in a paper for the MML workshop proceedings. The task organisers will write an overview paper that describes the task and summarises the different approaches taken, and analyses their results.

Important Dates
Submission Due: April 30 2022 (11:59pm AoE)
Notification: May 7 2022 (11:59pm AoE)
Camera-ready Due: May 14 2022 (11:59pm AoE)
Workshop: 27 May 2022

The shared task will consist of two subtasks:
ZS) Zero-shot transfer: Models are fine-tuned on the English NLVR2 data, and tested on MaRVL Indonesian, Mandarin, Swahili, Tamil, Turkish
FS) Few-shot transfer: Models are further fine-tuned on a few data points in the target language. This subtask corresponds to the most-shot setup of Bugliarello et al. (2022), wherein all the few-shot data points are used. In particular, performance is only reported in three languages: Indonesian, Mandarin and Turkish.

NB: we will *only* consider submissions that use pre-existing pre-trained models that are publicly available or new models that have been (pre)trained on publicly available data.

“Translate test” methods are accepted but will be ranked separately.

Submissions should be emailed to the organisers by the end of April 30, anywhere on Earth.
Submissions need to follow the jsonlines format, where languages are in ISO 639-2 codes:

{"concept": "39-Panci", "language": "id", "chapter": "Basic actions and technology", "id": "id-0", "prediction": true}

Files should be named as `{team-name}_{zs/fs}_{xl/tt}_{lang}.jsonl` to indicate the subtask (zero-shot or few-shot), whether it’s cross-lingual or translate-test transfer, and the target language.

Description Papers
Papers describing shared task submissions should consist of 4 to 8 pages of content plus additional pages of references, formatted according to the ARR format guidelines for ACL 2022. For shared task paper submission, it is not necessary to blind the team name and authors. Accepted papers will be published online in the ACL 2022 proceedings and will be presented at the MML workshop at ACL 2022. Writeups should be submitted through OpenReview, and are due by 30 April 2022 11:59pm [UTC-12h].

Emanuele Bugliarello (University of Copenhagen)
Kai-Wei Chang (UCLA)
Desmond Elliott (University of Copenhagen)
Spandana Gella (Amazon Alexa AI)
Aishwarya Kamath (NYU)
Liunian Harold Li (UCLA)
Fangyu Liu (University of Cambridge)
Jonas Pfeiffer (TU Darmstadt)
Edoardo M. Ponti (MILA Montreal)
Krishna Srinivasan (Google Research)
Ivan Vulić (University of Cambridge)
Yinfei Yang (Apple Research)
Da Yin (UCLA)

Please contact mml DOT wksp AT gmail DOT com if you have any questions

Related Resources

GUA-SPA at IberLEF 2023   GUA-SPA - Guarani-Spanish Code Switching Analysis at IberLEF 2023
FinCausal 2023   Call for Participation: Financial Document Causality Detection Shared Task (FinCausal 2023)
CoCo4MT Shared Task 2023   CoCo4MT Shared Task: First Call for Participation
FinTOC 2023   FNP-2023 Shared Task - FinTOC (Financial Document Structure Extraction)
SYMPTEMIST @ BioCreative/AMIA 2023   CFP: SYMPTEMIST Shared Task (BioCreative VIII run with AMIA 2023)
#SMM4H 2023   Social Media Mining for Health Applications Workshop & Shared Task 2023
WojoodNER 2023   Call for participation - Arabic NER Shared Task 2023
JCICE 2024   2024 International Joint Conference on Information and Communication Engineering(JCICE 2024)
MMNLG 2023   Workshop on Multimodal, Multilingual Natural Language Generation In conjunction with INLG/SIGDIAL 2023
GenChal 2023   The Generation Challenges @INLG'23