posted by user: grupocole || 738 views || tracked by 2 users: [display]

STAND4NP 2024 : STAND Workshop on Standardizing Tasks, meAsures and NLP Datasets

FacebookTwitterLinkedInGoogle

Link: https://stand4nlp.github.io/
 
When Jan 29, 2024 - Jan 29, 2024
Where Paris, France
Submission Deadline Jan 24, 2024
Categories    NLP   computational linguistics   artificial intelligene
 

Call For Papers


STAND Workshop on Standardizing Tasks, meAsures and NLP Datasets
https://stand4nlp.github.io/

Full-day workshop in Paris, France, January 29th 2024 (+ partial hybrid)

Abstract submission deadline: January 24th 2024, but earlier submissions are welcome

Scientific context:

The current lack of standardized practices and definitions in NLP systems hinders the progress of the field. Indeed, there is not always consensus on which evaluation methods are meaningful and fruitful, or which of their implementations are to be used with which parameters (eg. SacreBLEU, Post 2018).
In some cases, there is no general agreement on the very definition of a task.
This situation calls for work on standardizing NLP practices.

The International Organization for Standardization (ISO) has just created a dedicated working group on NLP (as a joint effort of the AI and Language committees), and 2 standards are already under way. Topics under consideration by the ISO standardization committees include NLP terminology, evaluation metrics, interoperability, annotation guidelines, good practices in NLP development/evaluation/corpora, documentation.

These topics are already heavily discussed in academia, and a number of informal guidelines have already been proposed. We believe that the creation of NLP standards can significantly benefit from the input of both NLP academics and industry NLP practitioners.
Reciprocally, NLP researchers would benefit from getting involved in the standardization effort, thus ensuring that academia's views are listened to, in particular in the context of the AI Act (the European regulation on AI that has been finalized in December), whose enforcement will strongly rely on those standards.

The STAND workshop is a research initiative whose goal is:

to foster discussion on existing standards, their creation and use
to assess the current needs of the community for standardization
to share experience on the impact on the research activities when lacking good practices
to collect existing good practices (and propose new ones)


We invite contributions from NLP practitioners from both the industry and academia, as well as standardization experts.

We invite two types of submission:
* short abstract: 1 page
* long abstract: 3 pages

Accepted submissions will be presented as posters. Authors accepted in the long-abstract track will be invited to submit a full paper (5-10 pages) after the workshop.
Topics for submissions include, but are not limited to:

Comparability and reproducibility of evaluation setup
Annotation guidelines
Evaluation metrics
Good practices for building, annotating and maintaining corpora
Good practices for system evaluation
Interoperability
Ethical guidelines
Guidelines for documenting corpora and models


Submission instructions:

Submissions are expected in PDF form by email at stand4nlp@inria.fr
All submissions should be formatted using the ACL 2023 style files https://2023.aclweb.org/calls/style_and_formatting/.


============

PROGRAM AT A GLANCE:

[09:00-10:00] Welcome, introduction to standardization, ongoing activities in NLP standardization, and the AI Act context
[10:15-11:50] Academic keynote (Joakim Nivre) and invited talks (Matt Post, other speaker TBC)
[11:50-13:30] Poster session (with boosters) & lunch
[13:30-14:40] Industry keynote (speaker TBC) and invited talk (Dirk Hovy)
[15:00-16:30] Moderator-led breakout discussions. Potential topics that will be discussed include:
- [sharing / drafting] Standardizing good practices for evaluation
- [sharing / drafting] Standardizing good practices for corpus management (collection, annotation, versioning)
- [sharing / drafting] Standardizing evaluation metrics (definitions, implementation, sharing scripts)
- [sharing / drafting] Standardizing annotation schemes (formats and guidelines)
- [debate] Explainability and ethics in NLP: what needs for standards?
- [debate] Comparing standardization needs with limitations of the state-of-the-art: how to bridge the gap?
- [debate] Towards standardizing translations of technical terminology in NLP: how to organize i18n?
[16:30-17:30] Reports from breakouts, definition of community-level actions & wrap-up. Example outcomes that are envisioned include:
- Collection and drafting of existing good practices
- Preparation of a joint submission for a position paper
- Creation of common repositories for evaluation scripts, corpus documentation


Participants to the workshop will be offered the opportunity to attend a standardization committee's meeting, which has been scheduled for the day after the workshop (January 30th). The outputs of that meeting will be used in direct support of the AI Act.

Remote access will be offered for part of the workshop only. In-person participation is recommended if possible.
Posters will be in-person only.


IMPORTANT DATES:
Abstract submission: Anytime by January 24
Notification of acceptance: Within a few days of submission
Workshop: January 29
Standardization committee meeting: January 30


ORGANISING COMMITTEE:
Lauriane Aufrant, Timothée Bernard, Maximin Coavoux, Yoann Dupont, Arnaud Ferré, Taras Holoyad, Rania Wazir

MORE INFORMATION
For the latest information see the workshop page at
https://stand4nlp.github.io/; for any questions contact stand4nlp@inria.fr.

Related Resources

IEEE Big Data - MMAI 2024   IEEE Big Data 2024 Workshop on Multimodal AI
SMM4H 2024   The 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks — Large Language Models (LLMs) and Generalizability for Social Media NLP
NIAI 2024   6th International Conference on Natural Language Processing, Information Retrieval and AI
MEIJ 2024   Mechanical Engineering: An International Journal
BIBC 2024   5th International Conference on Big Data, IOT and Blockchain
SNAM-Special Issue 2024   Datasets, Language Resources and Algorithmic Approaches on Online Wellbeing and Social Order in Asian Languages
NoDaLiDa/Baltic-HLT 2025   The Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies
SIGML 2025   6th International Conference on Signal Processing and Machine Learning
VCOI 2024   International Conference on Vision and Computational Intelligence
CSEA 2024   10th International Conference on Computer Science, Engineering and Applications