posted by user: grupocole || 582 views || tracked by 1 users: [display]

STAND4NP 2024 : STAND Workshop on Standardizing Tasks, meAsures and NLP Datasets


When Jan 29, 2024 - Jan 29, 2024
Where Paris, France
Submission Deadline Jan 24, 2024
Categories    NLP   computational linguistics   artificial intelligene

Call For Papers

STAND Workshop on Standardizing Tasks, meAsures and NLP Datasets

Full-day workshop in Paris, France, January 29th 2024 (+ partial hybrid)

Abstract submission deadline: January 24th 2024, but earlier submissions are welcome

Scientific context:

The current lack of standardized practices and definitions in NLP systems hinders the progress of the field. Indeed, there is not always consensus on which evaluation methods are meaningful and fruitful, or which of their implementations are to be used with which parameters (eg. SacreBLEU, Post 2018).
In some cases, there is no general agreement on the very definition of a task.
This situation calls for work on standardizing NLP practices.

The International Organization for Standardization (ISO) has just created a dedicated working group on NLP (as a joint effort of the AI and Language committees), and 2 standards are already under way. Topics under consideration by the ISO standardization committees include NLP terminology, evaluation metrics, interoperability, annotation guidelines, good practices in NLP development/evaluation/corpora, documentation.

These topics are already heavily discussed in academia, and a number of informal guidelines have already been proposed. We believe that the creation of NLP standards can significantly benefit from the input of both NLP academics and industry NLP practitioners.
Reciprocally, NLP researchers would benefit from getting involved in the standardization effort, thus ensuring that academia's views are listened to, in particular in the context of the AI Act (the European regulation on AI that has been finalized in December), whose enforcement will strongly rely on those standards.

The STAND workshop is a research initiative whose goal is:

to foster discussion on existing standards, their creation and use
to assess the current needs of the community for standardization
to share experience on the impact on the research activities when lacking good practices
to collect existing good practices (and propose new ones)

We invite contributions from NLP practitioners from both the industry and academia, as well as standardization experts.

We invite two types of submission:
* short abstract: 1 page
* long abstract: 3 pages

Accepted submissions will be presented as posters. Authors accepted in the long-abstract track will be invited to submit a full paper (5-10 pages) after the workshop.
Topics for submissions include, but are not limited to:

Comparability and reproducibility of evaluation setup
Annotation guidelines
Evaluation metrics
Good practices for building, annotating and maintaining corpora
Good practices for system evaluation
Ethical guidelines
Guidelines for documenting corpora and models

Submission instructions:

Submissions are expected in PDF form by email at
All submissions should be formatted using the ACL 2023 style files



[09:00-10:00] Welcome, introduction to standardization, ongoing activities in NLP standardization, and the AI Act context
[10:15-11:50] Academic keynote (Joakim Nivre) and invited talks (Matt Post, other speaker TBC)
[11:50-13:30] Poster session (with boosters) & lunch
[13:30-14:40] Industry keynote (speaker TBC) and invited talk (Dirk Hovy)
[15:00-16:30] Moderator-led breakout discussions. Potential topics that will be discussed include:
- [sharing / drafting] Standardizing good practices for evaluation
- [sharing / drafting] Standardizing good practices for corpus management (collection, annotation, versioning)
- [sharing / drafting] Standardizing evaluation metrics (definitions, implementation, sharing scripts)
- [sharing / drafting] Standardizing annotation schemes (formats and guidelines)
- [debate] Explainability and ethics in NLP: what needs for standards?
- [debate] Comparing standardization needs with limitations of the state-of-the-art: how to bridge the gap?
- [debate] Towards standardizing translations of technical terminology in NLP: how to organize i18n?
[16:30-17:30] Reports from breakouts, definition of community-level actions & wrap-up. Example outcomes that are envisioned include:
- Collection and drafting of existing good practices
- Preparation of a joint submission for a position paper
- Creation of common repositories for evaluation scripts, corpus documentation

Participants to the workshop will be offered the opportunity to attend a standardization committee's meeting, which has been scheduled for the day after the workshop (January 30th). The outputs of that meeting will be used in direct support of the AI Act.

Remote access will be offered for part of the workshop only. In-person participation is recommended if possible.
Posters will be in-person only.

Abstract submission: Anytime by January 24
Notification of acceptance: Within a few days of submission
Workshop: January 29
Standardization committee meeting: January 30

Lauriane Aufrant, Timothée Bernard, Maximin Coavoux, Yoann Dupont, Arnaud Ferré, Taras Holoyad, Rania Wazir

For the latest information see the workshop page at; for any questions contact

Related Resources

Ei/Scopus-AACIP 2024   2024 2nd Asia Conference on Algorithms, Computing and Image Processing (AACIP 2024)-EI Compendex
SNAM-Special Issue 2024   Datasets, Language Resources and Algorithmic Approaches on Online Wellbeing and Social Order in Asian Languages
IEEE Big Data - MMAI 2024   IEEE Big Data 2024 Workshop on Multimodal AI (Hybrid)
COLING 2025   International Conference on Computational Linguistics
SMM4H 2024   The 9th Social Media Mining for Health Research and Applications Workshop and Shared Tasks — Large Language Models (LLMs) and Generalizability for Social Media NLP
NLE Special Issue 2024   Natural Language Engineering- Special issue on NLP Approaches for Computational Analysis of Social Media Texts for Online Well-being and Social Order
ISEEIE 2024   2024 4th International Symposium on Electrical, Electronics and Information Engineering (ISEEIE 2024)
AISC 2024   12th International Conference on Artificial Intelligence, Soft Computing
IPM-LLMDQKG 2025   Special issue of Information Processing & Management on Large Language Models and Data Quality for Knowledge Graphs
CST 2024   11th International Conference on Advances in Computer Science and Information Technology