posted by organizer: emiel || 306 views || tracked by 2 users: [display]

EvalNLGEval 2020 : 1st Workshop on Evaluating NLG Evaluation, collocated with INLG

FacebookTwitterLinkedInGoogle

Link: https://evalnlg-workshop.github.io/
 
When Dec 18, 2020 - Dec 18, 2020
Where Dublin, Ireland (online conference)
Submission Deadline Sep 20, 2020
Notification Due Oct 20, 2020
Final Version Due Nov 20, 2020
Categories    natural language generation   natural language processing
 

Call For Papers

Workshop overview:
----------------------------------------------------------------------
This workshop is intended as a discussion platform on the status and the future of the evaluation of Natural Language Generation systems. Among other topics, we will discuss current evaluation quality, human versus automated metrics, and the development of shared tasks for NLG evaluation. The workshop also involves an ‘unshared task’, where participants are invited to experiment with evaluation data from earlier shared tasks.

Important dates:
----------------------------------------------------------------------
Call for workshop papers or abstracts - July 20, 2020
Submissions due - September 20, 2020
Notification of acceptance - October 20, 2020
Camera ready papers due - November 20, 2020
Workshop: December 18, 2020

Papers:
----------------------------------------------------------------------
We encourage a range of papers ranging from commentary and meta-evaluation of existing evaluation strategies to the suggestion of new metrics. We specifically place emphasis on the methodology and linguistic aspects of evaluation. We invite papers on any topic related to the evaluation of NLG systems, including (but not limited to):
Qualitative studies, definitions of evaluation metrics (e.g., readability, fluency, semantic correctness)
Crowdsourcing Strategies, qualitative tests for crowdsourcing (How to elucidate evaluation metrics?)
Looking at individual differences and cognitive biases in human evaluation (expert vs. non-expert, L1 vs L2 speakers)
Best practices for system evaluations (How does your lab choose models?)
Qualitative study/error analysis approaches
Demo: Systems that make the evaluation easier
Comparison of metrics across different NLG tasks (captioning, data2text, story generation, summarization…) or different languages (with a focus on low-resource languages)
Evaluation surveys
Position papers and commentary on trends in evaluation

We encourage the submission of “task proposals”, where authors can propose shared tasks for next year’s edition of the workshop.

Unshared Task:
----------------------------------------------------------------------
This year’s edition also features an unshared task: rather than working towards a specific goal, we encourage participants to use a specific collection of datasets, for any evaluation-related goal. For example: comparing a new evaluation method with existing ratings, or carrying out a subset analysis. This allows us to put the results from previous shared tasks in perspective, and helps us develop better evaluation metrics for future shared tasks. Working on the same datasets allows for more focused discussions at the workshop.

Datasets for this year’s edition are existing datasets with system outputs and human ratings. Participants may use any of these for their unshared task submission:
E2E NLG Challenge (http://www.macs.hw.ac.uk/InteractionLab/E2E/)
WebNLG Challenge 2017 (https://webnlg-challenge.loria.fr/challenge_2017/)
Surface Realization Shared Task (SRST) 2019 (http://taln.upf.edu/pages/msr2019-ws/SRST.html)

Submission Formats:
----------------------------------------------------------------------
Archival papers (up to 8 pages excluding references; shorter submissions are also welcome)
Non-archival abstract of papers within the topic accepted somewhere else or under submission at the main INLG 2020 (1-2 pages)
Demo papers (1-2 pages)

Organizers:
----------------------------------------------------------------------
Shubham Agarwal
Ondrej Dusek
Sebastian Gehrmann
Dimitra Gkatzia
Ioannis Konstas
Emiel van Miltenburg
Sashank Santhanam
Samira Shaikh

Contact: evalnlg.inlg@gmail.com

Related Resources

ACM--NLPIR--Ei Compendex and Scopus 2020   ACM--2020 4th International Conference on Natural Language Processing and Information Retrieval (NLPIR 2020)--Scopus, Ei Compendex
INLG CFW 2020   INLG 2020 Call for Workshop Proposals
ECIR 2021   European Conference on Information Retrieval
LREC 2020   12th Conference on Language Resources and Evaluation
SCAI 2020   9th International Conference on Soft Computing, Artificial Intelligence and Applications
INLG 2020   13th International Conference on Natural Language Generation
COMIT 2020   4th International Conference on Computer Science and Information Technology
EDML 2020   2nd Workshop on Evaluation and Experimental Design in Data Mining and Machine Learning (EDML 2020) @ ECML PKDD 2020
ENASE 2021   16th International Conference on Evaluation of Novel Approaches to Software Engineering
IJNGN 2020   International Journal of Next - Generation Networks