posted by user: grupocole || 3220 views || tracked by 4 users: [display]

BPPF 2021 : ACL-IJCNLP 2021 Workshop on Benchmarking: Past, Present and Future

FacebookTwitterLinkedInGoogle

Link: https://github.com/kwchurch/Benchmarking_past_present_future/blob/master/README.md
 
When Aug 5, 2021 - Aug 6, 2021
Where Bangkok
Submission Deadline Apr 26, 2021
Notification Due May 28, 2021
Final Version Due Jun 7, 2021
Categories    NLP
 

Call For Papers



Apologies for cross-postings
------------------------------------------------------
Call for Papers

Workshop on "Benchmarking: Past, Present and Future"

Co-located with ACL-IJCNLP 2021 to be held in Bangkok, on August 5-6, 2021

Webpage:
https://github.com/kwchurch/Benchmarking_past_present_future/blob/master/README.md

Important Dates

* April 26, 2021: Paper submission deadline
* May 28, 2021: Notification of acceptance
* June 7, 2021: Camera-ready papers due
* August 5-6, 2021: Workshop dates
--------------------------------------------------------

It is easier to talk about the past than the future. These days,
benchmarks evolve more bottom up (such as papers with code). There used to
be more top-down leadership from government (and industry, in the case of
systems, with benchmarks such as SPEC). Going forward, there may be more
top-down leadership from organizations like MLPerf and/or influencers like
David Ferrucci, who was responsible for IBM’s success with Jeopardy, and
has recently written a paper suggesting how the community should think
about benchmarking for machine comprehension (To Test Machine
Comprehension, Start by Defining Comprehension). Tasks such as reading
comprehension become even more interesting as we move beyond English.
Multilinguality introduces many challenges, and even more opportunities.

Keynote Talks

We have an amazing collection of invited talks, many with direct
first-hand knowledge of the history, and many insights for the future:

1. Past
a. John Makhoul
b. Mark Liberman: Reproducible Research and the Common Task Method c.
Ellen Voorhees

2. Present
a. Ming Zhou (Microsoft)
b. Hua Wu and Jing Liu (Baidu)
c. Neville Ryant DIHARD
d. Brian MacWhinney and Saturnino Haider, Dementia Challenge
e. Samuel Bowman (GLUE)
f. Douwe Kiela (https://dynabench.org/)
g. Eunsol Choi

3. Future
a. MLPerf Greg Diamos The 2021 SpeechNet Challenge
b. David Ferrucci
c. Ido Dagan

Submissions

We accept three types of submissions, long papers, short papers and
abstracts, all following the ACL2021 style, and the ACL submission policy:
https://www.aclweb.org/adminwiki/index.php?title=ACL_Policies_for_Submission,_Review_and_Citation

Long papers may consist of up to eight (8) pages of content, plus
unlimited references, short papers may consist of up to four (4) pages of
content; final versions will be given one additional page of content so
that reviewers' comments can be taken into account. Abstracts may consist
of up to two (2) pages of content, plus unlimited references but will not
be given any additional page upon acceptance. Submissions should be sent
in electronic forms, using the Softconf START conference management
system. The submission site will be announced on the workshop page once
available.

We invite original research papers from a wide range of topics, including
but not limited to:

1. What important technologies and underlying sciences need to be
fostered, now and in the future?
2. In each case, are there existing tasks/benchmarks that move the field
in the right direction?
3. Where are there gaps?
4. For the gaps, are there initial steps that are accessible, attractive,
and cost effective?
5. How large should a benchmark be?
a. How much data do we need to measure significant differences?
b. How much data do machines need to obtain good performance?
c. How much data do babies need to learn language?

Submissions are open to all, and are to be submitted anonymously. All
papers will be refereed through a double-blind peer review process by at
least three reviewers with final acceptance decisions made by the workshop
organizers.

The workshop is scheduled to last for one day either August 5th or 6th. If
you have any questions, contact us at
pc-benchmarking-ws-acl2021@googlegroups.com

Workshop organizers

Kenneth Church (Baidu USA)
Mark Liberman (University of Pennsylvania)
Valia Kordoni (Humboldt-Universität zu Berlin)


Related Resources

ACL-IJCNLP 2021   59t Annual Meeting of the Association for Computational Linguistcs and the 10th International Joint Conference on Natural Language Processing
CMCA 2022   11th International Conference on Control, Modelling, Computing and Applications
AIAPP 2022   9th International Conference on Artificial Intelligence and Applications
(IN)TANGIBLE HERITAGE(S) 2022   (IN)TANGIBLE HERITAGE(S): A conference on design, culture and technology – past, present, and future
NLP 2021   10th International Conference on Natural Language Processing
SfP 2021   Software for the Past (SfP 2021): 2nd Workshop on Digital Technologies to Study the Past and Present
ACM-Ei/Scopus-KDML 2021   2021 International Conference on Knowledge Discovery and Machine Learning (KDML 2021)
AM 2022   The Future of the Dutch Colonial Past (Amsterdam University Press, 2022)
TRANSLATON 2 2021   Translaton 2: The Present and Future of Translation Studies – Current Trends and New Directions.
DMA 2022   8th International Conference on Data Mining and Applications