posted by organizer: TomeEftimov || 1417 views || tracked by 3 users: [display]

BENCHMARK 2020 : GECCO 2020 Workshop - Good Benchmarking Practices for Evolutionary Computation


When Jul 8, 2020 - Jul 12, 2020
Where Cancun, Mexico
Submission Deadline Apr 3, 2020
Notification Due Apr 17, 2020
Final Version Due Apr 24, 2020
Categories    evolutionary algorithms   natural computing   benchmarking   artificial intelligence

Call For Papers

Workshop Description
Brace yourself for highly interactive workshop, with plenty of room for discussions and interaction. This is not just another mini-conference, but a platform to come together and to discuss recent progress and challenges in the area of benchmarking iterative optimization heuristics.
In the era of explainable and interpretable AI, it is increasingly necessary to develop a deep understanding of how algorithms work and how new algorithms compare to existing ones, both in terms of strengths and weaknesses. For this reason, benchmarking plays a vital role for understanding algorithms’ behavior. Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:

(i) most commonly-used benchmarks are too small and cover only a part of the problem space,

(ii) benchmarks lack the complexity of real-world problems, making it difficult to transfer the learned knowledge to work in practice,

(iii) we need to develop proper statistical analysis techniques that can be applied depending on the nature of the data,

(iv) we need to develop user-friendly, openly accessible benchmarking software. This enables a culture of sharing resources to ensure reproducibility, and which helps to avoid common pitfalls in benchmarking optimization techniques. As such, we need to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.

The topics of interest for this workshop include, but are not limited to:

Performance measures for comparing algorithms behavior;
Novel statistical approaches for analyzing empirical data;
Selection of meaningful benchmark problems;
Landscape analysis;
Data mining approaches for understanding algorithm behavior;
Transfer learning from benchmark experiences to real-world problems;
Benchmarking tools for executing experiments and analysis of experimental results.
The schedule will be designed to encourage a high level of interactivity — expect a real workshop rather than (yet another) mini-conference!

The workshop accepts two different types of submissions:

(i) regular workshop papers, which are published in the GECCO Companion Materials (with DOI and listed in dblp, Gopgle scholar, etc.). These submissions need to be made via the submission system, and undergo some (light) reviewing process.

(ii) informal position statements, suggestions for discussions, activities, etc. - we invite you to be creative! These contributions can be submitted by e-mail, and there are no formatting requirements whatsoever. Please share your ideas before June 1, 2020, AoE.

Note re. Regular Workshop Papers
If you want your position paper to appear in the GECCO Companion Materials (with DOI and listed in dblp, Gopgle scholar, etc.), please adhere to the GECCO formatting rules, which are specified here:

Workshop papers can be up to 8 pages long and need to be submitted via the GECCO submission system. They undergo a regular reviewing process.

Important deadlines

(all dates are strict, i.e., no extensions possible!):

Submission opening: February 27, 2020
Submission deadline: April 3, 2020
Notification of acceptance: around April 17, 2020
Camera-Ready Material: April 24, 2020
Author registration deadline: April 27, 2020
Please also note that, by GECCO rules, each accepted paper needs to have at least one author registered by the author registration deadline. If an author is presenting more than one paper at the conference, she/he does not pay any additional registration fees.

Note re. Informal Submissions
We particularly welcome position statements addressing or identifying open challenges in benchmarking optimization techniques. We also suggest topics for in-depth discussions. Please also consider to suggest alternative discussion formats - we want this to be a real workshop, not yet another mini-conference!

Please send your suggestions for presentations and/or discussions by e-mail to all five main contacts listed below. Please indicate the format of your suggested contribution (talk, discussion, breakout, brainstorming, etc.) and how much time you suggest for this activity.

William La Cava (
Boris Naujoks (
Pietro S. Oliveto (
Vanessa Volz (
Thomas Weise (

Thomas Bäck (Leiden University, The Netherlands)
Bilel Derbel (University of Lille, Lille, France)
Carola Doerr (CNRS researcher at Sorbonne University, Paris, France)
Tome Eftimov (Jožef Stefan Institute, Ljubljana, Slovenia)
Pascal Kerschke (University of Münster, Germany)
William La Cava (University of Pennsylvania, USA)
Manuel López-Ibáñez (University of Manchester, UK)
Katherine Malan (University of South Africa)
Boris Naujoks (TH Cologne, Germany)
Pietro S. Oliveto (University of Sheffield, UK)
Patryk Orzechowski (University of Pennsylvania, USA)
Mike Preuss (Leiden University, The Netherlands)
Jérémy Rapin (Facebook AI Research, Paris, France)
Ofer M. Shir (Tel-Hai College and Migal Institute, Israel)
Olivier Teytaud (Facebook AI Research, Paris, France)
Heike Trautmann (University of Münster, Germany)
Ryan J. Urbanowicz (University of Pennsylvania, USA)
Vanessa Volz (, Copenhagen, Denmark)
Markus Wagner (The University of Adelaide, Australia)
Hao Wang (LIACS, Leiden University, The Netherlands)
Thomas Weise (Institute of Applied Optimization, Hefei University, Hefei, China)
Borys Wróbel (Adam Mickiewicz University, Poland)
Aleš Zamuda (University of Maribor, Slovenia)

Related Resources

ICML 2020   37th International Conference on Machine Learning
BENCHMARK 2020   PPSN 2020 Workshop - Good Benchmarking Practices for Evolutionary Computation
IJCAI 2020   International Joint Conference on Artificial Intelligence
BEADO 2020   Special Session on Benchmarking of Evolutionary Algorithms for Discrete Optimization
ICDMML 2020   【EI SCOPUS】2020 International Conference on Data Mining and Machine Learning
PPSN 2020   Parallel Problem Solving from Nature
ISBDAI 2020   【Ei Compendex Scopus】2020 International Symposium on Big Data and Artificial Intelligence
GECCO 2020   Call for Papers - GECCO 2020
IEEE-CVIV 2020   2020 2nd International Conference on Advances in Computer Vision, Image and Virtualization (CVIV 2020)