posted by user: TomeEftimov || 3242 views || tracked by 7 users: [display]

BENCHMARK 2020 : PPSN 2020 Workshop - Good Benchmarking Practices for Evolutionary Computation


When Sep 5, 2020 - Sep 6, 2020
Where Leiden University, The Netherlands
Submission Deadline Jun 8, 2020
Notification Due Jun 15, 2020
Final Version Due Jun 22, 2020
Categories    evolutionary algorithms   natural computing   benchmarking   artificial intelligence

Call For Papers

Workshop Description
Brace yourself for highly interactive workshop, with plenty of room for discussions and interaction. This is not just another mini-conference, but a platform to come together and to discuss recent progress and challenges in the area of benchmarking iterative optimization heuristics.
In the era of explainable and interpretable AI, it is increasingly necessary to develop a deep understanding of how algorithms work and how new algorithms compare to existing ones, both in terms of strengths and weaknesses. For this reason, benchmarking plays a vital role for understanding algorithms’ behavior. Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:

(i) most commonly-used benchmarks are too small and cover only a part of the problem space,

(ii) benchmarks lack the complexity of real-world problems, making it difficult to transfer the learned knowledge to work in practice,

(iii) we need to develop proper statistical analysis techniques that can be applied depending on the nature of the data,

(iv) we need to develop user-friendly, openly accessible benchmarking software. This enables a culture of sharing resources to ensure reproducibility, and which helps to avoid common pitfalls in benchmarking optimization techniques. As such, we need to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.

The topics of interest for this workshop include, but are not limited to:

Performance measures for comparing algorithms behavior;
Novel statistical approaches for analyzing empirical data;
Selection of meaningful benchmark problems;
Landscape analysis;
Data mining approaches for understanding algorithm behavior;
Transfer learning from benchmark experiences to real-world problems;
Benchmarking tools for executing experiments and analysis of experimental results.

The schedule will be designed to encourage a high level of interactivity — expect a real workshop rather than (yet another) mini-conference!

We particularly welcome position statements addressing or identifying open challenges in benchmarking optimization techniques. We also suggest topics for in-depth discussions. Please also consider to suggest alternative discussion formats - we want this to be a real workshop, not yet another mini-conference!

Please send your suggestions for presentations and/or discussions by e-mail to all five main responsible organizers that are listed below:

Carola Doerr (
Tome Eftimov (
Pascal Kerschke (
Pietro S. Oliveto (
Mike Preuss (

There are no format requirements. You can submit your ideas by simple mail, or as PDF. Please indicate the format of your suggested contribution (talk, discussion, breakout, brainstorming, etc.) and how much time you suggest for this activity.

Please note that PPSN workshop papers are not published in the conference proceedings. However, if you want your contribution to be listed in the conference proceedings, please submit it to us before June 8, AoE. Notification of acceptance will be on June 15. PPSN early registration deadline is on June 22.

If you do not care about being listed in the conference proceedings, you can send us your ideas/contributions/position papers/suggested activities/... any time, ideally before July 31, 2020.

Thomas Bäck (Leiden University, The Netherlands)
Thomas Bartz-Beielstein (TH Cologne, Germany)
Jakob Bossek (The University of Adelaide, Adelaide, Australia)
Bilel Derbel (University of Lille, Lille, France)
Carola Doerr (CNRS researcher at Sorbonne University, Paris, France)
Tome Eftimov (Jožef Stefan Institute, Ljubljana, Slovenia)
Pascal Kerschke (University of Münster, Germany)
William La Cava (University of Pennsylvania, USA)
Arnaud Liefooghe (University of Lille, France)
Manuel López-Ibáñez (University of Manchester, UK)
Katherine Malan (University of South Africa)
Boris Naujoks (TH Cologne, Germany)
Pietro S. Oliveto (University of Sheffield, UK)
Patryk Orzechowski (University of Pennsylvania, USA)
Mike Preuss (Leiden University, The Netherlands)
Jérémy Rapin (Facebook AI Research, Paris, France)
Ofer M. Shir (Tel-Hai College and Migal Institute, Israel)
Olivier Teytaud (Facebook AI Research, Paris, France)
Heike Trautmann (University of Münster, Germany)
Ryan J. Urbanowicz (University of Pennsylvania, USA)
Vanessa Volz (, Copenhagen, Denmark)
Markus Wagner (The University of Adelaide, Australia)
Hao Wang (LIACS, Leiden University, The Netherlands)
Thomas Weise (Institute of Applied Optimization, Hefei University, Hefei, China)
Borys Wróbel (Adam Mickiewicz University, Poland)
Aleš Zamuda (University of Maribor, Slovenia)

Related Resources

ICAISC 2024   International Conference on Artificial Intelligence and Soft Computing
JCICE 2024   2024 International Joint Conference on Information and Communication Engineering(JCICE 2024)
ECTA 2023   15th International Conference on Evolutionary Computation Theory and Applications
IEEE Xplore-Ei/Scopus-CSPIT 2023   2023 Asia Conference on Communications, Signal Processing and Information Technology (CSPIT 2023) -EI Compendex
IEA/AIE 2024   37th International Conference on Industrial, Engineering & Other Applications of Applied Intelligent Systems
LREC-COLING 2024   The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation
Bench 2023   The 15th BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing
IEEE Xplore-Ei/Scopus-DMCSE 2023   2023 International Conference on Data Mining, Computing and Software Engineering (DMCSE 2023) -EI Compendex
GECCO 2024   The Genetic and Evolutionary Computation Conference (GECCO 2024)
evostar 2024   EvoStar 2024 - The Leading European Event on Bio‑Inspired Computation