Good Benchmarking Practices for Evolutionary Computation (BENCHMARKING'21)

This workshop continued our workshop series that we started in 2020 (BENCHMARK@GECCO with >75 participants and BENCHMARK@PPSN with >90 participants). The core theme is on benchmarking evolutionary computation methods and related sampling-based optimization heuristics, but each year, we will change the focus. For 2021, we aimed to have

• one session on general aspects of benchmarking (with 2-3 invited speakers and ample time for actual discussion)
• one session on reproducibility (see below for more details)

Session 1: General Aspects of Benchmarking Evolutionary Computation Methods

Benchmarking plays a vital role for understanding performance and search behavior of sampling-based optimization techniques such as evolutionary algorithms. Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:

1. most commonly-used benchmarks are too small and cover only a part of the problem space,
2. benchmarks lack the complexity of real-world problems, making it difficult to transfer the learned knowledge to work in practice,
3. we need to develop proper statistical analysis techniques that can be applied depending on the nature of the data,
4. we need to develop user-friendly, openly accessible benchmarking software. This enables a culture of sharing resources to ensure reproducibility, and which helps to avoid common pitfalls in benchmarking optimization techniques. As such, we need to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.

The topics of interest for this session of the workshop include, but are not limited to:

• Performance measures for comparing algorithms behavior;
• Novel statistical approaches for analyzing empirical data;
• Selection of meaningful benchmark problems;
• Landscape analysis;
• Data mining approaches for understanding algorithm behavior;
• Transfer learning from benchmark experiences to real-world problems;
• Benchmarking tools for executing experiments and analysis of experimental results.

Session 2: Understanding Reproducibility in Evolutionary Computation

Experimental studies are prevalent in Evolutionary Computation (EC), and concerns about the reproducibility and replicability of such studies has increased in recent years, following similar discussions in other scientific fields. In this workshop, we want to raise awareness of the reproducibility issue, shed light on the obstacles when trying to reproduce results, and discuss best practices in making results reproducible as well as reporting reproducibility results.

We invite submissions of papers repeating an empirical study published in a journal or conference proceedings, either by re-using, updating or reimplementing the necessary codes and datasets, irrespectively of whether this code was published in some form at the time or not.

The original study being reproduced should not be so recent as to make the reproduction attempt trivial. Ideally, we suggest looking at studies that are at least 10 years old. However, one of the criteria for acceptance is what can the GECCO community learn from the reproducibility study.

At least one of the co-authors of the submitted paper should be one of the co-authors of the original study. This condition makes sure that the reproducibility attempt is a fair attempt at reproducing the original work.

Program

The workshop took place on Sunday, July 11, 2021 at 11am with more than 75 attendants.

• 11:00-11:10: Welcome and Opening.
• 11:10-11:40: Workshop Keynote Relevance of Benchmarking for Industry or "What does it take for an algorithm to be applied in industrial contexts?" by Bernhard Sendhoff.
• 11:40-12:10: Workshop Keynote Benchmarking in Robotics by Emma Hart and Gusz Eiben (presented by Emma Hart).
• 12:10-12:50: Open Discussion.

Photos

Important Dates

Instructions for Authors

All relevant instructions regarding paper submission are available at https://gecco-2021.sigevo.org/Call-for-Workshop-Papers.

Chairs

• Jürgen Branke, University of Warwick, UK
• Carola Doerr, CNRS researcher at Sorbonne University, Paris, France
• Tome Eftimov, Jožef Stefan Institute, Ljubljana, Slovenia
• Pascal Kerschke, Westfälische Wilhelms-Universität Münster, Münster, Germany
• Manuel López-Ibáñez, University of Málaga, Spain and University of Manchester, Manchester, UK
• Boris Naujoks, TH Köln, Cologne, Germany
• Luís Paquete, University of Coimbra, Portugal
• Vanessa Volz, modl.ai, Copenhagen, Denmark
• Thomas Weise, Institute of Applied Optimization, Hefei University, Hefei, China

Hosting Event

The 2021 Genetic and Evolutionary Computation Conference (GECCO'21), Lille, France, July 10-14, 2021.

The Genetic and Evolutionary Computation Conference (GECCO) presents the latest high-quality results in genetic and evolutionary computation since 1999. Topics include: genetic algorithms, genetic programming, ant colony optimization and swarm intelligence, complex systems (artificial life, robotics, evolvable hardware, generative and developmental systems, artificial immune systems), digital entertainment technologies and arts, evolutionary combinatorial optimization and metaheuristics, evolutionary machine learning, evolutionary multiobjective optimization, evolutionary numerical optimization, real world applications, search-based software engineering, theory and more.

Current Related Events

• Analysing Algorithmic Behaviour of Optimisation Heuristics Workshop (AABOH'21)
• Special Session on Benchmarking of Computational Intelligence Algorithms (BOCIA'21)
• A similar benchmarking best practices workshop will be held at CEC 2021, which takes place from June 28 - July 1, 2021, in Kraków, Poland: Benchmarking@CEC-2021.

Past Related Events

• Special Issue on Benchmarking of Computational Intelligence Algorithms in the Applied Soft Computing Journal (2018-2020, Completed)
• 2020 Good Benchmarking Practices for Evolutionary Computation (BENCHMARK@PPSN'20) Workshop
• 2020 Good Benchmarking Practices for Evolutionary Computation (BENCHMARK@GECCO'20) Workshop
• 2019 Black Box Discrete Optimization Benchmarking (BB-DOB'19) Workshop
• 2019 Special Session on Benchmarking of Evolutionary Algorithms for Discrete Optimization (BEADO'19)
• 2018 Black-Box Discrete Optimization Benchmarking (BB-DOB@PPSN'18) Workshop
• 2018 Black-Box Discrete Optimization Benchmarking (BB-DOB@GECCO'18) Workshop
• 2018 International Workshop on Benchmarking of Computational Intelligence Algorithms (BOCIA'18)