2021 Genetic and Evolutionary Computation Conference (GECCO'21)

Lille, France, July 10/11, 2021

A platform to come together and to discuss recent progress and challenges in the area of benchmarking optimization heuristics.

This workshop continued our workshop series that we started in 2020 (BENCHMARK@GECCO with >75 participants and BENCHMARK@PPSN with >90 participants). The core theme is on benchmarking evolutionary computation methods and related sampling-based optimization heuristics, but each year, we will change the focus. For 2021, we aimed to have

GECCO'21 Conference Logo

Session 1: General Aspects of Benchmarking Evolutionary Computation Methods

Benchmarking plays a vital role for understanding performance and search behavior of sampling-based optimization techniques such as evolutionary algorithms. Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:

  1. most commonly-used benchmarks are too small and cover only a part of the problem space,
  2. benchmarks lack the complexity of real-world problems, making it difficult to transfer the learned knowledge to work in practice,
  3. we need to develop proper statistical analysis techniques that can be applied depending on the nature of the data,
  4. we need to develop user-friendly, openly accessible benchmarking software. This enables a culture of sharing resources to ensure reproducibility, and which helps to avoid common pitfalls in benchmarking optimization techniques. As such, we need to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.

The topics of interest for this session of the workshop include, but are not limited to:

  • Performance measures for comparing algorithms behavior;
  • Novel statistical approaches for analyzing empirical data;
  • Selection of meaningful benchmark problems;
  • Landscape analysis;
  • Data mining approaches for understanding algorithm behavior;
  • Transfer learning from benchmark experiences to real-world problems;
  • Benchmarking tools for executing experiments and analysis of experimental results.

Session 2: Understanding Reproducibility in Evolutionary Computation

Experimental studies are prevalent in Evolutionary Computation (EC), and concerns about the reproducibility and replicability of such studies has increased in recent years, following similar discussions in other scientific fields. In this workshop, we want to raise awareness of the reproducibility issue, shed light on the obstacles when trying to reproduce results, and discuss best practices in making results reproducible as well as reporting reproducibility results.

We invite submissions of papers repeating an empirical study published in a journal or conference proceedings, either by re-using, updating or reimplementing the necessary codes and datasets, irrespectively of whether this code was published in some form at the time or not.

The original study being reproduced should not be so recent as to make the reproduction attempt trivial. Ideally, we suggest looking at studies that are at least 10 years old. However, one of the criteria for acceptance is what can the GECCO community learn from the reproducibility study.

At least one of the co-authors of the submitted paper should be one of the co-authors of the original study. This condition makes sure that the reproducibility attempt is a fair attempt at reproducing the original work.


The workshop took place on Sunday, July 11, 2021 at 11am with more than 75 attendants.

  • 11:00-11:10: Welcome and Opening.
  • 11:10-11:40: Workshop Keynote Relevance of Benchmarking for Industry or "What does it take for an algorithm to be applied in industrial contexts?" by Bernhard Sendhoff.
  • 11:40-12:10: Workshop Keynote Benchmarking in Robotics by Emma Hart and Gusz Eiben (presented by Emma Hart).
  • 12:10-12:50: Open Discussion.


  • benchmarking_01
  • benchmarking_02
  • benchmarking_03
  • benchmarking_04
  • benchmarking_05
  • benchmarking_06
  • benchmarking_07

Important Dates

Paper Submission Opening:  11  February  2021
Paper Submission Deadline:  12  April  2021
Notification of Acceptance:  26  April  2021
Camera-Ready Copy Due:  3  May  2021
Author Registration:  3  May  2021
Conference Presentation:  11  July  2021

Instructions for Authors

All relevant instructions regarding paper submission are available at https://gecco-2021.sigevo.org/Call-for-Workshop-Papers.


Hosting Event

The 2021 Genetic and Evolutionary Computation Conference (GECCO'21), Lille, France, July 10-14, 2021.

The Genetic and Evolutionary Computation Conference (GECCO) presents the latest high-quality results in genetic and evolutionary computation since 1999. Topics include: genetic algorithms, genetic programming, ant colony optimization and swarm intelligence, complex systems (artificial life, robotics, evolvable hardware, generative and developmental systems, artificial immune systems), digital entertainment technologies and arts, evolutionary combinatorial optimization and metaheuristics, evolutionary machine learning, evolutionary multiobjective optimization, evolutionary numerical optimization, real world applications, search-based software engineering, theory and more.

Current Related Events

Past Related Events