at the Genetic and Evolutionary Computation Conference (GECCO 2020)
July 8-12, 2020, Cancún, Quintana Roo, Mexico
The Good Benchmarking Practices for Evolutionary Computation (BENCHMARK@GECCO) Workshop, a part of the Genetic and Evolutionary Computation Conference (GECCO) 2020, is cordially inviting contributions and the submission of original and unpublished research papers. Here you can download the BB-DOB@gecco Workshop Call for Papers (CfP) in PDF format and here as plain text file.
In the era of explainable and interpretable AI, it is increasingly necessary to develop a deep understanding of how algorithms work and how new algorithms compare to existing ones, both in terms of strengths and weaknesses. For this reason, benchmarking plays a vital role for understanding algorithms’ behavior. Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:
- most commonly-used benchmarks are too small and cover only a part of the problem space,
- benchmarks lack the complexity of real-world problems,
- we need to develop proper statistical analysis techniques that can be applied depending on the nature of the data, and
- we need to develop user-friendly, openly accessible benchmarking software.
This enables a culture of sharing resources to ensure reproducibility, and which helps to avoid common pitfalls in benchmarking optimization techniques. As such, we need to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.
The topics of interest for this workshop include, but are not limited to:
- performance measures for comparing algorithms behavior,
- novel statistical approaches for analyzing empirical data,
- the selection of meaningful benchmark problems,
- landscape analysis,
- data mining approaches for understanding algorithm behavior,
- transfer learning from benchmark experiences to real-world problems, and
- benchmarking tools for executing experiments and analysis of experimental results.
We particularly welcome position statements addressing or identifying open challenges in benchmarking optimization techniques. We also suggest topics for in-depth discussions. Please also consider to suggest alternative discussion formats - we want this to be a real workshop, not yet another mini-conference!
Papers can be up to 8 pages long, in ACM format and excluding references. We emphasize that we also welcome short position statements, which can be as short as one abstract. The schedule will be designed to encourage a high level of interactivity.
1. Important Dates
|Paper Submission Opening:||27||February||2020|
|Paper Submission Deadline:||3||April||2020|
|Camera-Ready Material Due:||24||April||2020|
|Author Registration Deadline:||24||April||2020|
Submission Page: https://ssl.linklings.net/conferences/gecco/
2. Instructions for Authors
Instructions regarding how to submit papers and the paper format are given at https://gecco-2020.sigevo.org/index.html/tiki-index.php?page=Workshops. Consider:
- The maximum paper length is eight pages (excluding references).
- The maximum abstract length is 200 words.
- Submitted papers must be anonymized in order to facilitate a double-blind review process.
- Follow https://gecco-2020.sigevo.org/index.html/Papers+Submission+Instructions.
- In the GECCO submission page, select "Workshop Paper" and in the field "Workshop" of the next form, select "Workshop Black Box Discrete Optimization Benchmarking".
- At least one author from each accepted paper must register for the conference by April 24, 2020, pay the conference fee, and be present at the conference to give an oral presentation.
3. Workshop History
This workshop emerged from an initiative at GECCO 2019, which was launched to consolidate the various activities around benchmarking. This was also preceded by the BB-DOB@PPSN, BB-DOB@GECCO, and BOCIA workshops in 2018 as well as the Benchmarking of Computational Intelligence Algorithms special issue in ASOC. We are proud that our workshop is organized by a large number of colleagues who jointly cover a broad spectrum of benchmarking aspects. Most of the organizers are experienced in workshop organization, and will support the junior fellow in making their first experiences.
4. List of Organizers
The workshop is co-organized by the following people (alphabetically ordered list). All of them are contributing equally to the workshop.
- Thomas Bäck, Leiden University, Leiden, The Netherlands
- Carola Doerr, CNRS and Sorbonne University, Paris, France
- Tome Eftimov, Stanford University, CA, USA and Jožef Stefan Institute, Ljubljana, Slovenia
- Pascal Kerschke, University of Münster, Münster, Germany
- William La Cava, University of Pennsylvania, Philadelphia, PA, USA
- Manuel López-Ibáñez, University of Manchester, Manchester, UK
- Boris Naujoks, Technical University of Cologne, Köln (Cologne), Germany
- Pietro S. Oliveto, University of Sheffield, Sheffield, UK
- Patryk Orzechowski, University of Pennsylvania, Philadelphia, PA, USA
- Mike Preuss, LIACS, Leiden University, Leiden, The Netherlands
- Jérémy Rapin, Facebook AI Research, Paris, France
- Ofer M. Shir, Tel-Hai College and Migal Institute, Israel
- Olivier Teytaud, Facebook AI Research, Paris, France
- Heike Trautmann, University of Münster, Münster, Germany
- Ryan J. Urbanowicz, University of Pennsylvania, Philadelphia, PA, USA
- Vanessa Volz, modl.ai, Copenhagen, Denmark
- Markus Wagner, The University of Adelaide, Adelaide, Australia
- Hao Wang, LIACS, Leiden University, Leiden, The Netherlands
- Thomas Weise, Institute of Applied Optimization, Hefei University, Hefei, China
- Borys Wróbel, Adam Mickiewicz University, Poland
- Aleš Zamuda, University of Maribor, Maribor, Slovenia
5. Hosting Event
The Genetic and Evolutionary Computation Conference (GECCO 2020) will present the latest high-quality results in genetic and evolutionary computation. Topics include genetic algorithms, genetic programming, evolution strategies, evolutionary programming, memetic algorithms, hyper-heuristics, real-world applications, evolutionary machine learning, evolvable hardware, artificial life, adaptive behavior, ant colony optimization, swarm intelligence, biological applications, evolutionary robotics, coevolution, artificial immune systems, and more. The full list of tracks is available at: Program Tracks
The GECCO 2020 Program Committee invites the submission of technical papers describing your best work in genetic and evolutionary computation. Full papers of at most 8 pages (excluding references) should present original work that meets the high-quality standards of GECCO. Accepted full papers appear in the ACM digital library as part of the Main Proceedings of GECCO. For full papers, a separate abstract needs to be submitted first by January 30, 2020. Full papers are due by the non-extensible deadline of February 6, 2020.
Each paper submitted to GECCO will be rigorously evaluated in a double-blind review process. Evaluation is done on a per-track basis, ensuring high interest and high expertise of the reviewers. Review criteria include the significance of the work, technical soundness, novelty, clarity, writing quality, relevance and, if applicable, sufficiency of information to permit replication.
Besides full papers, poster-only papers of at most 2 pages may be submitted. Poster-only papers should present original work that has not yet reached the maturity and completeness of research results that are published as full papers at GECCO. The review of poster-only papers follows the same double-blind process described above. Accepted poster-only papers will appear in the ACM digital library as part of the Companion Proceedings of GECCO. Poster-only papers are due by the non-extensible deadline of February 6, 2020, and no abstract needs to be submitted first.
By submitting a paper, the author(s) agree that, if their paper is accepted, they will:
- Submit a final, revised, camera-ready version to the publisher on or before the camera-ready deadline
- Register at least one author before April 17, 2020 to attend the conference
- Attend the conference (at least one author)
- Present the accepted paper at the conference
6. Related Events
- Good Benchmarking Practices for Evolutionary Computation (BENCHMARK@PPSN) Workshop
- Black Box Discrete Optimization Benchmarking (BB-DOB) Workshop
- Black-Box Discrete Optimization Benchmarking (BB-DOB@PPSN) Workshop
- Black-Box Discrete Optimization Benchmarking (BB-DOB@GECCO) Workshop
- Special Issue on Benchmarking of Computational Intelligence Algorithms in the Applied Soft Computing Journal
- International Workshop on Benchmarking of Computational Intelligence Algorithms (BOCIA)