Good Benchmarking Practices for Evolutionary Computation (BENCHMARK@PPSN) Workshop

at the Sixteenth International Conference on Parallel Problem Solving from Nature (PPSN XVI)

September 5-9, 2020 in Leiden, The Netherlands

https://sites.google.com/view/benchmarking-network/home/PPSN20
http://iao.hfuu.edu.cn/benchmark-ppsn20

The Good Benchmarking Practices for Evolutionary Computation Workshop (BENCHMARK@PPSN), a part of the Sixteenth International Conference on Parallel Problem Solving from Nature (PPSN XVI), is cordially inviting the submission of contributions. Here you can download the BB-DOB@PPSN Workshop Call for Papers (CfP) in PDF format and here as plain text file.

In the era of explainable and interpretable AI, it is increasingly necessary to develop a deep understanding of how algorithms work and how new algorithms compare to existing ones, both in terms of strengths and weaknesses. For this reason, benchmarking plays a vital role for understanding algorithms’ behavior. Even though benchmarking is a highly-researched topic within the evolutionary computation community, there are still a number of open questions and challenges that should be explored:

most commonly-used benchmarks are too small and cover only a part of the problem space,
benchmarks lack the complexity of real-world problems, making it difficult to transfer the learned knowledge to work in practice,
we need to develop proper statistical analysis techniques that can be applied depending on the nature of the data, and
we need to develop user-friendly, openly accessible benchmarking software.

This enables a culture of sharing resources to ensure reproducibility, and which helps to avoid common pitfalls in benchmarking optimization techniques. As such, we need to establish new standards for benchmarking in evolutionary computation research so we can objectively compare novel algorithms and fully demonstrate where they excel and where they can be improved.

The topics of interest for this workshop include, but are not limited to:

performance measures for comparing algorithms behavior,
novel statistical approaches for analyzing empirical data,
the selection of meaningful benchmark problems,
landscape analysis,
data mining approaches for understanding algorithm behavior,
transfer learning from benchmark experiences to real-world problems, and
benchmarking tools for executing experiments and analysis of experimental results.

Schedule

The workshop takes place Saturday, September 5, 16:00-17:30 (NL time zone). The workshop will be hybrid: onsite and online. Online participation will be possible for everyone, not only for people registered at PPSN! The schedule is as follows:

General introduction to the workshop [5 minutes]
Reproducibility, Replicability, etc [10 minutes + 15 minutes for discussion], host: Manuel López-Ibáñez
Performance Measures [10 minutes + 15 minutes for discussion], host: Jakob Bossek, building on chapter of the survey
Fairly Assessing Algorithms' Contribution to the State of the Art or how to incentivize the design of highly specialized algorithms [10 minutes + 15 minutes for discussion], host: Holger Hoos
Ongoing initiatives and how to get involved [10 minutes], host: Carola Doerr

This workshop is organized as part of the ImAppNIO Cost Action 15140.

Travel Grants

For colleagues affiliated with an institution in the following countries, ITC conference grants are available through COST action CA15140: Improving Applicability of Nature-Inspired Optimisation by Joining Theory and Practice (ImAppNIO): <https://imappnio.dcs.aber.ac.uk/stsms>. The grants can cover full travel cost to attend PPSN or the registration fees for online participation. Details: <https://imappnio.dcs.aber.ac.uk/itc-conference-grants> Eligible countries: Albania, Bosnia-Herzegovina, Bulgaria, Cyprus, Czech Republic, Estonia, Croatia, Hungary, Lithuania, Latvia, Luxembourg, Malta, Moldova, Montenegro, Poland, Portugal, Romania, Slovenia, Slovakia, Republic of North Macedonia, Republic of Serbia and Turkey.

Instruction for Authors

We particularly welcome position statements addressing or identifying open challenges in benchmarking optimization techniques. We also suggest topics for in-depth discussions. Please also consider to suggest alternative discussion formats — we want this to be a real workshop, not yet another mini-conference!

Please send your suggestions for presentations and/or discussions by e-mail to Carola.Doerr@mpi inf.mpg.de with CC to This email address is being protected from spambots. You need JavaScript enabled to view it., This email address is being protected from spambots. You need JavaScript enabled to view it., and This email address is being protected from spambots. You need JavaScript enabled to view it..

There are no format requirements. You can submit your ideas by simple mail, or as PDF. Please indicate the format of your suggested contribution (talk, discussion, breakout, brainstorming, etc.) and how much time you suggest for this activity.

Please note that PPSN workshop papers are not published in the conference proceedings. However, if you want your contribution to be listed in the conference proceedings, please submit it to us before June 8, anywhere on Earth. Notification of acceptance will be on June 15. At least one author from each accepted contribution must register for the conference by June 22, pay the conference fee, and be present at the workshop. The conference takes place from September 5 to 6, 2020.

If you do not care about being listed in the conference proceedings, you can send us your ideas/contributions/position papers/suggested activities/… any time, ideally before July 31, 2020.

For more information please contact Carola Doerr at This email address is being protected from spambots. You need JavaScript enabled to view it..

Workshop History

This workshop emerged from an initiative at GECCO 2019, which was launched to consolidate the various activities around benchmarking. This was also preceded by the BB-DOB@PPSN, BB-DOB@GECCO, and BOCIA workshops in 2018 as well as the Benchmarking of Computational Intelligence Algorithms special issue in ASOC. We are proud that our workshop is organized by a large number of colleagues who jointly cover a broad spectrum of benchmarking aspects. Most of the organizers are experienced in workshop organization, and will support the junior fellow in making their first experiences.

List of Organizers

The workshop is co-organized by the following people (alphabetically ordered list). All of them are contributing equally to the workshop.

Thomas Bäck, Leiden University, Leiden, The Netherlands
Thomas Bartz-Beielstein, Technical University of Cologne, Köln(Cologne), Germany
Jakob Bossek, The University of Adelaide, Adelaide, Australia
Bilel Derbel, University of Lille, Lille, France
Carola Doerr, CNRS and Sorbonne University, Paris, France
Tome Eftimov, Stanford University, CA, USA and Jožef Stefan Institute, Ljubljana, Slovenia
Pascal Kerschke, University of Münster, Münster, Germany
William La Cava, University of Pennsylvania, Philadelphia, PA, USA
Arnaud Liefooghe, University of Lille, Lille, France
Manuel López-Ibáñez, University of Manchester, Manchester, UK
Boris Naujoks, Technical University of Cologne, Köln (Cologne), Germany
Pietro S. Oliveto, University of Sheffield, Sheffield, UK
Patryk Orzechowski, University of Pennsylvania, Philadelphia, PA, USA
Mike Preuss, LIACS, Leiden University, Leiden, The Netherlands
Jérémy Rapin, Facebook AI Research, Paris, France
Ofer M. Shir, Tel-Hai College and Migal Institute, Israel
Olivier Teytaud, Facebook AI Research, Paris, France
Heike Trautmann, University of Münster, Münster, Germany
Ryan J. Urbanowicz, University of Pennsylvania, Philadelphia, PA, USA
Vanessa Volz, modl.ai, Copenhagen, Denmark
Markus Wagner, The University of Adelaide, Adelaide, Australia
Hao Wang, LIACS, Leiden University, Leiden, The Netherlands
Thomas Weise, Institute of Applied Optimization, Hefei University, Hefei, China
Borys Wróbel, Adam Mickiewicz University, Poland
Aleš Zamuda, University of Maribor, Maribor, Slovenia

Hosting Event

The Sixteenth International Conference on Parallel Problem Solving from Nature (PPSN XVI)
September 5-9, 2020, Leiden, The Netherlands
https://ppsn2020.liacs.leidenuniv.nl/

The Sixteenth International Conference on Parallel Problem Solving from Nature (PPSN XVI) will be held in Leiden, The Netherlands on September 5-9, 2020. Leiden University and the Leiden Institute of Advanced Computer Science (LIACS) are proud to host the 30th anniversary of PPSN.

PPSN brings together researchers and practitioners in the field of Natural Computing, the study of computing approaches which are gleaned from natural models. The field includes, but is not limited to, areas such as Amorphous Computing, Artificial Life, Artificial Ant Systems, Artificial Immune Systems, Artificial Neural Networks, Cellular Automata, Evolutionary Computation, Swarm Computing, Self-Organizing Systems, Chemical Computation, Molecular Computation, Quantum Computation, Machine Learning and Artificial Intelligence approaches using Natural Computing methods.

PPSN XVI will feature workshops and tutorials covering advanced and fundamental topics in the field of Natural Computing, as well as algorithm competitions. The keynote talks will be given by the world- renowned researchers in their fields.

Following PPSN’s unique tradition, all accepted papers will be presented during poster sessions and will be included in the proceedings. The proceedings will be published in the Lecture Notes in Computer Science (LNCS) series by Springer. Prospective authors are invited to contribute their high-quality original results in the field of natural computation as papers of no more than 12 pages plus references. The format follows Springer Verlag LNCS style.

https://ppsn2020.liacs.leidenuniv.nl/