On July 8 (Cancún time, July 9 in Chinese time), our Good Benchmarking Practices for Evolutionary Computation (BENCHMARK@GECCO) workshop took place as part of the Genetic and Evolutionary Computation Conference (GECCO) in form of an online meeting. It was a great success and more than 70 international researchers from all over the world took part in it. There were many discussions and talks, which will have a lasting impact and which probably have started quite some new efforts in our community. It is clear that all of us strive for improving the way metaheuristic algorithms and optimization problems are investigated. Good research requires sound and reproducible experimentation. Our report Benchmarking in Optimization: Best Practice and Open Issues, published on July 7 on Arxiv, was one step into that direction. Together with the input from the workshop, it will further be improved. Other community efforts discussed in this workshop include the benchmarking network and the IOHprofiler. All-in-all, many questions and topics have been raised, so we now have lots of exciting work to do to answer, research, and implement them.
The program of our workshop was as follows:
- Discussion: What is the purpose of benchmarking? (Organizer: Boris Naujoks, joint work with Thomas Bartz-Beielstein and Carola Doerr, Session Chair: Tome Eftimov, slides@google / slides@iao)
- Discussion: What are good benchmarking practices / how to do benchmarking / what not to do? (Organizer: William La Cava, Session Chair: Pietro S. Oliveto, slides@google)
- Presentation: "Benchmark with Facile Adjustment of Difficulty for Many-Objective Genetic Programming and Its Reference Set"(Speaker: Makoto Ohki, Session Chair: Thomas Weise)
- Discussion: Open questions/issues in benchmarking / What to do next? (Organizer: Vanessa Volz, Session Chair: Carola Doerr, slides@google / slides@iao)