Call for Papers Call for Papers

2021 IEEE Congress on Evolutionary Computation (CEC 2021)

June 28-July 1, 2021 in Kraków, Poland


The Special Session on Benchmarking of Computational Intelligence Algorithms (BOCIA), as part of the 2021 IEEE Congress on Evolutionary Computation (CEC 2021), is cordially inviting the submission of original and unpublished research papers. Here you can download the BOCIA Special Session Call for Papers (CfP) in PDF format and here as plain text file.

Computational Intelligence (CI), including Evolutionary Computation, Optimization, Machine Learning, and Artificial Intelligence, is a huge and expanding field which is rapidly gaining importance, attracting more and more interests from both academia and industry. It includes a wide and ever-growing variety of optimization and machine learning algorithms, which, in turn, are applied to an even wider and faster growing range of different problem domains. For all of these domains and application scenarios, we want to pick the best algorithms. Actually, we want to do more, we want to improve upon the best algorithm. This requires a deep understanding of the problem at hand, the performance of the algorithms we have for that problem, the features that make instances of the problem hard for these algorithms, and the parameter settings for which the algorithms perform the best. Such knowledge can only be obtained empirically, by collecting data from experiments, by analyzing this data statistically, and by mining new information from it. Benchmarking is the engine driving research in the fields of Computational Intelligence for decades, while its potential has not been fully explored.

The goal of this special session is to solicit original works on the research in benchmarking: Works which contribute to the domain of benchmarking of algorithms from all fields of Computational Intelligence, by adding new theoretical or practical knowledge. Papers which only apply benchmarking are not in the scope of the special session.

This special session is technically supported by the IEEE CIS Task Force on Benchmarking.

CEC 2021 Conference Logo

1. Topics of Interest

This special session wants to bring together experts on benchmarking of Evolutionary Computation, Machine Learning, Optimization, and Artificial Intelligence. It provides a common forum for them to exchange findings, to explore new paradigms for performance comparison, and to discuss issues such as

  • mining of higher-level information from experimental results
  • modelling and visualization of algorithm behaviors and performance
  • statistics for performance comparison (robust statistics, PCA, ANOVA, statistical tests, ROC, …)
  • evaluation of real-world goals such as robustness, reliability, and implementation issues
  • theoretical results for algorithm performance comparison
  • comparison of theoretical and empirical results
  • new benchmark problems
  • automatic algorithm configuration and selection
  • the comparison of algorithms in "non-traditional" scenarios such as multi- or many-objective domains, parallel implementations, e.g., using GGPUs, MPI, CUDA, clusters, or running in clouds, large-scale problems or problems where objective function evaluations are costly, dynamic problems or where the objective functions involve randomized simulations or noise, deep learning, big data
  • comparative surveys with new ideas on dos and don'ts, i.e., best and worst practices, for algorithm performance comparison; tools for experiment execution, result collection, and algorithm comparison

All accepted papers in this session will be included in the Proceedings of the 2021 IEEE Congress on Evolutionary Computation published by IEEE Press and indexed by EI.

2. Important Dates

Paper Submission Deadline:  21  February  2021
Notification of Acceptance:  6  April  2021
Camera-Ready Copy Due:  23  April  2021
Author Registration:  23  April  2021
Conference Presentation:  28  June to  1  July  2021

More information can be found in the Important Dates section of the CEC 2021 website.

3. Instructions for Authors

All papers submitted to our special session must follow the CEC 2021 submission guidelines.

Each paper must be in PDF format and should have 6 to 8 pages, including figures, tables, and references. The style files for IEEE Conference Proceedings – U.S. letter size – should be used as templates for your submission (LaTeX or Word).

All papers must be submitted through the IEEE CEC 2021 online submission system (

Special session papers are treated the same as regular conference papers. Make sure to specify that your paper is for "SS-18. Benchmarking of Computational Intelligence Algorithms (BOCIA)".

A screenshot of the CEC submission system.

In order to participate in this special session, full or student registration of CEC 2021 is needed. All papers accepted and presented at CEC 2021 will be included in the conference proceedings published by IEEE Explore.

4. Chairs

  • Aleš Zamuda, University of Maribor, Slovenia
  • Thomas Weise, Institute of Applied Optimization, Hefei University, Hefei, China
  • Markus Wagner, University of Adelaide, Adelaide, SA, Australia

5. International Program Committee

6. Chair Biographies

Portrait of Aleš Zamuda

Dr. Aleš Zamuda is an Associate Professor and Senior Research Associate at University of Maribor (UM), Slovenia. He received Ph.D. (2012), M.Sc. (2008), and B.Sc. (2006) degrees in computer science from UM. He is management committee (MC) member for Slovenia at European Cooperation in Science (COST), actions CA15140 (ImAppNIO - Improving Applicability of Nature-Inspired Optimisation by Joining Theory and Practice) and IC1406 (formerly, cHiPSet - High-Performance Modelling and Simulation for Big Data Applications), and H2020 DAPHNE (Integrated Data Analysis Pipelines for Large-Scale Data Management, HPC, and Machine Learning) project General Assembly Member. He is IEEE Senior Member, IEEE Task Force on Benchmarking vicechair, IEEE Slovenia Section vicechair and IEEE Slovenia CIS chair, and member of IEEE CIS, IEEE GRSS, IEEE OES, ACM, and ACM SIGEVO. He is also vicechair for working groups on Benchmarking at ImAppNIO and DAPHNE and an editorial board member (associate editor) for Swarm and Evolutionary Computation (2019 IF=6.912). His areas of computer science applications include differential evolution, multiobjective optimization, evolutionary robotics, artificial life, and cloud computing; currently yielding h-index 20, 50 publications, and 1236 citations on Scopus. He won IEEE R8 SPC 2007 award, IEEE CEC 2009 ECiDUE, 2016 Danubuius Young Scientist Award, and 1% top reviewer at 2017 and 2018 Publons Peer Review Awards, including reviews for over 50 journals and 120 conferences.

Portrait of Prof. Dr. Thomas Weise

Prof. Dr. Thomas Weise obtained the MSc in Computer Science in 2005 from the Chemnitz University of Technology and his PhD from the University of Kassel in 2009. He then joined the University of Science and Technology of China (USTC) as PostDoc and subsequently became Associate Professor at the USTC-Birmingham Joint Research Institute in Intelligent Computation and Its Applications (UBRI) at USTC. In 2016, he joined Hefei University as Full Professor to found the Institute of Applied Optimization (IAO) of the School of Artificial Intelligence and Big Data. Prof. Weise has more than a decade of experience as a full time researcher in China, having contributed significantly both to fundamental as well as applied research. He has published more than 90 scientific papers in international peer reviewed journals and conferences. His book "Global Optimization Algorithms – Theory and Application" has been cited more than 900 times. He is Member of the Editorial Board of the Applied Soft Computing Journal and has acted as reviewer, editor, or program committee member at 80 different venues.

Portrait of Senior Lecturer Dr. Markus Wagner

Dr. Markus Wagner is a Senior Lecturer at the School of Computer Science, University of Adelaide, Australia. He has done his PhD studies at the Max Planck Institute for Informatics in Saarbrücken, Germany and at the University of Adelaide, Australia. His research topics range from mathematical runtime analysis of heuristic optimization algorithms and theory-guided algorithm design to applications of heuristic methods to renewable energy production, professional team cycling and software engineering. So far, he has been a program committee member 30 times, and he has written over 70 articles with over 70 different co-authors. He has chaired several education-related committees within the IEEE CIS, is Co-Chair of ACALCI 2017 and General Chair of ACALCI 2018.


7. Hosting Event

The 2021 IEEE Congress on Evolutionary Computation (CEC 2021), June 28-July 1, 2021 in Kraków, Poland.

IEEE CEC 2021 is a world-class conference that brings together researchers and practitioners in the field of evolutionary computation and computational intelligence from around the globe. The Congress will encompass keynote lectures, regular and special sessions, tutorials, and competitions, as well as poster presentations. In addition, participants will be treated to a series of social functions, receptions, and networking events.

IEEE CEC 2021 will be held in Kraków, Poland. A pearl amongst Polish cities, Kraków is one of the architectural treasures in the world on the UNESCO World Cultural Heritage List.

8. Links and News

9. Current Related Events

10. Past Related Events