From November 9 to 13, 2020, the Lorentz Center Workshop "Benchmarked: Optimization Meets Machine Learning" is jointly organized by Carola Doerr, Thomas Stützle, Mike Preuss, Marc Schoenauer, and Joaquin Vanschoren as an online event. It brings together experts from all fields of benchmarking and automated algorithm configuration and selection with focus on optimization and had more than 100 registered participants. In particular, the Benchmarked: Optimization Meets Machine Learning workshop, the goal is to discuss the impact of automated decision-making on heuristic optimization. More specifically, it is discussed how the possibility to automatically select and configure optimization heuristics changes the requirements for their benchmarking. The key objectives of this Lorentz Center workshop are:

  • to develop a joint vision on the next generation of benchmarking optimization heuristics in the context of automated algorithm selection and configuration, and
  • to design a clear road-map guiding the research community towards this vision.

It is discussed what an ideal benchmarking environment would look like, how such an "ideal tool" compares to existing software, and how we can close the gap by improving the compatibility between ongoing and future projects. The aim is to designing a full benchmarking engine that ranges from modular algorithm frameworks over problem instance generators and landscape analysis tools to automated algorithm configuration and selection techniques, all the way to a statistically sound evaluation of the experimental data.

In this setting, Prof. Thomas Weise organized a first and co-organizes a second breakout session on "Data Formats for Benchmarking." The rationale behind these specific sessions is that technical details are often ignored in research. However, technicalities such as the data format used for storing experimental results can nevertheless have a big influence on our research. The data format determines what information will be available after experiment. This includes what information is available for evaluation. But it also determines whether the experiment will be easy to replicate or whether the results can be validated. It also determines which tools we can use for evaluating the results. The data format may even determine how we can execute an experiment (in parallel? in a distributed fashion? can experiments be restarted?). The goal of the breakout session is to collect thoughts and ideas about suitable data formats for storing the output of experiments in optimization and machine learning. The aim is to collect a set of requirements for a good format and structure. If these are well understood, it may be possible to eventually define a simple and clear standard for the future – or at least some guidelines that can help researchers to not miss any detail that should be considered when storing experimental data.

Call for Papers Call for Papers

2021 IEEE Congress on Evolutionary Computation (CEC 2021)

June 28-July 1, 2021 in Kraków, Poland


The Special Session on Benchmarking of Computational Intelligence Algorithms (BOCIA), as part of the 2021 IEEE Congress on Evolutionary Computation (CEC 2021), took place on June 30, 2021 at 10am. Here you can download the BOCIA Special Session Call for Papers (CfP) in PDF format and here as plain text file.

Computational Intelligence (CI), including Evolutionary Computation, Optimization, Machine Learning, and Artificial Intelligence, is a huge and expanding field which is rapidly gaining importance, attracting more and more interests from both academia and industry. It includes a wide and ever-growing variety of optimization and machine learning algorithms, which, in turn, are applied to an even wider and faster growing range of different problem domains. For all of these domains and application scenarios, we want to pick the best algorithms. Actually, we want to do more, we want to improve upon the best algorithm. This requires a deep understanding of the problem at hand, the performance of the algorithms we have for that problem, the features that make instances of the problem hard for these algorithms, and the parameter settings for which the algorithms perform the best. Such knowledge can only be obtained empirically, by collecting data from experiments, by analyzing this data statistically, and by mining new information from it. Benchmarking is the engine driving research in the fields of Computational Intelligence for decades, while its potential has not been fully explored.

The goal of this special session was to solicit original works on the research in benchmarking: Works which contribute to the domain of benchmarking of algorithms from all fields of Computational Intelligence, by adding new theoretical or practical knowledge. Papers which only apply benchmarking are not in the scope of the special session.

This special session was technically supported by the IEEE CIS Task Force on Benchmarking.

Our team warmly welcomes Mrs. Yuanyuan LEI [雷园园], Mr. Zhiyang LIU [刘志洋] , Mr. Peng GAO [高鹏], and Mrs. Shuoyi RAN [冉烁依] to join our team as graduate students for the period from 2020 to 2023. Mr. LEI and Mrs. GAO will be supervised by Assoc. Prof. Dr. Xinlu Li, whereas Mr. LIU and Mrs. RAN will be jointly co-supervised by Prof. Dr. Thomas Weise and Assoc. Prof. Dr. Zhize Wu. We are looking forward to working with you. Welcome aboard!

Today, on September 29, 2020, I attended the Mid-Autumn Tea Party for People from All Walks of Life [合肥市各界人士国庆中秋茶话会] in the city hall of Hefei [合肥市]. Like last year’s celebration of the 70th Anniversary of the People's Republic of China, it was a very nice event. As the name suggests, the event was attended by people from a variety of different career paths, including business representatives, representatives from several different religions, party members, military leaders, researchers, and representatives of the international community of our city. The event began with four ceremonial speeches – the first of which was held by Prof. Dr. Chunmei WU [吴春梅], the president of our Hefei University [合肥学院院长]. Then, a very nice program with different songs followed. There were several professional opera singers giving great performances. I found the performance of the CPCC youth chorus especially nice, because they are normal employees of the city hall and government, but they sang absolutely perfectly. In summary, this was a very enjoyable afternoon and I even had the chance to chat with some new friends. I hope that I will be able to attend more such events in the future. (Postscript: And I indeed got the chance to attend the 2021 Hefei National Day Tea Party for People from All Walks of Life.)

In the afternoon of September 22, 2020, the Mid-Autumn Festival and National Holiday Celebration for Foreign Experts [2020年度在皖高层次外国专家迎中秋庆国庆活动] of our province Anhui [安徽] took place at the Anhui Museum of Innovation [安徽创新馆]. The event was co-organized by the Department of Science and Technology of the Province Anhui [安徽省科学技术厅], the Hefei Municipal Bureau of Science and Technology [合肥市科学技术局], the Public Relations Department of the Provincial Party Committee [安徽省委宣传部], and the museum itself. More than 30 foreign experts from more than 10 different countries attended this event – a very high number given the current international COVID-19 pandemic. Together with Mr. Lei HONG [洪磊] of our International Office, I attended this event as representatives of our Hefei University [合肥学院].

feed-image rss feed-image atom