Optimization is a technology which has the potential to become even more important than data mining and big data are now, as it can improve the efficiency of each and every aspect of an enterprises operation and products. Optimization means to find approximate solutions for hard problems. Optimization algorithms can find short routes in logistic planning scenarios, construction plans for work pieces that require only little amounts of material, or efficient schedules in production planning. They can help to reduce costs and pollution at the same time.

The term "optimization algorithms" implies a plural, and there are already many different families of algorithms, including exact methods, heuristics, metaheuristics sub-families such as evolutionary computation and swarm intelligence, as well as local search and hybrid algorithms. Each of these families contains dozens of general and specialized algorithms. We want to know how we can find out which method is best for which problem and when and why. First, using the best algorithm will give us the best solutions, which is what we want. Knowing why an algorithm is best could help us to create even better ones. Thus, benchmarking of optimization algorithms in a statistically sound, robust, and ideally automatic way seems to be an important topic to me. Yet it is entirely under-represented in literature. We want to emphasize how important it is, as for practical applications of optimization, we simply need to know that we are using the best and most reliable methods.

1. Projects

  • the optimizationBenchmarking.org, a framework for comparing and researching the performance of any optimization algorithm implemented in any programming language on any optimization problem
  • the TSP Suite, a framework for investigating the performance of solvers for the Traveling Salesman Problem (TSP), with a fast collection of state-of-the-art TSP solvers implemented in Java

2. Publications

  • Qi Qi, Thomas Weise, and Bin Li. Modeling Optimization Algorithm Runtime Behavior and its Applications. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'17) Companion, July 15-19, 2017, Berlin, Germany, New York, NY, USA: ACM Press, pages 115-116, ISBN: 978-1-4503-4939-0.
    doi:10.1145/3067695.3076042 / paper / poster / blog entry 1 / blog entry 2

  • Weichen Liu, Thomas Weise, Yuezhong Wu, and Qi Qi. Combining Two Local Searches with Crossover: An Efficient Hybrid Algorithm for the Traveling Salesman Problem. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'17), July 15-19, 2017, Berlin, Germany, New York, NY, USA: ACM Press, pages 298-305, ISBN: 978-1-4503-4920-8.
    doi:10.1145/3071178.3071201 / paper / slides / blog entry 1 / blog entry 2
    Indexing: EI:20173104005769

  • Thomas Weise. From Standardized Data Formats to Standardized Tools for Optimization Algorithm Benchmarking. In Newton Howard, Yingxu Wang, Amir Hussain, Freddie Hamdy, Bernard Widrow, and Lotfi A. Zadeh, editors, Proceedings of the 16th IEEE Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC'17), July 26-28, 2017, University of Oxford, Oxford, UK, pages 490-497. Los Alamitos, CA, USA: IEEE Computer Society Press, ISBN: 978-1-5386-0770-1.
    accepted as full paper.
    paper / slides / blog entry

  • Thomas Weise, Yuezhong Wu, Raymond Chiong, Ke Tang, and Jörg Lässig. Global versus Local Search: The Impact of Population Sizes on Evolutionary Algorithm Performance. Journal of Global Optimization 66(3):511-534, November 2016.
    doi:10.1007/s10898-016-0417-5 / pdf
    Indexing: EI:20160902030834, WOS:000386373700007, SCI

  • Yuezhong Wu, Thomas Weise, and Weichen Liu. Hybridizing Different Local Search Algorithms with Each Other and Evolutionary Computation: Better Performance on the Traveling Salesman Problem. In Proceedings of the 18th Genetic and Evolutionary Computation Conference (GECCO'16), Denver, Colorado, USA, July 20–24, 2016, pages 57-58, New York, NY, USA: Association for Computing Machinery (ACM). ISBN: 978-1-4503-4323-7
    doi:10.1145/2908961.2909001 / pdf / poster
    Indexing: EI:20163702804277, WOS:000383741800029

  • Weichen Liu, Thomas Weise, Yuezhong Wu, Dan Xu, and Raymond Chiong. An Improved Ejection Chain Method and Its Hybrid Versions for Solving the Traveling Salesman Problem. Journal of Computational and Theoretical Nanoscience 13(6):3601-3610, June 2016
    doi:10.1166/jctn.2016.5189
    Indexing: EI:20164002870059

  • Weichen Liu, Thomas Weise, Yuezhong Wu, and Raymond Chiong. Hybrid Ejection Chain Methods for the Traveling Salesman Problem. In Proceedings of the 10th International Conference on Bio-Inspired Computing – Theories and Applications (BIC-TA'15), Maoguo Gong, Linqiang Pan, Tao Song, Ke Tang, and Xingyi Zhang, editors, September 25-28, 2015, Hefei, Anhui, China, volume 562 of Communications in Computer and Information Science. Berlin/Heidelberg: Springer-Verlag, pages 268-282, ISBN 978-3-662-49013-6.
    doi:10.1007/978-3-662-49014-3_25 / pdf
    Indexing: EI:20160300001732, WOS:000369890300025

  • Dan Xu, Thomas Weise, Yuezhong Wu, Jörg Lässig, and Raymond Chiong. An Investigation of Hybrid Tabu Search for the Traveling Salesman Problem. In Proceedings of the 10th International Conference on Bio-Inspired Computing – Theories and Applications (BIC-TA'15), Maoguo Gong, Linqiang Pan, Tao Song, Ke Tang, and Xingyi Zhang, editors, September 25-28, 2015, Hefei, Anhui, China, volume 562 of Communications in Computer and Information Science. Berlin/Heidelberg: Springer-Verlag, pages 523-537, ISBN 978-3-662-49013-6.
    doi:10.1007/978-3-662-49014-3_47 / pdf
    Indexing: EI:20160300001756, WOS:000369890300047

  • Yuezhong Wu, Thomas Weise, and Raymond Chiong. Local Search for the Traveling Salesman Problem: A Comparative Study. In Proceedings of the 14th IEEE Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC'15), July 6-8, 2015, Beijing, China, pages 213-220, Los Alamitos, CA, USA: IEEE Computer Society Press, ISBN: 978-1-4673-7289-3.
    doi:10.1109/ICCI-CC.2015.7259388 / pdf
    Indexing: EI:20161202119039, WOS:000380466100034

  • Thomas Weise and Raymond Chiong. An Alternative Way of Presenting Statistical Test Results when Evaluating the Performance of Stochastic Approaches. Neurocomputing, 147:235-238, January, 2015.
    doi:10.1016/j.neucom.2014.06.071 / pdf
    Indexing: EI:20143600022800, WOS:000343337600020, SCI

  • Yan Jiang, Thomas Weise, Jörg Lässig, Raymond Chiong, and Rukshan Athauda. Comparing a Hybrid Branch and Bound Algorithm with Evolutionary Computation Methods, Local Search and their Hybrids on the TSP. In Proceedings of the IEEE Symposium on Computational Intelligence in Production and Logistics Systems (CIPLS'14), Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI'14), Orlando, FL, USA: Caribe Royale All-Suite Hotel and Convention Center, December 9-12, 2014, pages 148-155. Los Alamitos, CA, USA: IEEE Computer Society Press. ISBN 978-1-4799-5375-2.
    doi:10.1109/CIPLS.2014.7007174 / pdf
    Indexing: EI:20150700522268, WOS:000380487400021

  • Thomas Weise, Raymond Chiong, Ke Tang, Jörg Lässig, Shigeyoshi Tsutsui, Wenxiang Chen, Zbigniew Michalewicz, and Xin Yao. Benchmarking Optimization Algorithms: An Open Source Framework for the Traveling Salesman Problem. IEEE Computational Intelligence Magazine (CIM), 9(3):40-52, August 2014.
    Featured article and selected paper at the website of the IEEE Computational Intelligence Society (http://cis.ieee.org/).
    doi:10.1109/MCI.2014.2326101 / pdf
    Indexing: EI:20143117995681, WOS:000340271800004, SCI, Google Scholar

  • Thomas Weise, Brian M. Blake, and Steffen Bleul. Semantic Web Service Composition: The Web Service Challenge Perspective. In Athman Bouguettaya, Quan Z. Sheng, and Florian Daniel, editors, Web Services Foundations – Part 1: Foundations of Web Services, chapter 7, pages 161-187. Secaucus, NJ, USA: Springer-Verlag New York, Inc., first online September 2013 / Copyright 2014. ISBN: 978-1-4614-7517-0.
    doi:10.1007/978-1-4614-7518-7_7 / pdf
    Indexing: EI:20172603853294

  • Ke Tang, Zhenyu Yang, and Thomas Weise. Special Session on Evolutionary Computation for Large Scale Global Optimization at 2012 IEEE World Congress on Computational Intelligence (CEC@WCCI-2012). Technical Report, Hefei, Anhui, China: University of Science and Technology of China (USTC), School of Computer Science and Technology, Nature Inspired Computation and Applications Laboratory (NICAL), June 14, 2012.

  • Ajay Bansal, Srividya Bansal, Brian M. Blake, Steffen Bleul, and Thomas Weise. Overview of the Web Services Challenge (WSC): Discovery and Composition of Semantic Web Services. In Brian M. Blake, Liliana Cabral, Birgitta König-Ries, Ulrich Küster, and David Martin, editors, Semantic Web Services – Advancement through Evaluation, chapter 19, pages 297-312. April 2012, Berlin/Heidelberg: Springer-Verlag. ISBN: 978-3-642-28734-3.
    doi:10.1007/978-3-642-28735-0_19
    Indexing: EI:20172803915654

  • Brian M. Blake, Thomas Weise, and Steffen Bleul. WSC-2010: Web Services Composition and Evaluation. In Proceedings of the IEEE International Conference on Service-Oriented Computing and Applications (SOCA'10), pages 1-4, Perth, WA, Australia, December 13–15, 2010. ISBN: 978-1-4244-9802-4, Los Alamitos, CA, USA: IEEE Computer Society Press.
    doi:10.1109/SOCA.2010.5707190
    Indexing: EI:20111113747701

  • Thomas Weise, Li Niu, and Ke Tang. AOAB – Automated Optimization Algorithm Benchmarking. In Proceedings of the 12th Annual Conference Companion on Genetic and Evolutionary Computation (GECCO'10), pages 1479-1486, Portland, OR, USA: Portland Marriott Downtown Waterfront Hotel, July 7-11, 2010. ISBN: 978-1-4503-0073-5, New York, NY, USA: ACM Press.
    doi:10.1145/1830761.1830763
    Indexing: EI:20103513190754, WOS:000322071400001

  • Ke Tang, Xiaodong Li, Ponnuthurai Nagaratnam Suganthan, Zhenyu Yang, and Thomas Weise. Benchmark Functions for the CEC'2010 Special Session and Competition on Large-Scale Global Optimization. Technical Report, Hefei, Anhui, China: University of Science and Technology of China (USTC), School of Computer Science and Technology, Nature Inspired Computation and Applications Laboratory (NICAL), January 8, 2010.

  • Srividya Kona, Ajay Bansal, Brian M. Blake, Steffen Bleul, and Thomas Weise. WSC-2009: A Quality of Service-Oriented Web Services Challenge. In Birgit Hofreiter, editor, Proceedings of the 11th IEEE Conference on Commerce and Enterprise Computing (CEC'09), pages 487-490, Vienna, Austria: Vienna University of Technology, July 20–23, 2009. ISBN: 978-0-7695-3755-9, Piscataway, NJ, USA: IEEE Computer Society and Red Hook, NY, USA: Curran Associates, Inc.
    doi:10.1109/CEC.2009.80
    Indexing: EI:20094712467206, Google Scholar

  • Steffen Bleul, Thomas Weise, and Kurt Geihs. The Web Service Challenge – A Review on Semantic Web Service Composition. Electronic Communications of the EASST, 17(204):1-13, Special Issue: Michael Wagner, Dieter Hogrefe, Kurt Geihs, and Klaus David, editors, “Kommunikation in Verteilten Systemen 2009”, KiVS'09, Proceedings, Workshops der Wissenschaftlichen Konferenz Kommunikation in Verteilten Systemen 2009 (WowKiVS 2009), Kassel, Hesse, Germany: University of Kassel, Fachbereich 16: Elektrotechnik/Informatik, March 5, 2009. Potsdam, Germany: European Association of Software Science and Technology (EASST; Universität Potsdam, Institute for Informatics).
    doi:10.14279/tuj.eceasst.17.204

  • Ajay Bansal, Brian M. Blake, Srividya Kona, Steffen Bleul, Thomas Weise, and Michael C. Jäger. WSC-08: Continuing the Web Services Challenge. In Proceedings of IEEE Joint Conference on E-Commerce Technology (10th CEC) and Enterprise Computing, E-Commerce and E-Services (5th EEE) (CEC/EEE'08), pages 351-354, Washington, DC, USA, July 21–24, 2008. ISBN: 978-0-7695-3340-7, Piscataway, NJ, USA: IEEE Computer Society.
    doi:10.1109/CECandEEE.2008.146
    Indexing: EI:20091512027446, WOS:000265327500048, Google Scholar

  • Thomas Weise, Stefan Niemczyk, Hendrik Skubch, Roland Reichle, and Kurt Geihs. A Tunable Model for Multi-Objective, Epistatic, Rugged, and Neutral Fitness Landscapes. In Maarten Keijzer, Giuliano Antoniol, Clare Bates Congdon, Kalyanmoy Deb, Benjamin Doerr, Nikolaus Hansen, John H. Holmes, Gregory S. Hornby, Daniel Howard, James Kennedy, Sanjeev P. Kumar, Fernando G. Lobo, Julian Francis Miller, Jason H. Moore, Frank Neumann, Martin Pelikan, Jordan B. Pollack, Kumara Sastry, Kenneth Owen Stanley, Adrian Stoica, El-Ghazali, and Ingo Wegener, editors, Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation Conference (GECCO'08), pages 795-802, July 12-16, 2008, Renaissance Atlanta Hotel Downtown: Atlanta, GA, USA. ISBN: 978-1-60558-130-9, New York, NY, USA: ACM Press.
    doi:10.1145/1389095.1389252 / pdf / slides / implementation [Java]
    Indexing: EI:20085111786051

  • Thomas Weise, Steffen Bleul, and Kurt Geihs. Web Service Composition Systems for the Web Service Challenge – A Detailed Review. Kasseler Informatikschriften (KIS) 2007, 7, Kassel, Hesse, Germany: University of Kassel, Fachbereich 16: Elektrotechnik/Informatik, November 19, 2007.
    urn:nbn:de:hebis:34-2007111919638

3. Organized Events

  • Thomas Weise, Bin Li, Markus Wagner, Xingyi Zhang, and Jörg Lässig, eds. Special Issue on Benchmarking of Computational Intelligence Algorithms in Computational Intelligence – An International Journal published by Wiley Periodicals Inc., indexed by SCI and EI. Pre-Submission Deadline: May 5, 2018.
    website / Call for Papers (CfP)

  • Thomas Weise, Bin Li, Markus Wagner, Xingyi Zhang, and Jörg Lässig. International Workshop on Benchmarking of Computational Intelligence Algorithms (BOCIA) at the Tenth International Conference on Advanced Computational Intelligence (ICACI 2018), March 29-31, 2018 in Xiamen, China. The submission system will be open until December 1, 2017.
    website / Call for Papers (CfP)

  • Thomas Weise and Jörg Lässig. SITA-UBRI Joint Workshop on Sustainable Logistics, November 5, 2015, Hefei, Anhui, China

  • Thomas Weise and Jörg Lässig. Special Session on Benchmarking and Testing for Production and Logistics Optimization of the 2014 IEEE Symposium on Computational Intelligence in Production and Logistics at the 2014 IEEE Symposium Series on Computational Intelligence (SSCI 2014), December 9-12, 2014, Orlando, Florida, USA

  • M. Brian Blake, Thomas Weise, and Steffen Bleul. 2010 Web Service Challenge (WSC-2010) on Semantic Web Service Composition with Quality-of-Service at the Conference on Commerce and Enterprise Computing (CEC), November 11, 2010, Fudan University, Shanghai, China

4. Funding

  • National Natural Science Foundation of China (NSFC), Regular Project "Research on the Automatic Performance Analysis of Optimization,"  2017 to 2020, Principal Investigator is Dr. Thomas Weise

  • The University of Newcastle in Callaghan, NSW, Australia, Faculty Strategic Initiatives Research Fund (SIRF) Grant "A Framework for Next Generation Algorithm Benchmarking: Performance Testing and Community Building," 2014, Principal Investigators were Dr. Raymond Chiong, Dr. Thomas Weise, and Dr. Rukshan Athauda

  • The University of Newcastle in Callaghan, NSW, Australia, Faculty Strategic Initiatives Research Fund (SIRF) Grant "Intelligent Transportation Planning: Benchmarking of Novel Business Analytics Techniques using the Travelling Salesman Problem as a Test-Bed," 2013, Principal Investigators were Dr. Raymond Chiong, Dr. Thomas Weise, Dr. David Cornforth, and Prof. Zbigniew Michalewicz

5. Lectures and Courses

6. News

  1. International Workshop on Benchmarking of Computational Intelligence Algorithms (BOCIA)
  2. Workshop on Benchmarking of Computational Intelligence Algorithms approved for ICACI 2018
  3. Research Talk at the Ludwig-Maximilians-Universität Munich in Munich (München), Germany
  4. Research Talk at the Johannes Gutenberg University Mainz in Mainz, Germany
  5. Research Talk at the University of Cologne in Köln (Cologne), Germany
  6. Research Talk at the University of Leipzig in Leipzig, Germany
  7. Research Talk at the University of Applied Sciences Zittau/Görlitz in Görlitz, Germany
  8. Research Talk at the Otto von Guericke University Magdeburg in Magdeburg, Germany
  9. Research Talk at the Clausthal University of Technology in Clausthal-Zellerfeld, Germany
  10. Research Talk at the Technische Universität Ilmenau in Ilmenau, Germany
  11. Research Talk at the Technische Universität Ilmenau in Ilmenau, Germany
  12. Research Talk at the University of Applied Sciences Mittweida in Mittweida, Germany
  13. Research Talk at the Chemnitz University of Technology in Chemnitz, Germany
  14. Research Talk at the Chemnitz University of Technology in Chemnitz, Germany
  15. Research Talk at the Friedrich Schiller University in Jena, Germany
  16. Research Talk: Automating Scientific Research in Optimization
  17. From Standardized Data Formats to Standardized Tools for Optimization Algorithm Benchmarking
  18. New Beta Release of our optimizationBenchmarking.org Software for Automating Research in Optimization
  19. Two Papers accepted at GECCO 2017