Today, we released the new version 0.8.9 of our optimizationBenchmarking.org software framework for automating significant parts of the research in the fields of optimization, machine learning, and operations research. This new version comes much closer to our goal to reduce the workload of researchers and practitioners during the development of algorithms to allow them to spend more time on thinking.
Besides all the functionality offered by the previous releases, it introduces a new process for obtaining high-level conclusions about problem hardness and algorithm behaviors. This process takes the raw data from experiments together with meta-information about algorithm setups and problem instances as input. It applies a sequence of machine learning technologies, namely curve fitting, clustering, and classification, to find which features make a problem instance hard and which algorithm setup parameters cause which algorithm behavior. We just submitted an article about this new process for review.
Our software provides a set of very general tools for algorithm performance analysis (e.g., plotting runtime/quality and ECDF charts) as well as our new process. Since it takes data in form of text files, it can analyze the results of any optimization algorithm implemented in any programming language applied to any optimization problem. It produces human-readable reports either in form of LaTeX/PDF documents or as XHTML.
The software provides a user-friendly, web-based GUI which runs either on your local machine or a server in your lab comes. The software comes in three flavors:
- as Java executable, requiring that several tools are installed (Java, R with several packages, a LaTeX system installation),
- as Docker image, which only requires an installation of Docker. It can be started directly under Linux, Windows, and Mac OS with the single command
docker run -t -i -p 9999:8080/tcp optimizationbenchmarking/evaluator-guiand then is used by browsing to http://localhost:9999. (At first start, the image is downloaded), and - as command line program without GUI for integration in other software environments (with the same installation requirements as the GUI),
The New Process: Synopsis
Most optimization methods are anytime algorithms that start with a rough approximation of the solution of a problem and improve their optimization quality over time. Our system can compare the complete runtime behaviors of such algorithms. It can analyze all the algorithms (or algorithm setups) you used in your experiments and find whether there are groups of algorithms which behave differently from others. For example, there could be algorithms that are faster than others, that are slower but have better approximation quality, or that are generally worse. Once it has found such groups of algorithms, our system can propose possible reasons why an algorithm belongs to a certain group: The behavior of an algorithm is controlled by its parameters and our system generates decision trees that use these parameters to classify the algorithms to the behavior groups.
Our system can also do that same with problem instances, i.e., find groups of problem instances where the algorithms behave differently and the propose reasons for the existence of these groups based on the instance features.
This is achieved by applying several machine learning technologies in sequence, namely curve fitting to model the progress of algorithms over time, clustering of the obtained models, and classification of the algorithms and problem instances to the clusters based on their parameter and feature settings.
The Three Flavors of the Software
The GUI is a stand-alone server providing a web-based interface for managing your experimental data, creating meta data, and running the evaluator.
The dockerized version of the GUI comes as a, well, Docker image. This image has all the software requirements pre-installed, including the Java 8 OpenJDK, TeX Live as LaTeX installation, R, the required R packages, and ghostscript. Using the evaluator as Docker image requires more disk space, but apart from Docker itself, you don't need to install anything. This is the most convenient scenario, given that one would otherwise need to install R and several of its packages, TeX Live, and such and such, to fully enjoy all features of our system.
The command line interface (CLI) contains only the pure evaluator component and is to be executed manually. This may make sense when running it as component of another, non-Java process. If one wants to use the software inside a Java program, one can just use the libraries directly. If you want to know how to do that,
Anyway, both the GUI and the CLI come as stand-alone JAR archives and you can run them directly via java -jar evaluatorGui.jar (in case of the GUI) and java -jar evaluator.jar in case of the evaluator. In the dockerized GUI version, this is already automatically done for you and you can directly access the control center by browsing to http://localhost:9999 after starting the container.
All in all, we think that this beta version of our software is a milestone in the project. It provides researchers and practitioners with a strong tool to get deep insights into their experimental data, which can then help them to improve their algorithms by better understanding their strengths and weaknesses.