This repository contains materials and files related to Quality Profiles and Test Set Profiles, introduced by Giovanni Fasano, Christian Piermarini, and Massimo Roma in 2025 (see G. Fasano, C. Piermarini, M. Roma, “Benchmarking Optimization Algorithms with Quality Profiles and Test Set Profiles,” arXiv:2607.16798 [math.OC], 2026, https://arxiv.org/abs/2607.16798). The same material is also included in a paper that is currently under review at an international journal.
Quality Profiles are designed to benchmark solvers for numerical optimization problems, with particular emphasis on the accuracy of the solutions obtained by the solvers on a given test set. Test Set Profiles, on the other hand, provide a measure for assessing the consistency of a test set with respect to the solvers being compared. Both Quality Profiles and Test Set Profiles can be used to compare either exact or heuristic solvers, as well as solvers developed within smooth or derivative-free optimization frameworks.
This repository provides MATLAB code for generating Quality Profiles, Test Set Profiles, Performance Profiles (E. D. Dolan, J. J. Moré, “Benchmarking Optimization Software with Performance Profiles,” Mathematical Programming, 91, pp. 201–213, 2002), and Data Profiles (J. J. Moré, S. M. Wild, “Benchmarking Derivative-Free Optimization Algorithms,” SIAM Journal on Optimization, 20, pp. 172–191, 2009).
The general MATLAB code for generating Quality Profiles, Test Set Profiles, Performance Profiles, and Data Profiles is contained in the file Profiles_N_codes_vXX_github_revised.m, where vXX denotes the version of the code.
The repository also includes files named Profiles_N_codes_vXX_github_revised_FigureXXXX.m, where FigureXXXX denotes the number of the corresponding figure in the paper currently under review at an international journal. Once the review process has been completed, the final version of the paper and the name of the journal may be reported in this repository, in accordance with the applicable copyright regulations.
Finally, this repository contains two folders with the numerical results reported in the above-mentioned arXiv Technical Report. The folder names correspond to the relevant sections of the Technical Report. These results were obtained by applying the selected solvers to the test sets considered in the study. The data are made available to both practitioners and researchers, allowing them to conduct their own numerical experiments and/or reproduce the results presented in the arXiv Technical Report.
The owner of this repository is solely responsible for the implementations of Performance Profiles and Data Profiles provided herein.