A service provided by the WU Library and the WU IT-Services

Deriving Consensus Rankings from Benchmarking Experiments

Hornik, Kurt and Meyer, David (2006) Deriving Consensus Rankings from Benchmarking Experiments. Research Report Series / Department of Statistics and Mathematics, 33. Department of Statistics and Mathematics, WU Vienna University of Economics and Business, Vienna.

[img]
Preview
PDF
Download (176Kb) | Preview

Abstract

Whereas benchmarking experiments are very frequently used to investigate the performance of statistical or machine learning algorithms for supervised and unsupervised learning tasks, overall analyses of such experiments are typically only carried out on a heuristic basis, if at all. We suggest to determine winners, and more generally, to derive a consensus ranking of the algorithms, as the linear order on the algorithms which minimizes average symmetric distance (Kemeny-Snell distance) to the performance relations on the individual benchmark data sets. This leads to binary programming problems which can typically be solved reasonably efficiently. We apply the approach to a medium-scale benchmarking experiment to assess the performance of Support Vector Machines in regression and classification problems, and compare the obtained consensus ranking with rankings obtained by simple scoring and Bradley-Terry modeling.

Item Type: Paper
Additional Information: GfKl 2006, Berlin, Germany
Keywords: benchmark experiments / consensus rankings / Borda / Condorcet / symmetric difference / linear order / poset / linear programming
Divisions: Departments > Finance, Accounting and Statistics > Statistics and Mathematics
Depositing User: Repository Administrator
Date Deposited: 02 May 2006 16:23
Last Modified: 28 Feb 2017 15:18
URI: http://epub.wu.ac.at/id/eprint/1300

Actions

View Item