A service provided by the WU Library and the WU IT-Services

The design and analysis of benchmark experiments

Hothorn, Torsten and Leisch, Friedrich and Zeileis, Achim and Hornik, Kurt (2003) The design and analysis of benchmark experiments. Report Series SFB "Adaptive Information Systems and Modelling in Economics and Management Science", 82. SFB Adaptive Information Systems and Modelling in Economics and Management Science, WU Vienna University of Economics and Business, Vienna.

[img]
Preview
PDF
Download (628Kb) | Preview

Abstract

The assessment of the performance of learners by means of benchmark experiments is established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Cross-validation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the cross-validated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in practice, the theoretical results are applied to benchmark studies in a supervised learning situation based on artificial and real-world data.

Item Type: Paper
Keywords: Model Comparison / Performance / Hypothesis Testing / Cross-Validation / Bootstrap
Divisions: Departments > Finance, Accounting and Statistics > Statistics and Mathematics
Departments > Informationsverarbeitung u Prozessmanag. > Informationswirtschaft
Departments > Informationsverarbeitung u Prozessmanag. > Produktionsmanagement > Taudes
Departments > Marketing > Service Marketing und Tourismus
Depositing User: Repository Administrator
Date Deposited: 11 Nov 2003 15:34
Last Modified: 27 Feb 2017 11:14
URI: http://epub.wu.ac.at/id/eprint/758

Actions

View Item