Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals

Fissler, Tobias and Frongillo, Rafael and Hlavinova, Jana and Rudloff, Birgit ORCID: (2021) Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals. Electronic Journal of Statistics, 15 (1). pp. 1034-1084. ISSN 1935-7524

Available under License Creative Commons: Attribution 4.0 International (CC BY 4.0).

Download (619kB) | Preview


We introduce a theoretical framework of elicitability and identifiability of set-valued functionals, such as quantiles, prediction intervals, and systemic risk measures. A functional is elicitable if it is the unique minimiser of an expected scoring function, and identifiable if it is the unique zero of an expected identification function; both notions are essential for forecast ranking and validation, and M- and Z-estimation. Our framework distinguishes between exhaustive forecasts, being set-valued and aiming at correctly specifying the entire functional, and selective forecasts, content with solely specifying a single point in the correct functional. We establish a mutual exclusivity result: A set-valued functional can be either selectively elicitable or exhaustively elicitable or not elicitable at all. Notably, since quantiles are well known to be selectively elicitable, they fail to be exhaustively elicitable. We further show that the classes of prediction intervals and Vorob’ev quantiles turn out to be exhaustively elicitable, hence not selectively elicitable, but still selectively identifiable. In particular, we provide a mixture representation of elementary exhaustive scores, leading the way to Murphy diagrams. We establish that the shortest prediction interval and those specified by an endpoint or midpoint in general fail to be elicitable with respect to either notion, unless an endpoint is given via a quantile. We end with a comprehensive literature review on common practice in forecast evaluation of set-valued functionals.

Item Type: Article
Additional Information: T. Fissler received financial support via his Chapman Fellowship from Imperial College London for parts of this work. R. Frongillo was supported in part by the U.S. National Science Foundation under Grant No. 1657598. B. Rudloff acknowledges support from the OeNB anniversary fund, project number 17793.
Keywords: consistency, convex level sets, elicitability, Identifiability, M-estimation, prediction intervals, random sets, Vorob’ev quantiles
Divisions: Departments > Finance, Accounting and Statistics > Statistics and Mathematics
Version of the Document: Published
Depositing User: Gertraud Novotny
Date Deposited: 23 Mar 2021 13:44
Last Modified: 23 Mar 2021 13:44
Related URLs:


View Item View Item


Downloads per month over past year

View more statistics