Significance Tests for the Measure of Raw Agreement

von Eye, Alexander and Mair, Patrick and Schauerhuber, Michael (2006) Significance Tests for the Measure of Raw Agreement. Research Report Series / Department of Statistics and Mathematics, 42. Department of Statistics and Mathematics, WU Vienna University of Economics and Business, Vienna.

[img]
Preview
PDF
document.pdf

Download (365kB)

Abstract

Significance tests for the measure of raw agreement are proposed. First, it is shown that the measure of raw agreement can be expressed as a proportionate reduction-in-error measure, sharing this characteristic with Cohen's Kappa and Brennan and Prediger's Kappa_n. Second, it is shown that the coefficient of raw agreement is linearly related to Brennan and Prediger's Kappa_n. Therefore, using the same base model for the estimation of expected cell frequencies as Brennan and Prediger's Kappa_n, one can devise significance tests for the measure of raw agreement. Two tests are proposed. The first uses Stouffer's Z, a probability pooler. The second test is the binomial test. A data example analyzes the agreement between two psychiatrists' diagnoses. The covariance structure of the agreement cells in a rater by rater table is described. Simulation studies show the performance and power functions of the test statistics. (author's abstract)

Item Type: Paper
Keywords: rater agreement / raw agreement coefficient / Stouffer's Z / binomial test / Monte Carlo Simulation
Divisions: Departments > Finance, Accounting and Statistics > Statistics and Mathematics
Depositing User: Repository Administrator
Date Deposited: 03 Nov 2006 11:32
Last Modified: 22 Oct 2019 00:41
URI: https://epub.wu.ac.at/id/eprint/1336

Actions

View Item View Item

Downloads

Downloads per month over past year

View more statistics