Affordable Access

A mixture model approach to indexing rater agreement.

Authors
Type
Published Article
Journal
The British journal of mathematical and statistical psychology
Publication Date
Volume
55
Issue
Pt 2
Pages
289–303
Identifiers
PMID: 12473229
Source
Medline
License
Unknown

Abstract

Raters are an important potential source of measurement error when assigning targets to categories. Therefore, psychologists have devoted considerable attention to quantifying the extent to which ratings agree with each other. Two main approaches to analysing rater agreement data can be distinguished. While the first approach focuses on the development of summary statistics that index rater agreement, the second models the association pattern among the observers' ratings. With the modelling approach three groups of models can be distinguished: latent class models, simple quasisymmetric agreement models, and mixture models. This paper discusses a class of mixture models that is defined by its characteristic of having a quasi-symmetric log-linear representation. This class of models has two interesting properties. First, the simple quasi-symmetric agreement models can be shown to be members of this class. Therefore, the results of a rater agreement analysis based on a simple quasi-symmetric agreement model may be interpreted in the mixture model framework. Second, since the mixture models readily provide a familiar measure of rater reliability, it is possible to obtain a model-based estimate of rater reliability from the simple quasi-symmetric agreement models. The suggested class of mixture models will be illustrated using data from a persuasive communication study in which three raters classified respondents on the basis of their elicited cognitive responses.

Statistics

Seen <100 times