Affordable Access

Controlling for chance agreement in the validation of medical expert systems with no gold standard: PNEUMON-IA and RENOIR revisited.

Authors
  • Martín-Baranera, M
  • Sancho, J J
  • Sanz, F
Type
Published Article
Journal
Computers and biomedical research, an international journal
Publication Date
Dec 01, 2000
Volume
33
Issue
6
Pages
380–397
Identifiers
PMID: 11150233
Source
Medline
License
Unknown

Abstract

In the validation of medical expert systems, agreement among different human specialists on a random sample of cases may be taken as a substitute to a missing gold standard. Distance measures between pairs of experts, extensively described in previous studies, do not take into account the influence of chance-expected agreement. A weighted kappa index, with three different weighting schemes, is proposed as an alternative to be applied in the general situation of N cases assessed by E experts about K possible diagnoses, each of them qualified with one of G ordinal categories. A hierarchical cluster analysis, applied to the kappa matrices generated, allows for the classification of the expert system among clinical specialists, providing a relative assessment of its diagnostic ability. The above methodology is applied to the validation of two medical expert systems, PNEUMON-IA and RENOIR.

Report this publication

Statistics

Seen <100 times