Affordable Access

deepdyve-link
Publisher Website

Validation of natural language processing to determine the presence and size of abdominal aortic aneurysms in a large integrated health system.

Authors
  • McLenon, Myra1
  • Okuhn, Steven2
  • Lancaster, Elizabeth M3
  • Hull, Michaela M4
  • Adams, John L4
  • McGlynn, Elizabeth4
  • Avins, Andrew L5
  • Chang, Robert W6
  • 1 Softek Illuminate, Inc, Overland Park, Kan.
  • 2 Division of Vascular Surgery, Department of Surgery, Veterans Affairs San Francisco Healthcare System, San Francisco, Calif; Division of Vascular Surgery, Department of Surgery, University of California, San Francisco, San Francisco, Calif.
  • 3 Division of Vascular Surgery, Department of Surgery, University of California, San Francisco, San Francisco, Calif.
  • 4 Kaiser Permanente Center for Effectiveness and Safety Research, Pasadena, Calif.
  • 5 Division of Research, Kaiser Permanente Northern California, Oakland, Calif; Department of Medicine, University of California, San Francisco, San Francisco, Calif; Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, Calif.
  • 6 Division of Research, Kaiser Permanente Northern California, Oakland, Calif; Division of Vascular Surgery, Department of Surgery, The Permanente Medical Group, South San Francisco, Calif. Electronic address: [email protected]
Type
Published Article
Journal
Journal of vascular surgery
Publication Date
Aug 01, 2021
Volume
74
Issue
2
Identifiers
DOI: 10.1016/j.jvs.2020.12.090
PMID: 33548429
Source
Medline
Keywords
Language
English
License
Unknown

Abstract

Previous studies of the natural history of abdominal aortic aneurysms (AAAs) have been limited by small cohort sizes or heterogeneous analyses of pooled data. By quickly and efficiently extracting imaging data from the health records, natural language processing (NLP) has the potential to substantially improve how we study and care for patients with AAAs. The aim of the present study was to test the ability of an NLP tool to accurately identify the presence or absence of AAAs and detect the maximal abdominal aortic diameter in a large dataset of imaging study reports. Relevant imaging study reports (n = 230,660) from 2003 to 2017 were obtained for 32,778 patients followed up in a prospective aneurysm surveillance registry within a large, diverse, integrated healthcare system. A commercially available NLP algorithm was used to assess the presence of AAAs, confirm the absence of AAAs, and extract the maximal diameter of the abdominal aorta, if stated. A blinded expert manual review of 18,000 randomly selected imaging reports was used as the reference standard. The positive predictive value (PPV or precision), sensitivity (recall), and the kappa statistics were calculated. Of the randomly selected 18,000 studies that underwent expert manual review, 48.7% were positive for AAAs. In confirming the presence of an AAA, the interrater reliability of the NLP compared with the expert review showed a kappa value of 0.84 (95% confidence interval [CI], 0.83-0.85), with a PPV of 95% and sensitivity of 88.5%. The NLP algorithm showed similar results for confirming the absence of an AAA, with a kappa of 0.79 (95% CI, 0.799-0.80), PPV of 77.7%, and sensitivity of 91.9%. The kappa, PPV, and sensitivity of the NLP for correctly identifying the maximal aortic diameter was 0.88 (95% CI, 0.87-0.89), 88.8%, and 88.2% respectively. The use of NLP software can accurately analyze large volumes of radiology report data to detect AAA disease and assemble a contemporary aortic diameter-based cohort of patients for longitudinal analysis to guide surveillance, medical management, and operative decision making. It can also potentially be used to identify from the electronic medical records pre- and postoperative AAA patients "lost to follow-up," leverage human resources engaged in the ongoing surveillance of patients with AAAs, and facilitate the construction and implementation of AAA screening programs. Copyright © 2021 Society for Vascular Surgery. Published by Elsevier Inc. All rights reserved.

Report this publication

Statistics

Seen <100 times