Affordable Access

Access to the full text

What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask

  • Kohane, Isaac S1
  • Aronow, Bruce J2
  • Avillach, Paul1
  • Beaulieu-Jones, Brett K1
  • Bellazzi, Riccardo3, 4
  • Bradford, Robert L5
  • Brat, Gabriel A1
  • Cannataro, Mario6, 6
  • Cimino, James J7
  • García-Barrio, Noelia8
  • Gehlenborg, Nils1
  • Ghassemi, Marzyeh9
  • Gutiérrez-Sacristán, Alba1
  • Hanauer, David A10
  • Holmes, John H11
  • Hong, Chuan1
  • Klann, Jeffrey G1, 12
  • Loh, Ne Hooi Will13
  • Luo, Yuan14
  • Mandl, Kenneth D15
  • And 19 more
  • 1 Harvard Medical School, Boston, MA , (United States)
  • 2 University of Cincinnati, Cincinnati, OH , (United States)
  • 3 University of Pavia, Pavia , (Italy)
  • 4 ICS Maugeri, Pavia , (Italy)
  • 5 University of North Carolina at Chapel Hill, Chapel Hill, NC , (United States)
  • 6 University Magna Graecia of Catanzaro, Catanzaro , (Italy)
  • 7 University of Alabama at Birmingham, Birmingham, AL , (United States)
  • 8 12 de Octubre University Hospital, Madrid , (Spain)
  • 9 University of Toronto, Toronto, ON , (Canada)
  • 10 University of Michigan Medical School, Ann Arbor, MI , (United States)
  • 11 University of Pennsylvania, Philadelphia, PA , (United States)
  • 12 Massachusetts General Hospital, Boston, MA , (United States)
  • 13 National University Health Systems, Singapore , (Singapore)
  • 14 Northwestern University, Chicago, IL , (United States)
  • 15 Boston Children's Hospital, Boston, MA , (United States)
  • 16 Assistance Publique - Hôpitaux de Paris, Paris , (France)
  • 17 Université de Paris, Paris , (France)
  • 18 University of Michigan, Ann Arbor, MI , (United States)
  • 19 University of Kansas Medical Center, Kansas City, KS , (United States)
  • 20 Wake Forest School of Medicine, Winston Salem, NC , (United States)
  • 21 National University of Singapore, Singapore , (Singapore)
  • 22 The Children's Hospital of Philadelphia, Philadelphia, PA , (United States)
  • 23 The University of Pennsylvania, Philadelphia, PA , (United States)
  • 24 Medical College of Wisconsin, Milwaukee, WI , (United States)
Published Article
Journal of Medical Internet Research
JMIR Publications Inc.
Publication Date
Mar 02, 2021
DOI: 10.2196/22219
PMID: 33600347
PMCID: PMC7927948
PubMed Central
External links


Coincident with the tsunami of COVID-19–related publications, there has been a surge of studies using real-world data, including those obtained from the electronic health record (EHR). Unfortunately, several of these high-profile publications were retracted because of concerns regarding the soundness and quality of the studies and the EHR data they purported to analyze. These retractions highlight that although a small community of EHR informatics experts can readily identify strengths and flaws in EHR-derived studies, many medical editorial teams and otherwise sophisticated medical readers lack the framework to fully critically appraise these studies. In addition, conventional statistical analyses cannot overcome the need for an understanding of the opportunities and limitations of EHR-derived studies. We distill here from the broader informatics literature six key considerations that are crucial for appraising studies utilizing EHR data: data completeness, data collection and handling (eg, transformation), data type (ie, codified, textual), robustness of methods against EHR variability (within and across institutions, countries, and time), transparency of data and analytic code, and the multidisciplinary approach. These considerations will inform researchers, clinicians, and other stakeholders as to the recommended best practices in reviewing manuscripts, grants, and other outputs from EHR-data derived studies, and thereby promote and foster rigor, quality, and reliability of this rapidly growing field.

Report this publication


Seen <100 times