Affordable Access

Understanding metric-related pitfalls in image analysis validation.

Authors
  • Reinke, Annika1
  • Tizabi, Minu D2
  • Baumgartner, Michael3
  • Eisenmann, Matthias4
  • Heckmann-Nötzel, Doreen2
  • Kavur, A Emre5
  • Rädsch, Tim6
  • Sudre, Carole H7
  • Acion, Laura8
  • Antonelli, Michela9
  • Arbel, Tal10
  • Bakas, Spyridon11
  • Benis, Arriel12
  • Blaschko, Matthew B13
  • Buettner, Florian14
  • Cardoso, M Jorge15
  • Cheplygina, Veronika16
  • Chen, Jianxu17
  • Christodoulou, Evangelia4
  • Cimini, Beth A18
    • And 56 more
  • 1 German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz Imaging, Germany and Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany. , (Germany)
  • 2 German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany. , (Germany)
  • 3 German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Germany. , (Germany)
  • 4 German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. , (Germany)
  • 5 HI Applied Computer Vision Lab, Division of Medical Image Computing; German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany. , (Germany)
  • 6 German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz Imaging, Germany. , (Germany)
  • 7 MRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK and School of Biomedical Engineering and Imaging Science, King's College London, London, UK.
  • 8 Instituto de Cálculo, CONICET - Universidad de Buenos Aires, Buenos Aires, Argentina. , (Argentina)
  • 9 School of Biomedical Engineering and Imaging Science, King's College London, London, UK and Centre for Medical Image Computing, University College London, London, UK.
  • 10 Centre for Intelligent Machines and MILA (Quebec Artificial Intelligence Institute), McGill University, Montreal, Canada. , (Canada)
  • 11 Division of Computational Pathology, Dept of Pathology & Laboratory Medicine, Indiana University School of Medicine, IU Health Information and Translational Sciences Building, Indianapolis, USA and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Richards Medical Research Laboratories FL7, Philadelphia, PA, USA. , (India)
  • 12 Department of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel and European Federation for Medical Informatics, Le Mont-sur-Lausanne, Switzerland. , (Switzerland)
  • 13 Center for Processing Speech and Images, Department of Electrical Engineering, KU Leuven, Leuven, Belgium. , (Belgium)
  • 14 German Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Germany, German Cancer Research Center (DKFZ) Heidelberg, Germany, Goethe University Frankfurt, Department of Medicine, Germany, Goethe University Frankfurt, Department of Informatics, Germany, and Frankfurt Cancer Insititute, Germany. , (Germany)
  • 15 School of Biomedical Engineering and Imaging Science, King's College London, London, UK.
  • 16 Department of Computer Science, IT University of Copenhagen, Copenhagen, Denmark. , (Denmark)
  • 17 Leibniz-Institut für Analytische Wissenschaften - ISAS - e.V., Dortmund, Germany. , (Germany)
  • 18 Imaging Platform, Broad Institute of MIT and Harvard, Cambridge, Massachusetts, USA.
  • 19 Centre for Statistics in Medicine, University of Oxford, Oxford, UK.
  • 20 Center for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA.
  • 21 Instituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Universitaria, Ciudad Autónoma de Buenos Aires, Argentina. , (Argentina)
  • 22 Universitat Pompeu Fabra, Barcelona, Spain and University of Adelaide, Adelaide, Australia. , (Australia)
  • 23 Fraunhofer MEVIS, Bremen, Germany and Radboud Institute for Health Sciences, Radboud University Medical Center, Nijmegen, The Netherlands. , (Germany)
  • 24 Department of Computing, Imperial College London, London, UK.
  • 25 German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Germany, Faculty of Mathematics and Computer Science, Heidelberg University, Heidelberg, Germany, and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany. , (Germany)
  • 26 Now with: Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Leipzig University, Leipzig, Germany, DFG Cluster of Excellence "Physics of Life", Technische Universität (TU) Dresden, Dresden, Germany, and Center for Systems Biology , Dresden, Germany. , (Germany)
  • 27 Department of Surgery, Perelman School of Medicine, Philadelphia, PA, USA and General Robotics Automation Sensing and Perception Laboratory, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA, USA.
  • 28 Princess Margaret Cancer Centre, University Health Network, Toronto, Canada, Department of Medical Biophysics, University of Toronto, Toronto, Canada, Department of Computer Science, University of Toronto, Toronto, Canada, and Vector Institute for Artificial Intelligence, Toronto, Canada. , (Canada)
  • 29 Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, The Netherlands. , (Netherlands)
  • 30 German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing and HI Applied Computer Vision Lab, Germany. , (Germany)
  • 31 Laboratoire Traitement du Signal et de l'Image - UMR_S 1099, Université de Rennes 1, Rennes, France and INSERM, Paris Cedex, France. , (France)
  • 32 Department of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USA.
  • 33 Max-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany and University of Potsdam, Digital Engineering Faculty, Potsdam, Germany. , (Germany)
  • 34 Department of Computing, Faculty of Engineering, Imperial College London, London, UK and Department AIBE, Friedrich-Alexander-Universität (FAU), Erlangen-Nürnberg, Germany. , (Germany)
  • 35 IHU Strasbourg, Strasbourg, France. , (France)
  • 36 Google Health DeepMind, London, UK.
  • 37 Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany. , (Germany)
  • 38 Translational Image-guided Oncology (TIO), Institute for AI in Medicine (IKIM), University Medicine Essen, Essen, Germany. , (Germany)
  • 39 Helmholtz AI, München, Germany. , (Germany)
  • 40 Lunit, Seoul, South Korea. , (North Korea)
  • 41 German Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Germany. , (Germany)
  • 42 Centre for Biomedical Image Analysis and Faculty of Informatics, Masaryk University, Brno, Czech Republic. , (Czechia)
  • 43 Cell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany. , (Germany)
  • 44 Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA.
  • 45 Electrical Engineering, Vanderbilt University, Nashville, TN, USA.
  • 46 Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands. , (Netherlands)
  • 47 Department of Surgery, University Health Network, Philadelphia, PA, Canada. , (Canada)
  • 48 German Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing and HI Helmholtz Imaging, Germany and Pattern Analysis and Learning Group, Department of Radiation Oncology, Heidelberg University Hospital, Heidelberg, Germany. , (Germany)
  • 49 Physical Sciences, Sunnybrook Research Institute, Toronto, Canada and Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada. , (Canada)
  • 50 Google, Mountain View, USA.
  • 51 School of Computer Science and Engineering, University of New South Wales, Sydney, Australia. , (Australia)
  • 52 Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland. , (Switzerland)
  • 53 Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, The Netherlands. , (Netherlands)
  • 54 Information Systems Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland and Medical Faculty, University of Geneva, Geneva, Switzerland. , (Switzerland)
  • 55 MILA (Quebec Artificial Intelligence Institute), Montréal, Canada. , (Canada)
  • 56 Department of General, Visceral and Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany. , (Germany)
  • 57 Allen Institute for Cell Science, Seattle, WA, USA.
  • 58 Tissue Image Analytics Laboratory, Department of Computer Science, University of Warwick, Coventry, UK.
  • 59 ARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland and Department of Radiation Oncology, University Hospital Bern, University of Bern, Bern, Switzerland. , (Switzerland)
  • 60 Simula Metropolitan Center for Digital Engineering, Oslo, Norway and UiT The Arctic University of Norway, Tromsø, Norway. , (Norway)
  • 61 NVIDIA GmbH, München, Germany. , (Germany)
  • 62 Institute for Computational Biomedicine, Heidelberg University, Heidelberg. Germany and Faculty of Medicine, Heidelberg University Hospital, Heidelberg, Germany. , (Germany)
  • 63 Informatics Institute, Faculty of Science, University of Amsterdam, Amsterdam, The Netherlands. , (Netherlands)
  • 64 Google Health, Google, CA, USA.
  • 65 National Institutes of Health Clinical Center, Bethesda, MD, USA.
  • 66 Institute of Information Systems Engineering, TU Wien, Vienna, Austria. , (Austria)
  • 67 Research Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland and Neurocenter Oulu, Oulu University Hospital, Oulu, Finland. , (Finland)
  • 68 School of Engineering, The University of Edinburgh, Edinburgh, Scotland.
  • 69 Department of Development and Regeneration and EPI-centre, KU Leuven, Leuven, Belgium and Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands. , (Belgium)
  • 70 Parietal project team, INRIA Saclay-Île de France, Palaiseau, France. , (France)
  • 71 National Institute of Allergy and Infectious Diseases, National Institutes of Health, Bethesda, MD, USA.
  • 72 German Cancer Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group and HI Helmholtz Imaging, Germany. , (Germany)
  • 73 German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems and HI Helmholtz Imaging, Germany, Faculty of Mathematics and Computer Science and Medical Faculty, Heidelberg University, Heidelberg, Germany, and National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Medical Center Heidelberg, Germany. , (Germany)
Type
Published Article
Journal
ArXiv
Publication Date
Sep 25, 2023
Identifiers
PMID: 36945687
Source
Medline
Keywords
Language
English
License
Unknown

Abstract

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

From MEDLINE®/PubMed®, a database of the U.S. National Library of Medicine. This record was last updated on 10/28/2023 and may not reflect the most current and accurate biomedical/scientific data available from NLM. The corresponding record at NLM can be accessed at https://www.ncbi.nlm.nih.gov/pubmed/36945687
Report this publication

Statistics

Seen <100 times