Article Text

Download PDFPDF
Trustworthy medical AI systems need to know when they don’t know
  1. Thomas Grote
  1. Ethics and Philosophy Lab; Cluster of Excellence: Machine Learning: New Perspectives for Science, University of Tübingen, Tübingen, Germany
  1. Correspondence to Thomas Grote, Ethics and Philosophy Lab, University of Tübingen, Tübingen 72076, Germany; thomas.grote{at}

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

There is much to learn from Durán and Jongsma’s paper.1 One particularly important insight concerns the relationship between epistemology and ethics in medical artificial intelligence (AI). In clinical environments, the task of AI systems is to provide risk estimates or diagnostic decisions, which then need to be weighed by physicians. Hence, while the implementation of AI systems might give rise to ethical issues—for example, overtreatment, defensive medicine or paternalism2—the issue that lies at the heart is an epistemic problem: how can physicians know whether to trust decisions made by AI systems? In this manner, various studies examining the interaction of AI systems and physicians have shown that without being able to evaluate their trustworthiness, especially novice physicians become over-reliant on algorithmic support—and ultimately are led astray by incorrect decisions.3–5

This leads to a second insight from the paper, namely that even if some (deep learning-based) AI system happens to be opaque, it is still not built on the moon. To assess its trustworthiness, AI developers or physicians have different sorts of higher order evidence at hand. Most importantly, …

View Full Text


  • Funding TG is supported by the Deutsche Forschungsgemeinschaft (BE5601/4-1; Cluster of Excellence ‘Machine Learning—New Perspectives for Science’, EXC 2064, project number 390727645).

  • Competing interests None declared.

  • Provenance and peer review Commissioned; internally peer reviewed.

  • Note, while Durán and Jongma emphasise that physicians should not blindly defer to algorithmic decisions due to choices of value, my account is confined to the epistemic part of his paper.

  • For a more detailed take on UQ, see ref 7.

Linked Articles