Table 1

Changing modes of interaction and their entanglement with different normative notions

Normative notionsInteraction modes
Clinician–AI-DSSClinician–patientPatient–AI-DSS
Agency Description: introduction of AI-DSS competence and authority alongside clinician competence.
Challenge: spectrum ranges from augmentation to replacement.
Consequence: need to prioritise and to mediate between clinician and AI-DSS judgements in case they diverge.
Description: introduction of additional evidence at the centre of shared decision-making.
Challenge: finding context-sensitive modes of participation in collaborative evaluation of outputs.
Consequence: need for enhanced and standardised sensitivity to clinician and patient attitudes and expectations about roles in decision-making informed by AI-DSS.
Description: interplay between AI-DSS ideally ameliorating individual patient care and patient data learning and refining AI-DSS.
Challenge: sharing of large amounts of health data as a necessary condition for using medical AI.
Consequence: need for mechanisms to exercise informational self-determination through awareness of available options, data flow controllability and privacy by design.
Trustworthiness Description: room for errors across data collection, model training and implementation; significance of AI-DSS reliability, validity and attunement to task.
Challenge: user-friendliness, safety and efficacy, potential burden of proof for clinician who deviates from AI-DSS recommendations.
Consequence: need for clinician training to evaluate reliability, validity and attunement.
Description: perceived transformations to clinical decision-making processes.
Challenge: shifts and extensions of conceptions of clinician competence and communication, expectation that human clinician remains in the loop.
Consequence: need for new forms of counselling and communication pathways.
Description: emerging conditions of trustworthiness for deployment of AI-driven technology from the patient perspective.
Challenge: varying degrees of background knowledge and openness for AI-driven clinical tools, possibility that contact to human clinicians becomes a luxury.
Consequence: need for provision of clear information on opportunities, challenges, data protection, procedures and addressees for damage claims, and humans in the loop.
Transparency Description: AI uncovering correlations without necessarily eliciting underlying causal relations.
Challenge: lack of evidence on how and why AI-DSS arrives at a given output.
Consequence: need for safety and efficacy validation and certification markers.
Description: black box issues coupled with information asymmetries between clinician and patient.
Challenge: asymmetries in ability to assess and reflect on AI-DSS outputs, challenges in attributability of recommendations.
Consequence: providing information and education about AI-driven clinical decision-making, clarifying the role and competence of the clinician relative to the AI-DSS.
Description: black box issues aggravated by lack of clinical and technical background.
Challenge: making the meaning, quality and limitations of AI-DSS outputs transparent to patients.
Consequence: need for provision of clear information on the nature of AI-DSS outputs, for example, through visualisations and innovative patient interfaces.
Responsibility Description: partial shift of responsibility for diagnosis, recommendations and decision-making from clinician towards AI-DSS.
Challenge: problem of many hands in AI development and implementation, complicating attributions of responsibility for malfunctions.
Consequence: need for clear and context-sensitive principles for how ethical and legal responsibility distributes across multiagential structures.
Description: changing nature of clinician responsibility in view of informational asymmetries between clinician and patient.
Challenge: collaboratively assessing risk-benefit ratios.
Consequences: need for training and education for clinicians and patients towards responsible utilisation of AI.
Description: consideration of and reliance on AI-DSS outputs when patients make decisions on their own authority,5 paternalism versus new forms of individual choices.
Challenge: mediating between self-interest, solidaric participation and backlashes at the level of justice.
Consequence: need for education and counselling.
  • AI-DSS, artificial intelligence-driven decision support system.