Disagreement 1: views about whether algorithmic bias exists

The majority of participants took a bias-critical view. These experts were mainly clinicians and regulators, with a small number from other roles. These participants noted potential harms of algorithmic bias, ranging from medico-technical harms (eg, compromising patient safety) to broader social and public health harms (eg, perpetuating health inequity). Participants implied that such harms could undermine the promise of healthcare AI.

And …the researchers … discovered that, in this risk assessment, many essentially produced by the algorithm, there was a bias against African Americans. … if you are black American… you are certainly more likely to be mislabelled as low risk if you actually were high risk. Informant 22, regulatory expert

Some participants taking the bias-critical view also critiqued misconceptions about AI or computer systems that could lead people to dismiss the problem of algorithmic bias. The misconceptions include misguided faith in AI systems as objective and infallible.

For people who are less informed—have this view that kind of computers or machines or programs are sort of objective and they are black and white in their coding and so they are right or they are wrong and they won’t make mistakes, so to speak. I guess, my position on it is that some of the fallibilities that these systems have are based on the same problems that the human brain has in that it can only work with what’s been put into it. Informant 45, consumer representative

I think that there’s subjectivity in a lot of [AI] that we act like it’s very objective but it’s not. We’re making all kinds of assumptions in our machine learning, algorithms, in the default or the parameters that we set. Informant 56, researcher in bioinformatics

Two other views were expressed by participants. A small group of participants from all roles conveyed a bias-apologist view. On this view, bias was recognised as a problem to some extent, but not seen as an ethical concern—or at least, did not raise any more ethical concerns than the status quo. Some participants argued that bias in AI is unintentional. In contrast to the bias-critical view, the bias-apologist view emphasised the potential benefits of AI, despite the presence of bias. Participants expressing this view argued that bias was an ongoing problem even among clinicians—and AI bias was preferable to human bias.

Obviously an algorithm will have biases as a human has biases, but those biases are able to be measured. Informant 1, screening manager

First of all I’d say, what, and doctors aren’t biased? I’m sorry. Don’t get me started on doctor bias. We’re used to hurting people the old way, with biased old doctors. Now we’ve got a new set of biases, what are we going to do with them? But it is a valid point. Informant 55, entrepreneur

A third small group expressed a bias-denial view. These participants included developers, entrepreneurs and screening programme managers. No regulatory or legal expert subscribed to this view. Participants who expressed this view argued that bias existed in other industries but not medicine, or existed in other countries but not in Australia.

I don’t know that bias is one that we’ve seen any evidence that is being realised here [in Australia]. I think it’s more of a concern in health systems like the US. Informant 2, entrepreneur

Now imaging, you mentioned imaging? I don’t think there’s any good data or bad data in imaging. An image is just an image that you create from a machine, or like a CT scan or an x-ray. And then you feed that to the machine and it’ll detect and whatever it finds, it finds. Informant 52, developer

Some participants expressing the bias-denial view argued that bias was not likely to occur in medical diagnosis because imaging technologies are objective and the human body is universal, a direct contrast to the bias-critical view that claimed AI systems are not always objective or infallible.

I can only tell you that in radiology, if we do a C-T scan on your brain and on my brain, it will look exactly the same with a variation of black and white pixels. It doesn’t matter if you’re black, or white, or green, or blue, or whether you’re Australian or Chinese, a rib fracture is a rib fracture and the same is true for a vertebrae fracture or a pneumothorax. There is no bias there. Informant 59, clinician

Actually, I’d have no idea with the pathology samples what a person looks like so we don’t even know that. And I don’t know, you’d have to ask the doctor. I think the internal organs from people from different race, backgrounds, are probably pretty similar but I don’t know. Informant 50, developer

Disagreement 2: views about acting to mitigate algorithmic injustice

Participants who expressed the bias-denial view and some who expressed the bias-apologist view implied that no action was required to address bias because according to their views, bias was not a problem in healthcare AI. Some ‘bias-apologist’ participants, mainly developers and entrepreneurs, agreed that bias was a problem but proposed that it was unintentional (see figure 1). Consequently, these developers and entrepreneurs deemed that they were not responsible for the creation of bias, and therefore, held no obligation to redress it. Some ‘bias apologist’ participants who were all entrepreneurs, researchers or clinicians involved in AI development, argued that they were just working with what was available, implying that research and development should not be hampered by concerns about bias.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Views about algorithmic bias as a problem.

We’re definitely using biased data sets. So part of my response is that we’re using the data we can get. Informant 23, developer and researcher in bioinformatics

Bias-critical and some bias-apologist participants subscribed to the action-required view, which explicitly calls for action to address the problem of bias. There were two subgroups within this view, distinguished by their views on whose responsibility it is to mitigate bias (see figure 1). In one subgroup, participants, mainly developers and data scientists, maintained that mitigating bias was the responsibility of experts outside science and technology, as illustrated by the statement:

I think people do understand that there is a problem. But as always the question is how do we—I mean from a science point of view the way to address it is to make sure that we have good representation, that we have a wide enough number of samples from all possible classes, or some groups and so on. But how do we ensure that? And that goes outside technology. That goes outside science really … we need more data, and more of the right data. But how we do it I think is outside science and technology really. Informant 51, developer

In contrast, the second subgroup consisting of experts from various roles argued that all professional stakeholders, including data scientists and developers, shared responsibility for and played a role in mitigating AI bias.

So we know about racism, so we expect that we need it to be tested. We know about sexism, so we know it needs to—we know ageism, and we know that that needs [to be tested]. Informant 21, entrepreneur

You’re only as good as the data. And the data is Caucasian predominantly. So—this is another element why human element needs to be in there to, sort, of take the context into account for doing—for assessing the prediction. I think, here, in Australia there’s a special issue with the Indigenous population, for example, where there’s certain diseases that’s much more prevalent in the Indigenous community and it’s just not understood why. Informant 24, researcher in bioinformatics

Both of these two positions—the view that everyone, including data scientists, were responsible, and the view that responsibility lay outside data science—were seen in both bias-critical and bias-apologist informants: there was no observable relationship between these positions.

Those participants who expressed a view that action was required offered mechanisms that fell broadly into two categories, sociolegal approaches and data science mechanisms. Participants who suggested data science mechanisms were predominantly clinicians, developers and researchers. The mechanisms included data governance, validation tests and use of synthetic data sets, among others (see figure 2).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Sociolegal and data science approaches to mitigate bias described by participants.

If it remained impossible to remove bias in AI algorithms, some participants recommended open disclosure of limitations: that is, that when communicating about the predictions or recommendations generated by AI, to always openly acknowledge that these were biased or partial.

So hugely important not to have bias, and hugely important to recognise the bias more than anything else … when we publish these trials. We don’t say, this algorithm is great at reading chest x rays, we say this algorithm is great at reading chest x rays in Caucasian males over the age of 70, for example. And for clinicians, we talked a bit before about how much they need to understand about AI. They don’t need to understand anything about AI before they understand everything about statistics. Because how that data is presented, how there are confounders how there is bias, that part is more interesting than the AI. Informant 54, clinician

Other participants, predominantly regulatory and legal experts, discussed sociolegal approaches—in addition to data science mechanisms—to mitigate algorithmic bias. These approaches included consumer and patient engagement, representation of marginalised groups, incorporation of equity considerations into sampling methods, and legal recourse.

You might essentially violate [human rights] if you create an algorithm that produces biased results, so I think—I don’t think—I don’t know if there have been any cases yet, but I think you could not only end up with ethical problem, but also with a legal problem, infringement of human rights, so from legal perspective. So when people talk about bias as an ethical principle, I think they often forget that it’s also a legal issue, so that’s as far as current law is concerned. Informant 32, regulatory expert

In addition to describing different approaches to bias mitigation, participants identified a wide range of stakeholders who may be responsible for addressing AI bias, and these included developers, healthcare workers, manufacturers and vendors, policy makers and regulators, AI researchers and consumers. The numerous responsibilities ascribed, which varied in scope, illustrated the complexity of algorithmic bias and potential ways to address it. See figure 3 for a summary. We emphasise that this is a summary of the views expressed by participants rather than our own normative position.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Summary of informants’ perspectives on stakeholder responsibilities regarding bias mitigation.

Noting the diversity of possible solutions, as well as the type of stakeholders that should be involved in these solutions, participants identified the need for interdisciplinary collaboration among experts. In particular, an interdisciplinary approach could capture the sociolegal and technical dimensions of the problem and causes of bias, as well as mechanisms to correct causes of and mitigate the impact of bias.

That’s where you need clinicians working with the model developers at the very outset, okay, to say if we’re going to use this tool in a certain way this is—these are the peculiarities of our population, so we need to make sure that we sample the population adequately to make sure that we’re not producing biased data. … So we can’t have I think model developers—model developers by themselves, or data scientists just in isolation. You’ve got to have clinicians and others who have a real understanding of the population that we’re dealing with. Informant 41, clinician

So when we find technical solutions probably we will get next to standards level, to set standards that are [the] minimum standard that each AI application has to meet to … satisfy this fairness criteria and then when we have standards we can then validate—put them in laws and say you will not be liable, let’s say, for breach of discrimination … if you comply with standards in the industry. Informant 32, regulatory and legal expert

Disagreement 3: views on the role of social identities in addressing bias

One particularly challenging issue when discussing mechanisms to address bias was the role of information regarding complex social identities (eg, race/ethnicity). Participants had divergent views about the role of social data (eg, information about ethnicity or income). A small number of participants, mostly researchers and developers, took an exclusionary view of social data in AI development. This view meant that in order to minimise algorithmic bias, social identities subject to prejudice should be masked or excluded in datasets.

If you are, for example, putting the ethnicity of certain groups as a variable, as a basic variable, and you put much weight of this variable, the whole algorithm will be biased towards or away from these groups. But actually you don’t need to put that because these classifications are human classifications, they are not scientific classifications. Informant 4, developer

Some participants took the inclusionary view of social data in AI development, as well as in addressing algorithmic bias. This heterogeneous group included entrepreneurs, clinicians, consumer representatives and regulatory experts. Several reasons to support the inclusionary view were provided. First, social information could make the dataset more robust and representative.

If you learn through things like race and ethnicities, if it captures employment status, if we don’t have things even such as housing status, sort of include information around sort of behaviours, substance abuse, alcohol abuse, et cetera, that those things aren’t there and EMR [electronic medical records], as we know, sometimes can omit the sort of information unless you have good structured templates where those things are listed and you have to actually include them, okay but we know that often people don’t necessarily provide all that information when they’re admitting a patient or even subsequently. Then, yeah—then that’s right, you’re not going to have a particularly accurate model or that they have a model that’s biased because it doesn’t include those other features. Informant 41, clinician

One of the ones I notice often when I’m looking at surveys or things that ask me about my features, because I’m taking part in a research study, you know, if you look at lists of gender and sexuality, like, asexuality, for example, will be missed from a lot of “pick-an-option” about your sexuality. Right? So, again, people don’t choose it, it doesn’t end up in the system, it doesn’t end up in the coding, it’s more invisible again and it becomes a bit of a cycle, unfortunately. Informant 45, consumer representative

Another reason that supported the inclusionary view was to enable healthcare systems to identify which groups were underserved.

And we also need to know their sociodemographic, and so there is publicly available data that you can connect up but there are people who don’t turn up and they’re the ones that are missing out often. So we need to get the census data, we need to get everything and go, “Well, okay, who are we missing?” So I think we can figure out who’s not connected with our system. They’re either not connected because there’s a reason or they’re well. Informant 31, regulatory expert

Some participants who supported the inclusionary view discussed caveats, such as increased (unjust) surveillance.

I get the argument of yes, there is some truth to that argument, that the more data you have, the more—I mean again, to give you an analogy, you could make similar arguments in the biomedical sphere as well. You could say we would get to cures or medical treatments faster if researchers were allowed to do whatever they wanted in the research phase. Just trial this drug that may kill people on people in prison. That would work. But we don’t do it for really good reasons and that is because we don’t accept that level of harm to innocent people. And that’s true also of, I think, research in this area and again, we’ve talked a bit about facial recognition. We know that facial recognition is much less accurate when it’s used on people with darker skin. And in an area I care deeply about, policing, there’s an argument, okay, we’ll make the training data sets a high proportion of those people with darker skin. But if the outcome of that is that those people will be policed more aggressively, that’s really hard to justify. So I think it’s quite a complex area… You’ve got to actually weigh the costs and benefits and specifically consider the uses to which the application may be put. Informant 37, regulatory expert

A substantial number of participants, mainly developers and regulatory experts, took neither the exclusionary nor the inclusionary view. This uncertain view about the role of social data in AI development took into account the challenges of collecting sensitive and complex information, and translating this into useful data for developing AI algorithms. Data science relies on categories and classifications that are discrete, a feature that is not always possible for social data.

When you say, “What about the culturally and linguistically diverse population?” Right, but for useful purposes you do need information to be much more granular, and that requires people to self-identify and that can be very tricky. I mean, most people don’t simply have one background, they have multiple—I’ve got 10, so what do you use? Informant 10, regulatory expert

We don’t capture all the data we need and even if we could capture it all actually making sense of it all and understanding which of these many, many variables actually matter is really unclear. Informant 38, developer

Thus, the findings show lack of consensus on how to manage, operationalise or capture complex social identities to mitigate algorithmic bias. The divergent views illustrate the tension between ensuring diversity in datasets and the potential harm of collecting data points marked by historical and continuing prejudice or discrimination.