Artificial Intelligence Bias in Healthcare

Artificial intelligence (AI) has the potential to revolutionize healthcare delivery. With applications in decision support, patient care, and disease management, it is fast becoming an industry standard. AI helps clinicians work smarter while improving patient outcomes, from the machine learning algorithms that read patient scans more accurately, to natural language processing facilitating searches through unstructured data in electronic health records.

Yet, this advancement brings with it the very real possibility that AI tools, trained with data that reflects historical and contemporary cognitive and societal biases, could unwittingly perpetuate or even amplify bias in healthcare delivery.

The growing use of AI across many industries has sparked many recent debates on bias and fairness. As AI-powered tools gain momentum, stakeholders have a critical window of opportunity to develop approaches for addressing biases in algorithms. Implementing safeguards will be vital in ensuring AI helps improve patient outcomes across diverse communities and avoids perpetuating today’s inequalities.

Human Bias and How It Affects Healthcare Decision Making

While there are many real and potential benefits from using AI in healthcare, an important risk is flawed or controversial decision making caused by human bias embedded in AI outputs. Bias occurs when we discriminate against a particular group—either consciously through explicit preconceived ideas such as racism and sexism, unconsciously through ingrained thoughts based on assumptions or stereotypes, or inadvertently through use of data skewed towards a particular segment of the population. Consider the following examples:

• Patients of color in emergency rooms are less likely to receive pain management compared to white patients. A systematic review of pain management studies found that black patients were 40% less likely to receive pain medication in U.S. emergency departments as compared to white patients. Hispanic patients were 25% less likely. “Emergency room clinicians may be choosing which patients get pain relief based on conscious, unconscious, and implicit bias as well as negative stereotypes based upon race, ethnicity, and class,” said Dr. Carmen Green, a professor of anesthesiology at the University of Michigan schools of medicine and public health in Ann Arbor.
  
  
• Stigma and bias can sometimes impact the treatment of obese patients. The clinical complexity of obesity may be overlooked when health providers oversimplify and assume that these patients cannot conform to lifestyle changes such as modifying their diet or exercising to control their weight. Providers may also erroneously attribute other health issues to the obesity instead of investigating other potential causes, thus delaying diagnoses and treatment.

How Cognitive Bias Can Influence AI

The intent of AI is to help healthcare providers make more objective decisions and provide more efficient care. AI technology relies on data to train machine algorithms to make decisions. If you want to teach a machine to estimate a factor such as disease prevalence across various demographics, you feed it millions of data records and tell it how to identify the target groups and contributing factors. From this, the machine learns how to make relevant distinctions. If the underlying data is inherently biased or doesn’t contain a diverse representation of target groups, the AI algorithms cannot produce accurate outputs.

For example, if the data used for an AI technology is gathered only from academic medical centers, then the resulting AI model will learn less about patient populations that do not typically seek care at academic medical centers. Similarly, if using a military health data source, the AI model will yield less about the female population given that most service members are male. Such biased data can lead to delayed or inappropriate care delivery that results in harmful patient outcomes. Therefore, it is important to ensure data used to train AI is representatively diverse to help mitigate potential harm to the public (and especially to historically marginalized populations).

The Human Cost of Biased AI in Healthcare

Bias built into artificial intelligence has societal, legal, and monetary implications. Take the recent case of UnitedHealth Group's Optum division. The organization was unaware of the biases in the data used to develop a product marketed and sold to hospital systems to help identify vulnerable people requiring care management. The algorithm used past spending data to predict severity of illness without considering broader societal factors that result in racial inequities in the amount of care received. Since less money was spent historically on black patients with the same level of need as their white counterparts, the algorithm falsely assigned black patients to the same level of risk as healthier white patients. A study of this particular biased algorithm determined that it could underestimate the number of black patients in need of greater care by half. Altering the algorithm to remedy the bias would increase the percentage of black patients receiving additional help from 17.7 percent to 46.5 percent.

While the potential human costs of biased AI should remain our central concern, we should also recognize the legal and financial implications that inappropriate, inaccurate, or discriminatory algorithms can have for organizations. UnitedHealth did not meet the expectations of their client base who relied on the company to provide accurate information on which patients to target for care management services. New York regulators have recently called upon UnitedHealth to either prove that the algorithm is not discriminatory or stop using it. If the company is found to be in violation of New York’s antidiscrimination laws, it could face financial penalties from the state. 

To realize the promise of AI in healthcare, we must work collaboratively to address bias.

With the use of AI rapidly expanding throughout healthcare, it is imperative that stakeholders take steps to address algorithmic bias now. If the goal is to ensure that AI creates more equitable healthcare solutions than what we have today, then we need better mechanisms to balance its risks and benefits. It will require collaboration between data scientists, healthcare providers, consumers, and regulators to address the complex issues of AI bias; algorithm fairness and accuracy; ethics and safety; and governance and oversight.

Next in the series: Identifying Bias in Hospital Length of Stay Algorithm