How to Mitigate Bias in Healthcare Algorithms

Healthcare systems are steadily adopting technologies driven by data analytics. Evidence suggests artificial intelligence (AI) and machine learning (ML) will increasingly play a role in patient care. This is, of course, cause for excitement—these technologies have the potential to advance healthcare in various subtle and revolutionary ways. 

But there's also cause for concern. The racial, gender, and socioeconomic biases too often present in AI models have the potential to harm already marginalized populations. Left unaddressed, models based on biased data may reinforce or magnify the inequities in the healthcare system.

To create an ethical AI ecosystem, organizations must commit to analyzing AI for bias and fairness and ensuring transparency and accountability. As AI and healthcare continually evolve, there's no one-size-fits-all solution. Instead, the problem must be addressed through interdisciplinary collaboration and from multiple angles including federal regulation, data governance, and data science accountability.

We propose the following controls as an initial foundation and guideline for arriving at a comprehensive approach for eliminating bias in AI: 

Interdisciplinary Collaboration with Patient Advocates

As University of Chicago Medicine’s (UCM) attempt to develop a length of stay algorithm suggests, developers should begin their bias mitigation efforts by seeking partnerships with two key stakeholders: diversity, equity, and inclusion professionals, and physicians with a deep understanding of the patient population(s) in question. 

UCM’s story is far from the only one that supports this point. In another case, a Canadian health AI startup developed a model to analyze speech for early identification of neurological diseases such as Alzheimer’s, Parkinson’s, and multiple sclerosis. The model was trained exclusively on speech data from English-speaking Canadians. When the tool was trialed with clinicians on a broader set of subjects, the model incorrectly classified patients from certain groups as likely to have early onset neurological disease. This included individuals who spoke a dialect other than Canadian English and those who were learning English as a second language.

Clinicians were immediately able to discern the differences between variable dialects and actual signs of neurological disease but this early version of the model was not. This type of collaboration between developers and clinicians should be established as a requirement prior to commercialization and clinical integration of applicable AI technologies. 

Federal Regulation on Quality Assurance of Health Data Protocols

Federal agencies now have a vital opportunity to lead the way in demonstrating how to incorporate diversity, equity, and inclusion as core components of the quality assurance protocols for model development, operations, and maintenance. In June 2019, the American Medical Association called on the FDA to do just that, asking the agency to develop better guardrails against bias in AI that may exacerbate health disparities, particularly when deploying AI tools on vulnerable populations.

In December 2019, Senators Cory Booker and Ron Wyden wrote letters to the Centers for Medicare and Medicaid Services (CMS) and the Federal Trade Commission (FTC) requesting insights into what practices the agencies were implementing to combat errors related to AI bias. Those same senators teamed with Rep. Yvette Clarke to introduce the Algorithmic Accountability Act, which directs the FTC to enforce private sector implementation of bias assessments. While these actions are valuable steps forward, we must continue to strengthen partnerships between private industry and regulators to ensure that the pace of technological integration does not outpace federal responses.

Data Governance to Ensure Diversity

Often, the data used to develop algorithms is not representative of diverse populations. In genomics research, representation bias is an issue that has been acknowledged for over a decade. In 2009, 96% of participants in genome-wide association studies were of European descent. By 2016, representation of non-European ancestry had increased by 20 percent, but that boost largely came from Asian participants. African and Latin American ancestry representation had barely changed in the seven years since the initial analysis.

AI models, genomics research, precision medicine, and other technologies must be built upon representatively diverse datasets if we are to use them to diagnose and treat diverse patient populations. The fact that it takes so long to achieve even modest gains in the diversity of genomic data demonstrates the potential risks of pushing forward with broad AI integration before these challenges are resolved. 

Make Data Science Accountable 

To address AI bias in a meaningful way, we must use quality control strategies to prevent unfair differential outcomes between diverse populations. To this end, the Alan Turing Institute recommends several checklists and toolkits that can help find potential biases during the development cycle. Google researchers have contributed to the effort with published research around bias in natural language processing algorithms and addressing geographic diversity in unrepresentative datasets.

Tools like these are a good start but they do not specifically address the challenges of AI in the healthcare industry. To map a comprehensive path for ethical and equitable AI development, we’ll need to expand upon such frameworks. The result of such efforts will be an AI ecosystem that reduces the risk for disparate impacts on vulnerable populations and limits the risk of legal and financial implications related to poorly developed and implemented AI. 

What Does AI-enabled Healthcare Look Like in the Future?

To date, the integration of AI technologies hasn't reliably narrowed the disparities in our healthcare system. Producing solutions that can fulfill the promise of AI in healthcare will require a sustained effort to develop comprehensive controls. Unbiased AI has the potential to improve care quality, foster patient engagement, predict optimal therapeutic options, and more. AI could enable us to make great leaps in reducing inequality and improving care across diverse and marginalized communities. If, instead, it merely places a veneer of objectivity onto care and diagnosis decisions, we will have failed.