Identifying Bias in Hospital Length of Stay Algorithm

Hospitals can be dangerous places for patients. While in the hospital, patients run the risk of contracting healthcare-acquired infections such as sepsis and Clostridioides difficile or experiencing other adverse events such as falls. Hospitals have limited resources, requiring efficient use of beds and clinician time. For these reasons and more, it is in the best interest of patients, hospitals, and public health to limit any hospital stay to no longer than necessary. 

Recognizing the need to support shorter lengths of stay, Dr. John Fahrenbach, a data scientist at the University of Chicago Medicine (UCM), developed a machine learning model that used clinical characteristics to identify patients most suitable for discharge after 48 hours. Using this tool, the hospital could ensure the timely release of specific patients by allocating and prioritizing care management resources, including discharge planning, home health services, and clinician or patient administrative assistance.

During the development process, Dr. Fahrenbach’s team determined that including zip codes as a feature increased the model’s predictive accuracy. After introducing zip codes into the model, however, a team member who reviewed the output raised concerns. “We know Chicago’s patient population and knew something was off when stratifying the model by race,” said Dr. Fahrenbach.

The zip codes in UCM’s service area reflect the city’s racial geography. Trends in the data showed patients from predominantly Caucasian and relatively affluent zip codes were more likely to have shorter lengths of stay than others. Conversely, the zip codes correlating to longer lengths of stay were predominantly African American and limited income. Had the model been deployed, it would have resulted in prioritizing care management services to a population with less need for those services, diverting them from patients more at risk and in greater need of enhanced care management. 

To confirm the model perpetuated historical biases in the UCM patient population, the team tested it for racial bias by stratifying the results by race. This test of the model confirmed the concerns, and the team halted the development process.

Dr. Fahrenbach’s team met with UCM’s diversity, inclusion, and equity department to develop approaches to avoid such biases in the future. Within UCM’s diversity department, the team collaborated with Dr. Marshall Chin, physician, professor of healthcare ethics, and expert in social determinants of health. Dr. Chin observed that the data analytics team “had good intentions to improve efficiency and patient safety, but problems arose because they initially did not have an equity lens when developing the algorithm.”

According to Dr. Chin, healthcare AI needs to apply “an equity lens at all stages of development and deployment to avoid negative unintended consequences—and better yet, have a positive effect on health equity.” The diversity department and the data science team worked together to create “equity checkpoints” to identify sources of bias and unfairness in future AI applications at each stage of development: conceptualization, development, deployment, and monitoring. At each equity checkpoint, the team asks themselves if they are improving the health outcomes of all patients or unfairly treating any at-risk patient group.

With no off-the-shelf products for facilitating equity checkpoints in healthcare-focused AI applications, Dr. Chin and Dr. Fahrenbach developed their own equity checkpoint system using a few general principles: 

• Examine the algorithm with bias in mind
  Audit your algorithm for potential sources of bias and outputs that contribute to unequal outcomes among patient populations.
  
  
• Look for flaws in your data that could lead to bias
  Evaluate your data in search of flaws that could result in biased algorithmic outputs. For example, is the available data for socially at-risk patient populations less accurate or complete than the data available for more advantaged populations?
  
  
• Take steps to ensure fairness
  Consider how to ensure fairness across different patient populations when the models are developed and deployed. This could include adjusting thresholds within the model to ensure equal outcomes and allocation of healthcare resources.
  
  
• Use continuous monitoring after the algorithm is deployed
  After deployment, developers and users should continue to monitor data for inequalities that may have been missed during development or can only be detected in a real practice setting.
  
  
• Listen to providers and patients
  Across all stages of development and deployment, developers should conduct discussions with both providers and patients. The voice of patients—the end users of clinical care—is especially important. 

“The AI field is fluid and evolving quickly. You need to run equity checks on off-the-shelf AI algorithms since you usually can’t modify these AI algorithms. If major equity issues are found, this should be addressed immediately before deployment,” said Dr. Fahrenbach. This case demonstrates the importance of safeguards in AI development to prevent the deployment of an algorithm with potentially unfair and even harmful effects on patients.

This case also illustrates just how crucial it is to account for factors such as social risk and other social determinants of health when developing healthcare AI. Appropriate representation of socially at-risk patient populations in machine learning datasets is essential to ensuring these tools serve to improve health equity.

As an example, the National Institutes of Health’s All of Us initiative aims to collect genetic and health data from one million Americans—particularly those from socially at-risk groups that have been historically underrepresented in medical studies and datasets. By 2020, half of the participants in the program were racial and ethnic minorities, potentially expanding knowledge about people who have historically been overlooked in clinical research.

With data such as those collected from the All of Us initiative, researchers can identify and address occurrences of underdiagnosis in smaller health system datasets by comparing them to these types of broadly representative datasets. To advance health equity and access for patients from marginalized communities, datasets must enlist criteria for determining how patients in these communities differ from the population overall.

Finally, the UCM experience demonstrates that, to address algorithmic bias, we must cultivate a strong feeling of psychological safety within development teams. Psychological safety is a concept rooted in organizational theory that seeks to empower team members to feel safe bringing up ideas or concerns that may differ from the thinking of the overall team. The issue with including zip codes as a variable in the length of stay algorithm was initially raised by a member of the development team. Without UCM’s culture of psychological safety, this team member may not have felt empowered to speak up and raise a potentially harmful issue with the model outputs.

UCM’s success with identifying and addressing bias in its length of stay algorithm demonstrates it is possible to put in place protocols that ensure the needs of socially at-risk patient populations are addressed when designing, deploying, and evaluating AI algorithms in healthcare. Ultimately, the length of stay algorithm was not deployed. But it did provide a useful learning experience for implementing processes that will enable the UCM team to identify sources of bias when developing healthcare AI models in the future.

Want to learn more about bias in healthcare AI? Read our next blog, How to Mitigate Bias in Healthcare Algorithms.