As government leaders increasingly recognize investments in artificial intelligence (AI) as a national imperative, the number of public-private partnerships focused on using machine intelligence to address some of the nation’s most pressing problems has steadily grown. Booz Allen has been dedicated to leveraging our public-sector alliances to connect professional, academic, and citizen data scientists with public interest initiatives since we launched the first Data Science Bowl in 2015. 

This competition to harness data science for social good has become a tradition we’ve repeated five times, with partners including Children’s National Hospital, the National Cancer Institute, and the Hatfield Marine Science Center. With this legacy in mind, Booz Allen is proud to have been part of the winning team for the largest-ever healthcare-focused AI competition, the Centers for Medicare & Medicaid Services (CMS) AI Health Outcomes Challenge.

For the Challenge, Booz Allen teamed up with Austin-based health data science startup ClosedLoop.ai. “Booz Allen and ClosedLoop were discussing collaboration opportunities long before the Health Outcomes Challenge,” says Booz Allen Senior Vice President John Larson. “We were already aware of the power of their platform, and, when the opportunity arose to work together to further CMS innovation initiatives, we were excited to join forces.”

In defining the Challenge, CMS asked competitors to investigate “how AI tools—such as deep learning and neural networks—can be used to accelerate development of AI solutions for predicting patient health outcomes for Medicare beneficiaries for potential use in CMS Innovation Center innovative payment and service delivery models.” The ClosedLoop-Booz Allen strategy to meet this goal merged three essential elements into a winning solution:

Data handling and preprocessing remain a key challenge in applying AI and deep learning to healthcare applications. Leveraging ClosedLoop’s platform, the team was able to quickly (1) identify high-value data features; (2) perform data cleaning, integration, and feature engineering; and (3) manage datasets used for different modeling experiments. ClosedLoop’s platform allowed our data scientists to hit the ground running and shortened the turnaround time for methods exploration and benchmarking.

In line with the Challenge objectives, the team employed advanced methods to achieve key goals:

• Maximize model accuracy: The research aimed at improving healthcare through predictive analysis is expanding rapidly, making it challenging for organizations to determine the optimal methodology for their use cases. Using the ClosedLoop platform, our team of data scientists quickly built and tested various models. The team’s goal was to test traditional machine learning libraries (e.g., XGBoost) while also experimenting with newer methods, such as Booz Allen’s deep learning methodology. Our work demonstrated that the individual XGBoost and deep learning models both performed well independently, but, when assembled together, these models produced the team’s best result.
  
  
• Support model explainability: In many settings, and especially in healthcare, it is critical for decision makers to understand the variables driving model predictions. Beyond accuracy, the transparency of a model is critical to increasing the adoption of AI-enabled decision support capabilities. To provide explainable predictions, our team combined Shapley Additive Explanation (SHAP) values for XGBoost with an attention mechanism from the deep learning model to reveal the input variables most influential to the final model output (e.g., recent diagnoses, demographic data, environmental factors).
  
  
• Mitigate algorithmic bias: To create the best model for predicting adverse events and mortality, we placed a strong emphasis on developing an ethical AI solution. Using several procedures—including subpopulation model performance analysis, U.S.-population-to-CMS-dataset demographic comparisons, and model unfairness corrections —we investigated how sensitive our model predictions were to patient demographics and other attributes. Through this analysis, we provided multiple recommendations for mitigating discrepancies, including attribute masking techniques and synthetic data mitigation.

Many organizations are struggling to realize the anticipated value from their AI investments. Part of our team’s strategy for pulling together a winning submission was to engage CMS data experts, clinicians, and health system administrators to review model outputs and user interfaces. This approach ensured that our solution presented precisely the information health professionals needed to optimize their decision making. The collaboration with domain experts resulted in the development and display of a host of relevant end-user data, including contextualized risk scores with associated confidence qualifications and patient-specific intervention recommendations.

The Challenge served as validation for an approach Booz Allen has successfully used to improve the veteran experience; combat fraud, waste, and abuse; augment health monitoring; and provide numerous other AI-enabled solutions to our clients. As Booz Allen and ClosedLoop continue to collaborate, we will use and evolve these practices to deliver impactful operational AI to federal health agencies.