Advances in High-Performance Scientific Computing

What do bacterial systems, weather processes, and galaxy formation have in common? All are studied using high-performance computing environments, where clusters of processors power accelerated analysis of large volumes of data. High-performance computers¹ (HPC) support complex and data-intensive computational methods, such as deep learning algorithms used to decode functional magnetic resonance images (fMRI) at the National Institutes of Health’s HPC Biowulf, or detailed simulations of the Sun’s atmosphere performed with the specialized algorithm Bifrost at NASA’s Pleiades supercomputer, or hosting of genomic data analysis applications critical to the COVID-19 and mpox response at the Centers for Disease Control Advanced Molecular Detection program’s SciComp. 

Recently, scientific disciplines have pushed computational boundaries. Drivers of this trend include researcher innovation, growing data volumes, and progress in artificial intelligence (AI), as well as advances in hardware (such as increasingly powerful graphics processing units [GPU] and field programmable gate arrays [FPGA]) and cloud service provider (CSP) services.

This growth and innovation are reflected in demand and federal funding—over half a billion dollars per year are allocated to scientific computing across agencies, with around $150 million directed to HPC resources. More than ever, ensuring scientists are fully equipped to use HPCs is critical, both to enable cutting-edge research and to maximize the effective use of resources. 

¹High-Performance Computers

High-performance computers can be “on-premise,” meaning institutionally owned and operated, or composed of computing resources accessed through cloud service providers, or a hybrid combination of the two.

Using HPCs is a nontrivial undertaking, and, as research utilization grows, HPC accessibility, usability, and support are becoming increasingly critical to ensuring that users can remain focused on advancing science rather than on solving IT challenges.

Overcoming these challenges by employing strategies that enhance HPC adoption and reduce utilization barriers will ensure effective use of investments, accelerate scientific discovery, and help maintain national leadership in emerging technologies that are critical to the economy of the future—such as AI and the bioeconomy.

Scientists can face technical challenges in effectively using HPCs due to the high skill floor needed for these computing resources. Other challenges stem from a lack of awareness of how an HPC can support research. Additional barriers can emerge from policy requirements, security concerns, and other limitations.

Significant technical skills are required to use HPCs. To prioritize and make effective use of HPCs, prospective users must understand:

• What to HPC: Identify which parts of their analysis stand to gain the most from HPC resources.
• How to HPC: Learn new tools and adapt their techniques, and optimize code to efficiently consume HPC resources and maximize benefits. 
• The Cost of HPC: Learn to budget platform-specific payment models and expenses (e.g., CSP pay-as-you-go and data egress costs).  

Scientists may be underinformed about HPC opportunities and may not appreciate the advantages an HPC could offer their research. Reasons for this include:

• Work in subfields where complex computation is comparatively new and there is little precedent for HPC use 
• Reservations regarding the advantages HPCs offer and fears about developing dependence on non-owned/-controlled environments
• Enterprise solutions that are less appealing than engaging within their communities, even if computing resources are scarce
• Movement of data to an external platform which can require additional security and privacy considerations
• Multiple competing high-priority responsibilities (particularly during crisis-response scenarios) that have escalated concerns about upskilling time

Barriers arising from infrastructure and policies can thwart even experienced HPC users. Scaling up can lead to growing pains for researchers. Demand for HPCs grows with data volumes, and competition for limited resources gets fiercer with time. Challenges include:

• On-premises HPCs that lack specialized hardware (such as GPUs or FPGAs) or software due to incompatibility or security requirements that preclude non-stable builds 
• Uploading of data to HPCs that is cumbersome when uploading from laboratory instruments or slower networks 
• Security requirements for handling sensitive data, such as personally identifiable information, that necessitate controls not always provided by default on HPCs
• Lack of comprehensive and transparent cost-sharing models for HPC resources, which can lead to continued provisioning of individualized resources, encouraging data and technology silos  
• Critically, obstacles to reproducibility of results across varying computing environments

To address these challenges, new strategies must be employed to empower HPC use by enhancing accessibility, spreading awareness of HPC capabilities, and measuring their performance with respect to scientific impact. Key principles include:

HPCs should be assessed for the effectiveness of services and tools that are provided to efficiently onboard and guide scientists with diverse backgrounds and varying HPC expertise. Examples of support services and tools include:

• Liaisons, who bring scientific understanding and HPC expertise, to provide tailored guidance for leveraging an HPC for diverse research applications
• Experts and automated tools that can identify bottlenecks in algorithms, assist with code refactoring, and guide resource provisioning
• Demonstrations, working groups, tutorials, and templates to support scientists when approaching common problems (e.g., training knowledgebase at Pleiades)
• Databases of previously run job descriptions (including data volume and algorithms) to serve as references for compute and cost estimations
• Comprehensive pre-installed software and avenues for integrating new software and computational tools
• Automated data and software cataloging for ease of discoverability and use in HPC workflows
• Support for tools to implement security policies with appropriate resource provisioning, data storage solutions, and permissions (e.g., Open Policy Agent or Sentinel) and strategies or infrastructure for accommodating sensitive data
• Quantifiable measurement of migration efficiency using metrics such as number of consultations to determine HPC suitability, users trained on common tasks and tools, and lines of code refactored to optimize analytical workflows for HPC deployment as well as accuracy of cost estimates
• Tools and knowledge bases to aid with code migration, environment setup, and cost assessment

HPCs enable scaling to more complex computational methods and larger datasets. To support scientists seeking to scale, HPC administrators should provide guidance for developing comprehensive project and data management plans, including long-term data storage. This includes data governance frameworks for clear data storage, access, usage, and retention guidelines across platforms, ensuring privacy and security.

Scientists should not be unnecessarily bound to a particular HPC environment—HPCs should be standardized, and pipelines modularized where possible to allow for easy migration between platforms. This standardization and modularization would promote beneficial competition between HPC environments and improve the dissemination of methods and data. Specifically, standardization should be considered for:

• Templates or containers to enable cross-platform compatibility wherever possible
• User interfaces, bash scripts, directory structures, and other conventions to ensure scientists develop transferable skills
• Data formats and transfer protocols to enhance interoperability and integration of raw, intermediary, and processed data and metadata
• Billing practices to enable budgeting and informed comparison of services

Administrators of on-premises HPCs should facilitate the seamless use of cloud computing via approaches such as cloud bursting (CSP on-demand utilization from the HPC) to augment constrained computational resources. In addition, on-premises HPC stakeholders should collaboratively reevaluate and redesign their HPC operating model (e.g., administration, data management, support structure, cost model) when considering cloud-based HPC to align with cloud requirements and support effective adoption. 

By focusing on advertising scientific impact and measuring outreach based on researcher engagement and knowledge attainment, HPC outreach can be better aligned to scientists. For example, introductory and specialized training curricula could be created expressly for HPC tasks. Summer schools (e.g., the International HPC Summer School) or short-term research or training experiences for junior researchers (e.g., the Supercomputer Institute’s Technical Summer Program) are important opportunities for experts to share specialized skills and knowledge with interested groups and are ideal for prospective researchers to get hands-on experience working with HPCs.

Certifications could also incentivize guided studies of HPC and CSP environments and services, especially if standardized and transferable. Hackathon competitions (e.g., HackHPC), composed of small teams of researchers and led by experienced HPC staff, allow researchers to get hands-on experience using HPCs in a supportive and fun environment, while HPC staff gain direct exposure to the needs and methodologies of the researchers. In addition, these events can be used to develop tools for future use at HPCs. 

In addition, dedicated online user forums can serve as avenues for informally asking questions and sharing information. These platforms further facilitate exchange among scientists from different fields who may have overlapping needs on HPCs. Free trials or credits can attract new scientific users. And collaborative research networks facilitate knowledge transfer, data sharing, standards development and adoption, and continued collaboration for technological advancements.   

Scientific output-focused metrics can also serve to measure HPC scientific alignment and impact. While traditional metrics of computing power (e.g., number of nodes and compute cores) are of undeniable importance to HPCs, in scientific computing, there are many other critical factors for scientists:

• Assign author-impact metrics to HPCs to quantify publication output and impact. 
• Track HPC usage by subfield to help researchers gauge subfield-specific tool availability and user support. 
• Track metrics related to early career engagement to better gauge their long-term impacts—for example, what percentage of resources is allocated to early-career researchers or how many advanced degrees are supported.
• Provide for transparent reporting of HPC resources allocated to indefinite or long-term work, in comparison to resources available for shorter term projects.
• Capture time and cost efficiencies gained by analytical workflow HPC optimization and deployment.
• Advertise HPC-enabled science advancement success stories.

HPCs are an essential part of accelerating scientific progress. Supporting and empowering scientists using HPCs is critical—and through work to enhance accessibility, promote awareness in the scientific community, and align performance goals with scientific impact, HPCs can aid in the realization of this priority.