High Performance Computing (HPC)
HPC refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical computer or workstation. HPC is used to solve very complex problems and perform research activities through computer modeling, simulation, and analysis. Common industries that rely on HPC for advanced computations and data processing are healthcare, energy, finance, and engineering. HPC allows scientist, researchers, and engineers to innovate with unprecedented speed and accuracy; it is a crucial tool in today’s data-driven world.
However, with great power comes great responsibility. An HPC environment is highly complex and demanding, which leads to several key challenges in areas such as scalability, performance, storage, cost-efficiency, ease of management, and security. Deploying, utilizing, and managing a HPC environment can be far too overwhelming for any given organization.
In this excerpt, we will take a look at how AWS Scale-Out Compute Architecture (SOCA) solution attempts to solve some of these challenges in the HPC realm.
Scale-Out Computing on AWS (SOCA)
AWS SOCA is a cloud-based solution that immensely simplifies the process of deploying and managing HPC workloads. It provides a fully functional HPC, customer-ready environment that can be deployed quickly. Everything in the diagram above can be easily deployed simultaneously in your environment through AWS SOCA (follow the SOCA installation guide). It features a job scheduler, shared storage, and pre-configured user/admin UI portals. With SOCA, users can dynamically scale necessary resources based on workload and demand, optimize performance, manage storage solutions, and take advantage of AWS’s robust security. The convenience, efficiency, and speed of SOCA allow organizations to more easily solve complex problems, analyze massive amounts of data, and rapidly accelerate the pace of productivity.
The chart below shows how AWS SOCA is able to address the overarching challenges of using HPC.
| Topic | HPC challenge | SOCA approach | AWS services to evaluate |
|---|---|---|---|
| Scalability | Fixed hardware limits the number of jobs that can run. | Provision compute for queued workloads and release it when jobs finish; account quotas and capacity still apply. | Amazon EC2, EC2 Auto Scaling |
| Performance | Different simulations require different compute and networking resources. | Match instance types and interconnects to the workload, then benchmark the application. | Amazon EC2, Elastic Fabric Adapter |
| Storage | Workloads need shared data with different throughput and retention requirements. | Select storage for the application’s access pattern. | Amazon S3, Amazon EFS, FSx for Lustre |
| Cost control | Idle capacity and storage can continue to incur costs. | Review utilization, interruption tolerance and spending limits. | EC2 Spot Instances, AWS Budgets, Cost Explorer |
| Management | Provisioning, scheduling and monitoring add operational work. | Use the workload manager and web interface; retain owners for updates and support. | AWS Systems Manager, Amazon OpenSearch Service |
| Security | Shared resources require controlled access and network boundaries. | Configure identity, permissions, network access and encryption for the deployment. | AWS IAM, Amazon VPC, AWS KMS |
Summary
High-Performance Computing (HPC) is an essential tool in today’s data-driven world, powering innovation and research in key industries across the globe. Yet, HPC systems are so intensely complicated and often face major challenges even amongst the most capable, expert practitioners.
AWS offers a complete and robust solution to these challenges with its Scale-Out Compute Architecture. The environment combines compute provisioning, a job scheduler, shared storage and user interfaces. Deployment still requires workload sizing, security configuration, cost review and ongoing operations.
Before deploying, review the AWS SOCA architecture guidance and test a representative job against your performance and cost targets.
