Click here to learn more about Gilad David Maayan.
Highly complex processes like gene analyses, artificial intelligence (AI), and mechanical load simulations demand large amounts of computing power. A few years ago, these processes were implemented using expensive supercomputers. Fortunately, companies can now leverage cloud-based High-Performance Computing (HPC).
HPC can process massive amounts of data in the most efficient way and solve complex problems much faster compared to the previous generation of supercomputers. Additionally, you can lower your costs because there is no need for buying and maintaining expensive hardware.
Why HPC Is Important
Cloud-based HPC is designed to handle extremely extensive, highly complex, and very specific problems in the quickest possible way. For this reason, public cloud providers are including HPC resources in their service portfolio.
Azure Virtual Machines (VMs), for example, provide support for HPC workloads. This includes support for different VM sizes like A9 and A8. In addition, Azure supports the N-series VMs with Graphical Processing Unit (GPU) support and H-series VMs like H16nr and H16r.
The H-series VMs can integrate with a Remote Direct Memory Access (RDMA) network. RDMA networks are designed to transfer large amounts of data between memory chips on multiple servers.
Microsoft also provides existing HPC applications in the Azure Marketplace. These include MATLAB DCS, Ansys CFD, Autodesk Maya, Arnold on Azure Batch, 3ds Max, Batch AI, and more.
The Difference Between HPC and Traditional Public Cloud Resources
Companies usually start using cloud-based HPC resources when their IT department capacities are fully utilized. Instead of expanding their on-premises infrastructure, they pay for HPC resources on an on-demand basis. This hybrid cloud model of using the public cloud as a supplement to on-premises IT resources is referred to as cloud bursting.
However, the costs of cloud-based HPC resources are much higher compared to traditional virtual machines. You have to use these resources efficiently to fully leverage their benefits. There are three basic aspects that determine when you should use traditional cloud resources and when you need specific HPC technology:
- The amount of processed data
- The available time
- The complexity of the process
Unfortunately, there is no easy way to compare the effectiveness of traditional public cloud to high performance computing resources. You should consult with your public cloud provider. They can help you choose the most suitable option based on their experience. If necessary, they can also help you set up the required cloud resources right away.
For example, AI-based algorithms for healthcare can determine if a patient needs immediate surgery or not, based on an analysis of thousands of MRI images. HPC is required to provide information as quickly as possible.
HPC Network Connection
To fully benefit from HPC capabilities, companies have to connect to the public cloud through their own Wide Area Network (WAN). The network bandwidth depends on the expected amount of data they want to transfer to the cloud.
Surprisingly, it can be much smaller than expected. For example, simulations of chemical materials require a transfer of roughly 100 to 200 MB of data to the cloud. However, only 10 GB of this data has to be transferred back to the company. As a result, companies may set up a low-bandwidth network connection.
In the oil and gas industry, the situation is different. Scientists calculate images of the subsurface using seismic methods. As a result, they get data sets of hundreds of gigabytes or even terabytes. Data transfer of this size can take a very long time. For instance, transferring a terabyte of data at a speed of 100 Mbit/s takes more than 22 hours.
A network connection with a bandwidth of 10 Gbit/s can reduce the data transfer time to just 13 minutes. However, a complete transfer of data to the cloud is not always necessary. For example, edge computing devices reduce the size by preprocessing the data before it is transferred to the cloud.
HPC Software and Certificates
High-performance computing applications demand dedicated HPC middleware and software. Companies that use on-premises HPC resources can keep the software they already have. However, this software must be supported by the cloud. Companies have to ensure that their HPC cloud provider takes over the required software operations.
In addition to software, companies have to be aware of special certifications required by the industry. For example, the automotive industry requires a TISAX 3 (Trusted Information Security Assessment Exchange) certificate for any IT-related operations. This certificate is a recognized auditing standard for information security. Automobile manufacturers that get this certificate prove they adhere to top-level IT security standards.
HPC Security and Data Protection
To maintain a high level of security and data protection, you have to ensure that your HPC software meets all the necessary standards. HPC often processes business-critical or personal data. For example, medical image storage contains personal data evaluation.
To ensure the safety of your data, choose a cloud provider that meets the highest data protection and security requirements. The location of the cloud provider is also critical for data protection reasons. Cloud providers that have data centers in Europe are required to comply with GDPR laws.
Conclusion
In many cases, companies prefer to use high-performance computing in the public cloud instead of their own on-premises resources. By doing so, companies solve extremely demanding tasks in the shortest possible time. As a result, they get a competitive advantage at a reasonable cost. But to properly assess their needs, they must first define their end goal, the amount of required data, and the type of network connection.