Cloud Computing Köln

Azure Managed Lustre—Parallel file system for HPC and AI workloads

Today, Microsoft is announcing the public preview of Azure Managed Lustre, a new addition to the storage offerings in our Azure HPC + AI solution. Lustre is an open-source parallel file system renowned for high-performance computing (HPC) and is adept at large-scale cluster computing. Azure Managed Lustre provides the high-performance storage of Lustre with the control and consistency of Microsoft’s cloud. As a result, customers can focus on their business goals, whether that’s building a fraud detection system based on SAS analytics, decoding the human genome to create the next breakthrough in medicine, leading the frontiers of energy exploration with seismic data processing and sustainable energy solution, or predicting the climate and weather patterns impacting human life on planet earth.

Lustre as an Azure Managed Service

With Lustre as a managed service, users can now focus on building and running their HPC and AI applications without managing an advanced parallel file system. HPC and AI workloads can seamlessly migrate to the cloud, retaining their compatibility with Lustre and protecting existing automation and platform investments developed when previously run on premises. Azure Managed Lustre enables a fast on-demand deployment of clusters serving global regions, alleviating advance planning while meeting the compliance and data residency requirements. With a palette of performance options, Azure Managed Lustre delivers an elastic solution where users can deploy independent and exclusive clusters with predictable performance eliminating the noisy-neighbor problem commonly experienced in the on-premises shared infrastructures.

The public preview includes two durable SSD-based SKUs which deliver a choice in performance options for mission critical workloads: 125 MB/s and 250 MB/s for every provisioned TiB of capacity. During the preview, you can create cluster sizes up to 128 TiB by default, with an option to scale up to 768 TiB upon request. Azure Managed Lustre is built on the highly durable Azure managed disks with locally-redundant storage consisting of 3 replicas so even if one or two replicas experience issues you still have tolerance against failures. Azure Managed Lustre delivers POSIX-compliant Lustre version 2.15 (LTS), which offers several performance improvements. We focused on capabilities that enable users to consume Azure Managed Lustre easily without worrying about how to build and deploy the Lustre clients on their compute VMs and containers. With the Azure Managed Lustre preview, customers can choose from:

Use Azure HPC images prebuilt with Lustre client packages for Ubuntu 18.04 and 20.04 (or)
Download Lustre client packages from packages.microsoft.com for Linux distros – Ubuntu 18.04, 20.04 & 22.04; RHEL 7.8 & RHEL 8.0

Data tiering using hierarchical storage management allows users to import and export data between Azure Managed Lustre and Azure Blob. This capability enables users to define data archival, retention and protection based on pre-defined policies. Users of large data sets can import the data which is hot and relevant for their active data processing in Azure Managed Lustre clusters and archive/retain the remaining in Azure Blob. This enables them to keep their run-time costs low. Additionally, data tiering to Azure Blob allows the users to leverage its global presence and availability to instantiate Azure Managed Lustre clusters in multiple regions on demand. Integration with Azure Blob additionally facilitates multiprotocol data access via NFS, HDFS and REST. Modern containerized applications in AI/machine learning (ML) and analytics can now run on Azure Kubernetes Service leveraging the CSI driver for Azure Managed Lustre. For example, analytics applications built on top of SAS Viya can seamlessly leverage the high per-core storage performance that they require by integrating with the Azure Managed Lustre CSI driver.

Purpose-built for HPC and AI workloads

We are thankful to our private preview customers who helped us build and perfect the product while addressing their key needs. We are committed to building and supporting Azure Managed Lustre as it helps you with your journey of running your high-performance applications in Azure.

“We work with customers who have many different types of HPC workloads, and we frequently hit storage performance constraints because their projects can involve anything from thousands to tens of thousands of files, all of which need to be read and written in parallel. When we encounter storage bottlenecks or throttling errors on our HPC platforms, we know it is time to turn to Azure Managed Lustre. With Azure Managed Lustre we can add extremely performant storage that can readily keep up with the HPC compute, and automatically sync input and results data with persistent blob storage. Because it can be started and stopped on demand, within a few minutes, Azure Managed Lustre is quite cost efficient and pays for itself by giving us the ability to scale up or scale out our HPC compute, generating results much faster.”—Felipe Ayora, Director—Research and Advanced Computing, BizData.

“The new Azure Managed Lustre file system is a real game changer. It accelerates HPC deployment of our software and eliminates the need to run persistent storage servers. Setup is simple, and parallel i/o performance is fast enough to support very high-throughput workloads.”—Bill Shea, CEO Sharp Reflections.

Next Steps

To trial Azure Managed Lustre for free, complete the registration form. Learn more about how to use Azure Managed Lustre and its various supported features from our documentation.

Learn more about HPC and AI solutions

Read the Azure Managed Lustre documentation.
Read our Azure Managed Lustre technical blog.
Visit our Azure HPC hub for more technical content developed for HPC.
Read about our Azure HPC + AI solution.

Quelle: Azure

Exploring mTLS setup to send a client certificate to the backend and OCSP validation

In our previous blog we discussed what mutual transport layer security (mTLS) is and what some of its use cases are. In this blog I want to discuss two of those use cases. First, how to send client certificate to the backend application server and validate the setup by curl command and second how to set up OCSP validation and do verification by openssl commands.

Insert client certificate as HTTP header

In some cases, backend applications may need a client certificate that is received by Application Gateway. Client certificates can serve different purposes as per the need of the backend applications. Some backend servers may need client certificate information for audit purposes or may want to issue token or cookie to a client certificate. In that case we may have to supply the client certificate to the backend. One way to solve this is by supplying the certificate in base64 encoded format within a nonstandard HTTP (Hypertext Transfer Protocol) header. Please note, for security purposes and to prevent header injections, backend server must accept the custom header from trusted Application Gateway. Let’s discuss first how to send client certificate to backend application as custom http header. To achieve that you can set up a rewrite rule to send client certificate as HTTPS header.

Find more details on how to set up a rewrite rule in our rewrite URL and query string with Azure Application Gateway documentation.

Below is the rewrite rule that you can create to send client certificate to the backend as an HTTP header. Setup rewrite action as below.

Above is screenshot of Create rewrite set explaining values that need to be populated for Rewrite rule.

Once a rewrite rule is created you can verify if the backend server is receiving client certificate in the HTTP header. To test the setup prerequisite is to have openssl and curl tool installed in your machine. You should have access to the client certificate and client private key.

Verification steps to check client certificate in custom HTTP header:

Capture the client certificate output.

more client.crt

Above is screen shot showing client certificate output.

Run the following commad to send a request to Application Gateway:

curl -vk HTTPS://<yourdomain.com> –key client.key –cert client.crt

In the backend server you should see the header you created in the Application gateway rewrite rule. You will have to run network capturing tools like tcpdump at the backend server.

Above screenshot shows Client certificate that backend has received.

Above you can see the X-Client-cert header received by backend that we have created in the rewrite rule. This header has the client certificate that we have sent. The backend server can extract this value and use it based on the desired use case.

OCSP

Online certificate status protocol (OCSP) is now supported by Application gateway. Let’s discuss here how to setup OCSP and validate the setup with openssl command. With OCSP support you can verify the status of the client certificate in real time. This can prevent man-in-the-middle attacks by ensuring that the certificate being present is still valid and has not been compromised. You can get more details about OCSP in RFC 2560. It is easy to setup. When a client initiates a connection to an Application Gateway configured with mutual TLS authentication, not only can the certificate chain and issuer's distinguished name be validated, but revocation status of the client certificate can be checked with OCSP (Online Certificate Status Protocol). During validation, the certificate presented by the client will be looked up via the defined OCSP responder defined in its Authority Information Access (AIA) extension. In the event the client certificate has been revoked, the application gateway will respond to the client with an HTTP 400 status code and reason. If the certificate is valid, the request will continue to be processed by application gateway and forwarded on to the defined backend pool.

Please check this OCSP link to enable this capability. I have summarized the PowerShell command to setup OCSP.

$AppGw = Get-AzApplicationGateway -Name "ApplicationGateway01" -ResourceGroupName "ResourceGroup01"

$profile = Get-AzApplicationGatewaySslProfile -Name "SslProfile01" -ApplicationGateway $AppGw

Set-AzApplicationGatewayClientAuthConfiguration -SslProfile $profile -VerifyClientCertIssuerDN -VerifyClientRevocation OCSP

Once you have set up OCSP, you can verify your client certificate with the OCSP endpoint using openssl command.

openssl ocsp -issuer <ca-bundle> -cert client.crt -text -url <HTTP://FQDN>

Ca-bundle—certificate authority (CA) that has issued the certificate (uploaded per the link step 8 from our previous blog)

Client.crt—Client certificate

url—This will be OCSP endpoint URL address. If you do not know what the URL is you can find the OCSP endpoint of client certificate by using following command:

openssl x509 -in client.crt -text | grep -I OCSP

OCSP—URL:HTTP://ocsp.sectigo.com

Above is screen shot of openssl command showing status of client certificate verification.

You should see the following response if certificate is valid:

Response verify OK

client.crt: good

After verification of your client certificate through OCSP endpoint, you can verify the traffic by sending a request to Application Gateway that has OCSP check-enabled.

curl -vk HTTPS://yourdomain.com –key client. Key –cert client.crt

In case the certificate is not a valid client certificate, OCSP will respond with either “revoked” or “unknown”. Below is the error for “unknown” certificate.

Conclusion

In this blog we have discussed two cases that application gateway supports. You have learned how to send client certificate to backend as HTTP header and verify the setup by using curl command. Also, you have learned how to set up OCSP and verify the setup by openssl command line.

Learn more and get started with Azure Application Gateway

What is Azure Application Gateway | Microsoft Learn
Overview of mutual authentication on Azure Application Gateway | Microsoft Learn
Frequently asked questions about Azure Application Gateway | Microsoft Learn
Overview of mutual authentication on Azure Application Gateway | Microsoft Learn

Quelle: Azure