Kategorie: Microsoft Azure

Observability from cloud to edge in Azure

This post is co-authored by Rahul Bagaria, Principal Product Manager, Azure Monitor Customer Success

Our customers are transforming their digital environments, whether migrating workloads to Azure, building new cloud-native apps, or unlocking new scenarios at the edge. As they combine these strategies to meet their business needs, they must also maintain their existing environments. It’s critical that customers can monitor the health, performance, and security of their mission-critical systems, and an observability solution is a best practice for streamlined well-governed management.

With Azure Monitor, our approach to observability centers on simplicity. We know from our customers how important it is to have monitoring available out-of-the-box and to easily get started with samples and recommendations including effective alerts, optimal queries, and customizable reports. To ensure that you can monitor your largest production workloads anywhere in a trustworthy manner, we have designed our data platform to handle thousands of terabytes of data per day easily and reliably. We also understand that our customers have their workloads in Azure, their on-premises datacenters, edge, and multi-cloud environments, and thus extend Azure Monitor through Azure Arc so that our customers have the flexibility to run their applications anywhere and monitor or manage them effectively.

With 99 percent of the top 1,000 Azure customers using advanced capabilities of Azure Monitor, large enterprises using Azure offer a master class in well-monitored environments across cloud, on-premises datacenters, and the edge. As customers including Ernst & Young, Nokia, LinkedIn, Mercedes Benz, and Asos.com trust Azure Monitor for their observability needs, several themes emerge for companies of all sizes to be successful in building a well-monitored environment.

Ability to observe at any level across the stack and get deep insights

Rich insights: You can take advantage of curated visualizations, reports, and diagnostic tools for specific resources, using insights from Azure Monitor. Application Insights provides application performance management (APM) capabilities, and you can use VM insights, Container insights, or Network Insights (and many more) for infrastructure monitoring.
Distributed tracing: Tracing is a key pillar of observability, and with Azure Monitor, you can easily correlate transactions end-to-end from apps to dependencies to infrastructure. There are multiple topology views built-in like Application Map, VM Map, and Network Map for you to visualize the architecture or drill down on transactions. With our investments in OpenTelemetry, we are starting to add support for vendor-agnostic tracing as well.
Log analytics: The heart of Azure Monitor is our powerful centralized logs platform which stores together all logs in Azure across monitoring, security, and management. You can even send your own custom logs, define schemas and transforms, and take advantage of the full power of log analytics. You get a rich query language (KQL) capable of correlations, troubleshooting, analytics, and even AIOps. This platform also powers the security monitoring and SIEM scenarios in Azure with Microsoft Sentinel and Defender for Cloud.

Open and extensible platform for partners and customers to innovate

Partners and integrations: While Azure Monitor provides rich observability and analytics capabilities, as a customer you have a choice to use any monitoring or analytics solution that suits your requirements, and we will fully support that from the platform. We integrate data from Azure resources with partner solutions like Datadog and Elastic (now natively available in Azure) using the same underlying platform that powers our experiences. We also provide hooks for you to export data, connect alerts with ITSM systems, or shift left by incorporating monitoring within your DevOps toolchains.
Open-source solutions: If you need to use open-source metrics or logging solutions alongside Azure Monitor, we support multiple CNCF (Cloud Native Compute Foundation) projects. You can seamlessly scrape Prometheus metrics for Kubernetes clusters with Container insights, and there is a Logstash output plugin for sending custom logs to Azure Monitor.
Operational dashboarding: Beyond the dashboarding and reporting capabilities natively available in Azure, Grafana provides a very rich single-pane-of-glass visualization solution for multi-cloud environments. Azure Monitor is natively part of the core Grafana software with a variety of dashboarding templates that you can build on.

Enterprise-ready for mission-critical scenarios

Privacy: Azure Monitor is fully GDPR compliant and does not collect any PII out-of-the-box. We do support data purging on request. Lockbox protection allows you to control access to any data you are collecting requests during support incidents.
Security: For all your logs in Azure Monitor, we provide data encryption at rest with customer-managed keys (CMK) in your Azure Key Vaults. There is even more security at the infrastructure level with 256-bit AES encryptions. You can take advantage of private links support to connect securely to any of your private network endpoints. To ensure further trust and data security, we recommend you configure agents to use at least Transport Layer Security (TLS) 1.2, and if needed, ingest Azure Active Directory authenticated logs into Azure Monitor.
Compliance: We provide many capabilities to help you meet any compliance requirements in your organization, industry, or geography. You can leverage activity logs and audit logs for security compliance and retain or archive specific data for long durations (up to 7 years) as needed, with additional support for data immutability (in Azure Storage). Overall, we comply with most of the data residency and sovereignty requirements and are even targeting to support Schrems II very soon.

For a more in-depth look at Azure and observability, you can learn more about monitoring best practices for your cloud and edge environments with our dedicated guidance cookbooks; we welcome you to reach out to us with any questions or feedback on our Tech Community.
Quelle: Azure

Azure Spring Cloud Enterprise is now available in preview

When we launched Azure Spring Cloud with VMware in 2019, we set out to solve common challenges developers, IT operators, and DevOps teams face when running Spring Boot applications at scale. Since then we’ve had the opportunity to work with many customers to help them adopt the service including Bosch, Digital Realty, Kroger, Liantis, Morgan Stanley, National Life, Raley’s, and Swiss Re. They value the fully managed infrastructure of Azure Spring Cloud that lets them focus on their apps, while the service manages dynamic scaling, security patching, out-of-the-box instrumentation for monitoring, and more.

Many organizations are running thousands of Spring Boot applications on-premises and need advanced capabilities to accelerate their Spring modernization projects. Based on our learnings from customer engagements, we built a new Azure Spring Cloud tier—Enterprise—that we announced at SpringOne 2021. Azure Spring Cloud Enterprise includes commercially supported Spring runtime components to help enterprise customers ship faster and unlock Spring’s full potential. We are thankful to the many customers and partners who shared their learnings and helped shaped Enterprise tier, and we are excited to announce that Azure Spring Cloud Enterprise is now available in preview for all customers.

Azure Spring Cloud Enterprise represents our continued collaboration with VMware to combine Microsoft’s cloud platform expertise with VMware’s innovative Tanzu portfolio. We’re also committed to making it an application platform where you can deploy polyglot applications that are inherently portable across any Azure service, any cloud, or any on-premises system. With Azure Spring Cloud Enterprise, you gain productivity and access to Spring experts for Spring app development and deployments. Azure Spring Cloud Enterprise builds on top of all the features available in the Standard tier, including the ability to leverage the broader Azure ecosystem to supercharge your Spring Boot applications.

Figure 1: Azure Spring Cloud tier selection now includes Enterprise

Ship faster

Deploy and manage Spring and polyglot applications

The fully managed VMware Tanzu Build Service in Azure Spring Cloud Enterprise automates container creation, management, and governance at enterprise scale using open source Cloud Native Buildpacks and commercial VMware Tanzu Buildpacks. Tanzu Build Service offers a higher-level abstraction for building apps and provides a balance of control that reduces the operational burden on developers and supports enterprise IT operators who manage applications at scale. You can configure what Buildpacks to apply and build Spring applications and polyglot applications that run alongside Spring applications on Azure Spring Cloud.

Tanzu Buildpacks make it easier to build Spring, Java, NodeJS, Python, Go, and .NET Core applications and configure application performance monitoring agents such as Application Insights, New Relic, Dynatrace, AppDynamics, and Elastic.

Effortlessly route client requests to applications

You can easily manage and discover request routes and APIs exposed by applications using the fully managed Spring Cloud Gateway for VMware Tanzu and API portal for VMware Tanzu.

Spring Cloud Gateway for Tanzu effectively routes diverse client requests to applications in Azure Spring Cloud, Azure, and/or on-premises, and addresses cross-cutting considerations for applications behind the Gateway such as securing, routing, rate limiting, caching, monitoring, resiliency, and hiding applications. You can configure:

Single sign-on integration with your preferred identity provider without any additional code or dependencies.
Dynamic routing rules to applications without any application redeployment.
Request throttling without any backing services.

API portal for VMware Tanzu provides API consumers the ability to find and view API route details exposed by Spring Cloud Gateway for Tanzu and test API requests.

Figure 2: Fully managed Spring Cloud Gateway for Tanzu routes diverse client requests to applications in Azure Spring Cloud, Azure, and/or on-premises systems

Figure 3: API portal for VMware Tanzu visualizes APIs that are accessible from Spring Cloud Gateway for Tanzu and other OpenAPI-compliant sources

Flexible and configurable VMware Tanzu components

With Azure Spring Cloud Enterprise, customers can use fully managed VMware Tanzu components on Azure. Customers can select which VMware Tanzu components they want to use in their environment during Enterprise instance creation. Tanzu Build Service, Spring Cloud Gateway for Tanzu, API portal for VMware Tanzu, Application Configuration Service for VMware Tanzu, and VMware Tanzu Service Registry are available during the preview.

VMware Tanzu components deliver increased value to customers such that you can:

Grow your enterprise-grade application portfolio from a few applications to thousands with end-to-end observability while delegating operational complexity to Microsoft and VMware.
Lift and shift Spring applications across Azure Spring Cloud and any other compute environment.
Control your build dependencies, deploy polyglot applications, and deploy Spring Cloud middleware components as needed.

Microsoft and VMware will continue to add more enterprise-grade features, including Tanzu components such as Application Live View for VMware Tanzu, Application Accelerator for VMware Tanzu, and Spring Cloud Data Flow for VMware Tanzu¹.

Unlock Spring’s full potential with Long-Term Support (LTS)

Azure Spring Cloud Enterprise includes VMware Spring Runtime Support for application development and deployments. This support gives you access to Spring experts, enabling you to unlock the full potential of the Spring ecosystem and develop and deploy applications faster.

Figure 4: Do more with Spring framework through world-class support for Spring projects

Typically, open source Spring project minor releases are supported for a minimum of 12 months from the date of the initial release. In Azure Spring Cloud Enterprise, Spring project minor releases will receive commercial support for a minimum of 24 months² from the date of initial release through the VMware Spring Runtime Support entitlement. This extended support ensures the security and stability of your Spring application portfolio even after the open source end-of-life dates.

Figure 5: Commercial support timeline for Spring Boot

Fully integrated into the Azure and the Java ecosystem

Azure Spring Cloud, including Enterprise tier, runs on Azure in a fully managed environment. You get all the benefits of Azure and the Java ecosystem, and the experience is familiar and intuitive.

Common development practices
Azure ecosystem

Create service instances using a provisioning tool
Azure Portal, CLI, ARM Template, Bicep, or Terraform

Automate environments and application deployments
GitHub, Azure DevOps, GitLab, and Jenkins

Monitor end-to-end using any tool and platform
Application Insights, Azure Log Analytics, Splunk, Elastic, New Relic, Dynatrace, or AppDynamics

Connect Spring applications and interact with your cloud services
Spring integrations with Azure services for data, messaging, eventing, cache, storage, and directories

Securely load app secrets and certificates
Azure Key Vault

Use familiar development tools
IntelliJ, VS Code, Eclipse, Spring Tool Suite, Maven, or Gradle

For example, after you create your Enterprise service instance and deploy your applications, you can easily monitor with Application Insights or any other application performance management tools of your choice.

Figure 6: Application Transactions visible through Application Insights Application Map

Get started today

Azure Spring Cloud Enterprise delivers even more productivity, and you can leverage Spring experts to make your projects even more successful. We would love to see you try Enterprise and share your feedback—get started today.

You can also learn more about the Azure Spring Cloud Enterprise preview announcement by VMware.

¹The Azure Spring Cloud Enterprise roadmap is not confirmed and is subject to change.

²You can find the current support timelines for Spring projects.
Quelle: Azure

Microsoft Azure’s first Azure HPC + AI Day

Welcome to our first Azure HPC + AI Day! We’re excited to host the first event, which will soon become an annual tradition. Join us on February 24 and 25 to hear directly from the Microsoft Azure team and earn points by attending sessions to get one of its kind Azure HPC + AI Day swag. Tune into a customer panel and learn how to start up an environment with our hands-on lab. It’s bound to be a great time, and we can’t wait to meet you there.

Register now for Azure HPC + AI Day.

Join us from anywhere

For full access to the entire event, make sure to register for the full experience, where you can tune in to live sessions, watch our on-demand content, check out some additional resources, try to win some swag, and take part in our hands-on labs.

Can’t get to the full event? Many of our sessions on February 24 will also be broadcast on LinkedIn. Watch from anywhere, and still get a chance to hear what’s coming and have the opportunity to ask our customers questions in the panel.

Look forward to these sessions

Day one

Start time

Title

9:00 – 9:30 AM PT

Azure HPC + AI keynote

9:30 – 10:00 AM PT

AI Infra Platform strategy

10:00 – 10:30 AM PT

Azure HPC + AI software strategy

10:45 – 11:45 AM PT

Topic breakouts sessions

12:00 – 1:00 PM PT

Customer panel

Topic breakouts

Title

Building a successful SaaS solution for Massive Scale Actuarial Modelling: An end-to-end Azure solution

Cloud versus on-premises considerations

How to onboard as a HPC partner

Day two

Start time

Title

9:00 – 9:45 AM PT

HPC Operations—Azure, the best place to run HPC in the cloud

10:00 AM – 12:00 PM PT

IT Operator hands-on lab using on az-hop

10:00 AM – 11:30 PM PT

Journey from on-premises to cloud: Story by Ansys

12:15 – 2:15 PM PT

Engineer hands-on lab using az-hop

Hear directly from customers

Join us on February 24 at 12:00 PM PT to hear directly from our customers about their experiences. Available on both platforms, be ready to hear about their experiences working in the cloud, the incredible projects they’ve been working on, and what the future holds. Don’t forget to ask questions.

Customers present

Jellyfish Pictures
Vestas
University of Bath
Kensington Mortgages
AMD

Time to roll up your sleeves—with help

On February 25, get ready to roll up your sleeves and work in Azure. We’ll have two hands-on labs guided by Microsoft experts. In the IT operator hands-on lab, you'll get the chance to understand and build from scratch an end-to-end high-performance computing (HPC) cluster in Azure with autoscaling compute nodes, a web portal, a lustre file system, and remote visualization nodes. Later in the day in the Engineering hands-on lab, you’ll then get the opportunity to run a typical HPC workload and visualize the end results.

Big moments in 2021

Jellyfish Pictures needed to enable secure remote access to immense computing power to render visual effects and animation sequences. They were able to use Azure to gain burst rendering on up to 90,000 processor cores in the cloud, gaining a 70 percent boost in capacity and the freedom to work from anywhere.

Global leader in sustainable energy solutions Vestas Wind Systems A/S wanted to optimize wind energy production by reducing the negative impact of turbine wakes. Working with Microsoft Azure and minds.ai, Vestas is now able to generate simulations to help wind farms mitigate wake effect, generate more wind energy, and build a more sustainable and prosperous energy future.

AMD needed more computing power and burst capacity to easily handle spikes in demand, optimize its own processes, and accelerate in market. Drawing on the power and flexibility it gained with Azure HPC, AMD has eliminated weeks and even months of delay, reducing procurement times and overall job times. It’s matching and exceeding on-premises performance and is now getting daily visibility into reports.

In the midst of COVID-19 the need for a way to get faster test results, Sensyne Health developed its MagnifEye solution, a mobile app that uses a device’s camera to capture the LFT stick image and read it in tenths of seconds with a stunning 99.6 percent accuracy rate.

Previous launches in 2021

NDm A100 v4 series virtual machines now generally available

In November, we announced the general availability of the new NDm A100 v4 series virtual machines (VMs). This new series built on the ND A100 v4 announced in June and expanded Azure’s leadership-class AI supercomputing scalability in the public cloud. The new high-memory NDm A100 v4 for data-intensive GPU compute workloads reaffirms Microsoft’s commitment to rapidly adopting and shipping the latest scale-up and scale-out GPU accelerator technologies to the public cloud.

Read more in the launch announcement.

Learn more

Register for Azure HPC + AI Day.
Learn more about Azure HPC + AI.
Learn more about our recently launched Digital Certification program with Cap Gemini focusing on NIVIDA GPU-powered Azure virtual machines.
Azure HBv3 virtual machines for HPC, now up to 80 percent faster with AMD Milan-X CPUs.

Quelle: Azure

Advancing Azure Virtual Machine availability monitoring with Project Flash

“As we head into the fourth calendar year of the Advancing Reliability blog series, empowering organizations to run their workloads reliably on Azure remains one of our top priorities. We continually invest in evolving the Azure platform to help achieve this on a daily basis. Your ability to monitor virtual machine (VM) availability in a robust and comprehensive way is paramount to ensuring that your applications are available and resilient. For today’s post in the series, I have asked Program Manager, Pujitha Desiraju, from our Azure Core Platform Fundamentals Engineering team to talk about the latest observability enhancements for VM availability monitoring, as well as planned investments to deliver the best monitoring experience.”—Mark Russinovich, CTO, Azure

 

This post was co-authored by Principal Software Engineering Manager, Gaurav Jagtiani.

Flash, as the project is internally known, is a collection of efforts across Azure Engineering, that aims to evolve Azure’s virtual machine (VM) availability monitoring ecosystem into a centralized, holistic, and intelligible solution customers can rely on to meet their specific observability needs. Today, we’re excited to announce the completion of the project’s first two milestones—the preview of VM availability data in Azure Resource Graph, and the private preview of a VM availability metric in Azure Monitor.

What is Project Flash?

Project Flash derives its name from our commitment to building robust and rapid ways to monitor virtual machine (VM) availability as comprehensively as possible—a key prerequisite for efficient application performance. It’s our mission to ensure you can:

Consume accurate and actionable data on VM availability disruptions (for example, VM reboots and restarts, application freezes due to network driver updates, and 30-second host OS updates), along with precise failure details (for example, platform versus user-initiated, reboot versus freeze, planned versus unplanned).
Analyze and alert on trends in VM availability for quick debugging and month-over-month reporting.
Periodically monitor data at scale and build custom dashboards to stay updated on the latest availability states of all resources.
Receive automated root cause analyses (RCAs) detailing impacted VMs, downtime cause and duration, consequent fixes, and similar—all to enable targeted investigations and post-mortem analyses.
Receive instantaneous notifications on critical changes in VM availability to quickly trigger remediation actions and prevent end-user impact.
Dynamically tailor and automate platform recovery policies, based on ever-changing workload sensitivities and failover needs.

With these goals in mind, we’ve divided our execution strategy into two phases—a near-term phase to meet critical current needs, and a long-term phase to deliver the best VM availability monitoring experience. This two-phased approach helps us continually bridge gaps, iterate on service quality, and learn from your feedback at every step along the way.

Announcing new monitoring options

For the first phase, we are providing different options to enable convenient access to VM availability data to address a range of observability needs. We aim to maintain data consistency with similar rigorous quality standards across all of these existing features and solutions, like Resource Health or Activity Log, to deliver a consistent view agnostic of the solution you choose.

Introducing at-scale analysis for VM availability

Today, we’re excited to reach our first Project Flash milestone—with the preview release of VM availability states in Azure Resource Graph for at-scale programmatic consumption.

Azure Resource Graph is a service in Azure that is extensively adopted for its efficient ability to query across many subscriptions, all at once and at low latencies. We’re currently emitting VM availability states (Available, Unavailable, and Unknown) to the Health Resources table in Azure Resource Graph, so you can perform complex Kusto Query Language (KQL) queries for sieving through large datasets at once. This functionality is handy for tracking historical changes in VM availability, for building custom dashboards, and for performing detailed investigations across numerous resource properties spread across multiple tables.

Figure 1: Azure Resource Graph Explorer Window with query and results, to demonstrate fetching data from the HealthResources table.

We are planning to add failure details and degraded VM scenarios to the Health Resources table in Azure Resource Graph, later this year. These details will ensure you are properly informed on the cause and impact of any failures—so you can either failover, reboot in place, or take the appropriate mitigations to prevent end-user impact.

Navigate to Azure Resource Graph Explorer on the Azure portal to get started with any of the KQL queries published for the Health Resources table.

Introducing VM availability metric in Azure Monitor

We’re also pleased to announce the private preview of an out-of-box VM availability metric in Azure Monitor, for a curated metric alerting and monitoring experience.

Metrics in Azure Monitor are great for monitoring and analyzing time series representations of VM availability for quick and easy debugging, receiving scoped alerts on concerning trends, catching early indicators of degraded availability, correlating with other platform metrics, and more.

The metric allows you to track the pulse of your VMs—during expected behavior, the metric displays a value of 1. In response to any VM availability disruptions, the metric dips to a 0 for the duration of impact. In case of an Azure infrastructure outage, we will emit nulls represented as a dotted line on the portal.

Figure 2: Screenshot of VM availability metric as seen on Metrics Explorer in the Azure portal, with occasional dips to reflect VM availability disruptions.

We released the private preview of the metric as phase one of our rollout plan, and are currently collecting customer feedback, to further improve our offering. We are planning to add failure details such as metric dimensions and platform logs next year, to allow you to precisely alert on failure scenarios that are impactful.

Coming soon

The two monitoring options introduced above are just the beginning for Project Flash! We will continue to build upon our existing solutions by improving data quality and failure attribution. In parallel, we are designing two new monitoring offerings to meet your latency and mitigation needs, while also investing heavily in the underlying platform to make our fault detection more resilient and comprehensive.

Azure Event Grid for instantaneous notifications

Successfully running business-critical applications requires hyper-awareness of any VM availability impacting event, so remediation actions can be triggered instantaneously to prevent end-user impact. To support you in your daily operations, we are planning to design a notification mechanism that leverages the low-latency technology of Azure Event Grid. This will allow you to simply subscribe to an Event Grid system topic, and route scoped events via event handlers to any downstream tooling, instantaneously.

Automate and tailor platform recovery policies

Considering the numerous ongoing investments to improve your VM availability monitoring experience, Project Flash intends to empower you even further by providing you knobs to customize recovery policies triggered by the platform, in response to cases of VM availability disruptions.

One such knob we are designing is the ability to opt-out of Service Healing for single-instance VMs, in response to a specific set of unanticipated Availability disruptions. This knob will be made available via the portal or at the time of VM deployment and can be updated dynamically. Note that leveraging this feature will render the usual Azure Virtual Machine availability SLAs ineffective.

In the future, we will explore introducing knobs to also opt-out of other applicable recovery policies (for example, Live Migration or Tardigrade), to ensure you can easily adapt to your ever-changing mitigation needs.

Ongoing platform quality investments

While the first phase is designed to meet your current observability needs, we remain focused on our long-term goal of delivering a world-class observability experience surrounding VM availability. We are extremely excited for all the data enrichments and technology advancements that will contribute to this experience, so here’s an early look at our roadmap of planned investments:

Fault detection and attribution: We are continuously evolving our underlying infrastructure to detect and attribute failures both precisely and instantaneously—so that we can reduce unknown or missing health status reports, emit actionable failure details, and handle platform recovery customizations. This remains our top investment area on which we continue to iterate every cycle.
Root cause analysis (RCA) automation: We are planning to implement easy tracking mechanisms for every unique VM downtime, along with automatic construction and emission of detailed downtime RCA statements to reduce manual tracking and churn on your end.
AIOps integration: We are looking to leverage the tremendous advancements being made in AIOps across Microsoft, for enabling smart insights and anomaly detection and diagnosis across the multitude of data points on VM Availability.
Centralized and cohesive user experience: We acknowledge that a consequence of our near-term approach is that across our different services we have multiple monitoring, alerting, and recovery tools which may lead to a confusing and disparate experience for you. This is a problem we intend to solve with our final phase. Our north star goal is to provide end-users access to distinct and necessary representations of VM availability, consolidated within Azure Monitor, and categorized according to common usage patterns for discoverability, ease of use and intuitive onboarding.

Learn more

This list is certainly not exhaustive as we have multiple enrichments planned as part of our long-term strategy. To reiterate, our intention with Project Flash is to make VM availability monitoring extremely intuitive, comprehensive, and seamless—so you are always prepared for and informed about any changes in the health of your workloads, ultimately to maintain your own SLAs and business promises.

We will continue to share updates on Project Flash through blogs like this, to ensure you stay up to date on the latest. Stay tuned!
Quelle: Azure

Genomic analysis on Galaxy using Azure CycleCloud

Cloud computing and digital transformation have been powerful enablers for genomics. Genomics is expected to be an exabase-scale big data domain by 2025, posing data acquisition and storage challenges on par with other major generators of big data. Embracing digital transformation offers a practically limitless ability to meet the genomic science demands in both research and medical institutions. The emergence of cloud-based computing platforms such as Microsoft Azure has paved the path for online, scalable, cost-effective, secure, and shareable big data persistence and analysis with a growing number of researchers and laboratories hosting (publicly and privately) their genomic big data on cloud-based services.

At Microsoft, we recognize the challenges faced by the genomics community and are striving to build an ecosystem (backed by OSS and Microsoft products and services) that can facilitate genomics work for all. We’ve focused our efforts on three main core areas—research and discovery in genomic data, building out a platform to enable rapid automation and analysis at scale, and optimized and secure pipelines at a clinical level. One of the core Azure services that has enabled us to leverage high performance compute environment to perform genomic analysis is Azure CycleCloud.

Galaxy and Azure CycleCloud

Galaxy is a scientific workflow, data integration, and data analysis persistence and publishing platform that aims to make computational biology accessible to research scientists that do not have computer programming or systems administration experience. Although it was initially developed for genomic research, it is largely domain agnostic and is now used as a general bioinformatics workflow management system. Galaxy system is used for accessible, reproducible, and transparent computational research.

Accessible: Programming experience is not required to easily upload data, run complex tools and workflows, and visualize results.
Reproducible: Galaxy captures information so that you don't have to; any user can repeat and understand a complete computational analysis, from tool parameters to the dependency tree.
Transparent: Users share and publish their histories, workflows, and visualizations via the web.
Community-centered: Inclusive and diverse users (developers, educators, researchers, clinicians, and more) are empowered to share their findings.

Azure CycleCloud is an enterprise-friendly tool for orchestrating and managing high-performance computing (HPC) environments on Azure. With Azure CycleCloud, users can provision infrastructure for HPC systems, deploy familiar HPC schedulers, and automatically scale the infrastructure to run jobs efficiently at any scale. Through Azure CycleCloud, users can create different types of file systems and mount them to the compute cluster nodes to support HPC workloads. With dynamic scaling of clusters, the business can get the resources it needs at the right time and the right price. Azure CycleCloud automated configuration enables IT to focus on providing service to the business users.

Deploying Galaxy on Azure using Azure CycleCloud

Galaxy is used by most academic institutions that conduct genomic research. Most institutions that already use Galaxy want to stick to it because it provides multiple tools for genomic analysis as a SaaS platform. Users can also deploy custom tools onto Galaxy.

Galaxy users generally use the SaaS version of Galaxy as part of UseGalaxy resources. UseGalaxy servers implement a common core set of tools and reference genomes and are open to anyone to use. All information on its usage is available on the Galaxy Platform Directory.

However, there are some research institutions that intend to deploy Galaxy in-house as an on-premises solution or a cloud-based solution. The remainder of this article describes how to deploy and run Galaxy on Microsoft Azure using Azure CycleCloud and grid engine cluster. The solution was built during the Microsoft hackathon (October 12 to 14, 2021) with code implementation assistance from Azure HPC Specialist, Jerry Morey. The architectural pattern described below can help organizations to deploy Galaxy in an Azure environment using CycleCloud and a scheduler of choice.

As a pre-requisite, genomic data should be available in a storage location, either cloud or on-premises. Azure CycleCloud should be deployed using the steps described in the “Install CycleCloud using the Marketplace image” documentation.

Cluster deployment that is truly supported by Galaxy on the cloud is called the unified method. In this method, the copy of Galaxy on the application server is the same copy as the one on the cluster nodes. The most common method to do this would be to put Galaxy in a network file system (NFS) somewhere that is accessible by the application server and the cluster nodes. This is the most common deployment method for Galaxy.

An admin user can SSH into Azure CycleCloud virtual machines or Galaxy server virtual machines to perform admin-related activities. It is recommended to close the SSH port when in production. Once the Galaxy server is running on a node, end users (researchers) can load the portal on their end device to perform analysis tasks which include loading data, installing, uploading tools, and more.

Access to functionalities (such as installing and deleting tools versus the usage of tools for analysis) are controlled by parameters defined in galaxy.yml that resides in the Galaxy server. Once a user accesses a functionality, they are converted to jobs that are submitted to the grid engine cluster for further execution.

Deployment scripts are available to ease deployment. These scripts can be used to deploy the latest version of Galaxy on Azure CycleCloud.
Following are the steps to use the deployment scripts:

Git clone this project (The project is in active development, so cloning the latest release is recommended).

git clone –b release_21.09 https://github.com/themorey/galaxy-gridengine.git

Upload project to CC locker.

cd galaxy-gridengine

Modify files (if needed)

cyclecloud locker list

Azure cycle Locker (az://mystorageaccount/cyclecloud

cyclecloud project upload "Azure cycle Locker"

Import cluster template to CC.

cyclecloud import_cluster <cluster-name> -c <galaxy-folder-name> -f templates/gridengine-galaxy2.txt

NOTE: Substitute <cluster-name> with a name for your cluster—all lower case, no spaces.

Navigate to CC Portal to configure and start the cluster.

Wait for 30 to 45 minutes for the Galaxy server to be installed.

To check if the server is installed correctly, SSH into Galaxy server node and check galaxy.log in /shared/home/<galaxy-folder-name> directory.

This deployment was adopted by a leading United States-based academic medical center. The Microsoft Industry Solutions team helped deploy this solution on the customer’s Azure tenant. Researchers at the center tested to assess the parity of this solution to existing Galaxy deployment on their on-premises HPC environment. They were able to successfully test the deployed Galaxy server that used Azure CycleCloud for job orchestration. Several common bioinformatics tools such as bedtools, fastqc, bcftools, picard, and snpeff were installed and tested. Galaxy supports local user by default. As part of this engagement, a solution to integrate their corporate active directory was tested and deployed. The solution was found to be on par with their on-premises deployment. With the increased number of execute nodes and size of those nodes, they found that the jobs were executed in less time.

For more information, support, or guidance related to the content in this blog, we recommend you reach out to your Microsoft sales representative.

Learn more

Learn more about Microsoft Genomics solutions.

Microsoft Genomics service on Azure.
Azure CycleCloud—HPC Cluster and Workload Management.
Galaxy on Azure deployment scripts.

Quelle: Azure

IoT adoption remains strong in the Asia-Pacific region as organizations broaden usage

The Asia-Pacific region has long been a strong manufacturing base and the sector continues to be a strong adopter of the Internet of Things (IoT). But as the latest Microsoft IoT Signals report shows, IoT is now much more widely adopted across verticals, and across the globe, with smart spaces—a key focus for many markets in the Asia-Pacific region—becoming one of the leading application areas.

The newest edition of this report provides encouraging reading for organizations in the Asia-Pacific region. The global study of over 3,000 business decision-makers (BDMs), developers, and internet technology decision-makers (ITDMs) across ten countries—including Australia, China, and Japan—shows that IoT continues to be widely adopted for a range of uses and is seen as critical to business success by a large majority. Further, rather than slowing growth which some might have feared, the COVID-19 pandemic is driving even greater investment across different industries as IoT becomes more tightly integrated with other technologies.

Across the Asia-Pacific region, the research shows that organizations in Australia report the highest rate of IoT adoption at 96 percent—beating both Italy (95 percent) and the United States (94 percent)—and that organizations in China are adopting IoT for more innovative use cases and have the highest rates of implementation against emerging technology strategies. In Japan, it found that companies are using IoT more often to improve productivity and optimize operations. Below we dive into three key trends that emerge for organizations in this region.

1. A greater focus on planning IoT projects pays off

Whilst IoT projects in the region take slightly longer to reach fruition, it seems that this reflects a more thoughtful and diligent approach which appears to be paying off. By thinking through and taking time upfront to determine the primary business objectives for success, organizations in the Asia-Pacific region report high levels of IoT adoption (96 percent in Australia), importance (99 percent of companies in China say IoT is critical to business success), and overall satisfaction (99 percent and 97 percent in China and Australia respectively). These objectives are broadly in line with global findings, with quality assurance and cloud security consistently mentioned across all three countries in this region. Organizations in Australia and Japan adopt IoT to help with optimization and operational efficiencies: in Australia, the focus is on energy optimization (generation, distribution, and usage); and in Japan, it is on manufacturing optimization (agile factory, production optimization, and front-line worker). Those in Australia and China also tend to do more device monitoring as part of IoT-enabled condition-based maintenance practices.

Companies in the region report that these varied use cases are delivering significant benefits in terms of more operational efficiency and staff productivity, improved quality by reducing the chance of human error, and greater yield by increasing production capacity.

2. Emerging technologies accelerate IoT adoption

Of the organizations surveyed, the 88 percent that are set to either increase or maintain their IoT investment in the next year are more likely to incorporate emerging technologies such as AI, edge computing, and digital twins into their IoT solutions. And in the Asia-Pacific region, awareness of these technologies tends to be higher than in other markets.

Organizations in China are far more likely than their counterparts elsewhere to have strategies that address these three areas. They lead all other countries when it comes to implementing against AI and edge computing strategies, and a staggering 98 percent of companies in Australia that are aware of digital twins say they have a specific strategy for that technology. More significantly, their experience with these technologies is driving greater adoption of IoT across the region, with around eight in ten organizations working to incorporate them into their IoT solutions.

3. Industry-specific IoT solutions drive a broader range of benefits

The IoT Signals report analyzed several industries in-depth, all well represented in the Asia-Pacific region. Organizations in Australia, for instance, should note that energy, power, and utility companies use IoT to help with grid automation (44 percent) and maintenance (43 percent), while oil and gas companies tend to apply it more to workplace and employee safety (45 percent and 43 percent respectively). Energy companies are also much more likely to use AI in their IoT solutions than other industries (89 percent of organizations versus 79 percent for all verticals). The benefits of IoT being seen by organizations in these sectors include increases in operational efficiency, increases in production capacity, and increases in customer satisfaction.

In Japan, where manufacturing makes up an important part of the market, we find that there are more IoT projects in the usage stage (26 percent) than in other sectors, mainly focused on bolstering automation. Manufacturing organizations are using these IoT solutions to ensure quality, facilitate industrial automation, and monitor production flow. In doing so, they benefit from improved operational efficiency and greater production capacity, driving competitive advantage. In this industry, it’s not technology that poses a challenge but the huge business transformation that takes extra time and thought, often due to legacy systems and processes.

China, of course, has always been an innovator when it comes to devices, so its manufacturing sector will see the same impacts. But smart spaces—as in other countries in the Asia Pacific region—are getting a lot of attention, and this is where we see the highest levels of IoT adoption (94 percent) and overall satisfaction (98 percent). It also has the strongest indications of future growth with 69 percent planning to use IoT more in the next two years. It’s also the industry sector where the highest proportion of organizations are implementing IoT against AI strategies. The top applications of IoT in smart spaces are around productivity and building safety, where organizations can benefit from improved operational efficiency and personal safety.

Learn more

It’s clear from the report that IoT is here to stay, and the diligent approach taken by organizations across the Asia Pacific region is paying off. For a more detailed exploration of how businesses in this region and across the globe are leveraging IoT, as well as drilldowns into topics such as security, implementation strategy, and sustainability, make sure you download the full Microsoft IoT Signals report.
Quelle: Azure