Kategorie: Microsoft Azure

How Azure Lighthouse enables management at scale for service providers

Extending Azure Resource Manager with delegated resource management

Today, Erin Chapple, Corporate Vice President, Microsoft Azure, announced the general availability of Azure Lighthouse, a single control plane for service providers to view and manage Azure across all their customers. Inspired by Azure partners who continue to incorporate infrastructure-as-code and automation into their managed service practices, Azure Lighthouse introduces a new delegated resource concept that simplifies cross-tenant governance and operations.

Granular access, better automation, and simplified customer onboarding

Powering Azure Lighthouse is an Azure Resource Manager capability called delegated resource management. Delegated resource management lets customers delegate permissions to service providers over scopes, including subscriptions, resource groups, and individual resources, which enable service providers to perform management operations on their behalf. After customers delegate resources to a service provider, the provider can provide access to users or accounts in provider’s tenant within the constraints specified by the customer, using the standard role-based access control (RBAC) mechanisms. The standard RBAC mechanisms work as if customer resources were resources in provider’s own subscriptions. Finally, delegated resource management works consistently regardless of the licensing construct service providers and their customers might choose—enterprise agreement (EA), cloud solution provider (CSP), and pay-as-you-go.

“Azure delegated resource management enables Nordcloud customers to easily provide secure access. It simplifies onboarding new managed services customers, ensuring our high security and compliance standards are met.”

Ilja Summala, Group CTO, Nordcloud

Cross-tenant management at scale, with enhanced visibility and governance

Delegated management uniquely supports management-at-scale and automation patterns of service providers, whether those providers are managed services partners acting on behalf of customers or central IT teams of enterprises with multiple Azure tenants. Partners can now manage tens of thousands of resources from thousands of distinct customers from their own Azure portal or CLI context. Because customer resources are visible to service providers as Azure resources in their own tenant, service providers can easily automate status monitoring, and applying create, update, change, delete (CRUD) changes across the resources of many customers from a single location.

Everything relevant to Azure resource management, from the Azure portal to services such as Azure Policy, Resource Graph, Log Analytics feature of Azure Monitor, or Update Management, all honor delegated resource management. What’s more, both customers and service providers can see who took actions on the resources from the activity log, increasing accountability for both parties, with protection of the privacy of individual service provider identities. That’s because the newly built resource provider, Microsoft Managed Services, enables Azure services to determine if a call was made from a resource’s home tenant or from a service provider’s tenant.

Our partners have several options for how they use these new capabilities. Since the Azure Lighthouse portal experiences have corresponding APIs, PowerShell, Azure CLI, REST APIs, or client SDKs, it’s easy to integrate into other cloud management portals, ITSM tools, or monitoring tools.

How our partners use Azure Lighthouse

Examples from two of our expert partners, Rackspace and Sentia, highlight the power of Azure Lighthouse and delegated resource management:

Rackspace is enhancing security and response capabilities using Azure Lighthouse in three steps:

Utilizing Azure Resource Graph and cross-tenant queries to quickly detect which customers have impacted images or hosts deployed
Applying an in-guest audit policy across all customers’ managed estates to verify host settings relating to impact/vulnerability
Using update management to report on impacted systems and schedule targeted hot fixes

Sentia pivoted CI/CD pipeline to use declarative Azure Resource Manager templates for provisioning management artifacts across all customers who are under Azure CSP licensing construct. Sentia’s managed services offer is now 90 percent based on Resource Manager templates, which simplifies deployments dramatically, automating monitoring, governance, and management tasks at scale, across customers. 

Continued Azure Resource Manager investments for our partners

Azure Lighthouse and delegated resource management are just the latest of the platform investments we continue to make for our partners. Together with Azure managed applications and custom providers, they enable comprehensive management-at-scale capability for partners and customers. To hear more, watch my demo at Microsoft Build 2019. Some of the other management innovations we’ve made include the following:

Partners can build cross-tenant experiences into their solutions with minimal development, since Azure Resource Manager APIs and Azure Resource Graph queries are now enhanced with tenant context.
Service providers and ISVs can extend and serve-up their IP natively within Azure using custom providers. Imagine end-customers raising service requests to service providers from within Azure, thanks to the ability of custom provider to integrate ITSM tools’ capabilities natively to Azure.
Customers can purchase applications developed by partners from the Azure Marketplace that come with management out of the box provided by service providers. Underlying application resources are protected from the customer while they use the new managed application UI to interact with an application safely. Service providers are given full access to the application to maintain, update, and provide application support for the customer from managed application center.

“We are delighted to see the adoption of the new Azure Lighthouse capabilities into Veeam’s Backup-as-a-Service offerings, representing a natural extension of our cloud-based business offerings. This partnership is a great opportunity for our managed services providers to easily extend Backup-as-a-Service offerings by Veeam using Azure Lighthouse, in order to manage their Azure customers at scale.”

Tim FitzGerald, Vice President, North America Cloud, Ingram Micro Inc.

When Azure as a platform does more for our partners, our partners can focus more on providing differentiated services and higher value to our joint customers. That is how partners make more possible on Azure. We look forward to hearing your feedback on Azure Lighthouse and delegated resource management.
Quelle: Azure

Announcing preview of Azure Data Share

In a world where data volume, variety, and type are exponentially growing, organizations need to collaborate with data of any size and shape. In many cases data is at its most powerful when it can be shared and combined with data that resides outside organizational boundaries with business partners and third parties. For customers, sharing this data in a simple and governed way is challenging. Common data sharing approaches using file transfer protocol (FTP) or web APIs tend to be bespoke development and require infrastructure to manage. These tools do not provide the security or governance required to meet enterprise standards, and they often are not suitable for sharing large datasets. To enable enterprise collaboration, we are excited to unveil Azure Data Share Preview, a new data service for sharing data across organizations.

Simple and safe data sharing

Data professionals in the enterprise can now use Azure Data Share to easily and safely share big data with external organizations in Azure Blob Storage and Azure Data Lake Storage. New services will continue to come online. As a fully managed Azure service, Azure Data Share does not require infrastructure to set up and it scales to meet big data sharing demands. The intuitive interface makes sharing easy and productive, directly from the Azure portal. With just a few clicks data professionals choose which data to share and who to share it with. They can schedule the service to automatically share new or changed data pertaining to specific datasets, as well as stop future updates from flowing through at any time. With Azure Data Share, data professionals have greater control over each data sharing relationship and can govern use by associating term of use with each data share created. To receive the data, recipients must agree to the terms of use specified.

Alongside governance, security is fundamental in Azure Data Share and leverages core Azure security measures to help protect the data.

Enabling data collaboration

Azure Data Share maximizes access to simple and safe data sharing for organizations in many industries. For example, retailers can leverage Azure Data Share to easily share sales inventory and demographic data for demand forecasting and price optimization with their suppliers.

In the finance industry, Microsoft collaborated with Finastra, a multi-billion dollar company and provider of the broadest portfolio of financial services software in the world today that spans retail banking, transaction banking, lending, and treasury and capital markets. Finastra is fully integrating Azure Data Share with their open platform, FusionFabric.cloud, to enable seamless distribution of premium datasets to a wider ecosystem of application developers across the FinTech value chain. These datasets have been curated by Finastra over several years, and by leveraging the data distribution capabilities of Azure Data Share, ingestion by app developers and other partners requires simple wrangling, significantly reducing the go to market timeframe and unlocking net new revenue potential for Finastra.

“Our decision to integrate Azure Data Share with Finastra’s FusionFabric.cloud platform is now a great way to further accelerate innovation via an expanded open ecosystem. Our partnership with Microsoft truly provides us with limitless opportunities to drive transformation in Financial Services.”

– Eli Rosner, Chief Product and Technology Officer, Finastra

Next steps

Industries of all types need a simple and safe way to share data. Azure Data Share opens up new opportunities for innovation and insights to drive greater business impact.

Watch the video about Azure Data Share.
Get started with documentation.
Start using Azure Data Share in the Azure portal.

Quelle: Azure

Enhancing Microsoft's commercial marketplace for partners

As we head into the global partner conference Microsoft Inspire on July 14-18, 2019, a big focus is on rethinking how we make it easier for customers to discover, try, and buy cloud-based software and services from our partners. Today, we're excited to announce new tools, commerce options, and a rewards program through the Microsoft commercial marketplace that help partners leverage this important distribution channel.

Today, we're excited to announce new tools, commerce options, and a rewards program through the Microsoft commercial marketplace that makes it easier than ever for our partners to grow their business through this important distribution channel.

Commercial marketplace as a new distribution channel

Many people think of a commercial marketplace as a simple catalog of offer listings which are often difficult to navigate.  For customers, they are often linked off to a different experience for trial and purchase. Publishers and partner selling solutions are challenged by how to differentiate their solutions to stand out in the volume of offers.

We are working with our partner community to ensure the commercial marketplace experiences deliver a new distribution channel to drive their business growth. For example, Microsoft AppSource targets business decision makers while Azure Marketplace targets IT and developers. This includes having the commerce capabilities and solution supply to capture the rising customer demand in online enterprise software purchases.

Microsoft’s commercial marketplace has at its core, one product catalog, which includes both Microsoft cloud software and services as well as software and services from our partners built on top of and to connect with one or more cloud services offered by Microsoft (Microsoft 365, Dynamics 365, Microsoft Power Platform, and Azure) publishing as transactable offers. This is not just for independent software vendors (ISVs) creating repeatable intellectual property (IP). The commercial marketplace experiences also support offers from managed service providers (MSPs) and consulting services from systems integrators (SIs) such as one-day assessments, migration offers, and more.

Customers can discover, try, and buy solutions from the marketplace in one of three ways:

Direct from the publishers
Through our field sales teams who retire quota for selling eligible partner solutions, or
Through our global distribution channel, where we now also pay the channel a 10 percent incentive to sell marketplace publisher solutions with a transactable SaaS offer, and who participate in the IP co-sell program. 

Customers are looking for quicker buying experiences where they can purchase Microsoft products AND solutions from our partners – together in one place, with one transaction, on a unified invoice, which the commercial marketplace provides.

Using the commercial marketplace as a strategic distribution channel will require partners to think about their business model in new and different ways, which can provide significant new revenue streams. For instance, any publisher can continue to list or trial their solution in Microsoft AppSource or Azure Marketplace, but the impact will likely be similar to what they face today, where the customer discovery experience is crowded due to volume of offers and the publisher struggles to differentiate their solution. However, when a publisher chooses to transact in Microsoft’s commercial marketplace, they get access to a whole new set of benefits and ways to sell:

Gain access to a global reseller channel with over 70,000 cloud solution providers (CSP) in over 140 countries who receive an incentive directly from Microsoft when they resell publisher solutions.
Provides simplified deal-making with custom contract amendments.
Centralized partnership experience via Partner Center for the commercial marketplace onboarding, lead sharing, deal registration, benefits, incentives, sales analytics, and investments.
New go-to-market (GTM) benefits via marketplace rewards that unlocks GTM benefits for publishers as they reach various transaction thresholds.

A single onboarding and management experience

Whether a customer buys direct through Microsoft field sellers or through CSP, each of these channels is accessible and managed by partners through a single ingestion point known as Partner Center. Within Partner Center, publishers can publish marketplace offers and manage their engagements, while resellers can bundle Microsoft software and services with publisher’s software and services. This simplifies customer, publisher, and reseller engagement with one transaction and one invoice.

New commerce options

To accompany this new publisher experience, we’ve released new commerce capabilities that partners of all sizes are already starting to benefit from such as ESRI with site-based SaaS, Barracuda, and Trend Micro who use custom business models for their SaaS-based applications. Approved Contact, Crossware, and MongoDB are also using the per-seat SaaS capabilities and managed services from long-time Microsoft partners like Ingram Micro.

These new commerce enhancements allow publishers to customize their offers to meet customer needs and scale through the global reach of Microsoft’s customer and channel communities.

Marketplace rewards

We’re also sharing marketplace rewards, which is a new benefits program which will enhance the success of publishers with transactable offers in the commercial marketplace. Through the program publishers can unlock sales, marketing, and technical benefits to help accelerate their success. As a publisher’s business grows they’ll continue to unlock more benefits designed to provide support at every stage of their growth. This comes with a new badging program for Microsoft AppSource and Azure Marketplace that will quickly direct customers to partner solutions they can trust, which will work with cloud services from Microsoft. We will be publishing additional details on the program next week during Microsoft Inspire.

With these capabilities, publishers will be able to create new revenue streams, reach new customers in new markets, and grow their business faster than ever before.

Next steps

Learn more about how to onboard and publish your offers at Partner Center, how to list them on Microsoft AppSource and Azure Marketplace, and how to take advantage of the new go-to-market services and onboarding resources.

Visit the Microsoft Inspire site, which will be updated with materials, photos, and keynote replays for more highlights from the event.
Quelle: Azure

Two ways to share Azure Advisor recommendations

If your IT organization is like most, you probably work with many different people across many different teams. When it comes to common IT tasks like optimizing your cloud workloads, you might need to interact with several resource owners or even complete a formal review process.

That’s why with Azure Advisor, we’ve made it easy to share recommendations with other people across your teams so you can follow best practices that help you get the most out of Azure. Advisor is a free Azure service that helps you optimize your Azure resources for high availability, security, performance, and cost by providing personalized recommendations based on your usage and configurations.

Here are two ways you can share your Advisor best practice recommendations with your teams.

1. Export a PDF or CSV of your Advisor recommendations

Probably the simplest way to share your Advisor recommendations is by exporting an Advisor recommendation report as a PDF or CSV through the Advisor UI in the Azure portal.

This report shows a summary of your Advisor recommendations by category, subscription, and potential business impact. Then you can easily share it with other teams so the resource owners can take action and optimize their resources for high availability, security, performance, and cost.

If you want to provide a specific view of a subset of your recommendations, you can use the UI filters or drill down into specific categories and recommendations. The recommendation report will only contain what you see on the screen when you generate it, which can help you focus on the most critical optimizations.

2. Use the Advisor API to integrate with your ticketing system or dashboards

The other way to share your Advisor recommendations with other people in your organization is via the Advisor REST API. Using this API, you can connect Advisor with your organization’s ticketing system and assign remediation work, set up an internal working dashboard your teams can review and action, or leverage Advisor’s recommendation data any way you choose.

The visual above shows just one way you can use the Advisor API with your ticketing application to share Advisor recommendations with your teams. Some setup is required, but once this scenario is complete, you can start remediating your recommendations more programmatically which will save you time as you optimize your resources.

This more advanced approach tends to work best for larger organizations, organizations managing a large number of Azure subscriptions and resources that are generating a large number of recommendations, and organizations that have a fairly sophisticated IT practice in place, since it scales well with the size of your deployments.

Visit the Advisor API documentation to learn more.

Get started with Advisor

Visit Advisor in the Azure portal to get started reviewing, sharing, and remediating your recommendations. For more in-depth guidance, visit the documentation. Let us know if you have a suggestion for Advisor by submitting an idea to the Azure Advisor feedback forum.
Quelle: Azure

Analyze AI enriched content with Azure Search’s knowledge store

Through integration with Cognitive Services APIs, Azure Search has long had the ability to extract text and structure from images and unstructured content. Until recently, this capability was used exclusively in full text search scenarios, exemplified in demos like the JFK files which analyzes diverse content in JPEGs and makes it available for online search. The journey from visual unstructured content, to searchable structured content is enabled by a feature called cognitive search. This capability in Azure Search is now extended with the addition of a knowledge store that saves enrichments for further exploration and analysis beyond search itself.

The knowledge store feature of Azure Search, available in preview, refers to a persistence layer in cognitive search that describes a physical expression of documents created through AI enrichments. Enriched documents are projected into tables or hierarchical JSON, which you can explore using any client app that is able to access Azure Storage. In Azure Search itself, you define the physical expression or shape of the projections in the knowledge store settings within your skillset.

Customers are using a knowledge store (preview) in diverse ways, such as to validate the structure and accuracy of enrichments, generate training data for AI models, and ad-hoc analysis of their data.

For example, the Metropolitan Museum of Art opened access to all images of public domain works in its collection. Enriching the artworks with cognitive search and the knowledge store allowed us to explore the latent relationships within the artworks on different dimensions like time and geography. Questions like how have images of family groups changed over time, or when were domestic animals included in paintings, are now answerable when you are able to identify, extract, and save the information in a knowledge store (preview).

With the knowledge store, anyone with an Azure subscription can apply AI to find patterns, insights, or create dashboards over previously inaccessible content.

What is the knowledge store (preview)?

Cognitive search is the enrichment of documents with AI skills before they are added to your search index. The knowledge store allows you to project the already enriched documents as objects (blobs) in JSON format or tabular data in table storage.

As part of your projection, you can shape the enriched document to meet your needs. This ensures that the projected data aligns with your intended use.

When using tabular projections, a knowledge store (preview) can project your documents to multiple tables while preserving the relationships between the data projected across tables. The knowledge store has several other features like allowing you to save multiple unrelated projections of your data. You can find more information about a knowledge store (preview) in the overview documentation.

Data visualization and analytics

Search enables you to find relevant documents, but when you’re looking to explore your data for corpus wide aggregations or want to visualize changes over time you need your data represented in a form other than a search index.

Leveraging Power BI’s integration with Azure tables, gets your dashboard started with only a few clicks. To identify insights from the enriched documents over dimensions like time or space, simply project your enriched documents into tables, validate that Power BI recognizes the relationships and you should now have your data in a format that is ready to consume within the visuals.

When you create a visual, any filters work, even when your data spans related tables. As an example, the art dashboard was created on the open access data from the MET in the knowledge store and the Art Explorer site uses the search index generated from the same set of enrichments.

The art explorer site allows you to find art works and related works while the Power BI report gives you a visual representation of the corpus and allows you to slice your data along different dimensions. You now can answer questions like “How does body armor evolve over time?”

In this example, a knowledge store (preview) enabled us to analyze the data ad-hoc. In another example, we may for instance enrich invoices or business forms, project the structured data to a knowledge store (preview), and then create a business-critical report.

Improving AI models

A knowledge store (preview) can also help improve the cognitive search experience itself as a data source for training AI models deployed as a custom skill within the enrichment pipeline. Customers deploying an AI model as a custom skill can project a slice of the enriched data shaped to be the source of their machine learning (ML) pipelines. A knowledge store (preview) now serves as a validator of the custom skill as well as a source of new data that can be manually labeled to retrain the model. While the enrichment pipeline operates on each document individually, corpus level skills like clustering require a set of documents to act on. A knowledge store (preview) can operate on the entire corpus to further enrich documents with skills like clustering and save the results back in a knowledge store (preview) or update the documents in the index.

Getting started

To start using a knowledge store (preview) you will need to:

Add a knowledge store (preview) configuration to your skillset.
Optionally, add a shaper skill to the skillset to define the shape of the projected enrichment.
Add a projection for tables, objects, or both to a knowledge store (preview). You may project the output of the shaper skill, or elements from the enriched document directly.

A knowledge store (preview) enables the use of your enriched data in new or improved models, visualizing and exploring the data in tools like Power BI and app based experiences merging the raw and enriched data. We will continue to add more capabilities and updates over the coming months.

For a detailed walkthrough, see the knowledge store (preview) getting started guide.
Quelle: Azure

Reducing overall storage costs with Azure Premium Blob Storage

In this blog post, we will take a closer look at pricing for Azure Premium Blob Storage, and its potential to reduce overall storage costs for some applications.

Premium Blob Storage is Azure Blob Storage powered by solid-state drives (SSDs) for block blobs and append blobs. For more information see, “Azure Premium Blob Storage is now generally available.” It is ideal for workloads that require very fast storage response times and/or has a high rate of operations. For more details on performance see, “Premium Block Blob Storage – a new level of performance.”

Azure Premium Blob Storage utilizes the same ‘pay-as-you-go’ pricing model used by standard general-purpose V2 (GPv2) hot, cool, and archive. This means customers only pay for the volume of data stored per month and the quantity of operations performed.

The current blob pricing can be found on the Azure Storage pricing page. You will see, data storage gigabyte (GB) pricing decreases for colder tiers, while the inverse is true for operation prices where operations per 10,000 pricing decreases for hotter tiers. Premium data storage pricing is higher than hot data storage pricing. However, read and write operations pricing for premium are lower than hot read and write operations. This means premium blob storage is meant to store data that is transacted upon frequently and is not intended for storing infrequently or rarely accessed data.

Given the lower operations costs, is there a point where premium, not only provides better performance, but also costs less than standard (GPv2) hot?

To answer this question, I created the graph below, which shows the relative total monthly cost of storing 1 Terabytes (TiB) of data in standard (GPv2) hot and premium, varying the operations per second performed on this 1TiB of data using a 70/30 split between read and write operations.

As you can see in the graph above, the estimated total monthly cost for premium becomes less than standard (GPv2) hot between 40 to 50 operations per second for each 1TiB of data. This means customers will save money for workloads with high rate of operations by using premium even if they do not require the better performance provided by premium.

Next steps

To get started with Premium Blob Storage, you provision a ‘Block Blob’ storage account in your subscription, and start creating containers and blobs using the existing Blob Service REST API or any existing tools such as AzCopy or Azure Storage Explorer.

Conclusion

We are very excited about Azure Premium Blob Storage providing low and consistent latency, and the potential cost savings for applications with high rate of operations. We look forward to hearing your feedback at premiumblobfeedback@microsoft.com or feel free to share your ideas and suggestions for Azure Storage on our feedback forum. To learn more about Azure Blob Storage please visit our product page.
Quelle: Azure

What’s new in Azure Firewall

This post was co-authored by Anitha Adusumilli, Principal Program Manager, Azure Networking. 

Today we are happy to share several key Azure Firewall capabilities as well as update on recent important releases into general availability (GA) and preview.

Multiple public IPs soon to be generally available
Availability Zones now generally available
SQL FQDN filtering now in preview
Azure HDInsight (HDI) FQDN tag now in preview
Central management using partner solutions

Azure Firewall is a cloud native firewall-as-a-service offering which enables customers to centrally govern and log all their traffic flows using a DevOps approach. The service supports both application and network level filtering rules and is integrated with the Microsoft Threat Intelligence feed for filtering known malicious IP addresses and domains. Azure Firewall is highly available with built-in auto scaling.

Multiple public IPs soon to be generally available

You can now associate up to 100 public IP addresses with your firewall. This enables the following scenarios:

DNAT – You can translate multiple standard port instances to your backend servers. For example, if you have two public IP addresses, you can translate TCP port 3389 (RDP) for both IP addresses.
SNAT – Additional ports are available for outbound SNAT connections, reducing the potential for SNAT port exhaustion.

Figure one – Sample Azure Firewall Public IP configuration with multiple public IPs.

Currently, Azure Firewall randomly selects the source public IP address to use for a connection. If you have any downstream filtering on your network, you need to allow all public IP addresses associated with your firewall. Explicit SNAT configuration is on our roadmap. See our documentation "Deploy an Azure Firewall with multiple public IP addresses using Azure PowerShell" for more information.

Multiple public IPs GA will be available in all public regions by July 12, 2019. It is currently supported using REST APIs, templates, PowerShell and Azure CLI. Portal support will be available shortly.

Availability Zones now generally available

Azure Firewall can be configured during deployment to span multiple Availability Zones for increased availability. With Availability Zones, your availability increases to 99.99 percent uptime. For more information, see the Azure Firewall Service Level Agreement (SLA). The 99.99 percent uptime SLA is offered when two or more Availability Zones are selected.

You can also associate Azure Firewall to a specific zone just for proximity reasons, using the service standard 99.99 percent SLA.

There's no additional cost for a firewall deployed in an Availability Zone. However, there are additional costs for inbound and outbound data transfers associated with Availability Zones. For more information, see Bandwidth pricing details.

 
Figure two – Creating Azure Firewall with 99.99 percent SLA

 

SQL FQDN filtering now in preview

You can now configure SQL FQDNs in Azure Firewall application rules. This allows you to limit access from your VNets to only the specified SQL server instances. The capability is available as a preview in all Azure regions.

Using this capability, you can filter traffic from your virtual networks (VNets) to Azure SQL Database, Azure SQL Data Warehouse, Azure SQL Managed Instance, or SQL IaaS instances deployed in your VNets.

During preview, SQL FQDN filtering is supported in proxy-mode only, port 1433. If you are using non-default ports for SQL IaaS traffic, you can configure those ports in the Firewall application rules. If you are using SQL in redirect mode, which is default for clients connecting within Azure, you can filter access using the SQL service tag as part of Azure Firewall network rules.

SQL FQDN filtering is currently available using REST APIs, templates, and Azure CLI. The portal will be available shortly.

Figure three – Creating Azure Firewall Application rule for SQL FQDN

Azure HDInsight (HDI) FQDN tag now in preview

We recently announced the availability of a FQDN Tag for Azure HDInsight (HDI). This tag is in public preview in all Azure public regions.

VNet-deployed Azure services like HDI have outbound infrastructure dependencies on other Azure services, for example, Azure Storage. To protect your data from exfiltration risk, you might want to use Azure Firewall to restrict outbound access for HDI clusters and allow access to only your data.  In addition, you should also allow access to the HDI infrastructure traffic.

FQDN tags for Azure Firewall allow services like HDI to pre-configure their infrastructure dependencies, for example, Azure Storage account FQDNs used by HDI. Instead of using network level service tags in the Azure Firewall to allow HDI outbound dependencies, you can get much more granular control to restrict outbound traffic for HDI by using the FQDN tags.

Figure four – Creating Azure Firewall Application rule for HDI FQDN tag

Central management using partner solutions

Azure Firewall public REST APIs can be used by third party security policy management tools to provide a centralized management experience for Azure Firewalls, Network Security Groups (NSGs), and network virtual appliances (NVAs).

Barracuda Cloud Security Guardian, now generally available in Azure Marketplace, automatically deploys and configures Barracuda's Cloud Generation WAF/Firewall, or Microsoft's Azure Firewall.
AlgoSec CloudFlow central management capability for Azure Firewall and NSGs is now public preview. For more information you can watch this video.
Tufin Orca, now public preview, automates the discovery, development and enforcement of a unified security policy across Kubernetes and Azure Firewall. For more information you can watch this video.

Next steps

For more information on everything we covered above please see the following blogs, documentation, and videos.

Azure Firewall Documentation
May blog: Azure Firewall and network virtual appliances

Azure Firewall central management partners:

AlgoSec CloudFlow
Barracuda Cloud Security Guardian
Tufin Orca

Quelle: Azure

Migrate to Azure HDInsight in as little as 12 weeks

Recent announcements in the open source ecosystem have led customers of prominent open source analytics technology companies to explore options with Microsoft Azure and to this end, we now have new and compelling offers aimed at helping on-premises open source analytics workloads to migrate to Azure! We are always trying to find ways to make it easier for our customers to move to the cloud and with this offer, our customers can now realize even greater savings and accelerate migration.

Azure HDInsight is an easy, cost-effective, fully managed and fully supported open source cloud-scale analytics service that can process massive amounts of data securely. We are pleased to offer customers who want to migrate their on-premises analytics workloads to the cloud, the HDInsight migration accelerator program which offers the following benefits:

Mitigate risk – Engagement led by Microsoft engineering and bolstered by custom accelerators enabling rapid migration of on-premises workloads to Azure HDInsight with minimal risk.
Get to production fast – Accelerated timeline of 12-17 weeks due to a structured engagement starting with an assessment all the way till go-live.

3. Discounted – A limited-time offer to take advantage of end-of-financial year discounts!
Quelle: Azure

CGG speeds geoscience insights on Azure HPC

A leader in geosciences, CGG has been imaging the earth’s subsurface for more than 85 years. Their products and solutions help clients locate natural resources. When a customer asked whether the cloud could speed the creation of the high-resolution reservoir models that they needed, CGG turned to Azure high performance computing (HPC).

CGG is a team of acknowledged imaging experts in the oil and gas industry. Their subsurface imaging centers are internationally acclaimed, and their researchers are regularly recognized with prestigious international awards for their outstanding technical contributions to the industry. CGG was an early innovator in seismic inversion, the technique of converting seismic survey reflection data into a quantitative description of the rock properties in an oil and gas reservoir. From this data, highly detailed models can be made, like the following image showing the classification of rock types in a 3D grid for an unconventional reservoir interval. These images are invaluable in oil and gas exploration. Accurate models help geoscientists reduce risk, drill the best wells, and forecast production better.

The challenge is to get the best possible look beneath the surface. The specialized software is compute-intensive, and complex models can take hours or even days to render.

Figure 1. An advanced seismic model, using CGG’s Jason RockMod.

Azure provides the elasticity of compute that enables our clients to rapidly generate multiple high-resolution rock property realizations with Jason RockMod for detailed reservoir models.”

– Joe Jacquot, Strategic Marketing Manager, CGG GeoSoftware

The challenge: prove it in the cloud

An oil and gas company in Southeast Asia asked CGG to work with them to find the optimum way to deploy reservoir characterization technology for their international team. The customer also wondered if they could take advantage of the cloud to save on hardware costs in their datacenter, where they already ran leading-edge CGG geoscience software solutions. They knew the theory of the cloud’s elasticity and on-demand scalability were familiar with its use in oil exploration, but they didn’t know if their applications would perform as well in the cloud as they did on-premises.

In a rapidly moving industry, performance gains translate to speedier decision making, accelerated exploration, and development timelines, so the company asked CGG to provide a demonstration. As geoscience experts, CGG wanted to show their software in the best possible light. They turned to the customer advisors at AzureCAT (led by Tony Wu) for help creating a proof of concept that demonstrated how the cloud capacity could help the company get insights faster.

Demo 1: CGG Jason RockMod

CGG Jason RockMod overcomes the limitations of conventional reservoir characterization solutions with a geostatistical approach to seismic inversion. The outcome is accurate reservoir models so in depth that geoscientists can use them to predict field reserves, fluid flow patterns, and future production.

The company wanted to better understand the scale of the RockMod simulation in the cloud, and wondered whether the cloud offered any real benefits compared to running the simulations on their own workstations.

Technical teams from CGG and AzureCAT looked at RockMod as a big-compute workload. In this type of architecture, computationally intensive operations such as simulations are split across CPUs in multiple computers (10 to 1,000s). A common pattern is to administer a cluster of virtual machines, then schedule and monitor the HPC jobs. This do-it-yourself approach would enable the company to set up their own cluster environment in Azure virtual machine scale sets. The number of VM instances can automatically scale per demand or on a defined schedule.

RockMod creates a family of multiple equi-probable simulation outputs, called realizations. In the proof of concept tests for their customer, CGG ran RockMod on a Microsoft Windows virtual machine on Azure. It served as the master node for job scheduling and distribution of multiple realizations across several Linux-based HPC nodes in a virtual machine scale set. Scale sets support up to 1,000 VM instances or 300 custom VM images, more than enough scale for multiple realizations.

An HPC task generates the realizations, and the speed depends on the number of CPU cores or non-hyperthreading more than the processor speed. AzureCAT recommended non-hyperthreading cores as the best choice for application performance. From a storage perspective, the realizations are not especially demanding. During testing, the storage IO rate went up to a few hundred megabytes per second (MB/s). Based on these considerations, CGG chose the cost-effective DS14V2 virtual machine with accelerated networking, which has 16 cores and 7 GB per core.

“Azure provided the ideal blend of HPC and storage performance and price, along with technical support that we needed to successfully demonstrate our high-end reservoir characterization technology on the cloud. It was a technical success and paved the way for full-scale commercial deployment.”

– Joe Jacquot, Strategic Marketing Manager, CGG GeoSoftware

Figure 2. Azure architecture of CGG Jason RockMod.

Benchmarks and benefits

A typical project is about 15 GB in size and includes more than three million seismic data traces. Some of the in-house workstations could render a single realization within a day or two, but the goal was to run 30 realizations by taking advantage of the cloud’s scalability. The initial small scale test ran one realization on one HPC node, which took just under 12 hours to complete. That was within the target range, and now the CGG team needed to see what would happen when they ran at scale.

To test the linear scalability of the job, they first ran eight realizations on eight nodes. The results were nearly identical to the small scale test run. The one realization to one node formula seemed to work. For the sake of comparison, they tried running 30 realizations on just eight nodes. That didn’t work so well, the tests were nearly four times slower.

The final test ran 30 realizations on 30 nodes, one realization to one node. The results were similar to the small scale test, and the job was completed in just over 12 hours. This scenario was tested several times to validate the results which were consistent. The test was a success.

Demo 2: CGG InsightEarth

In their exploration and development efforts, the company also used CGG InsightEarth, a software suite that accelerates 3D interpretation and visualization.

The interpretation teams at the company were using several InsightEarth applications to locate hydrocarbon deposits, faults and fractures, and salt bodies all of which can be difficult to interpret. They asked CGG to compare the performance of the InsightEarth suite on Azure to their workstations on premises. Their projects were typically 15 GB in size, with more than three million seismic data traces.

InsightEarth is a powerful, memory-intensive application. To get the best performance, the application must run on a GPU-based computer, and to get the best performance from GPUs, efficient memory access is critical. To meet this requirement on Azure, the team of engineers from CGG and AzureCAT looked at the specialized, GPU-optimized VM sizes that are available. The Azure NV-series virtual machines are powered by NVIDIA Tesla M60 GPUs and the NVIDIA GRID technology with Intel Broadwell CPUs, which are suited for compute-intensive visualizations and simulations.

The GRID license gives the company the flexibility to use an NV instance as a virtual workstation for a single user, or to give 25 users concurrent access to the VM running InsightEarth. The company wanted the collaborative benefits of the second model. Unlike a physical workstation, a cloud-based GPU virtual workstation would allow them all to view the data after running a pre-processing or interpretation step because all the data was in the cloud. This would streamline their workflow, eliminating the need to move the data back on premises for that task.

The initial performance tests ran InsightEarth on an NV24 VM on Azure. The storage IO demand was moderate, with rates around 100 MB/s. However, the VM’s memory size proved to be a bottleneck. The amount of data that could be loaded from the memory was limited, and the performance wasn’t as good as the more powerful GPU setup used on premises.

Next, the team ran a similar test using an ND-series VM. This type is specifically designed to offer excellent performance for AI and deep learning workloads, where huge data volumes are used to train models. They chose an ND24-size VM with double the amount of the memory and the newer NVIDIA Tesla P40 GPUs. This time, the results were considerably better than NV24.

From an infrastructure perspective, storage also matters when deploying high-performance applications on Azure. The team implemented SoftNAS, a type of storage that supports both the Windows and Linux VMs used in the overall solution. SoftNAS also suits the lower to mid-range storage IO demands for these simulation and interpretation workloads.

Figure 3. Azure Architecture for InsightEarth

Summary

GPUs and memory-optimized VMs provided the performance that the company needed for both Jason Rockmod and InsightEarth. Better yet, Azure gave them the scale they needed to run multiple realizations in parallel. In addition, they deployed GPUs to both software solutions, giving the CGG engineers a standard front end to work with. They set up access to the Azure resources through an easy-to-use portal based on Citrix Cloud, which also runs on Azure.

CGG’s customer, the oil and gas company, was delighted with the results. Based on the results of the benchmark tests, the oil and gas company decided to move all of their current CGG application workload to Azure. The cloud’s elasticity and Azure’s pay-per-use model were a compelling combination. Not only did the Azure solution perform, it proved to be more cost-effective compared to the limited scalability they could achieve with their computing power on-premises.

Company:
CGG

Microsoft Azure CAT Technical Lead:
Tony Wu

Quelle: Azure