da Agency

Learn how to deliver insights faster with Azure Synapse Analytics

Today, it’s even more critical to have a data-driven culture. Analytics and AI play a pivotal role in helping businesses make insights-driven decisions—decisions to transform supply chains, develop new ways to interact with customers, and evaluate new offerings.

Many organizations are turning to cloud analytics solutions to quickly create a data-driven culture, accelerate time to insight, reduce costs, and maximize ROI. Join us on Wednesday, June 17, 2020, from 10:00 AM–11:00 AM Pacific Time for Azure Synapse Analytics: How It Works, a virtual event where you’ll hear directly from Microsoft Azure customers. They’ll explain how they’re using the newest Azure Synapse capabilities to deliver insights faster, bring together an entire analytics ecosystem in a central location, reduce costs, and transform decision-making.

In technical demos, customers will show how they combine data ingestion, data warehousing, and big data analytics in a single cloud-native service with Azure Synapse. If you’re a data engineer trying to wrangle multiple data types from multiple sources to create pipelines, or a database administrator with responsibilities over your data lake and data warehouse, you’ll see how all this can be simplified in a code-free environment.

Customers will also demonstrate how Power BI provides a graphical complement to Azure Synapse with built-in Power BI authoring, giving their employees access to unprecedented insights from enterprise data—in seconds, through beautiful visualizations.

Companies have demonstrated significant cost reductions with cloud analytics solutions. Compared to on-premises solutions, these solutions:

Require lower implementation and maintenance costs.
Reduce analytics project development time.
Provide access to more frequent innovation.
Deliver higher levels of security and business continuity.
Help ensure better competitive advantage and higher customer satisfaction.

With cloud analytics, organizations pay for data and analytics tools only when needed, pausing consumption when not in use. Businesses can reallocate budget previously spent on hardware and infrastructure management to optimizing processes and launching new projects. In fact, customers average a 271 percent ROI with Azure Synapse—savings that come from lower operating costs, increased productivity, reallocating staff to higher-value activities, and increasing operating income due to improved analytics. Analytics in Azure is up to 14 times faster and costs 94 percent less than other cloud providers.

BI specialists, data engineers, and other IT and data professionals all use Azure Synapse to build, manage, and optimize analytics pipelines, using a variety of skillsets and in multiple industries. The Azure Synapse studio provides a unified workspace for data prep, data management, data warehousing, big data, and AI tasks.

Data engineers can use a code-free visual environment for managing data pipelines.
Database administrators can automate query optimization and easily explore data lakes.
Data scientists can build proofs of concept in minutes.
Business analysts can securely access datasets and use Power BI to build dashboards in minutes—all while using the same analytics service.

At the Azure Synapse Analytics: How It Works event, you’ll learn how to access and analyze all your data, from your enterprise data lake to multiple data warehouses and big data analytics systems, with blazing speed. With Azure Synapse, data professionals can query both relational and non-relational data using the familiar SQL language, using either serverless or provisioned resources.

Of course, trust is critical for any cloud solution. Customers will share how they take advantage of advanced Azure Synapse security and privacy features such as automated threat detection and always-on data encryption. They help ensure that data stays safe and private by using column-level security and native row-level security, as well as dynamic data masking to automatically protect sensitive data in real time.

Attend the Azure Synapse Analytics: How It Works virtual event on June 17, 2020, to learn how to deliver:

Powerful insights.
Unprecedented ROI.
Unified experience.
Limitless scale.
Unmatched security.

Register early for a chance to win a Microsoft Surface Go tablet (three winners total). Winners will be selected at random. NO PURCHASE NECESSARY. Open to any registered event attendee 18 years of age or older. Void in Cuba, Iran, North Korea, Sudan, Syria, Region of Crimea, and where prohibited. Sweepstakes ends June 17, 2020. See the Official Rules.  
Quelle: Azure

da Agency

Office Licensing Service and Azure Cosmos DB part 1: Migrating the production workload

This post is part 1 of a two-part series about how organizations use Azure Cosmos DB to meet real world needs, and the difference it’s making to them. In part 1, we explore the challenges that led the Microsoft Office Licensing Service team to move from Azure Table storage to Azure Cosmos DB, and how it migrated its production workload to the new service. In part 2, we examine the outcomes resulting from the team’s efforts.

The challenge: Limited throughput and other capabilities

At Microsoft, the Office Licensing Service (OLS) supports activation of the Microsoft Office client on millions of devices around the world—including Windows, Mac, tablets, and mobile. It stores information such as machine ID, product ID, activation count, expiration date, and more. OLS is accessed by the Office client more than more than 240 million times per day by users around the world, with the first call coming from the client upon license activation and then every 2-3 days thereafter as the client checks to make sure the license is still valid.

Until recently, OLS relied on Azure Table storage for its backend data store, which contained about 5 TB of data spread across 18 tables—with separate tables used for different license categories such as consumer, enterprise, and OEM pre-installation.

In early 2018, after years of continued workload growth, the OLS service began approaching the point where it would require more throughput than Table storage could deliver. If the issue wasn’t addressed, the inherent throughput limit of Table storage would begin to threaten overall service quality to the detriment of millions of users worldwide.

Danny Cheng, a software engineer at Microsoft, who leads the OLS development team explains:

“Each Table storage account has a fixed maximum throughput and doesn’t scale past that. By 2018, OLS was running low on available storage throughput, and, given that we were already maintaining each table in its own Table storage account, there was no way for us to get more throughput to serve more requests from our customers. We were being throttled during peak usage hours for the OLS service, so we had to find a more scalable storage backend soon.

In looking for a long-term solution to its storage needs, the OLS team wanted more than just additional throughput. We wanted the ability to deploy OLS in different regions around the world, as a means of minimizing latency by putting copies of the service closer to where our users are. But with Table storage, geo-replication capabilities are fairly limited.”

The OLS team also wanted better disaster recovery. With Table storage, they were storing all data in multiple regions within the United States. All reads and writes went to the primary region, and there were no SLAs in place for replication to the two backup regions, which could take up to 60 minutes. If the primary region became unavailable, human intervention would be required and data loss would be likely.

“If a region were to go down, it would be a real panic situation—with 30 to 60 minutes of downtime and a similar window for data loss,” says Cheng.

The solution: A lift-and-shift migration to Azure Cosmos DB

The OLS team chose to move to Azure Cosmos DB, which offered a lift-and-shift migration path from Table storage—making it easy to swap-in a premium backend service with turnkey global distribution, low latency, virtually unlimited scalability, guaranteed high availability, and more.

“At first, when we realized we needed a new storage backend, it was intimidating in that we didn’t know how much new code would be needed,” says Cheng. “We looked at several storage options on Azure, and Azure Cosmos DB was the only one that met all our needs. And with its Table API, we wouldn’t even need to write much new code. In many ways, it was an ideal lift-and-shift—delivering the scalability we needed and lots of other benefits with little work.”

Design decisions

In preparing to deploy Azure Cosmos DB, the OLS team had to make a few basic design decisions:

Consistency level, which gave the team options for addressing the fundamental tradeoffs between read consistency and latency, availability, and throughput.

“We picked strong consistency because some of our business logic requires reading from storage immediately after writing to it,” explains Cheng.

Partition key, which dictates how items within an Azure Cosmos DB container are divided into logical partitions—and determines the ultimate scalability of the data store.

“With the Azure Cosmos DB Table API, partition keys naturally map to what we had in Table storage—so we were able to reuse the same partition key,” says Cheng.

Migration process

Although Azure Cosmos DB offered a data migration tool, its use at that time would have entailed some downtime for the OLS service, which wasn’t an option. (Note: Today you can do live migrations without downtime.) To address this, the OLS team built a data migration solution that consisted of three components:

A Data Migrator that moves current data from Table storage to Azure Cosmos DB.
A Dual Writer that writes new database changes to both Table storage and Azure Cosmos DB.
A Consistency Checker that catches any mismatches between Table storage and Azure Cosmos DB.

The Data Migrator component is based on the same one provided to Microsoft customers by the Azure Cosmos DB team.

“To solve the downtime problem, we added Dual Writer and Consistency Checker components, which run on the same production servers as the OLS service itself,” explains Cheng.

The OLS team completed the migration process in late 2019. Today, Azure Cosmos DB is deployed to the same three regions as Table storage, which the team did to mimic the Table storage topology as closely as possible during the migration. Similarly, North Central US is the primary (read/write) region while the other two regions are currently read-only. The Azure Cosmos DB environment has 18 tables containing 5 TB of data and consumes about 1 million request units per second (RU/s), which are the units used to reserve guaranteed database throughput in Azure Cosmos DB.

Now that migration is complete, the team plans to turn on multi-master capabilities, which will write-enable all regions instead of just the primary one. Tying into this, the team also plans to scale out globally by replicating its backend store to additional regions around the world—as a means of improving latency from the perspective of the Office client by putting copies of the OLS data closer to where its users are.

In part 2 of this series, we examine the outcomes resulting from the team’s efforts to build its new Office Licensing Service on Azure Cosmos DB.

Get started with Azure Cosmos DB today

Visit Azure Cosmos DB.

See Introduction to Azure Cosmos DB Table API.

Quelle: Azure

da Agency

Office Licensing Service and Azure Cosmos DB part 2: Improved performance and availability

This post is part 2 of a two-part series about how organizations use Azure Cosmos DB to meet real world needs, and the difference it’s making to them. In part 1, we explored the challenges that led the Microsoft Office Licensing Service team to move from Azure Table storage to Azure Cosmos DB, and how it migrated its production workload to the new service. In part 2, we examine the outcomes resulting from the team’s efforts.

Strong benefits with minimal effort

The Microsoft Office Licensing Service (OLS) team’s migration from Azure Table storage to Azure Cosmos DB was simple and straightforward, enabling the team to meet all its needs with minimal effort.

An easy migration

In moving to Azure Cosmos DB, thanks to its Table API, the OLS team was able to reuse most of its data access code, and the migration engine they wrote to avoid any downtime was fast and easy to build.

Danny Cheng, a software engineer at Microsoft, who leads the OLS development team explains:

“The migration engine was the only real ‘new code’ we had to write. And the code samples for all three parts are publicly available, so it’s not like we had to start from scratch. All in all, the migration tooling we developed took three developers about four weeks each.”

Virtually unlimited throughput

Today, database throughput is no longer an issue for the OLS team. With Table storage, the team faced a throughput limit of 20,000 operations per second per storage account, which forced them to maintain each of their 18 tables in a different storage account to achieve maximum throughput. The team now maintains one Azure Cosmos DB account, which has no upper limit on throughput and can support more than 10 million operations per second per table—all dedicated and backed by SLAs.

Guaranteed high availability

Azure Cosmos DB gives the OLS team a 99.999 percent read availability SLA for all multi-region accounts. This has led to a significant increase in storage quality-of-service (QoS), as illustrated in the following metrics captured using internally developed tooling.

“During peak traffic hours, Azure Cosmos DB delivers much better storage QoS than we were seeing with Table storage,” says Cheng. “Today we’re seeing five nines, when in the past we were at about three nines.”

Automatic failover

The OLS team can now configure automatic or manual failovers to help protect against the unlikely event of a regional outage, with all SLAs maintained. The team can also prioritize failover order for its multi-region accounts and can manually trigger failover to test the end-to-end availability of OLS.

“We’ve configured automatic failover, but the service is so reliable that we haven’t needed it yet,” says Cheng.

Lower latency

Table storage provided the OLS team with no upper bounds on latency. In contrast, Azure Cosmos DB provides single-digit latency for reads and writes, backed with a guarantee of <10 millisecond latency for reads and writes at the 99th percentile, at any scale, anywhere in the world. The following metrics illustrate the differences in latency that the OLS service is seeing between Table storage and Azure Cosmos DB. (DbTable is Azure Table storage and CosmosDbTable is the Azure Cosmos DB Table API.)

Turnkey data distribution

With Table storage, options for global distribution were limited. What’s more, the OLS team couldn’t implement failover on its own. With Azure Cosmos DB, the team now enjoys distribution  to any number of regions—including multi-master capabilities, which when enabled will let any regions accept write operation.

“Just by clicking on the map, data can be automatically replicated to any Azure region in the world,” says Cheng. “This feature is very convenient, and we plan to put it to use soon.”

Other technical benefits

In addition to the above, Azure Cosmos DB provides the OLS team with some additional advantages over Table storage:

Automatic indexing. With Table storage, primary indexes are limited to PartitionKey and RowKey, and there are no secondary indexes. Azure Cosmos DB provides automatic and complete indexing on all properties by default, with no index management.

Faster query times. With Table storage, query execution uses the index for the primary key and scans otherwise. With Azure Cosmos DB, queries can take advantage of automatic indexing on all properties for faster query times.

Consistency. With Table storage, the OLS team was limited to strong consistency within the primary region and eventual consistency within the secondary region. With Azure Cosmos DB, they can choose from well-defined consistency levels, enabling them to optimize tradeoffs between read consistency and latency, availability, and throughput while they were designing the solution.

Get started with Azure Cosmos DB today

Visit Azure Cosmos DB.
See Introduction to Azure Cosmos DB Table API.

Quelle: Azure

da Agency

Automating cybersecurity guardrails with new Zero Trust blueprint and Azure integrations

In our day-to-day work, we focus on helping customers advance the security of their digital estate using the native capabilities of Azure. In the process, we frequently find that using Azure to improve an organization’s cybersecurity posture can also help these customers achieve compliance more rapidly.

Today, many of our customers in regulated industries are adopting a Zero Trust architecture, moving to a security model that more effectively adapts to the complexity of the modern environment, embraces the mobile workforce, and protects people, devices, applications, and data wherever they’re located.

Regardless of where the request originates or what resource it accesses, Zero Trust teaches us to “never trust, always verify.” In a Zero Trust model, every access request is strongly authenticated, authorized within policy constraints, and inspected for anomalies before granting access. This approach can aid the process of achieving compliance for industries that use NIST-based controls including financial services, defense industrial base, and government.

A Zero Trust approach should extend throughout the entire digital estate and serve as an integrated security philosophy and end-to-end strategy, across three primary principles: (1) verify explicitly, (2) enforce least privilege access, and (3) assume breach.

Use the Azure blueprint for faster configuration of Zero Trust

The Azure blueprint for Zero Trust enables application developers and security administrators to more easily create hardened environments for their application workloads. Essentially, the blueprint will help you implement Zero Trust controls across six foundational elements: identities, devices, applications, data, infrastructure, and networks.

Using the Azure Blueprints service, the Zero Trust blueprint will first configure your VNET to deny all network traffic by default, enabling you to extend it and/or set rules for selective traffic based on your business needs. In addition, the blueprint will enforce and maintain Azure resource behaviors and configuration in compliance with specific NIST SP 800-53 security control requirements using Azure Policy.

The blueprint includes Azure Resource Manager templates to deploy and configure Azure resources such as Virtual Network, Network Security Groups, Azure Key Vault, Azure Monitor, Azure Security Center, and more. If you’re working with applications that need to comply with FedRAMP High or DoD Impact Level 4 requirements or just want to improve the security posture of your cloud deployment, the blueprint for Zero Trust is designed to help you get there faster.

The Azure blueprint for Zero Trust is currently in preview with limited support. To learn more and find instructions to deploy into Azure, see Azure blueprint for Zero Trust. For more information, questions, and feedback, please contact us at Zero Trust blueprint feedback.

In addition to this new blueprint, we’re announcing two new integrations with Azure to bring faster authorization and increased flexibility to the public sector and regulated industries:

Accelerate risk management for Azure deployments with Xacta

Increasing the speed with which cloud-based initiatives achieve authorization is a critical part of modernization. Often this process is highly manual and lacks the ability to provide a clear picture for continuous monitoring

Xacta now integrates with Azure Policy and Azure Blueprints, enabling customers to centrally manage compliance policies, track their compliance status, and more easily enforce policies to ensure ongoing compliance. For example, Xacta streamlines and automates many labor-intensive tasks associated with key security frameworks such as the NIST Risk Management Framework (RMF), NIST Cybersecurity Framework (CSF), FedRAMP, and ISO 27001.

Through this new integration, Azure Policy automatically generates a significant portion of the required accreditation package directly into Xacta, instantiating a risk management framework and reducing the manual effort required of risk professionals, freeing up their time to focus on critical risk decisions.

Enable continuous monitoring of containers using Anchore

Customers using containers to achieve greater flexibility within regulated environments commonly encounter security and governance challenges. To address those challenges, Anchore recently announced their support for Windows containers, delivering more choice for public sector agencies and enterprises developing container-based applications and implementing broad DevSecOps initiatives. Anchore Enterprise 2.3 performs deep image inspection of Windows container images, helping teams establish policy-based approaches to container compliance without compromising velocity.

Whether you’re using containers today or evaluating services, such as Azure Kubernetes Service, you can count on us to continue to provide world-class cybersecurity technology, controls, and best practices to help you accelerate both security and compliance.

Learn more

To learn more about how to implement Zero Trust architecture on Azure, read the six-part blog series on the Azure Government Dev blog. You may also want to bookmark the Security blog to keep up with our coverage on security matters and follow us at @MSFTSecurity for the latest news and updates on cybersecurity.
Quelle: Azure

da Agency

Use Azure Firewall for secure and cost-effective Windows Virtual Desktop protection

This post was co-authored by Pavithra Thiruvengadam, Program Manager, Windows Virtual Desktop

Work from home policies require many IT organizations to address fundamental changes in capacity, network, security, and governance. Many employees aren’t protected by the layered security policies associated with on-premises services while working from home. Virtual desktop infrastructure (VDI) deployments on Azure can help organizations rapidly respond to this changing environment.  However, you need a way to protect inbound or outbound internet access to and from these VDI deployments.

Windows Virtual Desktop is a comprehensive desktop and application virtualization service running in Azure. It’s the only VDI that delivers simplified management, multi-session Windows 10, and optimizations for Office 365. You can deploy and scale your Windows desktops and apps on Azure in minutes and get built-in security and compliance features. In this post, we explore how to use Azure Firewall for secure and cost-effective Windows Virtual Desktop protection.

Windows Virtual Desktop components

The Windows Virtual Desktop service is delivered in a shared responsibility model:

Customer-managed RD clients connect to Windows desktops and applications from their favorite client device from anywhere on the internet.
Microsoft-managed Azure service handles connections between RD clients and Windows Virtual Machines in Azure (including Windows 10 multi-session).
Customer-managed virtual network in Azure hosts Windows 10 multi-session virtual machines in host pools.

Windows Virtual Desktop doesn’t require you to open any inbound access to your virtual network. However, to ensure platform connectivity between customer-managed virtual machines and the service, a set of outbound network connections must be enabled for the host pool virtual network. While these dependencies can be configured using Network Security Groups, this configuration is limited to network-level traffic filtering only. For application-level protection, you can use Azure Firewall or a third party network virtual appliance (NVA). For best practices to consider before deploying an NVA, see Best practices to consider before deploying a network virtual appliance.

Host pool outbound access to Windows Virtual Desktop

Azure Firewall is a cloud-native firewall as a service (FWaaS) offering that allows you to centrally govern and log all your traffic flows using a DevOps approach. The service supports both application and network-level filtering rules and is integrated with the Microsoft Threat Intelligence feed for filtering known malicious IP addresses and domains. Azure Firewall is highly available with built-in auto scaling.

Azure Firewall provides a Windows Virtual Desktop FQDN Tag to simplify host pool outbound access to Windows Virtual Desktop. Use the following steps to allow outbound platform traffic:

Deploy Azure Firewall and configure your Windows Virtual Desktop host pool subnet User Defined Route (UDR) to route all traffic via the Azure Firewall.
Create an application rule collection and add a rule to enable the WindowsVirtualDesktop FQDN tag. The source IP address range is the host pool virtual network, the protocol is https, and the destination is WindowsVirtualDesktop.

   

The set of required storage and service bus accounts for your Windows Virtual Desktop host pool is deployment specific and isn’t yet captured in the WindowsVirtualDesktop FQDN tag. Additionally, a network rule collection is needed to allow DNS access from your Active Directory Domain Services (ADDS) deployment and KMS access from your virtual machines to Windows Activation Service. To configure access for these additional dependencies, see Use Azure Firewall to protect Windows Virtual Desktop deployments.

Host pool outbound access to the internet

Depending on your organization needs, you may want to enable secure outbound internet access for your end users. As Windows Virtual Desktop sessions are running on customer-managed virtual machines, they are also subject to your virtual network security controls. In cases where the list of allowed destinations is well-defined (for example, Office 365 access), you can use Azure Firewall application and network rules to configure the required access. This routes end-user traffic directly to the internet for best performance.

If you want to filter outbound user internet traffic using an existing on-premises secure web gateway, you can configure web browsers or other applications running on the Windows Virtual Desktop host pool with an explicit proxy configuration. For example, see How to use Microsoft Edge command-line options to configure proxy settings. These proxy settings only influence your end-user internet access, allowing outbound traffic directly via Azure Firewall.

Next steps

For more information on everything we covered above please see the following blogs, documentation, and videos.

What is Windows Virtual Desktop?
Azure Firewall documentation.
Use Azure Firewall to protect Windows Virtual Desktop deployments.
Azure Firewall February 2020 blog: New Azure Firewall certification and features in Q1 CY2020.

Quelle: Azure

da Agency

Azure Virtual Machine Scale Sets now provide simpler management during scale-in

We recently announced the general availability of three features for Azure Virtual Machine Scale Sets. Instance protection, custom scale-in policy, and terminate notification provide new capabilities to simplify management of virtual machine instances during scale-in.

Azure Virtual Machine Scale Sets are a way to collectively deploy and easily manage a number of virtual machine (VM) instances in a group. You can also configure autoscaling rules for your scale set that enable you to dynamically increase or decrease the number of instances based on what the workload requires.

With these new features, you now have more control over gracefully handling the removal of instances during scale-in, enabling you to achieve better user experience for your applications and services. These new features are available across all Azure regions for public cloud as well as sovereign clouds. There is no extra charge for using these features with Azure Virtual Machine Scale Sets.

Let’s take a look at how these features provide you better control during scale-in.

Instance protection—protect one or more instances from scale-in

You can apply the policy Protect from scale-in to one or more instances in your scale set if you do not want these instances to be deleted when a scale-in occurs. This is useful when you have a few special instances that you would like to preserve while dynamically scaling in or out other instances in your scale set. These instances might be performing certain specialized tasks different from other instances in the scale set and you may want these special instances to not be removed from the scale set. Instance protection provides you the capability to enable such scenarios for your workload.

Protect one or more instances from scale-set actions

Instance protection also allows you to protect one or more of your instances from getting modified during other scale-set operations like reimage or upgrade. This can be done by applying the policy Protect from scale-set actions to specific instances. Applying this policy to an instance automatically also protects it from a scale-in.

Custom scale-in policy—configure the order of instance removal during scale-in

When one or more instances need to be removed form a scale set during scale-in, then instances are selected for deletion in such a way that the scale set remains balanced across availability zones and fault domains, if applicable. Custom scale-in policies allow you to further specify and control the order in which instances should be selected for deletion during scale-in. You can use the OldestVM scale-in policy to remove the oldest created instance first, or NewestVM scale-in policy to remove the newest created instance first. In both the scenarios, balancing across availability zones is given preference. If you have applied either of the protection policies to an instance, then it will not be picked up for deletion during scale-in.

Below are a couple examples of the scale-in order for a scale set with three availability zones and initial instance count 9. These examples assume that the VM with smallest instance ID was created first and that the VM associated with highest instance ID was created last. The VM instance enclosed in a dotted square represents that it has been protected using one of the instance protection policies. The cross indicates that the VM instance will be selected for deletion during scale-in.

 

Terminate notification—receive in-VM notification of instance deletion

When an instance is about to be deleted from a scale set, you may want to perform certain custom actions on the instance. Examples of these actions could be de-registering from the load balancer, or copying the logs, among others. When instance deletions are triggered by the platform, for example due to a scale-in, then these actions need to be programmatically performed to ensure that application does not get interrupted or useful logs are properly retained. With the terminate notification feature, you can configure your instances to receive in-VM notifications about upcoming instance deletion and pause the delete operation for 5 to 15 minutes to perform such custom actions on the instance.

The terminate notifications are sent through the Azure metadata service—Scheduled events—and can be received using a REST endpoint accessible from within the VM instance. Specific actions or scripts can be configured to run when an instance receives the terminate notification at the configured endpoint. Once these actions are completed and you do not want to wait for the pre-configured pause timeout to finish, then you can approve the deletion by issuing a POST call to the metadata service. This will allow deletion of the instance to continue.

Get started

You can enable these features for your scale set using REST API, Azure CLI, Azure PowerShell or Azure Portal. Below are the links to the documentation pages for detailed instructions.

Instance protection
Custom scale-in policy
Terminate notification

Quelle: Azure

da Agency

Azure Blob Storage enhancing data protection and recovery capabilities

Enterprises, partners, and IT professionals store business-critical data in Azure Blob Storage. We are committed to providing the best-in-class data protection and recovery capabilities to keep your applications running. Today, we are announcing the general availability of Geo-Zone-Redundant Storage (GZRS)—providing protection against regional disasters and Account failover—allowing you to determine when to initiate a failover instead.

Additionally, we are releasing two new preview features: Versioning and Point in time restore. These new functionalities expand upon Azure Blob Storage’s existing capabilities such as data redundancy, soft delete, account delete locking, and immutable blobs, making our data protection and restore capabilities even better.

Geo-Zone-Redundant Storage (GZRS)

Geo-Zone-Redundant Storage (GZRS) and Read-Access Geo-Zone-Redundant Storage (RA-GZRS) are now generally available offering intra-regional and inter-regional high availability and disaster protection for your applications.

GZRS writes three copies of your data synchronously across multiple Azure Availability zones, similar to Zone redundant storage (ZRS), providing you continued read and write access even if a datacenter or availability zone is unavailable. In addition, GZRS asynchronously replicates your data to the secondary geo pair region to protect against regional unavailability. RA-GZRS exposes a read endpoint on this secondary replica allowing you to read data in the event of primary region unavailability.

To learn more, see Azure Storage redundancy.

Account failover

Customer-initiated storage account failover is now generally available, allowing you to determine when to initiate a failover instead of waiting for Microsoft to do so. When you perform a failover, the secondary replica of the storage account becomes the new primary. The DNS records for all storage service endpoints—blob, file, queue, and table—are updated to point to this new primary. Once the failover is complete, clients will automatically begin reading from and writing to data to the storage account in the new primary region, with no code changes.

Customer initiated failover is available for GRS, RA-GRS, GZRS and RA-GZRS accounts. To learn more, see our Disaster recovery and account failover documentation.

Versioning preview

Applications create, update, and delete data continuously. A common requirement is the ability to access and manage both current and previous versions of the data. Versioning automatically maintains prior versions of an object and identifies them with version IDs. You can restore a prior version of a blob to recover your data if it is erroneously modified or deleted.

A version captures a committed blob state at a given point in time. When versioning is enabled for a storage account, Azure Storage automatically creates a new version of a blob each time that blob is modified or deleted.

Versioning and soft delete work together to provide you with optimal data protection. To learn more, see our documentation on Blob versioning.

Point in time restore preview

Point in time restore for Azure Blob Storage provides storage account administrators the ability to restore a subset of containers or blobs within a storage account to a previous state. This can be done by an administrator to a specific past date and time in the event of an application corrupting data, a user inadvertently deleting contents, or a test run of a machine learning model.

Point in time restore makes use of Blob Change feed, currently in preview. Change feed enables recording of all blob creation, modification, and deletion operations that occur in your storage account. Today we are expanding upon our Change feed preview by enabling four new regions and enabling support for two new blob event types: BlobPropertiesUpdated and BlobSnapshotCreated.

This improvement now captures change records caused by the SetBlobMetadata, SetBlobProperties, and SnapshotBlob operations. To learn more, read Change feed support in Azure Blob Storage (Preview).

Point in time restore is intended for ISV partners and customers who want to implement their own restore workflow on top of Azure Storage. To learn more, see Point in time restore.

Build it, use it, and tell us about it

These new capabilities provide greater data protection control for all users of Azure Storage. The general availability release of GZRS adds region-replicated zone redundancy types. Account failover allows customers to control geo-replicated failover for their storage accounts. In addition, the previews of Versioning and Point in time restore allow greater control of data protection and restoration to a previous date and time.

We look forward to hearing your feedback on these features and suggestions for future improvements through email at AzureStorageFeedback@microsoft.com. As a reminder, we love hearing all of your ideas and suggestions about Azure Storage, which you can post at the Azure Storage feedback forum.
Quelle: Azure

da Agency

Minecraft Earth and Azure Cosmos DB part 2: Delivering turnkey geographic distribution

This post is part 2 of a two-part series about out how organizations are using Azure Cosmos DB to meet real world needs and the difference it’s making to them. In part 1, we explored the challenges that led service developers for Minecraft Earth to choose Azure Cosmos DB and how they’re using it to capture almost every action taken by every player around the globe—with ultra-low latency. In part 2 we examine the solution’s workload and how Minecraft Earth service developers have benefited from building it on Azure Cosmos DB.

Geographic distribution and multi-region writes

Minecraft Earth service developers used the turnkey geographic distribution feature in Azure Cosmos DB to achieve three goals: fault tolerance, disaster recovery, and minimal latency—the latter achieved by also using the multi-master capabilities of Azure Cosmos DB to enable multi-region writes. Each supported geography has at least two service instances. For example, in North America, the Minecraft Earth service runs in the West US and East US Azure regions, with other components of Azure used to determine which is closer to the user and route traffic accordingly.

Nathan Sosnovske, a Senior Software Engineer on the Minecraft Earth services development team explains:

“With Azure available in so many global regions, we were able to easily establish a worldwide footprint that ensures a low-latency gaming experience on a global scale. That said, people mostly travel within one geography, which is why we have multi-master writes setup between all of the service instances in each geography. That’s not to say that a player who lives in San Francisco can’t travel to Europe and still play Minecraft Earth—it’s just that we’re using a different mechanism to minimize round-trip latency in such cases.”

Request units per second (RU/s) consumption

In Azure Cosmos DB, request units per second (RU/s) is the “currency” used to reserve guaranteed database throughput. For Minecraft Earth, a typical write request consumers about 10 RU/s, with an additional 2-3 RU/s used for background processing of the append-only event log, which is driven by Azure Service Bus.

“We’ve found that our RU/s usage scales quite linearly; we only need to increase capacity when we have a commensurate increase in write requests per second. At first, we thought we would need more throughput, but it turned out there was a lot of optimization to be done,” says Sosnovske. “Our original design handled request volumes and complexity relatively well, but it didn’t handle the case where the system would shard—that is, physically repartition itself internally—because of overall data volumes.”

The reason for this was because allocated RU/s are equally distributed across physical partitions, and the physical partition with the most current data was running a lot hotter than the rest.

“Fortunately, because our system is modeled as an append only log that gets materialized into views for the client, we very rarely read old data directly from Azure Cosmos DB,” explains Sosnovske. “Our data model was flexible enough to allow us to archive events to cold storage after they were processed them into views, and then delete them from Azure Cosmos DB using its Time to Live feature.”

Today, with the service’s current architecture, Sosnovske isn’t worried about scalability at all.

“During development, we tested the scalability of Azure Cosmos DB up to one million RU/s, and it delivered that throughput without a problem,” Sosnovske says.

Initial launch of Mindcraft Earth

Minecraft Earth was formally released in one geography in October 2019, with its global rollout across all other geographies completed over the following weeks. For Minecraft fans, Minecraft Earth provides a means of experiencing the game they know and love at an entirely new level, in the world of augmented reality.

And for Sosnovske and all the other developers who helped bring Minecraft Earth to life, the opportunity to extend one of the most popular games of all time into the realm of augmented reality has been equally rewarding.

“A lot of us are gamers ourselves and jumped on the opportunity to be a part of it all,” Sosnovske recalls. “Looking back, everything went pretty well—and we’re all quite satisfied with the results.”

Benefits of using Azure Cosmos DB

Although Azure Cosmos DB is just one of several Azure services that support Minecraft Earth, it plays a pivotal role.

“I can’t think of another way we could have delivered what we did without building something incredibly complex completely from scratch,” says Sosnovske. “Azure Cosmos DB provided all the functionality we needed, including low latency, global distribution, multi-master writes, and more. All we had to do was properly put it to use.”

Specific benefits of using Azure Cosmos DB to build the Minecraft Earth service included the following:

Easy adoption and implementation. According to Sosnovske, Azure Cosmos DB was easy to adopt.

“Getting started with Azure Cosmos DB was incredibly easy, especially within the context of the .NET ecosystem,”  Sosnovske says. “We simply had to install the Nuget package and point it at the proper endpoint. Documentation for the service is very thorough; we haven’t had any major issues due to misunderstanding how the SDK works.”

Zero maintenance. As part of Microsoft Azure, Azure Cosmos DB is a fully managed service, which means that nobody on the Minecraft Earth services team needs to worry about patching servers, maintaining backups, data center failures, and so on.

“Not having to deal with day-to-day operations is a huge bonus,” says Sosnovske. “However, this is really a benefit of building on Azure in general.”

Guaranteed low latency. A big reason developers chose Azure Cosmos DB was because it provides a guaranteed single-digit (<10ms) latency SLA for reads and writes at the 99th percentile, at any scale, anywhere in the world. In comparison, Table storage latency would have been higher—with no guaranteed upper bound.

“Azure Cosmos DB is delivering as promised, in that we’re seeing an average latency of 7 milliseconds for reads,” says Sosnovske.

Elastic scalability. Thanks to the elastic scalability provided by Azure Cosmos DB, the game enjoyed a frictionless launch.

“At no point was Azure Cosmos DB the bottleneck in scaling our service,” says Sosnovske. “We’ve done a lot of work to optimize performance since initial release and knowing that we wouldn’t hit any scalability limits as we did that work was a huge benefit. We may have paid a bit more for throughput then we had to at first, but that’s a lot better than having a service that can’t keep up with growth in user demand.”

Turnkey geographic distribution. With Azure Cosmos DB, geographic distribution was a trivial task for Minecraft Earth service developers. Adjustments to provisioned throughput (in RU/s) are just as easy because Azure Cosmos DB transparently performs the necessary internal operations across all the regions, continuing to provide a single system image.

“Turnkey geo-distribution was a huge benefit,” says Sosnovske. “We did have to think a bit more carefully about how to model our system when turning on multi-master support, but it was orders of magnitude less work than solving the problem ourselves.”

Compliance. Through their use of Time-to-Live within Azure Cosmos DB, developers can safely store location-based gameplay data for short periods of time without having to worry about violating compliance mandates like Europe’s General Data Protection Regulation (GDPR).

“It lets us drive workflows like ‘This player should only be able to redeem this location once in a given period of time,’ after which Azure Cosmos DB automatically cleans up the data within our set TTL,” explains Sosnovske.

In summarizing his experience with Azure Cosmos DB, Sosnovske says it was quite positive.

“Azure Cosmos DB is highly reliable, easy to use after you take the time to understand the basic concepts, and, best of all, it stays out of the way when you’re writing code. When junior developers on my team are working on features, they don’t need to think about the database or how data is stored; they can simply write code for a domain and have it just work.”

Get started with Azure Cosmos DB

Visit Azure Cosmos DB.
Learn more about Azure for Gaming.

Quelle: Azure

da Agency

Minecraft Earth and Azure Cosmos DB part 1: Extending Minecraft into our real world

This post is part 1 of a two-part series about how organizations use Azure Cosmos DB to meet real world needs and the difference it’s making to them. In part 1, we explore the challenges that led service developers for Minecraft Earth to choose Azure Cosmos DB and how they’re using it to capture almost every action taken by every player around the globe—with ultra-low latency. In part 2, we examine the solution’s workload and how Minecraft Earth service developers have benefited from building it on Azure Cosmos DB.

Extending the world of Minecraft into our real world

You’ve probably heard of the game Minecraft, even if you haven’t played it yourself. It’s the best-selling video game of all time, having sold more than 176 million copies since 2011. Today, Minecraft has more than 112 million monthly players, who can discover and collect raw materials, craft tools, and build structures or earthworks in the game’s immersive, procedurally generated 3D world. Depending on game mode, players can also fight computer-controlled foes and cooperate with—or compete against—other players.

In May 2019, Microsoft announced the upcoming release of Minecraft Earth, which began its worldwide rollout in December 2019. Unlike preceding games in the Minecraft franchise, Minecraft Earth takes things to an entirely new level by enabling players to experience the world of Minecraft within our real world through the power of augmented reality (AR).

For Minecraft Earth players, the experience is immediately familiar—albeit deeply integrated with the world around them. For developers on the Minecraft team at Microsoft, however, the delivery of Minecraft Earth—especially the authoritative backend services required to support the game—would require building something entirely new.

Nathan Sosnovske, a Senior Software Engineer on the Minecraft Earth services development team explains:

“With vanilla Minecraft, while you could host your own server, there was no centralized service authority. Minecraft Earth is based on a centralized, authoritative service—the first ‘heavy’ service we’ve ever had to build for the Minecraft franchise.”

In this case study, we’ll look at some of the challenges that Minecraft Earth service developers faced in delivering what was required of them—and how they used Azure Cosmos DB to meet those needs.

The technical challenge: Avoiding in-game lag

Within the Minecraft Earth client, which runs on iOS-based and Android-based AR-capable devices, almost every action a player takes results in a write to the core Minecraft Earth service. Each write is a REST POST that must be immediately accepted and acknowledged to avoid any noticeable in-game lag.

“From a services perspective, Minecraft Earth requires low-latency writes and medium-latency reads,” explains Sosnovske. “Writes need to be fast because the client requires confirmation on each one, such as might be needed for the client to render—for example, when a player taps on a resource to see what’s in it, we don’t want the visuals to hang while the corresponding REST request is processed. Medium-latency reads are acceptable because we can use client-side simulation until the backing model behind the service can be updated for reading.”

To complicate the challenge, Minecraft Earth service developers needed to ensure low-latency writes regardless of a player’s location. This required running copies of the service in multiple locations within each geography where Minecraft Earth would be offered, along with built-in intelligence to route the Minecraft Earth client to the nearest location where the service is deployed.

“Typical network latency between the east and west coasts of the US is 70 to 80 milliseconds,” says Sosnovske. “If a player in New York had to rely on a service running in San Francisco, or vice versa, the in-game lag would be unacceptable. At the same time, the game is called Minecraft Earth—meaning we need to enable players in San Francisco and New York to share the same in-game experience. To deliver all this, we need to replicate the service—and its data—in multiple, geographically distributed datacenters within each geography.”

The solution: An event sourcing pattern based on Azure Cosmos DB

To satisfy their technical requirements, Minecraft Earth service developers implemented an event sourcing pattern based on Azure Cosmos DB.

“We originally considered using Azure Table storage to store our append-only event log, but its lack of any SLAs for read and write latencies made that unfeasible,” says Sosnovske. “Ultimately, we chose Azure Cosmos DB because it provides 10 millisecond SLAs for both reads and writes, along with the global distribution and multi-master capabilities needed to replicate the service in multiple locations within each geography.”

With an event sourcing pattern, instead of just storing the current state of the data, the Minecraft Earth service uses an append-only data store that’s based on Azure Cosmos DB to record the full series of actions taken on the data—in this case, mapping to each in-game action taken by the player. After immediate acknowledgement of a successful write is returned to the client, queues that subscribe to the append-only event store handle postprocessing and asynchronously apply the collected events to a domain state maintained in Azure Blob storage. To optimize things further, Minecraft Earth developers combined the event sourcing pattern with domain-driven design, in which each app domain—such as inventory items, character profiles, or achievements—has its own event stream.

“We modeled our data as streams of events that are stored in an append-only log and mutate an in-memory model state, which is used to drive various client views,” says Sosnovske. “That cached state is maintained in Azure Blob storage, which is fast enough for reads and helps to keep our request unit costs for Azure Cosmos DB to a minimum. In many ways, what we’ve done with Azure Cosmos DB is like building a write cache that’s really, really resilient.”

The following diagram shows how the event sourcing pattern based on Azure Cosmos DB works:

Putting Azure Cosmos DB in place

In putting Azure Cosmos DB to use, developers had to make a few design decisions:

Azure Cosmos DB API. Developers chose to use the Azure Cosmos DB Core (SQL) API because it offered the best performance and the greatest ease of use, along with other needed capabilities.

“We were building a system from scratch, so there was no need for a compatibility layer to help us migrate existing code,” Sosnovske explains. “In addition, some Azure Cosmos DB features that we depend on—such as TransactionalBatch—are only supported with the Core (SQL) API. As an added advantage, the Core (SQL) API was really intuitive, as our team was already familiar with SQL in general.”

Read Introducing TransactionalBatch in the .NET SDK to learn more.

Partition key. Developers ultimately decided to logically partition the data within Azure Cosmos DB based on users.

“We originally partitioned data on users and domains—again, examples being inventory items or achievements—but found that this breakdown was too granular and prevented us from using database transactions within Azure Cosmos DB to their full potential,” says Sosnovske.”

Consistency level. Of the five consistency levels supported by Azure Cosmos DB, developers chose session consistency, which they combined with heavy etag checking to ensure that data is properly written.

“This works for us because of how we store data, which is modeled as an append-only log with a head document that serves as a pointer to the tail of the log,” explains Sosnovske. “Writing to the database involves reading the head document and its etag, deriving the N+1 log ID, and then constructing a transactional batch operation that overwrites the head pointer using the previously read etag and creates a new document for the log entry. In the unlikely case that the log has already been written, the etag check and the attempt to create a document that already existed will result in a failed transaction. This happened regardless of whether another request ‘beats’ us to writing or if our request reads slightly out-of-date data.”

In part 2 of this series, we examine the solution’s current workload and how Minecraft Earth service developers have benefited from building it on Azure Cosmos DB.

Get started with Azure Cosmos DB

Visit Azure Cosmos DB.
Learn more about Azure for Gaming.

Quelle: Azure

da Agency

How Azure VPN helps organizations scale remote work

In the weeks and months we have all been grappling with the global pandemic, there’s no doubt about the impact it has had on the lives of people everywhere. A shift to remote work is one of the widespread effects of the global pandemic, and we heard from organizations around the world who are looking for ways to enable more of their employees to work remotely for their safety and that of the community. With this shift, we’re working to address common infrastructure challenges businesses face when helping employees stay connected at scale.

Common challenges for businesses expanding secure, remote access

One of the major challenges while setting up remote access is providing workers/employees access to key internal resources, which may reside on-premises or Azure, for example, healthcare or government organizations who have sensitive patient or tax information in on-premises datacenters and other sensitive information in Azure.

Another challenge that the businesses around the world now face is how to quickly scale an existing VPN setup, which is typically targeted at a small portion of an organization’s workforce, to now accommodate all or most workers. Even within Microsoft, we’ve seen our typical remote access at 50,000+ employee spike to as high as 128,000 employees while we’re working to protect staff and our communities during the global pandemic.

How Azure VPN can help with secure, remote work at scale

The Azure network is designed to withstand sudden changes in the utilization of resources and can greatly help during periods of peak utilization. The Azure Point-to-Site (P2S) VPN Gateway solution is cloud-based and can be provisioned quickly to cater for the increased demand of users to work from home. It can scale up easily and be turned off just as easily.

Tips to help you get started with Azure VPN Gateway

Based on the customers we’ve been working with and best practices we’ve established over our years of work with enterprises, here are tips to help your own company get started with Azure VPN Gateway:

For scenarios where you need to access resources on-premises or in Azure, you can build a VPN Gateway in Azure and connect your existing VPN solution to Azure. This eliminates single point of failure to on-premises and provides nearly limitless scale. See Remote work using Azure VPN Gateway Point-to-Site to help you understand how to set up Azure VPN Gateway and integrate it with your existing setup.
Use Azure Active Directory (Azure AD), certificate-based authentication, or RADIUS authentication to authenticate users and to validate the status of their device before allowing them on VPN. You can review Create an Azure AD tenant for P2S OpenVPN protocol connections for more details.
We recommend split tunneling VPN traffic. This allows network traffic to go directly to public resources—such as Office 365 and Windows Virtual Desktops—and prevents internet traffic from having to go back to the corporate office, reducing overall load and bandwidth on your corporate internet links and on-premises VPN infrastructure.
To improve on-premises to Azure connectivity to support scale, you can work with your local telecommunications provider to temporarily increase connectivity to the internet. This can help scale your connectivity from your office or data center to Microsoft up to 10 Gbps.
Apply all available security updates to your VPN and firewall devices. The patching and updates for the Azure VPN gateway are managed by Microsoft. For your on-premises devices, please follow the guidance from the device vendor. We’ve brought together tips in this blog post.

How to get started

If you’re not currently using P2S tunnels, please review the following document, evaluate your scenario, and follow the instructions to start using Azure VPN services.
Quelle: Azure