Kategorie: Microsoft Azure

The network is a living organism

Organism, from Greek word Organismos, depicts a complex structure of living elements. But, what does a network have in common with organisms?

At Microsoft, we build and manage a hyper-scale global network that’s constantly growing and evolving. Supporting workloads such as Microsoft Azure, Bing, Dynamics, Office 365, OneDrive, Skype, Xbox, and soon LinkedIn, has stringent needs of reliability, security, and performance. Such needs make it imperative to continually monitor the pulse of the network, to detect anomalies, faults, and drive recovery at the millisecond level, much akin to monitoring a living organism.

Monitoring a large network that connects 38 regions, as of April 2017, hundreds of datacenters, thousands of servers with several thousand devices, and millions of components requires constant innovation and invention.

Figure 1. Microsoft global network

Figure 2. Illustration of a physical network in a datacenter

Four core principles drive the design and innovation of our monitoring services:

Speed and accuracy: It’s imperative to detect failures at the sub-second level and drive recovery of the same.
Coverage: From bit errors to bytes, to packets, to protocols, to components, to devices that make up the end-to-end network, our monitoring services must cover them all.
Scale: The services must process petabytes of logs, millions of events, and thousands of correlations that are spread over several thousand miles of connectivity across the face of the planet.
Optimize based on real user metrics: Our monitoring services must use metrics from a network topology level—within a rack, to a cluster, to a datacenter, to a region, to the WAN and the edge—and they must have the capability to zoom in and out.

We built innovations to proactively detect and localize a network issue, including PingMesh and NetBouncer. These services are always on and monitor the pulse of our network for latency and packet drops.

PingMesh uses lightweight TCP probes (consuming negligible bandwidth) for probing thousands of peers for latency measurement (RTT, or round trip time) and detects whether the issue is related to the physical network. RTT measurement is a good tool for detecting network reachability and packet-level latency issues.

After a latency deviation or packet drop is discovered, Netbouncer’s machine learning algorithms are then used to filter out transient issues, such as top-of-rack reboots for an upgrade. After completing temporal analysis in which we look at historical data and performance, we can confidently classify the incident as a network issue and accurately localize the faulty component. After the issue is localized, we can auto-mitigate it by rerouting the impacted traffic, and then either rebooting or removing the faulty component. In the following figure, green, yellow, or red visualize network latency ranges at the 99th percentile between a source-destination rack-pair.

Figure 3. Examples of network latency patterns for known failure modes

In some customer incidents, the incident might need deeper investigation by an on-call engineer to localize and find the root cause. We needed a troubleshooting tool to efficiently capture and analyze the life of a packet through every network hop in its path. This is a difficult problem because of the necessary specificity and scale for packet-level analysis in our datacenters, where traffic can reach hundreds of terabits per second. This motivated us to develop a service called Everflow—it’s used to troubleshoot network faults using packet-level analysis. Everflow can inject traffic patterns, mirror specific packet headers, and mimic the customer’s network packet. Without Everflow, it would be hard to recreate the specific path taken by a customer’s packet; therefore, it would be difficult to accurately localize the problem. The following figure illustrates the high-level architecture of Everflow.

Figure 4. Packet-level telemetry collection and analytics using Everflow

Everflow is one of the tools used to monitor every cable for frame check sequence (FCS) errors. The optical cables can get dirty from human errors like bending or bad placement, or simply aging of the cable. The following figure shows examples of cable bending and cable placed near fans that can cause an FCS error on this link.

Figure 5. Examples of cable bending, and cable placed near the fans that can cause an FCS error on this link

We currently monitor every cable and allow only one error for every billion packets sent, and we plan to further reduce this threshold to ensure link quality for loss-sensitive traffic across millions of physical cables in each datacenter. If the cable has a higher error rate, we automatically shut down any links with these errors. After the cable is cleaned or replaced, Everflow is used to send guided probes to ensure that the link quality is acceptable.

Beyond the datacenter, supporting critical customer scenarios on the most reliable cloud requires observing network performance end-to-end from Internet endpoints. The Azure WAN evolved to build a service called the Map of the Internet that monitors Internet performance and customer experience in real time. This system can disambiguate between expected client performance across wired and wireless connections, separates sustained issues from transient ones, and provides visibility into any customer perspective on demand. For example, it helps us to answer questions like, “Are customers in Los Angeles seeing high RTT on AT&T?”, “Is Taipei seeing increased packet loss through HiNet to Hong Kong?”, and “Is Bucharest seeing reliability issues to Amsterdam?” We use this service to proactively and reactively intervene on impact or risks to customer experiences and quickly correlate them to the scenario, network, and location at fault. This data also triggers automated response and traffic engineering to really minimize impact or mitigate ahead of time whenever possible.

Figure 6. Example of latency degradation alert with a peering partner

The innovation built to monitor our datacenters, and its connectivity is also leveraged to provide insights to our customers.

Typically, customers use our network services via software abstractions. Such abstractions, including virtual networks, virtual network interface cards, and network access control lists, hide the complexity and intricacies of the datacenter network. We recently launched Azure Network Watcher, a service to provide visibility and diagnostic capability of the virtual/logical network and related network resources.

Using Network Watcher, you can visualize the topology of your network, understand performance metrics of the resources deployed in the topology, create packet captures to diagnose connectivity issues, and validate the security perimeter of your network to detect vulnerabilities and for compliance/audit needs.

Figure 7. Topology view of a customer network

The following figure shows how a remote packet capture operation can be performed on a virtual machine.

Figure 8. Variable packet capture in a virtual machine

Building and operating the world’s most reliable and hyper-scale cloud is underpinned by the need to proactively monitor and detect network anomalies and take corrective action—much akin to monitoring a living organism. As the pace, scale, and complexity of the datacenters evolve, new challenges and opportunities emerge, paving the way for continuous innovation. We’ll continue to invest in networking monitoring and automatic recovery, while also sharing our innovations with customers to also help them manage their virtual networks.

References

PingMesh: Guo, Chuanxiong, Lihua Yuan, Dong Xiang, Yingnong Dang, Ray Huang, Dave Maltz, Zhaoyi Liu, et al. "Pingmesh: A large-scale system for data center network latency measurement and analysis." ACM SIGCOMM Computer Communication Review 45, no. 4 (2015): 139-152.

Everflow: Zhu, Yibo, Nanxi Kang, Jiaxin Cao, Albert Greenberg, Guohan Lu, Ratul Mahajan, Dave Maltz, et al. "Packet-level telemetry in large datacenter networks." In ACM SIGCOMM Computer Communication Review, vol. 45, no. 4, pp. 479-491. ACM, 2015.

Read more

To read more posts from this series please visit:

Networking innovations that drive the cloud disruption
SONiC: The networking switch software that powers the Microsoft Global Cloud
How Microsoft builds its fast and reliable global network
Lighting up network innovation
Azure Network Security
Microsoft's open approach to networking

Quelle: Azure

Announcing Azure CLI Shell (Preview); more Azure CLI 2.0 commands now generally Available

Following up on the generally available release of VM, ACS, Storage and Network commands in the new Azure CLI 2.0, we are today announcing a new Azure CLI interactive Shell preview mode release, in addition to the generally available release of following command modules: ACR, Batch, KeyVault, and SQL.

Interactive Shell

Azure CLI 2.0 provides an idiomatic command line interface that integrates natively with Bash and POSIX tools. It is easy to install, use and learn. You can use it to run one command at a time, as well as to run automation scripts composed of multiple commands, including other BASH commands. To support this, commands are not interactive and will error out when provided with incomplete or incorrect input.

However, there are circumstances when you might prefer an interactive experience, such as when learning the Azure CLI’s capabilities, command structures and output formats. Azure CLI Shell (az-shell) provides an interactive mode in which to run Azure CLI 2.0 commands. It provides autocomplete dropdowns, auto-cached suggestions combined with on the fly documentation, including examples of how each command is used. Azure CLI Shell provides an experience that makes it easier to learn and use Azure CLI commands.

We invite you to install and use the new interactive shell for Azure CLI 2.0. You can use it in a Docker image we created, or install it locally on your Mac or Windows machine. It works with your existing Azure CLI installations, and you can use the commands side-by-side in az-shell or another command shell of your choice (BASH on MAC/linux and cmd.exe on Windows).

New commands now generally available

Continuing with the momentum of our GA release of the first Azure CLI 2.0 command modules on Feb 27th, today we are also announcing that following command modules are now Generally Available: Azure Container Registry, Batch, KeyVault, and SQL. With this GA release, you can use these commands in production with full support from Microsoft through our Azure support channels or on GitHub. We don’t expect any breaking changes for these commands in future releases of Azure CLI 2.0.

Azure Container Registry enables developers to create and maintain Azure container registries to store and manage private Docker container images. Using the acr commands in Azure CLI 2.0, you can create and manage these registries right from the command line. After you create a registry, you can use other CLI commands to assign a service principal to it, manage admin credentials, and list the repositories within it.

Azure Batch service provides an environment developers can use to manage their compute resources, and to schedule jobs to run with specific resources and dependencies. Using the batch commands in Azure CLI 2.0, you can create Azure Batch accounts, applications, and application packages in that account. You can also create jobs, tasks and job schedules to run at specific times, and manage (create, update, delete) them directly from the command line.

Azure Key Vault helps safeguard cryptographic keys and secrets used by cloud applications and services. Developers and security administrators can generate keys, store and access them, set policies, and monitor their usage using this service. Using the keyvault commands in Azure CLI 2.0, you can create/delete a key vault, manage certificates, policies, import and create new keys, and set secrets to key vaults.

Azure SQL Database is a relational database-as-a-service using the Microsoft SQL Server engine. Using the SQL commands in Azure CLI 2.0, you can manage all aspects of this service from the command line: create/delete/update SQL server, create/delete/update databases and data warehouses and scale them individually by creating elastic pools and moving databases in and out of shared pools, etc.

In addition to the above commands being generally available, the new release also contains command modules for dev/test labs (lab) and monitoring (monitor) services that are now available in preview mode.

New features in Azure CLI 2.0

This release also contains some new features that will make working with the Azure CLI easier and more productive.

“Az find” is a new command for searching Azure CLI 2.0 commands based on simple text. As the number of commands and coverage of Azure services grows in Azure CLI 2.0, we recognize that it may become hard for developers to find the commands they need for specific tasks.

For example, the following command finds all Azure CLI 2.0 commands that contains the text “arm,” “template,” or “deploy.”

az find -q arm template deploy

`az monitor autoscale-settings get-parameters-template`
Scaffold fully formed autoscale-settings' parameters as json
template

`az group export`
Captures a resource group as a template.

`az group`
Manage resource groups and template deployments.

`az group deployment export`
Export the template used for the specified deployment.

`az group deployment create`
Start a deployment.

`az group deployment validate`
Validate whether the specified template is syntactically correct
and will be accepted by Azure Resource Manager.

`az vm capture`
Captures the VM by copying virtual hard disks of the VM and
outputs a template that can be used to create similar VMs.
For an end-to-end tutorial, see https://docs.microsoft.com/azure
/virtual-machines/virtual-machines-linux-capture-image.

`az keyvault certificate get-default-policy`
Get a default policy for a self-signed certificate
This default policy can be used in conjunction with `az keyvault
create` to create a self-signed certificate. The default policy
can also be used as a starting point to create derivative
policies. Also see: https://docs.microsoft.com/en-
us/rest/api/keyvault/certificates-and-policies

`az keyvault certificate create`
Creates a new certificate version. If this is the first version,
the certificate resource is created.
Create a Key Vault certificate. Certificates can also be used as a
secrets in provisioned virtual machines.

`az vm format-secret`
Format secrets to be used in `az vm create –secrets`
Transform secrets into a form consumed by VMs and VMSS create via
–secrets.

You can now set global defaults and scope for specific variables and resources that you need to use repeatedly within a command line session. You can set these defaults using the “az configure” command:

az configure –defaults group=MyResourceGroup

This sets the resource group to “MyResourceGroup” so that you don’t need to supply it as a parameter in subsequent commands that require this parameter. For example, you could then run the “vm show” command without explicitly specifying the resource group parameter:

az vm show -n MyLinuxVM

Name ResourceGroup Location
——— ————— ———-
MyLinuxVM MyResourceGroup westus2

You can also specify multiple defaults by listing them in <resource name=value> pairs in the “az configure” command, and you can reset them by simply setting an empty value in the configure command.

Start using Azure CLI 2.0 today!

Whether you are an existing CLI user, or you are starting a new Azure project, it’s easy to get started with the CLI directly, or use the interactive mode to master the command line with our updated docs and samples.

Azure CLI 2.0 is open source and on GitHub.

In the next few months, we’ll provide more updates. As ever, we want your ongoing feedback! Customers using the commands, that are now in GA release, in production can contact Azure Support for any issues, reach out via StackOverflow using the azure-cli tag, or email us directly at azfeedback@microsoft.com. You can also use the "az feedback" command directly from within the CLI to send us your feedback.
Quelle: Azure

New search analytics for Azure Search

One of the most important aspects of any search application is the ability to show relevant content that satisfies the needs of your users. Measuring relevance requires combining search results with the app side user interactions, and it can be hard to decide what to collect and how to do it. This is why we are excited to announce our new version of Search Traffic Analytics, a pattern on how to structure, instrument, and monitor search queries and clicks, that will provide you with actionable insights about your search application. You’ll be able to answer common questions, like most clicked documents or most common queries that do not result in clicks, as well as provide evidence for other situations, like deciding on the effectiveness of a new UI layout or tweaks on the search index. Overall, this new tool will provide valuable insights that will let you make more informed decisions.

Let’s expand on the scoring profile example. Let’s say you have a movies site and you think your users usually look for the newest releases, so you add a scoring profile with a freshness function to boost the most recent movies. How can you tell this scoring profile is helping your users find the correct movies? You will need information on what your users are searching for, the content that is being displayed and the content that your users select. When you have the data on what your users are clicking, you can create metrics to measure effectiveness and relevance.

Our solution

To obtain rich search quality metrics, it’s not enough to log the search requests; it’s also necessary to log data on what users are choosing as the relevant documents. This means that you need to add telemetry to your search application that logs what a user searches for and what a user selects. This is the only way you can have information on what users are really interested on and wether they are finding what they are looking for. There are many telemetry solutions available and we didn&#039;t invent yet another one. We decided to partner with Application Insights, a mature and robust telemetry solution, available for multiple platforms. You can use any telemetry solution to follow the pattern that we describe, but using Application Insights lets you take advantage of the Power BI template created by Azure Search.

The telemetry and data pattern consists of 4 steps:

1.    Enabling Application Insights
2.    Logging search request data
3.    Logging users’ clicks data
4.    Monitoring in Power BI desktop

Because it’s not easy to decide what to log and how to use that information to produce interesting metrics, we created a clear set schema to follow, that will immediately produce commonly asked for charts and tables out of the box on Power BI desktop. Starting today, you can access the easy to follow instructions on the Azure Portal and the official documentation.

Once you instrument your application and start sending the data to your instance of Application Insights, you will be able to use Power BI to monitor the search quality metrics. Upon opening the Power BI desktop file, you’ll find the following metrics and charts
•    Clickthrough Rate (CTR): ratio of users who click on a document to the number of total searches.
•    Searches without clicks: terms for top queries that register no clicks.
•    Most clicked documents: most clicked documents by ID in the last 24 hours, 7 days and 30 days.
•    Popular term-document pairs: terms that result in the same document clicked, ordered by clicks.
•    Time to click: clicks bucketed by time since the search query.

Operational Logs and Metrics

Monitoring metrics and logs are still available. You can enable and manage them in the Azure Portal under the Monitoring section.

Enable Monitoring to copy operation logs and/or metrics to a storage account of your choosing. This option lets you integrate with the Power BI content pack for Azure Search as well as your own custom integrations.

If you are only interested in Metrics, you don’t need to enable monitoring as metrics are available for all search services since the launch of Azure Monitor, a platform service that lets you monitor all your resources in one place.

Next steps

Follow the instructions in the portal or in the documentation to instrument your app and start getting detailed and insightful search metrics.

You can find more information on Application Insights here.  Please visit Application Insights pricing page to learn more about their different service tiers.
Quelle: Azure