April 2019 - Seite 79 von 85 - Cloud Computing Köln

The typical workflow we hear from customers – both ITOps and DevOps teams – is that alerts go to the appropriate team (on-call individual) based on some metadata such as subscription ID, resource groups, and more. The common alert schema makes this workflow more streamlined by providing a clear separation between the essential meta-data that is needed to route the alert, and the additional context that the responsible team (or individual) needs to debug and fix the issue.Azure Monitor alerts provides rich alerting capabilities on a variety of telemetry such as metrics, logs, and activity logs. Over the past year, we have unified the alerting experience by providing a common consumption experience including UX and API for alerts. However, the payload format for alerts remained different which puts the burden of building and maintaining multiple integrations, one for each alert type based on telemetry, on the user. Today, we are releasing a new common alert schema that provides a single extensible format for all alert types.

What’s the common alert schema?

With the common alert schema, all alert payloads generated by Azure Monitor will have a consistent structure. Any alert instance describes the resource that was affected and the cause of the alert, and these are described in the common schema in the following sections:

Essentials: A set of standardized fields which are common across all alert types. It describes what resource the alert is on, along with additional common alert metadata such as severity or description.
Alert context: A set of fields which describe the cause of the alert details that vary based on the alert type. For example, a metric alert would have fields like the metric name and metric value in the alert context, whereas an activity log alert would have information about the event that generated the alert.

How does it help me?

Find more information about the exact fields, versioning, and other schema related details.

How is this going to impact me?

If you consume alerts from Azure in any manner whether it be email, webhooks, external tools, or others you might want to continue reading.

Email: A consistent and detailed email template allowing you to not only diagnose issues at a glance, but also jump to the process of working on the incident through deeplinks to the alert details on the portal and the affected resource.
SMS: A consistent SMS template
Webhook, Logic Apps, Azure Functions: A consistent JSON structure, allowing you to easily build integrations across different alert types.

The new schema will also enable a more rich consumption experience across both the Azure portal and the Azure mobile app in the immediate future. You can learn more about the changes coming as part of this feature by visiting our documentation.

Why should I switch over from my existing integrations?

If you already have integrations with the existing schemas, the reason to switch over are many:

Consistent alert structure means that you could potentially have fewer integrations, making the process of managing and maintaining these connectors a much simpler task.
Payload enrichments like rich diagnostic information, ability to customize, and more would surface up only in the new schema.

How do I get this new schema?

To avoid breaking your existing integrations, the common alert schema is something you can opt-in to and opt-out of as well.

To opt-in or out from the Azure portal:

Open any existing or a new action in an action group.
Select Yes for the toggle to enable the common alert schema as shown.

If you wish to opt-in at scale, you can also use the action groups API to automate this process. Learn more about how to write integrations for the common alert schema and the alert context schemas for the different alert types.

As always, we would love to hear your feedback. Please continue to share your thoughts at azurealertsfeedback@microsoft.com
Quelle: Azure

2. April 2019

da Agency

Monitoring on Azure HDInsight Part 2: Cluster health and availability

This is the second blog post in a four-part series on Monitoring on Azure HDInsight. "Monitoring on Azure HDInsight Part 1: An Overview" discusses the three main monitoring categories: cluster health and availability, resource utilization and performance, and job status and logs. This blog covers the first of those topics, cluster health and availability, in more depth.

As a high-availability service, Azure HDInsight ensures that you can spend time focused on your workloads, not worrying about the availability of your cluster. To accomplish this, HDInsight clusters are equipped with two head nodes, two gateway nodes, and three ZooKeeper nodes, making sure there is no single point of failure for your cluster. Nevertheless, Azure HDInsight offers multiple ways to comprehensively monitor the status of your clusters’ nodes and the components that run on them. HDInsight clusters include both Apache Ambari, which provides health information at a glance and predefined alerts, as well as Azure Monitor logs integration, which allows the querying of metrics and logs as well as configurable alerts.

Apache Ambari

Apache Ambari, included on all HDInsight clusters, simplifies cluster management and monitoring cluster via an easy-to-use web UI and REST API. Today, Ambari is the best way to monitor the health and availability of a single HDInsight cluster in depth.

Dashboard

The Ambari dashboard contains widgets that show a handful of metrics to give you a quick overview of your HDInsight cluster’s health. These widgets show metrics such as the number of live DataNodes (worker nodes), JournalNodes (ZooKeeper nodes), NameNode (head nodes) uptime, as well as metrics specific to certain cluster types such as YARN ResourceManager uptime for Spark and Hadoop clusters.

The Ambari Dashboard, included on all Azure HDInsight clusters.

Hosts – View individual node status

The hosts tab allows you to drill down further and view status information for individual nodes in the cluster. This offers a view showing whether there are any active alerts for the current node as well as the status/availability of each individual component running on the node.

The Ambari Hosts view shows detailed status information for individual nodes in your cluster.

Ambari alerts

Ambari also provides several configurable alerts out of the box that can provide notification of specific events. The number of currently active alerts is shown in the upper-left corner of Ambari in a red badge containing the number of alerts.

Ambari offers many predefined alerts related to availability, including:

Alert Name

Description

DataNode Health Summary

This service-level alert is triggered if there are unhealthy DataNodes.

NameNode High Availability Health

This service-level alert is triggered if either the Active NameNode or Standby NameNode are not running.

Percent JournalNodes Available

This alert is triggered if the number of down JournalNodes in the cluster is greater than the configured critical threshold. It aggregates the results of JournalNode process checks.

Percent DataNodes Available

This alert is triggered if the number of down DataNodes in the cluster is greater than the configured critical threshold. It aggregates the results of DataNode process checks.

A full list of Ambari alerts that help monitor the availability of a cluster can be found in our documentation, “Availability and reliability of Apache Hadoop cluster in HDInsight.”

The detailed view for each alert shows a description of the alert, the specific criteria or thresholds that will trigger a warning or critical alert, and the check interval for the criteria. The thresholds and check interval can be configured for individual alerts.

The Ambari detailed alert view shows the description of the alert and the check interval and threshold for the alert to fire.

Email Notifications

Ambari also offers support for configuring email notifications. Ambari email notifications can be a good way to monitor alerts when managing many HDInsight clusters.

Configuring Ambari email notifications can be a useful way to be notified of alerts for your clusters.

Azure Monitor logs integration

Azure Monitor logs enables data generated by multiple resources such as HDInsight clusters, to be collected and aggregated in one place to achieve a unified monitoring experience.

As a prerequisite, you will need a Log Analytics Workspace to store the collected data. If you have not already created one, you can follow the instructions for creating a Log Analytics Workspace.

You can then easily configure an HDInsight cluster to send many workload-specific metrics to Log Analytics, such as YARN ResourceManager information for Spark/Hadoop clusters, broker topics, and controller metrics for Kafka clusters. You can even configure multiple HDInsight clusters to send metrics to the same Log Analytics Workspace so you can monitor all of your clusters in a single place. See how to enable Azure Monitor logs integration on your HDInsight cluster by visiting our documentation on using Azure Monitor logs to monitor HDInsight clusters.

Query metrics tables in the logs blade

Once Log Analytics Integration is enabled, which may take a few minutes, you can start querying the logs/metrics tables.

The Logs blade in a Log Analytics workspace lets you query collected metrics and logs across many clusters.

The computer availability tab in the logs blade of your Log Analytics Workspace lists a number of sample queries related to availability, such as:

Query Name

Description

Computers availability today

Chart the number of computers sending logs, each hour.

List heartbeats

List all computer heartbeats from the last hour.

Last heartbeat of each computer

Show the last heartbeat sent by each computer.

Unavailable computers

List all known computers that didn't send a heartbeat in the last 5 hours.

Availability rate

Calculate the availability rate of each connected computer.

Azure Monitor alerts

You can also set up Azure Monitor alerts that will trigger when the value of a metric or the results of a query meet certain conditions.

You can condition on a query returning a record with a value that is greater than or less than some thresholds, or even on the number of results returned by a query. For example, you could create an alert to send an email when one or more nodes haven’t sent a heartbeat in one hour (i.e. is presumed to be unavailable). You can create multiple conditions that need to be met in order for an alert to fire.

There are several types of actions you can choose to trigger when your alert fires, such as an email, SMS, push, voice, an Azure Function, a LogicApp, a Webhook, an ITSM, or an Automation Runbook. You can set multiple actions for a single alert. Find more information about these different types of actions by visiting our documentation, “Create and manage action groups in the Azure portal.”

Finally, you can specify a severity for the alert in addition to the name. The ability to specify severity is a powerful tool that can be used when creating multiple alerts. For example, you could create one alert to raise a Warning (Sev 1) alert if a single head node becomes unavailable and another alert that raises a Critical (Sev 0) alert in the unlikely event that both head nodes go down. Alerts can be grouped by severity when viewed later.

Azure Monitor alerts are an extremely customizable way to receive alerts for specific events.

Next steps

While HDInsight’s redundant architecture, designed for high availability, means that a single failure will never impact the functionality of your cluster, HDInsight makes sure that you are always informed about potential availability issues so they can be mitigated early on. Between Apache Ambari with Azure Monitor logs integration, and Apache Ambari with Azure Log Analytics integration, Azure HDInight will offer comprehensive solutions for both monitoring a cluster in depth or monitoring many clusters at a glance. You can learn more and see concrete examples in our documentation, “How To Monitor Cluster Availability With Ambari and Azure Monitor Logs.”

Try HDInsight now

We hope you will take full advantage of monitoring on HDInsight and we are excited to see what you will build with Azure HDInsight. Read this developer guide and follow the quick start guide to learn more about implementing these pipelines and architectures on Azure HDInsight. Stay up-to-date on the latest Azure HDInsight news and features by following us on Twitter #AzureHDInsight and @AzureHDInsight. For questions and feedback, reach out to AskHDInsight@microsoft.com.

About HDInsight

Azure HDInsight is an easy, cost-effective, enterprise-grade service for open source analytics that enables customers to easily run popular open source frameworks including Apache Hadoop, Spark, Kafka, and others. The service is available in 36 public regions and Azure Government and National Clouds. Azure HDInsight powers mission-critical applications in a wide variety of sectors and enables a wide range of use cases including ETL, streaming, and interactive querying.
Quelle: Azure

2. April 2019

da Agency

Open standards vs. open source: A basic explanation

What are open standards, exactly? You’ve probably heard the term thrown around, but why does it matter to your business? How does it relate to open source? What’s the difference?
Take a common example. Have you ever noticed that Wi-Fi seems to work the same with any router, phone or computer? We tend to take these types of standards for granted, but they bring huge benefits to our daily lives.
Imagine if there were no standards like Wi-Fi. Every business might have its own form of wireless technology. If your favorite coffee shop had a router made by Company X, and you owned a computer made by Company Y, you might have to find another coffee shop to check your email.
Even if each business had a functioning form of wireless internet, a lack of standards would make interoperability nearly impossible. Customers of every company would suffer.
Have you ever wondered how competing businesses all across the world somehow converge on one format for these things?
The answer is often open standards.
What are open standards?
An open standard is a standard that is freely available for adoption, implementation and updates. A few famous examples of open standards are XML, SQL and HTML.
Businesses within an industry share open standards because this allows them to bring huge value to both themselves and to customers. Standards are often jointly managed by a foundation of stakeholders. There are typically rules about what kind of adjustments or updates users can make, to ensure that the standard maintains interoperability and quality.
What is open source?
What is open source, then? The term may sound similar to open standards; but, in reality, it is fundamentally different.
At its core, open source code is created to be freely available, and most licenses allow for the redistribution and modification of the code by anyone, anywhere, with attribution. In many cases the license further dictates that any updates from contributors will also become free and open to the community. This allows a decentralized community of developers to collaborate on a project and jointly benefit from the resulting software.
How open standards and open source help prevent vendor lock-in
Both open source and open standards can help protect clients from vendor lock-in, but they do it in different ways.
Let’s start with an example of an open standard. A business might buy a PDF reader and editor from a vendor. Over time, the team could create a huge number of PDF documents. Maybe these documents become a valuable asset for the company. Since the PDF format is an open standard, the business would have no problem switching from one PDF software to another. There is no concern that it would be unable to access its documents. Even if the PDF reader software isn’t open source, the PDF format is an open standard. Everyone uses this format.
Now, let’s instead take a look at the benefits of open source. Imagine that a business had spent millions of dollars writing internal software code for a proprietary operating system. That business would no longer have the option of changing vendors. It would be stuck with that operating system, unless it wanted to make a significant investment re-writing that code to run on a different system.
Open source software could have prevented that issue. Because open source software does not belong to any particular business, clients are not locked-in to any particular provider.
In both of these examples, the client would be able to avoid vendor lock-in. In one case this is because a piece of closed software followed a common open standard. In the other case, it is because the software itself belonged to an open source community.
While these are fundamentally different things, both help foster innovation while also providing more options to customers. Learn more about the power of open technology from IBM.

More articles for beginners

What is Kubernetes?
Kubernetes vs Docker: Why not both?
What is multicloud?

The post Open standards vs. open source: A basic explanation appeared first on Cloud computing news.
Quelle: Thoughts on Cloud