Announcing new Stackdriver Logging features and expanded free logs limits

By Mary Koes and Deepak Tiwari, Product Managers 

When we announced the general availability of Google Stackdriver, our integrated monitoring, logging and diagnostics suite for applications running on cloud, we heard lots of enthusiasm from our user community as well as some insightful feedback:

Analysis – Logs based metrics are great, but you’d like to be able to extract labels and values from logs, too. 
Exports – Love being able to easily export logs, but it’s hard to manage them across dozens or hundreds of projects. 
Controls – Aggregating all logs in a single location and exporting them various places is fantastic, but you want control over which logs go into Stackdriver Logging. 
Pricing – You want room to grow with Stackdriver without worrying too much about the cost of logging all that data. 

We heard you, which is why today we’re announcing a variety of new updates to Stackdriver, as well as updated pricing to give you the flexibility to scale and grow.

Here’s a little more on what’s new.

Easier analysis with logs-based metrics 
Stackdriver was created with the belief that bringing together multiple signals from logs, metrics, traces and errors can provide greater insight than any single signal. Logs-based metrics are a great example. That’s why the new and improved logs-based metrics are:

Faster – We’ve decreased the time from when a log entry arrives until it’s reflected in a logs-based metric from five minutes to under a minute. 
Easier to manage – Now you can extract user-defined labels from text in the logs. Instead of creating a new logs based metric for each possible value, you can use a field in the log entry as a label. 
More powerful – Extract values from logs and turn them into distribution metrics. This allows you to efficiently represent many data points at each point in time. Stackdriver Monitoring can then visualize these metrics as a heat map or by percentile. 

The example above shows a heat map produced from a distribution metric extracted from a text field in log entries.

Tony Li, Site Reliability Engineer at the New York Times, explains how they use the new user defined labels applied to proxies help them improve reliability and performance from logs.

“With LBMs [Logs based metrics], we can monitor errors that occur across multiple proxies and visualize the frequency based on when they occur to determine regressions or misconfigurations.”
The faster pipeline applies to all logs-based metrics, including the already generally available count-based metrics. Distribution metrics and user labels are now available in beta.

Manage logs across your organization with aggregated exports 

Stackdriver Logging gives you the ability to export logs to GCS, PubSub or BigQuery using log sinks. We heard your feedback that managing exports across hundreds or thousands of projects in an organization can sometimes be tedious and error prone. For example, if a security administrator in an organization wanted to export all audit logs to a central project in BigQuery, she would have to set up a log sink at every project and validate that the sink was in place for each new project.

With aggregated exports, administrators of an organization or folder can set up sinks once to be inherited by all the child projects and subfolders. This makes it possible for the security administrator to export all audit logs in her organization to BigQuery with a single command:

gcloud beta logging sinks create my-bq-sink
bigquery.googleapis.com/projects/my-project/datasets/my_dataset
–log-filter=’logName= “logs/cloudaudit.googleapis.com%2Factivity”‘
–organization=1234 –include-children

Aggregated exports help ensure that logs in future projects will be exported correctly. Since the sink is set at the organization or folder level, it also prevents an individual project owner from turning off a sink.

Control your Stackdriver Logging pipeline with exclusion filters 
All logs sent to the Logging API, whether sent by you or by Google Cloud services, have always gone into Stackdriver Logging where they’re searchable in the Logs Viewer. But we heard feedback that users wanted more control over which logs get ingested into Stackdriver Logging, and we listened. To address this, exclusion filters are now in beta. Exclusion filters allow you to reduce costs, improve the signal to noise ratio by reducing chatty logs and manage compliance by blocking logs from a source or matching a pattern from being available in Stackdriver Logging.

The new Resource Usage page provides visibility into which resources are sending logs and which are excluded from Stackdriver Logging.

This makes it easy to exclude some or all future logs from a specific resource. In the example above, we’re excluding 99% of successful load balancer logs.

We know the choice and freedom to choose any solution is important, which is why all GCP logs are available to you irrespective of the logging exclusion filters, to export to BigQuery, Google Cloud Storage or any third party tool via PubSub. Furthermore, Stackdriver will not charge for this export, although BigQuery, GCS and PubSub charges will apply.

Starting Dec 1, Stackdriver Logging offers 50GB of logs per project per month for free 

You told us you wanted room to grow with Stackdriver without worrying about the cost of logging all that data, which is why on December 1 we’re increasing the free logs allocation to an industry-leading 50GB per project per month. This increase aims to bring the power of Stackdriver Logging search, storage, analysis and alerting capabilities to all our customers.

Want to keep logs beyond the free 50GB/month allocation? You can sign up for the Stackdriver Premium Tier or the logs overage in the Basic Tier. After Dec 1, any additional logs will be charged at a flat rate of $0.50/GB.

Audit logs, still free and now available for 13 months 
We’re also exempting admin activity audit logs from the limits and overage. They’ll be available in Stackdriver in full without any charges. You’ll now be able to keep them for 13 months instead of 30 days.

Continuing the conversation 

We hope this brings the power of Stackdriver Logging search, storage, analysis and alerting capabilities to all our customers. We have many more exciting new features planned, including a time range selector coming in September to make it easier to get visibility into the timespan of search results. We’re always looking for more feedback and suggestions on how to improve Stackdriver Logging. Please keep sending us your requests and feedback.

Interested in more information on these new features?

Discover new ways to extract value from your logs with the improved logs-based metrics.
Learn how to control log sinks across your organization using aggregated exports. 
Read more on controlling which logs you bring into your Stackdriver Logging account using exclusion filters. 
Find the details on our pricing model on the Stackdriver pricing page. 
Learn our best practices on managing logs and controlling costs. 
View your current logs usage on the Logging Resource Usage page. 
Estimate the charges at your current usage rate with the Stackdriver Billing Calculator.

Quelle: Google Cloud Platform

How Netcool Operations Insight delivers cognitive automation

In my last blog, I talked about some of the growing challenges facing operations teams looking to maximize the availability of services and applications while minimizing the cost of doing so.
Enter analytics and machine learning. Why would operations teams care about them?
Picture an incident first responder in an operations team – let’s call her Annette. She needs the most relevant events presented to her in meaningful context with as little noise as possible. She needs to be able to see the woods for the trees, so that she can resolve a problem indicated by an event as quickly as possible.
Now imagine Brock, a site reliability engineer with deep knowledge of an app, service or supporting technology. Brock may not have the time to author event reduction and correlation rules for Annette’s benefit. But the difference between him getting, say, a single incident SMS per day and a handful of notifications could mean a world of difference. It could mean he spends a day either grepping logfiles and examining irrelevant events or successfully preparing for his team’s next app rollout. Brock will want to know whether there are chronic problems in the managed environments that he doesn’t see because he might not have time to sift through the management data.
Neither Brock nor Annette are data scientists. They don’t need to care about machine learning. But they care about what such technologies can do for them. Netcool Operations Insight (NOI), delivered by the Netcool/OMNIbus, introduces machine learning technology to give Brock and Annette a data scientist in a box that aims to help solve operations management problems.
How NOI delivers insights
With NOI’s related event analytics, Brock can enable correlations that will group statistically related events into a single incident. For Annette, this means she can forego dealing with dozens of apparently unrelated events. Instead, she would see just a handful of true incidents, each containing relevant context for identifying the underlying cause. It can also mean that Brock gets a single SMS for, rather than many alerts throughout the day—or in the middle of the night.
NOI’s seasonality analysis shows Brock events that occur in a predictable pattern in time, helping him identify and remedy persistent problems in the managed infrastructure. When they go unidentified, chronic problems can lead to significant hidden costs from replugging the same hole time and time again.
NOI’s event and log search analysis provides Annette and Brock with critical contextual data from informational events and log files. For example, NOI can help identify an out-of-band configuration change as the root cause of a cascade of symptomatic events, all from the Netcool event console. Brock can look for hotspots for his application or service—say, an unreliable software module or a consistently faulty hardware model.
So why should Annette and Brock trust the output of these analytics? Neither are data scientists. In developing these capabilities, IBM development teams have ensured that, when an analytically derived insight is produced, the software can produce supporting evidence that the insight is valid. How do I know these events are correctly grouped? How do I know this event is chronic? Because the software can show me the event instances in history that support this conclusion. Annette and Brock don’t need a PhD in artificial intelligence to develop trust in the system and see that the generated insights are valid.
Adding cognitive and machine learning capabilities to Netcool helps the IT operations organization effectively deal with dramatically increasing numbers of events from highly complex hybrid environments. Along with integrated offerings such as Predictive Insights, Agile Service Manager and Runbook Automation, NOI helps companies to move into the area of cognitive automation. This means transforming IT operations from a people-led and technology-assisted approach to one that is technology-led and people-assisted.
To learn more, register for our webinar on the value predictive insight brings to IT operations. Check out the earlier posts in our IBM Operations Analytics blog series. And stay tuned for additional key learnings from our colleagues in coming weeks.
The post How Netcool Operations Insight delivers cognitive automation appeared first on Cloud computing news.
Quelle: Thoughts on Cloud