Drive higher utilization of Azure HDInsight clusters with Autoscale

We are excited to share the preview release of the Autoscale feature for Azure HDInsight. This feature enables enterprises to become more productive and cost-efficient by automatically scaling clusters up or down based on the load or a customized schedule. 

Let’s consider the scenario of a U.S. based health provider who is using Azure HDInsight to build a unified big data platform at corporate level to process various data for trend prediction or usage pattern analysis. To achieve their business goals, they operate multiple HDInsight clusters in production for real-time data ingestion, batch and interactive analysis.

Some clusters are customized to exact requirements, such as ISV/line of business applications and access control policies, which are subject to rigorous SLA requirements. Sizing such clusters is a hard problem by itself and operating them 24/7 at peak capacity is expensive. So once the clusters are created, IT admins either need to manually monitor the dynamic capacity requirements, scale the clusters up and down, or develop custom tools to do the same. These challenges prevent IT admins from being as productive as possible when building and operating cost-efficient big data analytics workloads.

With the new cluster Autoscaling feature, IT admins can have the Azure HDInsight service automatically monitor and scale the cluster up or down between a admin specified minimum and maximum number of nodes based on either actual load on the cluster or a customized schedule. IT admins can flexibly adjust the cluster size range or the schedule as the unique requirements of their workloads change. The Autoscale feature releases IT admins from having to build complex monitoring tools or worrying about wasted resources and high costs.

Benefits

Automatically make scaling decisions

Once Autoscale is enabled, you can rest assured that the service will take care of your cluster size.

In the load based mode: The cluster size will be scaled up exactly to how much more resources is needed by your applications, but never goes beyond the maximum number you set. Similarly, the cluster size will be scaled down to the minimum to meet your current resource requirements, but never goes below the minimum number of worker nodes you set.
In the schedule based mode: Cluster size will be scaled up and down based on the predefined schedule.  

All the above benefits release IT admins from worrying about wasted resources and allow enterprise to be cost effective and productive.

Pay for only what you need

Autoscale helps you achieve the balance between performance and cost efficiency. Scaling up the cluster lets you derive the business insight you need on time while scaling down the cluster removes the excess resources. Ultimately, Autoscale leads to higher utilization enabling you to pay for only what you need.

Customize to your own scenario

HDInsight Autoscale allows you to customize the scaling strategy based on your own scenario. In the load based mode, you can define the maximum and minimum based on your cost requirements. In the schedule based mode, you can define a schedule for each weekday to meet your own business objectives.

Monitor scaling history easily

The Autoscale feature gives you full visibility in to how the cluster has been scaled up or down. This enables you to further optimize the Autoscale configuration for higher utilization and workload performance.

Using the Azure portal, you can zoom in and out to check the cluster size over the past 90 days.

All the scaling events are also available in Azure Log Analytics. You can run queries to get all the details including when the scaling operation took place, how much resources were needed and how many worker nodes it scaled to. 

Get started

Read the HDInsight Autoscale documentation.
Learn the best practices for Autoscale and tune the settings to become more cost efficient.
Read this developer guide and follow the quick start guide to learn more about implementing open source analytics pipelines on Azure HDInsight.
Stay up-to-date on the latest Azure HDInsight news and features coming up in the near future by following us on Twitter #HDInsight and @AzureHDInsight.
For questions and feedback, please reach out to AskHDInsight@microsoft.com.

Quelle: Azure

Delivering end-to-end data analytics and data management solutions with Informatica

As more enterprises transition from on-premises data centers to the cloud, they increasingly need hybrid and multi-cloud solutions that can help them get the most from existing investments and take advantage of familiar, easy-to-use tooling.  Today, we’re extending our strategic partnership with Informatica, a leader in enterprise cloud data management, to meet these hybrid and multi-cloud needs. This includes the availability of Informatica Intelligent Cloud Services (IICS) and Master Data Management (MDM) on Google Cloud Platform (GCP), offering advanced data integration, data governance, data quality, and broader data management solutions for a seamless end-to-end data lifecycle management experience.In our conversations with enterprise customers across every industry, we frequently hear that data management and analytics are top of mind. Through our expanded collaboration with Informatica, we’re bringing these enterprises solutions that address their challenges in three key ways: data warehouse modernization for smart analytics and real-time insights, data management for marketing analytics, and data governance.Data warehouse modernizationWith the availability of IICS on Google Cloud Platform, customers will be able to easily and securely move data from their hybrid and multi-cloud applications and systems into GCP, and seamlessly and scalably analyze the data with smart analytics solutions like BigQuery, Cloud Dataproc and cloud AI capabilities.Existing Google Cloud customers will find that these new Informatica product integrations make GCP’s data management and data analytics solutions even easier to use. Our partnership will help accelerate data warehouse modernization by making data and schema migration, including ETL pipelines, seamless.In addition, with the availability of IICS on GCP, customers will be able to take advantage of our leading AI and ML capabilities. This means they can move data from multiple hybrid and multi-cloud systems into BigQuery, and use BigQuery ML or AutoML Tables to build machine learning models on their datasets. They can even fast-track data preparation with out-of-the-box data quality solutions for training AI models using cloud machine learning APIs or Cloud AutoML.Master data management for marketing analyticsEnterprise CMOs have told us they want to make decisions backed by data, but struggle with data fragmentation across systems such as CRM, POS, and billing. By bringing all their data together inside BigQuery and then applying Informatica’s Master Data Management solution, they get a single source of truth for customer data, and they can apply analytics that generate meaningful insights and improve the customer experience.For instance, by taking advantage of IICS on GCP, businesses can now build marketing-specific data lakes filled from sources such as Google Adwords, YouTube, DoubleClick, and more than 100 SaaS applications. This can help enterprises create a 360-degree view of their customers to enhance customer experiences, predict business outcomes, and improve campaign performance. Informatica MDM will also be available as a managed service within IICS on Google Cloud, offering better data governance and master data management, giving customers the ability to govern and understand all of their data across on-prem, public cloud, and GCP-native stores like Cloud Storage, BigQuery, and Cloud Spanner.Data GovernanceIICS Data Quality and Governance Cloud will make it easy for customers to explore, govern and manage data quality across a variety of on-premises systems and Google Cloud data stores such as Cloud Storage, BigQuery, and Cloud Spanner using a single pane of glass.  For Informatica customers moving to the cloud, GCP offers a secure, scalable, and reliable infrastructure to help build and operate mission-critical data analytics solutions. They can easily port their data pipelines and move data into GCP to realize the benefits of Google Cloud’s serverless, integrated, and intelligent data analytics services. Beyond analysis, Informatica customers will be able to apply Google Cloud’s industry-leading AI and machine learning capabilities to their data for predictive analytics so they can make their applications and business processes even more intelligent.“Informatica is committed to providing our customers with the broadest ecosystem support across the industry, and our new integration with Google Cloud creates an enhanced strategic alignment with a rapidly growing enterprise-ready cloud platform,” said Anil Chakravarthy, chief executive officer, Informatica. “Today, two major industry leaders are coming together to offer data integration innovations that power our customers’ digital transformations and expands our strategic partnership with Google Cloud.”Equinix, the world’s largest global interconnection platform, is already using Informatica and Google Cloud to help them connect digital businesses directly to their customers, clouds, employees and partners inside their more than 200 data centers. “One of our strategic goals is to deliver rich, on-demand business insights through big data and advanced analytics capabilities to help scale the organization and deliver a superior experience for our customers,” says Milind Wagle, CIO, Equinix. “As part of modernizing our technology in support of this goal, we selected Informatica and Google Cloud as strategic platforms in our data and analytics architecture stack. The strategic alignment between the Informatica and Google Cloud platforms, leveraging Equinix ECX Fabric, is helping us fast-track our enterprise digital transformation.”Informatica Intelligent Cloud Services (IICS) and Master Data Management (MDM) on Google Cloud Platform (GCP) will be available to customers through an early access program later in 2019.To learn more, visit informatica.com/gcp.
Quelle: Google Cloud Platform

Icann: Amazon darf .amazon-Domain nutzen

Wem gehört .amazon? Nachdem zahlreiche Amazonas-Staaten und der gleichnamige Onlinehändler jahrelang über die Nutzungsrechte der Top-Level-Domain gestritten haben, gibt es jetzt eine Entscheidung – gefallen dürfte sie den südamerikanischen Staaten nicht. (Amazon, DNS)
Quelle: Golem