Kategorie: Microsoft Azure

Public Preview: Azure Data Lake Tools for Visual Studio Code (VSCode)

We are pleased to announce the Public Preview of the Azure Data Lake (ADL) Tools for VSCode. The tools provide users with the best in class light weight, keyboard focused authoring experience for U-SQL as an alternative solution to the Data Lake Tools for Visual Studio.

By extending VSCode, leveraging the Azure Data Lake Java SDK for U-SQL job submission, and integrating with the Azure portal for job monitoring, the tools provide a cross-platform IDE. Users can run it smoothly in Window, Linux and Mac.

The ADL Tools for VSCode fully embrace the U-SQL language. User can enjoy the power of IntelliSense, Syntax Highlighting, and the Error Marker. It covers the core user scenarios of U-SQL scripting and U-SQL extensibility through custom code. The ADL Tools seamlessly integrate with ADL, which allows user to compile and submit jobs to ADLA.

What features are supported in ADL Tools for VSCode?

U-SQL Language Authoring

The ADL Tools for VSCode allows users to fully utilize the power of U-SQL: a language you’ll be comfortable with from Day One. It empowers users to enjoy the advantages of U-SQL: process any type of data, integrate with your custom code, as well as efficiently scale to any size of data.

U-SQL Scripting

U-SQL combines the declarative advantage of T-SQL and extensibility of C#. Users can create a VSCode U-SQL job in a file format with usql file extension, and leverage the full feature set of U-SQL language and its built-in C# expressions for U-SQL job authoring and submission.

U-SQL Language Extensibility

ADL Tools for VSCode enables user to fully leverage U-SQL extensibility (e.g. UDOs, UDFs, UDAGG) through custom code. User can do so either through registering assembly or using Code Behind feature.

Manage Assembly

The Register Assembly command allows users to register custom code assemblies into the ADLA Metadata Service so that users can refer to the UDF, UDO and UDAGG in their U-SQL scripts. This functionality allows users to package the custom code and share the functionality with others.

Code Behind

The easiest way to make use of custom code is to use the code-behind capabilities. Users can fill in the custom code for the script (e.g., Script.usql) into its code-behind file (e.g., Script.usql.cs). The advantage of code-behind is that the tooling takes care of the following steps for you when you submit your script:

It creates a .CS C# codebehind file and links it with the original U-SQL file.
It compiles the codebehind into an assembly under the codebehind folder.
It registers and unregisters the codebehind assembly as part of the script through an automatic prologue and epilogue.

Azure Data Lake Integration

The ADL Tools for VSCode integrate seamlessly with Azure Data Lake Analytics (ADLA). Azure Data Lake includes all the capabilities required to make it easy for developers, data scientists, and analysts to store data of any size, shape and speed, and do all types of processing and analytics across platforms and languages. U-SQL on ADLA offers Job as a Service with the Microsoft invented U-SQL language. Customers do not have to manage deployment of clusters but can simply submit their jobs to ADLA, an analytics platform managed by Microsoft.

ADLA – Metadata Navigation

Upon signing into Azure, users can view his / her ADLA Metadata entities through a list of customized VSCode command items. The workflow and steps to navigate through ADLA Metadata based on its hierarchy are managed through a set of command items.

ADLA – Job Submission

The ADL Tools for VSCode allow users to submit the U-SQL job into ADLA either through the Submit Job command in the command palette or the right click menu in U-SQL file.

Users can either output the job to ADLS or Azure Blob storage based on their needs. The U-SQL compilation and execution is performed remotely in ADLA.

How do I get started?

You need to first install Visual Studio Code and download the prerequisite files including JRE 1.8.x, Mono 4.2.x (for Linux and Mac), and .Net Core (for Linux and Mac). Then get the latest ADL Tools by going to the VSCode Extension repository or VSCode Marketplace and searching Azure Data Lake Tool for VSCode. Please visit the following link for more information.

For more information, check out the following links:

User Manual: Azure Data Lake Tools for VSCode
Tutorial: get started with Azure Data Lake Analytics

Learn more about today’s announcements on the Azure Data Lake Blog.

Discover more Azure service updates.

If you have questions, feedback, comments, or bug reports, please use the comments below or send a note to hdivstool@microsoft.com.
Quelle: Azure

Azure Data Lake Analytics now generally available

Today, we are pleased to announce that Azure Data Lake Analytics is generally available. Since we announced the public preview, Azure Data Lake has become one of the fastest growing Azure service now with thousands of customers. With the GA announcement, we are revealing improvements we’ve made to the service including making it more productive for end users and security and availability improvements to make it ready for production deployments.

What is Azure Data Lake?

Today’s Big data solutions have been driving some organizations from “rear-view mirror” thinking to forward-looking and predictive analytics. However, there has been adoption challenges and the widespread usage of big data has not yet occurred. Azure Data Lake was introduced to drive big data adoption by making big data easy for developers, data scientists, and analysts to store data of any size, shape and speed, and do all types of processing and analytics across platforms and languages. It removes the complexities of ingesting and storing all your data while making it faster to get up and running with big data. Azure Data Lake includes three services:

Azure Data Lake Store, a no limits data lake that powers big data analytics
Azure Data Lake Analytics, a massively parallel on-demand job service
Azure HDInsight, a full managed Cloud Hadoop and Spark offering

What is Azure Data Lake Analytics?

Azure Data Lake Analytics service is a new distributed analytics job service that dynamically scales so you can focus on your business goals, not on distributed infrastructure. Instead of deploying, configuring and tuning hardware, you write queries to transform your data and extract valuable insights. The analytics service can handle jobs of any scale instantly by simply setting the dial for how much power you need. You only pay for your job when it is running making it cost-effective.

Azure Data Lake Analytics also provides a unified big data developer platform that integrates language, runtime, tooling, development environments, resource management, extensibility security that makes developers and ISVs far more productive. It supports the entire end-to-end big data development lifecycle from authoring, to debugging, monitoring, and optimization.

Start in seconds, Scale instantly, Pay per job:

Our on-demand service will have you processing Big Data jobs within 30 seconds. There is no infrastructure to worry about because there are no servers, VMs, or clusters to wait for, manage or tune. You can instantly scale the analytic units (processing power) from one to thousands for each job with literally a single slider. You only pay for the processing used per job. This model dramatically simplifies the lives of developers who want to start working with big data.

Tutorial: get started with Azure Data Lake Analytics using Azure portal
Demo: Getting Started with Azure Data Lake

Develop massively parallel programs with simplicity:

U-SQL is a simple, expressive, and extensible language that allows you to write code once and automatically have it be parallelized for the scale you need.  U-SQL blends the declarative nature of SQL with the expressive power of C#. In other declarative SQL-based languages used for big data, their extensibility model is “bolted-on” and much harder to use. U-SQL allows developers to easily define and utilize user-defined types and user-defined functions defined in any .NET language.

Big data developers need to accommodate any type of data: images, audio, video, documents. However, to handle those kinds of data, there are many existing libraries that are not all readily accessible to big data languages. U-SQL can seamlessly reuse any .NET library either one that is locally developed or published in repositories such as NuGet to handle any type of data. Developers can also use code written in R or in Python in their U-SQL scripts. After the code is written, you can deploy it as a massively parallel program letting you easily scale out diverse workload categories such as ETL, machine learning, cognitive science, machine translation, imaging processing, and sentiment analysis by using U-SQL and leveraging existing libraries.

Tutorial: Get started with Azure Data Lake Analytics U-SQL language
Develop U-SQL User defined operators for Azure Data Lake Analytics jobs
U-SQL Language Reference
Video: Introducing U-SQL – A new language for Massive Data Processing
Video: U-SQL Query Execution
Video: U-SQL Extensibility

Debug and Optimize your Big Data programs with ease:

With the tools that exist today, developers face serious challenges as their data workloads increase. Understanding bottlenecks in performance and scale is challenging and requires experts in distributed computing and infrastructure. For example, developers must carefully account for the time & cost of data movement across a cluster and rewrite their queries or repartition their data to improve performance. Optimizing code and debugging failures in cloud distributed programs are now as easy as debugging a program in your personal environment. Our execution environment actively analyzes your programs as they run and offers recommendations to improve performance and reduce cost. For example, if you requested 1000 AUs for your program and only 50 AUs were needed, the system would recommend that you only use 50 AUs resulting in a 20x cost savings.

Today, we are also announcing the availability of this big data productivity environment in Visual Studio Code allowing users to have this type of productivity in a free cross-platform code editor that is available on Windows, Mac OS X, and Linux.

Tutorial: develop U-SQL scripts using Data Lake Tools for Visual Studio
Video: Data Lake Developer Tools
Video: Getting Started with Debugging U-SQL

Virtualize your analytics:

The power to act on all your data with optimized data virtualization of your relational sources such as Azure SQL Database, and Azure SQL Data Warehouse. Queries are automatically optimized by moving processing close to the source data, without data movement, thereby maximizing performance and minimizing latency.

Video: U-SQL Federated Query

Enterprise-grade Security, Auditing and Support:

Extend your on-premises security and governance controls to the cloud for meeting your security and regulatory compliance needs. Capabilities such as single sign-on (SSO), multi-factor authentication and seamless management of millions of identities is built-in through Azure Active Directory. Role Based Access control, and the ability to audit all processing and management operations are on by default. We guarantee a 99.9% enterprise-grade SLA and 24/7 support for your big data solution.

Overview of Security in Azure Data Lake

How do I get started?

To get started, customers will need to have an Azure subscription or a free trial to Azure. With this in hand, you should be able to get an Azure Data Lake Analytics up and running in seconds by going through this getting started guide. Also, visit our free Microsoft Virtual Academy course on Data Lake.

Free course: Microsoft Virtual Academy on Azure Data Lake
Overview of Azure Data Lake Analytics
Get started using Microsoft Azure Portal
Get started using Azure PowerShell
Get started using .NET SDK
Develop U-SQL Scripts using Data Lake Tools for Visual Studio
Use Data Lake Analytics interactive tutorial
Analyze weblogs using Data Lake Analytics
Get started with U-SQL
U-SQL reference
.NET SDK reference

Quelle: Azure

Azure Data Lake Store now generally available

Today, we are pleased to announce  that Azure Data Lake Store is generally available.  Since we announced the public preview, Azure Data Lake has become one of the fastest growing Azure service now with thousands of customers. With the GA announcement, we are revealing improvements we’ve made to the service including making it more secure and highly available to make it ready for production deployments.

What is Azure Data Lake?

Today’s Big data solutions have been driving some organizations from “rear-view mirror” thinking to forward-looking and predictive analytics. However, there has been adoption challenges and the widespread usage of big data has not yet occurred. Azure Data Lake was introduced to drive big data adoption by making big data easy for developers, data scientists, and analysts to store data of any size, shape and speed, and do all types of processing and analytics across platforms and languages. It removes the complexities of ingesting and storing all your data while making it faster to get up and running with big data. Azure Data Lake includes three services:

Azure Data Lake Store, a no limits data lake that powers big data analytics
Azure Data Lake Analytics, a massively parallel on-demand job service
Azure HDInsight, a full managed Cloud Hadoop and Spark offering

What is Azure Data Lake Store?

The value of a data lake resides in the ability to develop solutions across data of all types – unstructured, semi-structured and structured. This begins with the Azure Data Lake Store, the first cloud Data Lake for enterprises that is secure, massively scalable and built to the open HDFS standard.  With no limits to the size of data and the ability to run massively parallel analytics, you can now unlock value from all your analytics data. For example, data can be ingested in real-time from sensors and devices for IoT solutions, or from online shopping websites into the store.

Petabyte size files and Trillions of objects:

Prior to the Azure Data Lake Store, storing large datasets in the cloud has been a major challenge.  Artificial limits placed by object stores make them unsuitable to store large files that can be hundreds of terabytes in size such as high-resolution video, genomic and seismic datasets, medical data, and data from a wide variety of industries.  Azure Data Lake Store has revolutionary technology for analyzing and storing massive data sets.  A single Azure Data Lake Store account can store trillions of files where a single file can be greater than a petabyte in size which is 200x larger than other cloud stores.

This makes Data Lake Store ideal for storing any type of data including massive datasets like high-resolution video, genomic and seismic datasets, medical data, and data from a wide variety of industries.

Scalable throughput  for massively parallel analytics:

Data Lake Store is built for running large analytic systems that require massive throughput to process and analyze petabytes of data. Without redesigning your application or repartitioning your data at higher scale, Data Lake Store scales throughput to support any size of analytic workload. It provides massive throughput to run analytic jobs with thousands of concurrent executors that read and write hundreds of terabytes of data efficiently. You only need only focus on the application logic, and we automatically optimize the store for any throughput level.

HDFS for the Cloud:

Microsoft Azure Data Lake Store supports any application that uses the open Apache Hadoop Distributed File System (HDFS) standard. By supporting HDFS, you can easily migrate your existing Hadoop and Spark data to the cloud without recreating your

Use with Hadoop clusters
Use with Data Lake Analytics
Use with Stream Analytics
Use with Data Catalog
Use with Power BI

Always encrypted, Role-based security & Auditing:

Data Lake Store protects your data assets and extends your on-premises security and governance controls to the cloud easily.  Data is always encrypted; in motion using SSL, and at rest using service or user managed HSM-backed keys in Azure Key Vault.  Capabilities such as single sign-on (SSO), multi-factor authentication and seamless management of millions of identities is built-in through Azure Active Directory. You can authorize users and groups with fine-grained POSIX-based ACLs for all data in the Store enabling role-based access controls. Finally, you can meet security and regulatory compliance needs by auditing every access or configuration change to the system. Finally, we guarantee a 99.9% enterprise-grade SLA and 24/7 support for your big data solution.

Security overview
Access control lists
Secure massive datasets
Active Directory authentication
Video: Overview of Security in Azure Data Lake
Video: Developing with OAuth in Azure Data Lake
Video: Authorization in Azure Data Lake

How do I get started?

To get started, customers will need to have an Azure subscription or a free trial to Azure. With this in hand, you should be able to get an Azure Data Lake Analytics up and running in seconds by going through this getting started guide.

Also, visit our free Microsoft Virtual Academy course on Data Lake.

Free course: Microsoft Virtual Academy on Azure Data Lake
Video: Introduction to Azure Data Lake Store
What is Data Lake Store
Create account and upload data
Self-guided learning
Copy to and from Azure Blob Storage
Start with the REST API
Start with the .NET SDK
Start with the Java SDK
Security overview
Active Directory authentication

Quelle: Azure

Application Insights: Three of the latest features in Visual Studio

Application Insights telemetry is a powerful tool for detecting, triaging, and diagnosing issues in your web services. The Developer Analytics Tools features in Visual Studio integrate Application Insights data into your editing and debugging workflows. Check out three of the latest features we’ve added.

Operation timelines in Application Insights Search

Web services are driven by requests. So, when you’re using telemetry to diagnose an issue in your web service, it’s helpful to see the telemetry in the context of the request that triggered the exception, dependency call, or custom event. These sequences of related events are called “operations” in Application Insights.

The new Track Operation tab on each event in the Application Insights Search tool shows other events that occurred during the same operation.

The Track Operation tab makes it easy to piece together what happened in your service before a problem occurred by listing events chronologically. Timelines and event duration data for each event can help you spot slow dependencies and improve the performance of your service. The slowest event in each operation is marked with a flame icon to make it easy to find.

Request telemetry in CodeLens

Request telemetry for each of your ASP.NET controller methods is now shown in CodeLens. From the CodeLens indicator, you can see the number of requests in the last 24 hours along with the percentage of failed requests. By clicking the CodeLens indicator, you can also see the average response time for the request, plus comparisons between the last 24 hours and the prior 24 hours for each request metric.

By placing request telemetry right in your editor, it’s easy to spot production reliability and performance issues while you’re working in your codebase. Seeing how often a method has been requested in the last 24 hours can also provide useful context while making changes: “Does this method that I’m about to edit see a lot of usage in production?”

Learn more about Application Insights and CodeLens in the Azure documentation.

CodeLens for debug session telemetry

CodeLens can now also show telemetry from local debug sessions, even if you haven’t connected your application to the Application Insights service in Azure. In ASP.NET projects with the Application Insights SDK, you’ll see CodeLens indicators for exception and request telemetry from the most recent debug session. When you stop debugging and edit your code, you’ll have request response times, exception data, and more at your fingertips.

Get started

The Developer Analytics Tools features are included with Visual Studio 2015 and Visual Studio “15.” They’re also available as an extension on the Visual Studio Gallery.

To connect your application to the Application Insights service and enable production telemetry:

Right-click your project in the Solution Explorer and choose Add Application Insights Telemetry…
Follow the directions in the Application Insights Configuration window.

If you prefer, you can just add the Application Insights SDK to your application for debug session telemetry without connecting to the Application Insights service:

Right-click your project in the Solution Explorer and choose Add Application Insights Telemetry…
Follow the directions in the Application Insights Configuration window
Look for a link to “Just add the SDK to try local-only mode.”

Let us know in the comments below how you use the Developer Analytics Tools, and how else you’d like to use Application Insights telemetry from within Visual Studio.
Quelle: Azure

The Intelligent Data Lake

Advanced Analytics and Cognitive Intelligence on Petabyte sized files and trillions of objects with Azure Data Lake

Today we are announcing the general availability of Azure Data Lake, ushering in a new era of productivity for your big data developers and scientists. Fundamentally different from today’s cluster-based solutions, the Azure Data Lake services enable you to securely store all your data centrally in a “no limits” data lake, and run on-demand analytics that instantly scales to your needs. Our state-of-the-art development environment and rich and extensible U-SQL language enable you to write, debug, and optimize massively parallel analytics programs in a fraction of the time of existing solutions.

Before Azure Data Lake

Traditional approaches for big data analytics constrain the productivity of your data developers and scientists due to time spent on infrastructure planning, and writing, debugging, & optimizing code with primitive tooling. They also lack rich built-in cognitive capabilities like keyphrase extraction, sentiment analysis, image tagging, OCR, face detection, and emotion analysis. The underlying storage systems also impose challenges with artificial limits on file and account sizes requiring you to build workarounds. Additionally, your developer’s valuable time is spent either optimizing the system or you end up overpaying for unused cluster capacity. The friction in these existing systems is so high, it effectively prevents companies from realizing the business transformation that Big Data promises.

With Azure Data Lake

With thousands of customers, Azure Data Lake has become one of the fastest growing Azure services. You can get started on this new era of big data productivity and scalability with the general availability of Azure Data Lake Analytics and Azure Data Lake Store.

Azure Data Lake Store – the first cloud Data Lake for enterprises that is secure, massively scalable and built to the open HDFS standard.  With no limits to the size of data and the ability to run massively parallel analytics, you can now unlock value from all your unstructured, semi-structured and structured data.
Azure Data Lake Analytics – the first cloud analytics job service where you can easily develop and run massively parallel data transformation and processing programs in U-SQL, R, Python and .NET over petabytes of data. It has rich built-in cognitive capabilities such as image tagging, emotion detection, face detection, deriving meaning from text, and sentiment analysis with the ability to extend to any type of analytics. With Azure Data Lake Analytics, there is no infrastructure to manage, and you can process data on demand, scale instantly, and only pay per job.
Azure HDInsight – the only fully managed cloud Hadoop offering that provides optimized open source analytic clusters for Spark, Hive, Map Reduce, HBase, Storm, Kafka and R-Server backed by a 99.9% SLA. Today, we are announcing the general availability of R Server for HDInsight to do advanced analytics and predictive modelling with R+Spark. Further, we are introducing the public preview of Kafka for HDInsight, now the first managed cluster solution in the cloud for real-time ingestion with Kafka.

Data Lake Store – A No Limits Data Lake that powers Big Data Analytics

Petabyte sized files and trillions of objects

With Azure Data Lake Store your organization can now securely capture and analyze all data in a central location with no artificial constraints to limit the growth of your business. It can manage trillions of files where a single file can be greater than a petabyte in size – this is 200x larger file size than other cloud object stores. Without the limits that constrain other cloud offerings, Data Lake Store is ideal for managing any type of data; including massive datasets like high-resolution video, genomic and seismic datasets, medical data, and data from a wide variety of industries. Data Lake Store is an enterprise data lake that can power your analytics today and in the future.

“DS-IQ provides Dynamic Shopper Intelligence by curating data from large amounts of non-relational sources like weather, health, traffic, and economic trends so that we can give our customers actionable insights to drive the most effective marketing and service communications. Azure Data Lake was perfect for us because it could scale elastically, on-demand, to petabytes of data within minutes. This scalability and performance has impressed us, giving us confidence that it can handle the amounts of data we need to process today and, in the future, enable us to provide even more valuable, dynamic, context-aware experiences for our clients.” -William Wu, Chief Technology Officer at DS-IQ

Scalability for massively parallel analytics

Data Lake Store provides massive throughput to run analytic jobs with thousands of concurrent executors that read and write hundreds of terabytes of data efficiently. You are no longer forced to redesign your application or repartition your data because Data Lake Store scales throughput to support any size of workload. Multiple services like Data Lake Analytics, HDInsight or HDFS compliant applications can efficiently analyze the same data simultaneously.

“Ecolab has an ambitious mission to find solutions to some of the world’s biggest challenges – clean water, safe food, abundant energy and healthy environments.  “Azure Data Lake has been deployed to our water division where we are collecting real-time data from IoT devices so we can help our customers understand how they can reduce, reuse, and recycle water and at the same address one of the world’s most pressing sustainability issues. We’ve been impressed with Azure Data Lake because it allows us to store any amount of data we require and also lets us use our existing skills to analyze the data. Today, we have both groups who use open source technologies such as Spark in HDInsight to do analytics and other groups that use U-SQL, leveraging the extensibility of C# with the simplicity of SQL.” -Kevin Doyle, VP of IT, Global Industrial Solutions at Ecolab

Data Lake Analytics – A On-Demand Analytics Job Service to power intelligent action

Start in seconds, scale instantly, pay per job

Our on-demand service will have your data developer and scientist processing big data jobs to power intelligent action in seconds. There is no infrastructure to worry about because there are no servers, VMs, or clusters to wait for, manage or tune. Instantly apply or adjust the analytic units (processing power) from one to hundreds or even thousands for each job. Only pay for the processing used per job, freeing valuable developer time from doing capacity planning and optimizations required in cluster-based systems that can take weeks to months.

“Azure Data Lake is instrumental because it helps Insightcentr ingest IoT-scale telemetry from PCs in real-time and gives detailed analytics to our customers without us spending millions of dollars building out big data clusters by scratch. We saw Azure Data Lake as the fastest and most scalable way we can get bring our customers these valuable insights to their business.” -Anthony Stevens, CEO Australian start-up Devicedesk

Develop massively parallel analytic programs with simplicity

U-SQL is an easy-to-use, highly expressive, and extensible language that allows you to write code once and automatically have it be parallelized for the scale you need. Instead of writing low-level code dealing with clusters, nodes, mappers, and reducers, etc., a developer writes a simple logical description of how data should be transformed for their business using both declarative and imperative techniques as desired. The U-SQL data processing system automatically parallelizes the code – enabling developers to control the amount of resources devoted to parallel computation with the simplicity of a slider. The U-SQL language is highly extensible and can reuse existing libraries written in a variety of languages like .NET languages, R, or Python. You can massively parallelize the code to process petabytes of data for diverse workload categories such as ETL, machine learning, feature engineering, image tagging, emotion detection, face detection, deriving meaning from text, and sentiment analysis.

“Azure Data Lake allows us to develop quickly on large sets of data with our current developer expertise. We have been able to leverage Azure Data Lake Analytics to capture and process large marketing audiences for our dynamic marketing platforms." -McPherson White, Director of Development at PureCars

Run Big Cognition at Petabyte Scale

Furthermore, we’ve incorporated the technology that sits behind the Cognitive Services API inside U-SQL directly. Now you can process any amount of unstructured data, e.g., text, images, and extract emotions, age, and all sorts of other cognitive features using Azure Data Lake and perform query by content. You can join emotions from image content with any other type of data you have and do incredibly powerful analytics and intelligence over it. This is what we call ‘Big Cognition’. It’s not just extracting one piece of cognitive information at a time, not just about understanding an emotion or whether there’s an object in an image, but rather it’s about joining all the extracted cognitive data with other types of data, so you can do some really powerful analytics with it. We have demonstrated this capability at Microsoft Ignite and PASS Summit, by showing a Big Cognition demo in which we used U-SQL inside Azure Data Lake Analytics to process a million images and understand what’s inside those images. You can watch this demo here and try it yourself using a sample project on GitHub.

Debug and optimize your Big Data programs with ease

With today’s tools, developers face serious challenges debugging distributed programs. Azure Data Lake makes debugging failures in cloud distributed programs as easy as debugging a program in your personal environment using the powerful tools within Visual Studio. Developers no longer need to inspect thousands of logs on each machine searching for failures. When a U-SQL job fails, logs are automatically located, parsed, and filtered to the exact components involved in the failure and available as a visualization. Developers can even debug the specific parts of the U-SQL job that failed to their own local workstation without wasting time and money resubmitting jobs to the cloud. Our service can detect and analyze common performance problems that big data developers encounter such as imbalanced data partitioning and offers suggestions to fix your programs using the intelligence we’ve gathered in the analysis of over a billion jobs in Microsoft’s data lake.

Developers do a lot of heavy lifting when optimizing big data systems and frequently overpay for unused cluster capacity. Developers must manually optimize their data transformations, requiring them to carefully investigate how their data is transformed step-by-step, often manually ordering steps to gain improvements. Understanding performance and scale bottlenecks is challenging and requires distributed computing and infrastructure experts. For example, to improve performance, developers must carefully account for the time & cost of data movement across a cluster and rewrite their queries or repartition their data. Data Lake’s execution environment actively analyzes your programs as they run and offers recommendations to improve performance and reduce cost. For example, if you requested 1000 AUs for your program and only 50 AUs were needed, the system would recommend that you only use 50 AUs resulting in a 20x cost savings.

“We ingest a massive amount of live data from mobile, web, IoT and retail transactions. Data Lake gives us the ability to easily and cost effectively store everything and analyse what we need to, when we need to. The simplicity of ramping up parallel processing on the U-SQL queries removes the technical complexities of fighting with the data and lets the teams focus on the business outcomes. We are now taking this a step further and exposing the powerful Data Lake tools directly to our clients in our software allowing them to more easily explore their data using these tools.” -David Inggs, CTO at Plexure

Today, we are also announcing the availability of this big data productivity environment in Visual Studio Code allowing users to have this type of productivity in a free cross-platform code editor that is available on Windows, Mac OS X, and Linux.

Azure HDInsight introduces fully managed Kafka for real-time analytics and R Server for advanced analytics

At Strata + Hadoop World New York, we announced new security, performance and ISV solutions that build on Azure HDInsight’s leadership for enterprise-ready cloud Hadoop. Today, we are announcing the public preview of Kafka for HDInsight. This service lets you ingest massive amounts of real-time data and analyze that data with integration to Storm, Spark, for HDInsight and Azure IoT Hub to build end-to-end IoT, fraud detection, click-stream analysis, financial alerts, or social analytics solutions.

We are also announcing the general availability of R Server for HDInsight. Running Microsoft R Server as a service on top of Apache Spark, developers can achieve unprecedented scale and performance with code that combines the familiarity of the open source R language and Spark. Multi-threaded math libraries and transparent parallelization in R Server enables handling up to 1000x more data and up to 50x faster speeds than open source R—helping you train more accurate models for better predictions than previously possible. Newly available in GA is the inclusion of R Studio Server Community Edition out-of-the-box making it easy for data scientists to get started quickly.

“Milliman is among the world’s largest providers of actuarial and related products and services, with offices in major cities around the globe. R Server for HDInsight, offers the ability for our clients to be able to forecast risk over much larger datasets than ever before, improving the accuracy of predictions, in a cost-efficient way. The familiarity of the R Programming language to our users, as well as the ability to spin up Hadoop and Spark clusters within minutes, running at unprecedented scale and performance, is what really gets me excited about R Server for HDInsight.” -Paul Maher, Chief Technology Officer of the Life Technology Solutions practice at Milliman

Enterprise-grade Security, Auditing and Support

Enterprise-grade big data solutions must meet uptime guarantees, stringent security, governance & compliance requirements, and integrate with your existing IT investments. Data Lake services (Store, Analytics, and HDInsight) guarantee an industry-leading 99.9% uptime SLA and 24/7 support for all Data Lake services. They are built with the highest levels of security for authentication, authorization, auditing, and encryption to give you peace-of-mind when storing and analyzing sensitive corporate data and intellectual property. Data is always encrypted; in motion using SSL, and at rest using service or user managed HSM-backed keys in Azure Key Vault. Capabilities such as single sign-on (SSO), multi-factor authentication and seamless management of your on-premises identity & access management is built-in through Azure Active Directory. You can authorize users and groups with fine-grained POSIX-based ACLs for all data in the Store or with Apache Ranger in HDInsight enabling role-based access controls. Every access or configuration change is automatically audited for security and regulatory compliance requirements.

Supporting open source and open standards

Microsoft continues to collaborate with the open source community reflected by our contributions to Apache Hadoop, Spark, Apache REEF and our work with Jupyter notebooks. This is also the case with Azure Data Lake.

Azure Data Lake Analytics uses Apache YARN, the central part of Apache Hadoop to govern resource management and deliver consistent operations. To lead innovations to YARN, Microsoft has been a primary contributor to improve performance, scale, and made security innovations.

“Hortonworks and Microsoft have partnered closely for the past 5 years to further the Hadoop platform for big data analytics, including contributions to YARN, Hive, and other Apache projects.  Azure Data Lake services, including Azure HDInsight and Azure Data Lake Store, demonstrate our shared commitment to make it easier for everyone to work with big data in an open and collaborative way.” -Shaun Connolly, Chief Strategy Officer at Hortonworks

Data Lake Store supports the open Apache Hadoop Distributed File System (HDFS) standard. Microsoft has also contributed improvements to HDFS such as OAuth 2.0 protocol support.

Leadership

Both industry analysts and customers recognize Microsoft’s capabilities in big data. Forrester recently recognized Microsoft Azure as a leader in their Big Data Hadoop Cloud Solutions. Forrester notes that leaders have the most comprehensive, scalable, and integrated platforms. Microsoft specifically was called out for having a cloud-first strategy that is paying off.

Getting started with Data Lake

Data Lake Analytics and Store is generally available today. R Server for HDInsight is also generally available today. Kafka for HDInsight is in public preview. Try it today individually or as part of Cortana Intelligence Suite to transform your data into intelligent action.

Read the overview, pricing and getting started pages of Data Lake Analytics or attend the free course
Read the overview, pricing and getting started pages of Data Lake Store
Read the R Server, Kafka overview and pricing pages of HDInsight

@josephsirosh
Quelle: Azure