Kategorie: Microsoft Azure

Individually great, collectively unmatched: Announcing updates to 3 great Azure Data Services

As Julia White mentioned in her blog today, we’re pleased to announce the general availability of Azure Data Lake Storage Gen2 and Azure Data Explorer. We also announced the preview of Azure Data Factory Mapping Data Flow. With these updates, Azure continues to be the best cloud for analytics with unmatched price-performance and security. In this blog post we’ll take a closer look at the technical capabilities of these new features.

Azure Data Lake Storage – The no compromise Data Lake

Azure Data Lake Storage (ADLS) combines the scalability, cost effectiveness, security model, and rich capabilities of Azure Blob Storage with a high-performance file system that is built for analytics and is compatible with the Hadoop Distributed File System. Customers no longer have to tradeoff between cost effectiveness and performance when choosing a cloud data lake.

One of our key priorities was to ensure that ADLS is compatible with the Apache ecosystem. We accomplished this by developing the Azure Blob File System (ABFS) driver. The ABFS driver is officially part of Apache Hadoop and Spark and is incorporated in many commercial distributions. The ABFS driver defines a URI scheme that allows files and folders to be distinctly addressed in the following manner:

abfs[s]://file_system@account_name.dfs.core.windows.net/<path>/<path>/<filename>

It is important to note that the file system semantics are implemented server-side. This approach eliminates the need for a complex client-side driver and ensures high fidelity file system transactions.

To further boost analytics performance, we implemented a hierarchical namespace (HNS) which supports atomic file and folder operations. This is important because it reduces the overhead associated with processing big data on blob storage. This speeds up job execution and lowers cost because fewer compute operations are required.

The ABFS driver and HNS significantly improve ADLS’ performance, removing scale and performance bottlenecks.  This performance enhancement is now available at the same low cost as Azure Blob Storage.

ADLS offers the same powerful data security capabilities built into Azure Blob Storage, such as:

Encryption of data in transit and at rest via TLS 1.2
Storage account firewalls
Virtual network integration
Role-based access security

In addition, ADLS’ file system provides support for POSIX compliant access control lists (ACLs). With this approach, you can provide granular security protection that restricts access to only authorized users, groups, or service principals and provides file and object data protection.

ADLS is tightly integrated with Azure Databricks, Azure HDInsight, Azure Data Factory, Azure SQL Data Warehouse, and Power BI, enabling an end-to-end analytics workflow that delivers powerful business insights throughout all levels of your organization. Furthermore, ADLS is supported by a global network of big data analytics ISV’s and system integrators, including Cloudera and Hortonworks.

Next steps

Visit the Azure Data Lake Storage product page to learn more.
Access documentation, quick starts, and tutorials.
Find pricing information for Azure Data Lake Storage.
Get started with Azure Data Lake Storage now.

Azure Data Explorer – The fast and highly scalable data analytics service

Azure Data Explorer (ADX) is a fast, fully managed data analytics service for real-time analysis on large volumes of streaming data. ADX is capable of querying 1 billion records in under a second with no modification of the data or metadata required. ADX also includes native connectors to Azure Data Lake Storage, Azure SQL Data Warehouse, and Power BI and comes with an intuitive query language so that customers can get insights in minutes.

Designed for speed and simplicity, ADX is architected with two distinct services that work in tandem: The Engine and Data Management (DM) service. Both services are deployed as clusters of compute nodes (virtual machines) in Azure.

The Data Management (DM) service ingests various types of raw data and manages failure, backpressure, and data grooming tasks when necessary. The DM service also enables fast data ingestion through a unique method of automatic indexing and compression.

The Engine service is responsible for processing the incoming raw data and serving user queries. It uses a combination of auto scaling and data sharding to achieve speed and scale. The read-only query language is designed to make the syntax easy to read, author, and automate. The language provides a natural progression from one-line queries to complex data processing scripts for efficient query execution.

ADX is available in 41 Azure regions and is supported by a growing ecosystem of partners, including ISV’s and system integrators.

Next steps

Visit the Azure Data Explorer product page to learn more.
Access documentation, quick starts, and tutorials.
Find pricing information for Azure Data Explorer.
Get started with Azure Data Explorer now.

Azure Data Factory Mapping Data Flow – Visual, zero-code experience for data transformation

Azure Data Factory (ADF) is a hybrid cloud-based data integration service for orchestrating and automating data movement and transformation. ADF provides over 80 built-in connectors to structured, semi-structured, and unstructured data sources.

With Mapping Data Flow in ADF, customers can visually design, build, and manage data transformation processes without learning Spark or having a deep understanding of their distributed infrastructure.

Mapping Data Flow combines a rich expression language with an interactive debugger to easily execute, trigger, and monitor ETL jobs and data integration processes.

Azure Data Factory is available in 21 regions and expanding, and is supported by a broad ecosystem of partners including ISV’s and system integrators.

Next steps

Visit the Azure Data Factory product page to learn more.
Access documentation, quick starts, and tutorials.
Find pricing information on Azure Data Factory.
Learn more about Mapping Data Flow.
Get started and sign-up for the preview of Azure Data Factory – Mapping Data Flow.

Azure is the best place for data analytics

With these technical innovations announced today, Azure continues to be the best cloud for analytics. Learn more why analytics in Azure is simply unmatched.
Quelle: Azure

Azure Cost Management now general availability for enterprise agreements and more!

As enterprises accelerate cloud adoption, it is becoming increasingly important to manage cloud costs across the organization. Last September, we announced the public preview of a comprehensive native cost management solution for enterprise customers. We are now excited to announce the general availability (GA) of Azure Cost Management experience that helps organizations visualize, manage, and optimize costs across Azure.

In addition, we are excited to announce the public preview for web direct Pay-As-You-Go customers and Azure Government cloud.

With the addition of the Azure Cost Management, customers now have an always-on, low-latency solution to understand and visualize costs with the following features available in Cost Management:

Cost analysis

This feature allows you to track costs over the course of the month and offers you a variety of ways to analyze your data. To learn more about how to use cost analysis, please visit our documentation, “Quickstart: Explore and analyze costs with Cost analysis.”

Budgets

Use budgets to proactively manage costs and drive accountability within your organization. To learn more about using Azure budgets please visit our documentation, “Tutorial: Create and manage Azure budgets.”

Exports

Export all your cost data to an Azure storage account using our new exports feature. You can use this data in external systems and combine it with your own data to maximize your cost management capabilities. To learn more about using Azure exports please visit our documentation, “Tutorial: Create and manage exported data.”

New Azure APIs

As a part of this release we are also making the APIs mentioned below available for you to build your own cost management solutions. To learn more about developing on top of our new cost management functionality, please visit the Azure REST API documentation links below.

Usage Query – Develop advanced API query calls to learn the most about your organization’s usage and cost patterns.
Budgets – Create and view your budgets in an automated fashion.
Exports – Automate data export configuration.
Usage details by Management Group – Use this API to analyze your organization’s usage across multiple subscriptions.

Alerts (in preview)

View and manage all your alerts in one single place with the new alerts preview feature. In the release you can view budget alerts, monetary commitment alerts, and department spending quota alerts. You can also view active and dismissed alerts.

Getting started

Get started now on this end-to-end cost management and optimization solution that enables you to get the most value for every cloud dollar spent. Please visit the Azure Cost Management documentation page for tutorial and details on getting started.

What’s coming next?

We will continue to iterate additional Cost Management features, so can enjoy a more unified user experience with features like ability to save and schedule reports, additional capabilities in cost analysis, budgets, alerts, and exports, as well as show backs in the coming months.

Partners will also soon be able to leverage the benefits of cost management with our support for the Cloud Solution Provider (CSP) program. With Azure Cost Management, Microsoft is committed to continuing the investment in supporting a multi-cloud environment including Azure and AWS. Public preview for AWS is currently targeted for Q2 of the current calendar year. We plan continue to enhance this with support for other clouds in the near future.

Are you ready for the best part? Azure Cost Management is available for free to all customers and partners to manage Azure costs.

The Cloudyn portal will continue to be available to customers while we integrate all relevant functionality into native Azure Cost Management.

Follow us on Twitter @AzureCostMgmt for exciting cost management updates.
Quelle: Azure

Build your own deep learning models on Azure Data Science Virtual Machines

As a modern developer, you may be eager to build your own deep learning models but aren’t quite sure where to start. If this is you, I recommend you take a look at the deep learning course from fast.ai. This new fast.ai course helps software developers start building their own state-of-the-art deep learning models. Developers who complete this fast.ai course will become proficient in deep learning techniques in multiple domains including computer vision, natural language processing, recommender algorithms, and tabular data.

You’ll also want to learn about Microsoft’s Azure Data Science Virtual Machine (DSVM). Azure DSVM empowers developers like you with the tools you need to be productive with this fast.ai course today on Azure, with virtually no setup required. Using fast cloud-based GPU virtual machines (VMs), at the most competitive rates, Azure DSVM saves you time that would otherwise be spent in installation, configuration, and waiting for deep learning models to train.

Here is how you can effectively run the fast.ai course examples on Azure.

Running the fast.ai deep learning course on Azure DSVM

While there are several ways in which you can use Azure for your deep learning course, one of the easiest ways is to leverage Azure Data Science Virtual Machine (DSVM). Azure DSVM is a family of virtual machine (VM) images that are pre-configured with a rich curated set of tools and frameworks for data science, deep learning, and machine learning.

Using Azure DSVM, you can utilize tools like Jupyter notebooks and necessary drivers to run on powerful GPUs. In result saving time that would otherwise be spent installing, configuring, and troubleshooting any compatibility issues on your system. Azure DSVM is offered on both Linux and Windows editions. Azure VMs provides a neat extension mechanism that the DSVM can leverage, allowing you to automatically configure your VM to your needs.

Microsoft provides an extension to the DSVM specifically for the fast.ai course, making the process so simple you can answer a couple of questions and get your own instance of DSVM provisioned in a few minutes. The fast.ai extension installs all the necessary libraries you need to run the course Jupyter notebooks and also pull down the latest course notebooks from the fast.ai GitHub repository. So in a very short time, you’ll be ready to start running your course samples.

Getting started with Azure DSVM and fast.ai

Here’s how simple it is to get started:

1. Sign in or sign up for an Azure subscription

If you don’t have an Azure subscription you can start off with a free trial subscription to explore any Azure service for 30 days and access to a set of popular services free for 12 months. Please note that free trial subscriptions do not give access to GPU resources. For GPU access, you need to sign up for an Azure pay-as-you-go subscription or use the Azure credits from the Visual Studio subscriptions if you have one. Once you have created your subscription, you can login to the Azure portal.

2. Create a DSVM instance with fast.ai extension

You can now create a DSVM with the fast.ai extension by selecting one of the links below. Choose one depending on whether you prefer a Windows or a Linux environment for your course.

Linux (Ubuntu) edition of DSVM with fast.ai
Windows Server 2016 edition of DSVM with fast.ai

After answering a few simple questions in the deployment form, your VM is created in about five to 10 minutes and is pre-configured with everything you need to run the fast.ai course. While creating the DSVM, you can choose between a GPU-based or a CPU-only instance of the DSVM. A GPU instance will drastically cut down execution times when training deep learning models. This is largely what the course notebooks covers, so I recommend a GPU instance. Azure also offers low-priority instances including GPU at a significant discount which is as much as 80 percent on compute usage charges compared to standard instances. Though keep in mind, they can be preempted and deallocated from your subscription at any time depending on factors like the demand for these resources. If you want to take advantage of the deep discount, you can create preemptable Linux DSVM instance with the fast.ai extension.

3. Run your course notebooks

Once you have created your DSVM instance, you can immediately start using it to run all the code in the course examples by accessing Jupyter and the course notebooks that are preloaded in the DSVM.

You can find more information on how to get started with fast.ai for Azure on the course documentation page.

Next steps

You can continue your journey in machine learning and data science by taking a look at the Azure Machine Learning service which enables you to track your experiments. You can also use automated machine learning, build custom models, and deploy machine learning, deep learning models, or pipelines in production at scale with several sample notebooks that are pre-built in the DSVM. You can also find additional learning resources on Microsoft’s AI School and LearnAnalytics.

I look forward to your feedback and questions on the fast.ai forums or on Stack Overflow.
Quelle: Azure

Best practices to consider before deploying a network virtual appliance

A network virtual appliance (NVA) is a virtual appliance primarily focused on network functions virtualization. A typical network virtual appliance involves various layers four to seven functions like firewall, WAN optimizer, application delivery controllers, routers, load balancers, IDS/IPS, proxies, SD-WAN edge, and more. While the public cloud may provide some of these functionalities natively, it is quite common to see customers deploying network virtual appliances from independent software vendors (ISV). These capabilities in the public cloud enable hybrid solutions and are generally available through the Azure Marketplace.

What exactly is the network virtual appliance in the cloud?

A network virtual appliance is often a full Linux virtual machine (VM) image consisting of a Linux kernel and includes user level applications and services. When a VM is created, it first boots the Linux kernel to initialize the system and then starts up any application or management services needed to make the network virtual appliance functional. The cloud provider is responsible for the compute resources, while the ISV provides the image that represents the software stack of the virtual appliance.

Similar to a standard Linux distribution, the Linux kernel is integral to the NVA’s image and is provided by the ISV often customized. The kernel itself includes the drivers needed for all network and disk devices available to the virtual machine. The version and customizations made to the NVA’s kernel will often impact the performance and functionality of the virtual machine, for more information about Linux and accelerated networking see our documentation, “Create a Linux virtual machine with Accelerated Networking.” As new networking enhancements are made to the Azure platform such as performance improvements or even entirely new networking features, the ISV may need to update the software image to provide support for those enhancements. Often, this entails updating their version of the Linux kernel from the upstream Linux project. For the latest updates, see the Linux Kernel Archives website.

All NVA images published in the Azure Marketplace go through rigorous testing and onboarding workflows. As part of Azure’s continuous integration and deployment life cycle, NVA images are deployed and tested in a pre-production environment for any regression or issues. ISVs are responsible for publishing deployment guidelines and GithHub published Azure Resource Manager (ARM) templates for their specific products. Technical and performance specifications of the appliance are owned by the ISVs, while Microsoft owns the technical and performance specifications of the host environment. Technical support for the customer’s virtual appliance, it’s features, recommended OS version, kernel version, and security updates are provided by the ISV.

Pricing for NVA solutions may vary based on product types and publisher specifications. Software license fees and Microsoft Azure usage costs are charged separately through the Azure subscription. Learn more by visiting our list of Marketplace FAQs related to virtual appliance and Azure marketplace.

Below is an example of a hybrid network that extends an on-premises network to Azure. Demilitarized zone (DMZ) represents a perimeter network between on-premises and Azure, which includes NVAs.

Another example below shows an NVA with Azure Virtual WAN. For more details on how to steer traffic from a Virtual WAN hub to a network virtual appliance please visit our documentation, “Create Virtual Hub route table steer traffic to a Network Virtual Appliance.”

Common best practices

Microsoft continues to collaborate with multiple ISVs to improve cloud experience for Microsoft customers.

Azure accelerated networking support: Consider a virtual appliance that is available on one of the supported VM types with Azure’s accelerated networking capability. Accelerated networking enables single root I/O virtualization (SR-IOV) to a VM, greatly improving its networking performance. This high-performance path bypasses the host from the datapath, reducing latency, jitter, and CPU utilization for use with the most demanding network workloads on supported VM types. Accelerated networking is supported on most general purpose and compute-optimized instance sizes with two or more vCPUs. For a list of supported OS and additional information visit our documentation, “Create a Windows virtual machine with Accelerated Networking.” 
Multi-NIC support: A network interface (NIC) is the interconnection between a VM and a virtual network (VNet). A VM must have at least one NIC, but can have more than one depending on the size of the VM you create. Learn about how many NICs each VM size supports for Windows and Linux in our documentation, “Sizes for Windows virtual machines in Azure” or “Sizes for Linux virtual machines in Azure.” Many network virtual appliances require multiple NICs. With multiple NICs you can better manage your network traffic by isolating various types of traffic across the different NICs. A good example would be separating data plane traffic from the management plane and hence the VM supporting at least two NICs. A VM can only have as many network interfaces attached to it as the VM size supports. If you are considering adding a NIC after deploying the NVA, be sure to Enable IP forwarding on the NIC. The setting disables Azure's check of the source and destination for a network interface. Learn more about how to enable IP forwarding for a network interface.
HA Port with Azure Load Balancer: Azure Standard Load Balancer helps you load-balance TCP and UDP flows on all ports simultaneously when you're using an internal load balancer. A high availability (HA) port load balancing rule is a variant of a load balancing rule, configured on an internal Standard Load Balancer. You would want your NVA to be reliable and highly available, to achieve these goals simply by adding NVA instances to the back-end pool of your internal load balancer and configuring a HA ports load-balancer rule. For more information on HA port overview please visit our documentation, “High availability ports overview.”

Support for Virtual Machine Scale Sets (VMSS): Azure Virtual Machine Scale Sets let you create and manage a group of identical, load balanced VMs. The number of VM instances can automatically increase or decrease in response to a demand or a defined schedule. Scale sets provide high availability to your applications, and allow you to centrally manage, configure, and update a large number of VMs. Scale sets are built from virtual machines. With scale sets, the management and automation layers are provided to run and scale your applications. For more information visit our documentation, “What are virtual machine scale sets.”

As enterprises move ever demanding mission-critical workloads to the cloud, it is important to consider comprehensive networking services that are easy to deploy, manage, scale, and monitor. We are fully committed to providing you the best network virtual appliance experience that can provide all the benefits of cloud in conjunction with your network needs. Picking a virtual appliance can be an important decision when you are designing your network. We want to ensure you do so for ease of use, scale, and a better future together.

Additional links

Support for Linux and open source technology in Azure
Deploy highly available network virtual appliances
Azure Reference Architectures

Quelle: Azure

Microsoft Azure portal February 2019 update

This month we’re bringing you updates to several compute (IaaS) resources, the ability to export contents of lists of resources and resource groups as CSV files, an improvement to the layout of essential properties on overview pages, enhancements to the experience on recovery services pages, and expansions of setting options in Microsoft Intune.

Sign in to the Azure portal now and see for yourself everything that’s new. You can also download the Azure mobile app.

Here is a list of February updates to the Azure portal:

Compute (laaS)

Add a new virtual machine (VM) directly to an application gateway or load balancer
Migrate classic virtual machines (VMs) to Azure Resource Manager
Virtual machine scale sets (VMSS) password reset

Shell

Export as CSV in All resources and Resource groups
Layout change for essential properties on overview pages

Site Recovery

Azure Site Recovery UI updates

Other

Updates to Microsoft Intune

Let’s look at each of these updates in detail.

Compute (laaS)

Add a new VM directly to an application gateway or load balancer

We learned from you that a common scenario involves adding a new VM to a load balanced set such as setting up a SharePoint form or putting together a three-tier web application. You can now add a new VM to an existing load balancing solution during the VM creation process. When you specify networking parameters for your virtual machine, you can now choose to add it to the backend pool of an application gateway for HTTP and HTTPS traffic or load balancer Standard SKU for all TCP and UDP traffic.

Migrate classic VMs to Azure Resource Manager

The Azure Resource Manager (ARM) deployment model was released nearly three years ago, and many features have been added since then that are exclusive to ARM. The Azure platform supports migrating classic Azure Service Manager (ASM) resources to ARM, and you can now use the Azure portal to migrate existing infrastructure virtual machines, virtual networks, and storage accounts to the modern ARM deployment model.

Navigate to a classic virtual machine, and select Migrate to ARM from the Resource menu under Settings.

VMSS password reset

You can now use the portal to reset the password of virtual machine scale set instances.

Navigate to a virtual machine scale set in the Azure portal, and select Reset password.

Shell

Export as CSV in All resources and Resource groups

We have recently added the ability to export the contents of lists of resources and resource groups to a CSV (comma separated values) file.

This capability is available in the All resources screen:

It is also available also in the Resource groups screen:

We have added this capability to an instance of the Resource group screen, so you can download all the resources within a single resource group to a CSV file:

Layout change for essential properties on overview pages

We’ve changed the way that essential properties are laid out on overview pages so there’s less vertical scrolling required now. On standard wide screen resolutions, the essential properties (key/value) will be laid out horizontally rather than vertically to save vertical space. However, you will still get the vertical layout if the essential properties do not have enough horizontal space to avoid truncation and/or ellipsis of the important information.

Select Virtual Machines within the menu on the left.
Select any virtual machine.

Site Recovery

Azure Site Recovery UI updates

The new enhanced IaaS VM disaster recovery multiple tab experience lets you configure the replication with a single click. It’s as simple as selecting the Target region.

Select any virtual machine.
Select Disaster recovery within the menu located on the left.
Select Target region.
Select Review + Start replication.

We also now have a new immersive experience for Site Recovery infrastructure with the addition of an overview tab.

Select any Recovery service vault.
Select Site Recovery infrastructure under the subheading Manage.

Other

Updates to Microsoft Intune

The Microsoft Intune team has been hard at work on updates. You can find a complete list on the What’s new in Microsoft Intune page, including changes that affect your experience using Intune.

Did you know?

You can always test features by visiting the preview version of Azure portal.

Next steps

Thank you for all your terrific feedback. The Azure portal is built by a large team of engineers who are always interested in hearing from you.

We recently launched the Azure portal “how to” series where you can learn about a specific feature of the portal in order to become more productive using it. To learn more please watch the videos “How to manage multiple accounts, directories, and subscriptions in Azure” and “How to create a virtual machine in Azure.” Keep checking in on the Azure YouTube channel for new videos each week.

If you’re interested in learning how we streamlined resource creation in Microsoft Azure to improve usability, consistency, and accessibility, read the new Medium article, “Creation at Cloud Scale.” If you’re curious to learn more about how the Azure portal is built, be sure to watch the Microsoft Ignite 2018 session, “Building a scalable solution to millions of users.”

Don’t forget to sign in on the Azure portal and download the Azure mobile app today to see everything that’s new. Let us know your feedback in the comments section or on Twitter. See you next month.
Quelle: Azure