Kategorie: Microsoft Azure

Improving observability of your Kubernetes deployments with Azure Monitor for containers

Over the past few years, we’ve seen significant changes in how an application is thought of and developed, especially with the adoption of containers and the move from traditional monolithic applications to microservices applications. This shift also affects how we think about modern application monitoring, now with greater adoption of open source technologies and the introduction of observability concepts.

In the past, vendors owned the application and infrastructure, and as a result, they knew what metrics to monitor. With open source products growing in number, vendors do not own all the metrics, and custom metrics are extremely necessary with current monitoring tools. Unlike the monolith application, which is a single deployment unit with a simple status of healthy or not, modern applications will consist of dozens of different microservices with fractional n-states. This is due to the sophisticated deployment strategies and rollbacks where customers may be running different versions of the same services in production, especially on Kubernetes. Thus, embracing these shifts is essential in monitoring.

Custom metrics and open source technologies help improve the observability of specific components of your application, but you also need to monitor the full stack. Azure Monitor for containers embraces both observability through live data and collecting custom metrics using Prometheus, providing the full stack end-to-end monitoring from nodes to Kubernetes infrastructure to workloads.

Collecting Prometheus metrics and viewing using Grafana dashboards

By instrumenting Prometheus SDK into your workloads, Azure Monitor for containers can scrape the metrics exposed from Prometheus end-points so you can quickly gather failure rates, response per secs, and latency. You can use Prometheus to collect some of the Kubernetes infrastructure metrics that are not provided out of the box by Azure Monitor by configuring the containerized agent.

From Log Analytics, you can easily run a Kusto Query Language (KQL) query and create your custom dashboard in the Azure portal dashboard. For many customers using Grafana to support their dashboard requirements, you can visualize the container and Prometheus metrics in a Grafana dashboard.

Below is an example of a dashboard that provides an end-to-end Azure Kubernetes Service (AKS) cluster overview, node performances, Kubernetes infrastructure, and workloads.
  

If you would like to monitor or troubleshoot other scenarios, such as list of all workload live sites, or noisy neighbor issues on a worker node, you can always switch to Azure Monitor for container to view the visualizations included from the Grafana dashboard by clicking on Azure Monitor – Container Insights in the top right-hand corner.

  
Azure Monitor for containers provides the live, real-time data of container logs and Kubernetes event logs to provide observability as seen above. You can see your deployments immediately and observe any anomalies using the live data.

If you are interested in trying Azure Monitor for containers, please check the documentation. Once you have enabled the monitoring, and if you would like to try the Grafana template, please go to the Grafana gallery. This template will light up using the out-of-the-box data collected from Azure Monitor for containers. If you want to add more charts to view other metrics collected, you can do so by checking our documentation.

Prometheus data collection and Grafana are also supported for AKS Engine as well.

For any feedback or suggestions, please reach out to us through Azure Community Support or Stack Overflow.
Quelle: Azure

Azure Container Registry: preview of repository-scoped permissions

The Azure Container Registry (ACR) team is rolling out the preview of repository scoped role-based access control (RBAC) permissions, our top-voted item on UserVoice. In this release, we have a command-line interface (CLI) experience for you to try and provide feedback.

ACR already supports several authentication options using identities that have role-based access to an entire registry. However, for multi-team scenarios, you might want to consolidate multiple teams into a single registry, limiting each team’s access to their specific repositories. Repository scoped RBAC now enables this functionality.

Here are some of the scenarios where repository scoped permissions might come in handy:

Limit repository access to specific user groups within your organization. For example, provide write access to developers who build images that target specific repositories, and read access to teams that deploy from those repositories.

Provide millions of IoT devices with individual access to pull images from specific repositories.

Provide an external organization with permissions to specific repositories.

In this release, we have introduced tokens as a mechanism to implement repository scoped RBAC permissions. A token is a credential used to authenticate with the registry. It can be backed by username and password or Azure Active Directory(AAD) objects like Azure Active Directory users, service principals, and managed identities. For this release, we have provided tokens backed by username and password. Future releases will support tokens backed by Azure Active Directory objects like Azure Active Directory users, service principals, and managed identities. See Figure 1.

*Support for Azure Active Directory (AAD) backed token will be available in a future release.

Figure 1

Figure 2 below describes the relationship between tokens and scope-maps.

A token is a credential used to authenticate with the registry. It has a permitted set of actions which are scoped to one or more repositories. Once you have generated a token, you can use it to authenticate with your registry. You can do a docker login using the following command:

docker login –username mytoken –password-stdin myregistry.azurecr.io.

A scope map is a registry object that groups repository permissions you apply to a token. It provides a graph of access to one or more repositories. You can apply scoped repository permissions to a token or reapply them to other tokens. If you don't apply a scope map when creating a token, a scope map is automatically created for you, to save the permission settings.

A scope map helps you configure multiple users with identical access to a set of repositories.

Figure 2

As customers use containers and other artifacts for their IoT deployment, the number of devices can grow into the millions. In order to support the scale of IoT, Azure Container Registry has implemented repository based RBAC, using tokens (figure 3). Tokens are not a replacement for service principals or managed identities. You can add tokens as an additional option providing scalability of IoT deployment scenarios.

This article shows how to create a token with permissions restricted to a specific repository within a registry. With the introduction of token-based repository permissions, you can now provide users or services with scoped and time-limited access to repositories without requiring an Azure Active Directory identity. In the future, we will support tokens backed by Azure Active Directory objects. Check out this new feature and let us know your feedback on GitHub.

Figure 3

Availability and feedback

Azure CLI experience is now in preview. As always, we love to hear your feedback on existing features as well as ideas for our product roadmap.

Roadmap: For visibility into our planned work.

UserVoice: To vote for existing requests or create a new request.

Issues: To view existing bugs and issues, or log new ones.

ACR documents: For ACR tutorials and documentation.
Quelle: Azure

Announcing the general availability of the new Azure HPC Cache service

If data-access challenges have been keeping you from running high-performance computing (HPC) jobs in Azure, we’ve got great news to report! The now-available Microsoft Azure HPC Cache service lets you run your most demanding workloads in Azure without the time and cost of rewriting applications and while storing data where you want to—in Azure or on your on-premises storage. By minimizing latency between compute and storage, the HPC Cache service seamlessly delivers the high-speed data access required to run your HPC applications in Azure.

Use Azure to expand analytic capacity—without worrying about data access

Most HPC teams recognize the potential for cloud bursting to expand analytic capacity. While many organizations would benefit from the capacity and scale advantages of running compute jobs in the cloud, users have been held back by the size of their datasets and the complexity of providing access to those datasets, typically stored on long-deployed network-attached storage (NAS) assets. These NAS environments often hold petabytes of data collected over a long period of time and represent significant infrastructure investment.

Here’s where the HPC Cache service can help. Think of the service as an edge cache that provides low-latency access to POSIX file data sourced from one or more locations, including on-premises NAS and data archived to Azure Blob storage. The HPC Cache makes it easy to use Azure to increase analytic throughput, even as the size and scope of your actionable data expands.

Keep up with the expanding size and scope of actionable data

The rate of new data acquisition in certain industries such as life sciences continues to drive up the size and scope of actionable data. Actionable data, in this case, could be datasets that require post-collection analysis and interpretation that in turn drive upstream activity. A sequenced genome can approach hundreds of gigabytes, for example. As the rate of sequencing activity increases and becomes more parallel, the amount of data to store and interpret also increases—and your infrastructure has to keep up. Your power to collect, process, and interpret actionable data—your analytic capacity—directly impacts your organization’s ability to meet the needs of customers and to take advantage of new business opportunities.

Some organizations address expanding analytic throughput requirements by continuing to deploy more robust on-premises HPC environment with high-speed networking and performant storage. But for many companies, expanding on-premises environments presents increasingly daunting and costly challenges. For example, how can you accurately forecast and more economically address new capacity requirements? How do you best juggle equipment lifecycles with bursts in demand? How can you ensure that storage keeps up (in terms of latency and throughput) with compute demands? And how can you manage all of it with limited budget and staffing resources?

Azure services can help you more easily and cost-effectively expand your analytic throughput beyond the capacity of existing HPC infrastructure. You can use tools like Azure CycleCloud and Azure Batch to orchestrate and schedule compute jobs on Azure virtual machines (VMs). More effectively manage cost and scale by using low-priority VMs, as well as Azure Virtual Machine Scale Sets. Use Azure’s latest H- and N-series Virtual Machines to meet performance requirements for your most complex workloads.

So how do you start? It’s straightforward. Connect your network to Azure via ExpressRoute, determine which VMs you will use, and coordinate processes using CycleCloud or Batch—voila, your burstable HPC environment is ready to go. All you need to do is feed it data. Ok, that’s the stickler. This is where you need the HPC Cache service.

Use HPC Cache to ensure fast, consistent data access

Most organizations recognize the benefits of using cloud: a burstable HPC environment can give you more analytic capacity without forcing new capital investments. And Azure offers additional pluses, letting you take advantage of your current schedulers and other toolsets to ensure deployment consistency with your on-premises environment.

But here’s the catch when it comes to data. Your libraries, applications, and location of data may require the same consistency. In some circumstances, a local analytic pipeline may rely on POSIX paths that must be the same whether running in Azure or locally. Data may be linked between directories, and those links may need to be deployed in the same way in the cloud. The data itself may reside in multiple locations and must be aggregated. Above all else, the latency of access must be consistent with what can be realized in the local HPC environment.

To understand how the HPC Cache works to address these requirements, consider it an edge cache that provides low-latency access to POSIX file data sourced from one or more locations. For example, a local environment may contain a large HPC cluster connected to a commercial NAS solution. HPC Cache enables access from that NAS solution to Azure Virtual Machines, containers, or machine learning routines operating across a WAN link. The service accomplishes this by caching client requests (including from the virtual machines), and ensuring that subsequent accesses of that data are serviced by the cache rather than by re-accessing the on-premises NAS environment. This lets you run your HPC jobs at a similar performance level as you could in your own data center. HPC Cache also lets you build a namespace consisting of data located in multiple exports across multiple sources while displaying a single directory structure to client machines.

HPC Cache provides a Blob-backed cache (we call it Blob-as-POSIX) in Azure as well, facilitating migration of file-based pipelines without requiring that you rewrite applications. For example, a genetic research team can load reference genome data into the Blob environment to further optimize the performance of secondary-analysis workflows. This helps mitigate any latency concerns when you launch new jobs that rely on a static set of reference libraries or tools.

  
Azure HPC Cache Architecture

HPC Cache Benefits

Caching throughput to match workload requirements

HPC Cache offers three SKUs: up to 2 gigabytes per second (GB/s), up to 4 GB/s, and up to 8 GB/s throughput. Each of these SKUs can service requests from tens to thousands of VMs, containers, and more. Furthermore, you choose the size of your cache disks to control your costs while ensuring the right capacity is available for caching.

Data bursting from your datacenter

HPC Cache fetches data from your NAS, wherever it is. Run your HPC workload today and figure out your data storage policies over the longer term.

High-availability connectivity

HPC Cache provides high-availability (HA) connectivity to clients, a key requirement for running compute jobs at larger scales.

Aggregated namespace

The HPC Cache aggregated namespace functionality lets you build a namespace out of various sources of data. This abstraction of sources makes it possible to run multiple HPC Cache environments with a consistent view of data.

Lower-cost storage, full POSIX compliance with Blob-as-POSIX

HPC Cache supports Blob-based, fully POSIX-compliant storage. HPC Cache, using the Blob-as-POSIX format, maintains full POSIX support including hard links. If you need this level of compliance, you’ll be able to get full POSIX at Blob price points.

Start here

The Azure HPC Cache Service is available today and can be accessed now. For the very best results, contact your Microsoft team or related partners—they’ll help you build a comprehensive architecture that optimally meets your specific business objectives and desired outcomes.

Our experts will be attending at SC19 in Denver, Colorado, the conference on high-performance computing, ready and eager to help you accelerate your file-based workloads in Azure!
Quelle: Azure

Democratizing agriculture intelligence: introducing Azure FarmBeats

For an industry that started 12,000 years ago, there is a lot of unpredictability and imprecision in agriculture. To be predictable and precise, we need to align our actions with insights gathered from data. Last week at Microsoft Ignite, we launched the preview of Azure FarmBeats, a purpose-built, industry-specific solution accelerator built on top of Azure to enable actionable insights from data.

With AgriTechnica 2019 starting today, more than 450,000 attendees from 130 countries are gathering to experience innovations in the global agriculture industry. We wanted to take this opportunity to share more details about Azure FarmBeats.

Azure FarmBeats is a business-to-business offering available in Azure Marketplace. It enables aggregation of agriculture datasets across providers and generation of actionable insights by building artificial intelligence (AI) or machine learning (ML) models based on fused datasets. So, agribusinesses can focus on their core value-add rather than the undifferentiated heavy lifting of data engineering.

Figure 1: Overview of Azure FarmBeats

With the preview of Azure FarmBeats you can:

Assess farm health using vegetation index and water index based on satellite imagery.
Get recommendations on how many sensors to use and where to place them.
Track farm conditions by visualizing ground data collected by sensors from various vendors.
Scout farms using drone imagery from various vendors.
Get soil moisture maps based on the fusion of satellite and sensor data.
Gain actionable insights by building AI or ML models on top of fused datasets.
Build or augment your digital agriculture solution by providing farm health advisories.

As an example, here is how a farm populated with data appears in Azure FarmBeats:

Figure 2: Boundary, sensor locations, and sensor readings for a farm

Figure 3: Drone imagery and model-generated precision maps (soil moisture, sensor placement)

For a real-world example of how it works, take a look at our partnership with the United States Department of Agriculture (USDA). In a pilot, USDA is using Azure FarmBeats to collect data from multiple sources, such as sensors, drones, and satellites, and feeding it into cloud-based AI models to get a detailed picture of conditions on the farm.

Azure FarmBeats includes the following components:

 Datahub: An API layer that enables aggregation, normalization, and contextualization of various agriculture datasets across providers. You can leverage the following data providers:

Available now:

o Sensor: Davis Instruments, Teralytic

o Drone imagery: DJI, EarthSense, senseFly, SlantRange

Coming soon: DTN, Pessl

Datahub is designed as an API platform and we are working with many more providers – sensor, satellite, drone, weather, farm equipment – to integrate with FarmBeats, so you have more choice while building your solution.

Accelerator: A sample solution, built on top of Datahub, that jumpstarts your user interface (UI) and model development. This web application leverages APIs to demonstrate visualization of ingested sensor data as charts and visualization of model output as maps. For example, you can use this to quickly create a farm and easily get a vegetation index map or a sensor placement map for that farm.

While this preview is the culmination of years of research work and working closely with more than a dozen agriculture majors, it is just the beginning. It would not have been possible without the early feedback and validation from these organizations, and we take this opportunity to extend our sincere gratitude.

Azure FarmBeats is offered at no additional charge and you pay only for the Azure resources you use. You can get started by installing it from Azure Marketplace in Azure Portal. In addition, you can:

Get and stay informed with our documentation.
Seek help by posting a question on our support forum.
Provide feedback by posting or voting for an idea on our feedback forum.

With Azure FarmBeats preview, we are pioneering a cloud platform to empower every person and every organization in agriculture to achieve more, by harnessing the power of IoT, cloud, and AI. We are delighted to have you with us on this global transformational journey and look forward to your feedback on the preview.
Quelle: Azure

Sharing the DevOps journey at Microsoft

Today, more and more organizations are focused on delivering new digital solutions to customers and finding that the need for increased agility, improved processes, and collaboration between development and operation teams is becoming business-critical. For over a decade, DevOps has been the answer to these challenges. Understanding the need for DevOps is one thing, but the actual adoption of DevOps in the real world is a whole other challenge. How can an organization with multiple teams and projects, with deeply rooted existing processes, and with considerable legacy software change its ways and embrace DevOps?

At Microsoft, we know something about these challenges. As a company that has been building software for decades, Microsoft consists of thousands of engineers around the world that deliver many different products. From Office, to Azure, to Xbox we also found we needed to adapt to a new way of delivering software. The new era of the cloud unlocks tremendous potential for innovation to meet our customers’ growing demand for richer and better experiences—while our competition is not slowing down. The need to accelerate innovation and to transform how we work is real and urgent.

The road to transformation is not easy and we believe that the best way to navigate this challenging path is by following the footsteps of those who have already walked it. This is why we are excited to share our own DevOps journey at Microsoft with learnings from teams across the company who have transformed through the adoption of DevOps.

 

More than just tools

An organization’s success is achieved by providing engineers with the best tools and latest practices. At Microsoft, the One Engineering System (1ES) team drives various efforts to help teams across the company become high performing. The team initially focused on tool standardization and saw some good results—source control issues decreased, build times and build reliability improved. But over time it became clear that the focus on tooling is not enough, to help teams, 1ES had to focus on culture change as well. Approaching culture change can be tricky, do you start with quick wins, or try to make a fundamental change at scale? What is the right engagement model for teams of different sizes and maturity levels? Learn more about the experimental journey of the One Engineering System team.

Redefining IT roles and responsibilities

The move to the cloud can challenge the definitions of responsibilities in an organization. As development teams embrace cloud innovation, IT operations teams find that the traditional models of ownership over infrastructure no longer apply. The Manageability Platforms team in the Microsoft Core Service group (previously Microsoft IT), found that the move to Azure required rethinking the way IT and development teams work together. How can the centralized IT model be decentralized so the team can move away from mundane, day-to-day work while improving the relationship with development teams? Explore the transformation of the Manageability Platforms team.

Streamlining developer collaboration

Developer collaboration is a key component of innovation. With that in mind, Microsoft open-sourced the .NET framework to invite the community to collaborate and innovate on .NET. As the project was open-sourced over time, its scale and complexity became apparent. The project spanned over many repositories, each with its own structure using multiple different continuous integration (CI) systems, making it hard for developers to move between repositories. The .NET infrastructure team at Microsoft decided to invest in streamlining developer processes. That challenge was approached by focusing on standardizing repo structure, shared tooling, and converging on a single CI system so both internal and external contributors to the project would benefit. Learn more about the investments made by the .NET infrastructure team.

A journey of continuous learning

DevOps at Microsoft is a journey, not a destination. Teams adapt, try new things, and continue to learn how to change and improve. As there is always more to learn, we will continue to share the transformation stories of additional teams at Microsoft in the coming months. As an extension of this continuous internal learning journey, we invite you to join us on the journey and learn how to embrace DevOps and empower your teams to build better solutions, faster and deliver them to happier customers.

Resources

The DevOps journey at Microsoft
What is DevOps?
DevOps Solutions on Azure

Azure. Invent with purpose.
Quelle: Azure

10 user experience updates to the Azure portal

We’re constantly working to improve your user experience in the Azure portal. Our goal is to offer you a productive and easy-to-use single-pane-of glass where you can build, manage, and monitor your Azure services, applications, and infrastructure. In this post, I’d like to share the highlights of our latest experience improvements, including:

Improved portal home experience: increased focus and clarity to bring services and instances that are relevant to you front and center.
New service cards: new service hovercards that present contextual information relevant to each service.
Enhanced service browsing experience: simplified offering navigation by progressively disclosing services.
Extended Microsoft Learn integration: contextual integration of free training in key parts of the experience.
Improved instance browsing experience: updated experience for more than 70 services with improved performance, better filtering and sorting options, grouping, and to allow exporting your resource lists to a CSV file.
Improved Azure Resource Graph experience: re-use and share your queries via Resource Graph Saved Queries.
Automatic refresh in Azure Dashboard: set automatic refresh intervals for your dashboard.
Improved service icons: New icons re-designed for better visual consistency and reduced distractions.
Simplified settings panel: better separation between general settings and localization.
New landing page for Azure Mobile application: added a new landing page that brings important information.

Improved portal home experience

We have improved the Azure portal home page to increase focus and clarity and to make things that are important to you easily accessible.

  Figure 1 – simplified Azure portal home.

We’ve organized these into differentiated sections for ease of use:

Services and resources (dynamic): the top section has dynamic content that gets adjusted based on your usage without requiring any additional customizations. The more you use the portal, the more it adjusts to you!
Common entry points and useful info (static): the lower section contains static content with common entry points to provide quick access to main navigation flows that are always there, enabling users to develop muscle memory for repeated usage.

Figure 2 – sections of the home page.

The Azure services section provides quick access to the Azure Marketplace, a list of eight of the most-used Azure services, and access to browse the entire Azure offering. The list of services is populated by default with some of our most popular services and gets automatically updated with your most recently used services. The Recent resources section shows a list of your recently used resources. Both lists get updated as you use the product. Our goal is to bring relevant services and instances front and center without requiring customization. The more you use the product, the more useful it gets for you! The rest of the sections are static, providing important points of reference for navigation and access to key Azure products, services, content, and training.

The overall home experience has been streamlined by hiding the left navigation bar under an always present menu button in the top navigation bar:

Figure 3 – The menu button

The main motivation for this change is improving focus, reducing distractions and redundancy, and to enable more immersive experiences. Before this change, when you were immersed in a workload in the portal you always had two vertical menus side by side, the left navigation bar and the menu for the experience. The left navigation bar is still available with all its functionality, including favorites, through the menu button at the top bar, always only one click away.

Figure 4 – The new experience allows for more focus.

If you prefer the old visual, having the left navigation always present, you can always bring it back using the Portal Settings panel.

New service cards

We have added hover cards associated with each service that show contextual information and provide direct access to some of the most common workflows. These hover cards are displayed after the cursor is placed for about a second on a service tile. We used the same interaction pattern and design than Outlook uses for identities (users and groups) that are well established with our customer base.

Figure 5 – hover card for virtual machines.

The cards expose relevant contextual information and actions for a service, including:

Create an instance: this provides quick access to a very common flow, short circuiting going though intermediate screens to launch the creation.
Browse instances: browse the full list of instances of that service.
Recently used: the last three recently used instances of that service, providing direct contextual access.
Microsoft Learn content: specialized free training curated for that service. The curation has been done by the Microsoft Learn team based on usage data and customer feedback.
Links to documents: key documents to learn or use the product (quick starts, technical docs, pricing.)
Free offerings available: if the service has free options available, surface them.

Figure 6 – Anatomy of the card

The cards help improve on multiple aspects including more efficient customer journeys, better discoverability, and contextualized information, all presented in the context of one service. The card also helps customers of all levels of expertise: While new customers can benefit from Microsoft Learn content and free offerings advanced customers have a faster path the create instances or access their recently used instances of that service.

The card does not only show on the home page. It is available in every place we display a service like the left navigation bar, the all services list, as well as the Azure home page.

Extended Microsoft Learn integration

Microsoft Learn provides official high-quality free learning material for Microsoft technologies. In this portal update we have introduced several contextual integration points:

Service browsing: contextual integration at the service category level (compute, storage, web, etc.)
Service cards: contextual integration at the service level (virtual machine, Cosmos DB, etc.) available in Azure home page, left navigation, and service browsing experience.
Azure QuickStart center: integration of most popular trainings in the landing page
Azure home: direct access to the main Microsoft Learn entry point

Moving forward, the Azure portal and Microsoft Learn integration will continue to grow, to help you improve your Azure journey!

Enhanced service browsing experience

Azure is big and gets bigger every day. Navigating through Azure’s offering in the portal can be intimidating and challenging due to the vast set of available services. To make this easier, we’ve made the following updates:

Improved global search: improved performance and functionality when searching for services in the global search box in the top bar of the portal. This improved search is also always present and available in your portal session.
Improved service browsing experience: improved the All services experience adding an overview category supporting progressive disclosure of services, reducing visual clutter, and adding contextual Microsoft Learn content.

For service browsing, we introduced an overview category with the goal of progressively disclosing information.

Figure 7 – progressive disclosure of information and better discoverability

The new Overview category presents a list of 15 of Azure’s most popular services, curated Microsoft Learn training content, and access to key functionality like Azure QuickStart center and free offerings.

If the service that you are looking for is not available on this screen you can use the service search functionality, at the top left, or you can browse through the different categories available, at the left of the screen. When displaying a category, we are now surfacing contextual and free Microsoft Learn content to assist you in your Azure learning journey.

Figure 8 – service category with contextual and free Microsoft Learn integration. The training offered in this category is contextual and related to databases in this case.

Improved instance browsing experience

The resource instances browsing experience, going through the list of instances and services is one of the most common entry points for customers using the portal. We are introducing an updated experience that leverages the power of Azure Resource Graph to provide improved performance, better filtering and sorting options, better grouping, and allows exporting your resource lists to a CSV file.

Figure 9 – improved resource browsing experience

As of this month, this experience will be available for more than 70 services and over the next few months it will be rolled out across the entire platform.

Improved Azure Resource Graph experience

The Azure Resource Graph Explorer available in the portal enables you to write queries and create dashboards using the full power of Azure Resource Graph. Here is a video that shows how to use Resource Graph to write queries and create an inventory dashboard for your Azure subscriptions.

We have now introduced Azure Resource Graph Queries in the Azure portal as a new top-level resource. Basically, you can save any Kusto Query Language (KQL) query as a resource in your Azure subscription. Like any other resource you can share it with colleagues, set permissions, check activity logs, and tag it.

Figure 10 – Azure Graph Queries

Automatic refresh in Azure Dashboards

We have added automatic refresh to our Azure dashboards, allowing to automatically refresh your dashboards over several time intervals.

Figure 11 – Configuring automatic refresh

Improved service icons

We’ve updated all of the service icons in the Azure portal with a more consistent and modern look. All these icons have been designed together as a family to provide better visual consistency and reduce distractions.

Figure 12– Improved icons

Simplified settings panel

The settings panel has been simplified. The main reason for this change is that many customers could not find the “Language & region” settings in the previous design and were asking us for capabilities that were already available in the portal. This new design separates the general and the Language & region settings, the portal supports 18 languages and dozens of regional formats, which was a common source of confusion for many of our users.

Figure 13 – separation of general and localization settings

New landing page for Azure Mobile application

The Azure mobile app enables you to stay connected, informed, and in control of your Azure assets while on the go. The app is available for iOS and Android devices.

We have added a brand-new landing screen to the Azure Mobile App that brings all important information together as soon as you open the application. The new Home experience is composed of multiple cards with support for:

Azure services
Recent resources
Latest alerts
Service Health
Resource groups
Favorites

The home view is fully customizable, you can decide what sections to show and in which order to show them.

Figure 14 – new home in the Azure Mobile App

If you have not tried the Azure Mobile app yet, make sure to try it out.

Let us know what you think

We’ve gone through a lot of new capabilities and still did not cover everything that is coming up in this release! The team is always hard at work focusing on improving the experience and is always eager to get your feedback and learn how can we make your experience better.

Azure. Invent with purpose.
Quelle: Azure

Azure SQL Data Warehouse is now Azure Synapse Analytics

On November fourth, we announced Azure Synapse Analytics, the next evolution of Azure SQL Data Warehouse. Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs.

With Azure Synapse, data professionals can query both relational and non-relational data using the familiar SQL language. This can be done using either serverless on-demand queries for data exploration and ad hoc analysis or provisioned resources for your most demanding data warehousing needs. A single service for any workload.

In fact, it’s the first and only analytics system to have run all the TPC-H queries at petabyte-scale. For current SQL Data Warehouse customers, you can continue running your existing data warehouse workloads in production today with Azure Synapse and will automatically benefit from the new preview capabilities when they become generally available. You can sign up to preview new features like serverless on-demand query, Azure Synapse studio, and Apache Spark™ integration.

 

Taking SQL beyond data warehousing

A cloud native, distributed SQL processing engine is at the foundation of Azure Synapse and is what enables the service to support the most demanding enterprise data warehousing workloads. This week at Ignite we introduced a number of exciting features to make data warehousing with Azure Synapse easier and allow organizations to use SQL for a broader set of analytics use cases.

Unlock powerful insights faster from all data

Azure Synapse deeply integrates with Power BI and Azure Machine Learning to drive insights for all users, from data scientists coding with statistics to the business user with Power BI. And to make all types of analytics possible, we’re announcing native and built-in prediction support, as well as runtime level improvements to how Azure Synapse handles streaming data, parquet files, and Polybase. Let’s dive into more detail:

With the native PREDICT statement, you can score machine learning models within your data warehouse—avoiding the need for large and complex data movement. The PREDICT function (available in preview) relies on open model framework and takes user data as input to generate predictions. Users can convert existing models trained in Azure Machine Learning, Apache Spark™, or other frameworks into an internal format representation without having to start from scratch, accelerating time to insight.

We’ve enabled direct streaming ingestion support and ability to execute analytical queries over streaming data. Capabilities such as: joins across multiple streaming inputs, aggregations within one or more streaming inputs, transform semi-structured data and multiple temporal windows are all supported directly in your data warehousing environment (available in preview). For streaming ingestion, customers can integrate with Event Hubs (including Event Hubs for Kafka) and IoT Hubs.

We’re also removing the barrier that inhibits securely and easily sharing data inside or outside your organization with Azure Data Share integration for sharing both data lake and data warehouse data.

By using new ParquetDirect technology, we are making interactive queries over the data lake a reality (in preview). It’s designed to access Parquet files with native support directly built into the engine. Through improved data scan rates, intelligent data caching and columnstore batch processing, we’ve improved Polybase execution by over 13x.

Workload isolation

To support customers as they democratize their data warehouses, we are announcing new features for intelligent workload management. The new Workload Isolation functionality allows you to manage the execution of heterogeneous workloads while providing flexibility and control over data warehouse resources. This leads to improved execution predictability and enhances the ability to satisfy predefined SLAs.

COPY statement

Analyzing petabyte-scale data requires ingesting petabyte-scale data. To streamline the data ingestion process, we are introducing a simple and flexible COPY statement. With only one command, Azure Synapse now enables data to be seamlessly ingested into a data warehouse in a fast and secure manner.

This new COPY statement enables using a single T-SQL statement to load data, parse standard CSV files, and more.

COPY statement sample code:

COPY INTO dbo.[FactOnlineSales] FROM ’https://contoso.blob.core.windows.net/Sales/’

Safe keeping for data with unmatched security

Azure has the most advanced security and privacy features in the market. These features are built into the fabric of Azure Synapse, such as automated threat detection and always-on data encryption. And for fine-grained access control businesses can ensure data stays safe and private using column-level security, native row-level security, and dynamic data masking (now generally available) to automatically protect sensitive data in real time.

To further enhance security and privacy, we are introducing Azure Private Link. It provides a secure and scalable way to consume deployed resources from your own Azure Virtual Network (VNet). A secure connection is established using a consent-based call flow. Once established, all data that flows between Azure Synapse and service consumers is isolated from the internet and stays on the Microsoft network. There is no longer a need for gateways, network addresses translation (NAT) devices, or public IP addresses to communicate with the service.

Get started today

Businesses can continue running their existing data warehouse workloads in production today with generally available features on Azure Synapse.

Email the team to nominate yourself to try the preview features announced in this blog.
Visit the Azure Synapse Analytics page to learn more.
Get started with a free Azure Synapse Analytics account.
Register for the live virtual event with the Azure Synapse Analytics team.

Azure. Invent with purpose.

Quelle: Azure