Kategorie: Microsoft Azure

Azure IoT Edge runtime available for Ubuntu virtual machines

Azure IoT Edge is a fully managed service that allows you to deploy Azure and third-party services—edge modules—to run directly on IoT devices, whether they are cloud-connected or offline. These edge modules are container-based and offer functionality ranging from connectivity to analytics to storage—allowing you to deploy modules entirely from the Azure portal without writing any code. You can browse existing edge modules in the Azure Marketplace.

Today, we’re excited to offer the open-source Azure IoT Edge runtime preinstalled on Ubuntu virtual machines to make it even easier to get started, simulate an edge device, and scale out your automated testing.

Why use virtual machines?

Azure IoT Edge deployments are built to scale so that you can deploy globally to any number of devices and simulate the workload with virtual devices which is an important step to verify if your solution is ready for mass deployment. The easiest way to do this is by creating simulated devices with Azure virtual machines (VMs) running the Azure IoT Edge runtime to scale your testing from the earliest stages of development—even before you have production hardware.

Azure VMs are:
•    Scalable/automatable: deploy as many as you need
•    Persistent: cloud- managed, rather than locally
•    Flexible: any operating system and elastic resources
•    Easy to use: deploy with simple command line instructions or template

Azure IoT Edge on Ubuntu VM

On first boot, the Azure IoT Edge on Ubuntu VM preinstalls the latest version of the Azure IoT Edge runtime, so you will always have the newest features and fixes. It also includes a script to set the connection string and then restarts the runtime, which can be triggered remotely through the Azure VM portal or Azure command line, allowing you to easily configure and connect the IoT Edge device without starting a secure shell (SSH) or remote desktop session. This script will wait to set the connection string until the IoT Edge client is fully installed so that you don’t have to build that into your automation.

The initial offering is based on Ubuntu Server 16.04 LTS, but other operating systems and versions will be added based on user feedback. We’d love to hear your thoughts in the comments.

Getting started

You can deploy the Azure IoT Edge on Ubuntu VM through the Azure Marketplace, Azure Portal, or Azure Command-line. Let me show you how to use the Azure Marketplace and the Portal.

Azure Marketplace

The quickest way to set up a single instance is to use the Azure Marketplace:

1.  Navigate to the Marketplace with our short link or by searching “Azure IoT Edge on Ubuntu” on the Azure Marketplace

2.  Select “GET IT NOW” and then “Continue” on the next dialog.

3.  Once in the Azure Portal, select “Create” and follow the wizard to deploy the VM.

If it’s your first time trying out a VM, it’s easiest to use a password and enable the SSH in the public inbound port menu.
If you have a resource intensive workload, you should upgrade the virtual machine size by adding more CPUs and/or memory.

4.  Once the virtual machine is deployed, configure it to connect to your IoT Hub by:

Copying your device connection string from your IoT Edge device created in your IoT Hub (You can follow the “Register a new Azure IoT Edge device from the Azure portal” how-to guide if you aren’t familiar with this process)
Selecting your newly created virtual machine resource from the Azure Portal and opening the “run command” option

Select the “RunShellScript” option

Execute the script below via the command window with your device connection string:/etc/iotedge/configedge.sh “{device_connection_string}”
Select “Run”
Wait a few moments, and the screen should then provide a success message indicating the connection string was set successfully.

5.  Voila! Your IoT Edge on Ubuntu VM is now connected to IoT Hub

Azure portal

If you’re already working in the Azure portal, you can search for “Azure IoT Edge” and select “Ubuntu Server 16.04 LTS + Azure IoT Edge runtime” to begin the VM creation workflow. From there, complete steps 3-5 in the Marketplace instructions above.

 

If you’d like to learn how you can deploy these virtual machines at scale, check out the “Deploy from Azure CLI” section in the Run Azure IoT Edge on Ubuntu Virtual Machines article.

Now that you have created an IoT Edge device with your virtual machine and connected it to your IoT Hub, you can deploy modules to it like any other IoT Edge device. For example, if you go to the IoT Edge Module Marketplace and select the “Simulated Temperature Sensor,” you can deploy this module to the new device and see data flowing in just a few clicks! Next, try deploying your own workloads to the virtual machine and let us know how we can further simplify your IoT Edge testing experience in the comments section below or on user voice. 

Get started with Azure IoT Edge on Ubuntu virtual machines today!
Quelle: Azure

Controlling costs in Azure Data Explorer using down-sampling and aggregation

Azure Data Explorer (ADX) is an outstanding service for continuous ingestion and storage of high velocity telemetry data from cloud services and IoT devices. Leveraging its first-rate performance for querying billions of records, the telemetry data can be further analyzed for various insights such as monitoring service health, production processes, and usage trends. Depending on data velocity and retention policy, data size can rapidly scale to petabytes of data and increase the costs associated with data storage. A common solution for storage of large datasets for a long period of time is to store the data with differing resolution. The most recent data is stored at maximum resolution, meaning all events are stored in raw format. While the historic data is stored at reduced resolution, being filtered and/or aggregated. This solution is often used for time series databases to control hot storage costs.

In this blog, I’ll use the GitHub events public dataset as the playground. For more information read about how to stream GitHub events into your own ADX cluster by reading the blog, “Exploring GitHub events with Azure Data Explorer.” I’ll describe how ADX users can take advantage of stored functions, the “.set-or-append” command, and the Microsoft Flow Azure Kusto connector. This will help you to create and update tables with filtered, down-sampled, and aggregated data for controlling storage costs. The following are steps which I performed.

Create a function for down-sampling and aggregation

The ADX demo11 cluster contains a database named GitHub. Since 2016, all events from GHArchive have been ingested into the GitHubEvent table and now total more than 1 billion records. Each GitHub event is represented in a single record with event-related information on the repository, author, comments, and more.

Initially, I created the stored function AggregateReposWeeklyActivity which counts the total number of events in every repository for a given week.

.create-or-alter function with (folder = "TimeSeries", docstring = "Aggregate Weekly Repos Activity”)
AggregateReposWeeklyActivity(StartTime:datetime)
{
let PeriodStart = startofweek(StartTime);
let Period = 7d;
GithubEvent
| where CreatedAt between(PeriodStart .. Period)
| summarize EventCount=count() by RepoName = tostring(Repo.name), StartDate=startofweek(CreatedAt)
| extend EndDate=endofweek(StartDate)
| project StartDate, EndDate, RepoName, EventCount
}

I can now use this function to generate a down-sampled dataset of the weekly repository activity. For example, using the AggregateReposWeeklyActivity function for the first week of 2017 results in a dataset of 867,115 records.

Using Kusto query, create a table with historic data

Since the original dataset starts in 2016, I formulated a program that creates a table named ReposWeeklyActivity and backfills it with weekly aggregated data from the GitHubEvent table. The query runs in parallel ingestion of weekly aggregated datasets using the “.set-or-append” command. The first ingestion operation also creates the table that holds the aggregated data.

.show table GithubEvent details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

.show table ReposWeeklyActivity details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

Code sample:
using Kusto.Data.Common;
using Kusto.Data.Net.Client;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace GitHubProcessing
{
class Program
{
static void Main(string[] args)
{
var clusterUrl = "https://demo11.westus.kusto.windows.net:443;Initial Catalog=GitHub;Fed=True";
using (var queryProvider = KustoClientFactory.CreateCslAdminProvider(clusterUrl))
{
Parallel.For(
0,
137,
new ParallelOptions() { MaxDegreeOfParallelism = 8 },
(i) =>
{
var startDate = new DateTime(2016, 01, 03, 0, 0, 0, 0, DateTimeKind.Utc) + TimeSpan.FromDays(7 * i);
var startDateAsCsl = CslDateTimeLiteral.AsCslString(startDate);
var command = $@"
.set-or-append ReposWeeklyActivity <|
AggregateReposWeeklyActivity({startDateAsCsl})";
queryProvider.ExecuteControlCommand(command);

Console.WriteLine($"Finished: start={startDate.ToUniversalTime()}");
});
}
}
}
}

Once the backfill is complete, the ReposWeeklyActivity table will contain 153 million records.

Configure weekly aggregation jobs using Microsoft Flow and Azure Kusto connector

Once the ReposWeeklyActivity table is created and filled with the historic data, we want to make sure it stays updated with new data appended every week. For that purpose, I created a flow in Microsoft Flow that leverages Azure Kusto connector to ingest aggregation data on a weekly basis. The flow is built of two simple steps:

Weekly trigger of Microsoft Flow.
Use of “.set-or-append” to ingest the aggregated data from the past week.

For additional information on using Microsoft Flow with Azure Data Explorer see the Azure Kusto Flow connector.

Start saving

To depict the cost saving potential of down-sampling, I’ve used “.show table <table name> details” command to compare the size of the original GitHubEvent table and the down-sampled table ReposWeeklyActivity.

.show table GithubEvent details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

.show table ReposWeeklyActivity details
| project TableName, SizeOnDiskGB=TotalExtentSize/pow(1024,3), TotalRowCount

The results, summarized in the table below, show that for the same time frame the down-sampled data is approximately 10 times smaller in record count and approximately 180 times smaller in storage size.

 

Original data

Down-sampled/aggregated data

Time span

2016-01-01 … 2018-09-26

2016-01-01 … 2018-09-26

Record count

1,048,961,967

153,234,107

Total size on disk (indexed and compressed)

725.2 GB

4.38 GB

Converting the cost savings potential to real savings can be performed in various ways. A combination of the different methods are usually most efficient in controlling costs.

Control cluster size and hot storage costs: Set different caching policies for the original data table and down-sampled table. For example, 30 days caching for the original data and two years for the down-sampled table. This configuration allows you to enjoy ADX first-rate performance for interactive exploration of raw data, and analyze activity trends over years. All while controlling cluster size and hot storage costs.
Control cold storage costs: Set different retention policies for the original data table and down-sampled table. For example, 30 days retention for the original data and two years for the down-sampled table. This configuration allows you to explore the raw data and analyze activity trends over years while controlling cold storage costs. On a different note, this configuration is also common for meeting privacy requirements as the raw data might contain user-identifiable information and the aggregated data is usually anonymous.
Use the down-sampled table for analysis: Running queries on the down-sampled table for time series trend analysis will consume less CPU and memory resources. In the example below, I compare the resource consumption of a typical query that calculates the total weekly activity across all repositories. The query statistics shows that analyzing weekly activity trends on the down-sampled dataset is approximately 17 times more efficient in CPU consumption and approximately eight times more efficient in memory consumption.

Running this query on the original GitHubEvent table consumes approximately 56 seconds of total CPU time and 176MB of memory.

The same calculation on the aggregated ReposWeeklyActivity table consumes only about three seconds of total CPU time and 16MB of memory.

Next steps

Azure Data Explorer leverages cloud elasticity to scale out to petabyte-size data, depict exceptional performance, and handle high query workloads. In this blog, I’ve described how to implement down-sampling and aggregation to control the costs associated with large datasets.

To find out more about Azure Data Explorer you can:

Try Azure Data Explorer in preview now.
Find pricing information for Azure Data Explorer.
Access documentation for Azure Data Explorer.

Quelle: Azure

Actuating mobility in the enterprise with new Azure Maps services and SDKs

The mobility space is at the forefront of the most complex challenges faced by cities and urban areas today. The movement of people and things is as much a driver of opportunity as it is an agent of chaos, aggravating existing challenges of traffic, pollution, and unbalanced livelihoods. Today, Azure Maps is continuing to expand the offerings of our platform, introducing a new set of capabilities in the form of SDKs and cloud-based services, to enable enterprises, partners, and cities to build the solutions that will help visualize, analyze, and optimize these mobility challenges.

The services we’re introducing are designed exclusively for the needs of the modern enterprise customer – powerful, real-time analytics and seamless cross-screen experiences, fortified by robust security services.

First, we’re officially moving the following services from public preview to general availability: Route Range {Isochrones}, Get Search Polygon, and Satellite and Hybrid Imagery. Furthermore, we’re introducing multiple new services. We’re enhancing our map canvas by introducing a stunning set of natural earth map tiles and an image compositor to make interaction with our maps more aesthetic, useful, and powerful. We’re also introducing Spatial Operations services that offer powerful analytics used by mobility applications and other industries today, as well as a new Android SDK and a Web SDK that equip Azure customers with the tools necessary to make smarter, faster, more informed decisions. And because privacy and security are top of mind, Azure Maps is now natively integrated with Azure Active Directory, making access to our services more secure while enabling roles and restrictions to our customers.

These services will provide Azure customers with the ability to offload map data processing and hosting costs all while getting the benefits of a rich set of maps and mapping services, securely with the fastest map data refresh rate available today. This refresh rate of data and services is bolstered by the recently announced partnership with TomTom who has committed to moving their map-making workloads to the Azure cloud significantly shortening the time from impetus to end user.

Android SDK (Preview)

While the Azure Maps Web SDK can work within a web control on mobile platforms, many developers prefer native support to interoperate with other native components and have these capabilities in native code… Java != JavaScript. In support of our customers who rely on applications running on Android, Azure Maps is distributing an Android SDK, complete with rendering maps and traffic, drawing, event handling, and using the variety of our map canvases. You can also connect to other Azure Maps services such as Search and Routing though the Azure Maps services APIs.

Spatial Operations (Preview)

Data analysis is central to the Internet of Things (IoT). Azure Maps Spatial Operations will take location information and analyze it on the fly to help inform our customers of ongoing events happening in time and space, enabling near real-time analysis and predictive modeling of events. Today Spatial Operations includes the following services:

Geofencing. A geofence is an “invisible fence” around a particular area. These “fences” exist in coordinates in the shape of customizable polygons and can be associated with temporal constraints so that fences are evaluated only when relevant. Furthermore, you can store a geofence Azure Maps Data Service (more on that below). With Azure Event Grid integration, you can create notifications whenever the position of an object changes with respect to a geofence – including entry, exit, or changing proximity to a geofence.

This has multiple applications. In transportation and logistics, it can be used to create alerts when cargo ships have arrived at waypoints along routes – critical for creating delivery alerts as well as for anticipating incidents of piracy. For drones, a geofence could enforce the limitation of where a drone is permitted to fly.

However, geofencing isn’t limited to transportation. In agriculture, it can enable real-time notifications when a herd has left a field. In construction, you can receive a notification when expensive equipment leaves a construction site that it shouldn’t be leaving or when it is parked in an area that may cause damage to equipment. It can also be used to warn site visitors when entering hazardous zones on sites, as implemented by Scottish IoT technology firm, Tagdat. In services, vending machines in college dorms often disappear. The servicing company can be notified if the machine leaves the premises. Geofencing is an amazingly powerful service to provide notifications and analytics when objects are moving. And, even when objects aren’t moving and should be!

Another example of a customer that is using this service is GovQA who state, “We use geofencing to identify when a requester is trying to submit a request which is outside the customer's pre-defined limits. The limits are defined by the city or county during the set up of the system. This allows the city to correctly handle the request and communicate with the requester appropriately using rules and configurations within GovQA."

Buffer. Build an area around points, lines, and polygons based on a given distance. Define the area in proximity of powerlines that should be kept clear from vegetation or create route buffers in fleet management for managing route deviations.

Closest Point. Returns the closest points between a base point and a given collection of points. It can be used to quickly identify the closest stores, charging stations, or in mobility scenarios, it could be used to identify the closest devices.

Great-circle distance. Returns the great-circle or shortest distance between two points on the surface of a sphere. In the context of drone delivery services, this API can be used to calculate the distance between an origin and destination as the crow flies taking into account the curvature of the earth so that a accurate time estimate for the delivery can be taken into account to optimize operations.

Point in Polygon. Returns a Boolean indicating whether the location is inside a given set of Polygon and MultiPolygon geometries. For example, the point in polygon API can be used to determine whether a home for sale is in the preferred area of customers.

Data Service (Preview)

Data is an imperative for maps, and bringing customer data closer to the Azure Maps service will reduce latency, increase productivity, and create powerful, new scenarios to light up in your applications. As such, Azure Maps will now have the ability for customers to upload and store up to 50MB of geospatial data for use with other Azure Maps services, such as geofencing or image composition. 

Azure Active Directory Integration (Preview)

Security and role-based access control have been of paramount concern for the modern enterprise. As such, we’re proud to announce that Azure Active Directory (AD) is now a core capability of Azure Maps. Use Azure AD to protect your customers’ information and implement secure access by providing role-based access control (RBAC).  Whether you have public applications or applications requiring a login, Azure AD and Azure Maps will support your security needs by authenticating your applications and Azure AD user(s). Additionally, this Azure AD implementation supports managed identities for Azure resources which provide Azure services (Azure App service, Azure Functions, Virtual Machines, etc.) with an automatically managed identity that can be authorized for access to Azure Maps services. 

Azure Maps Web SDK 2.0

Today we’re announcing a new module for accessing Azure Maps services to use in conjunction with the Azure Map Control. The new Service Module allows you to natively work directly with the Azure Maps services. This new module, plus the aforementioned adoption of Azure Active Directory, warranted the need for creating a new version and encapsulating them into a single Web SDK. Henceforth, we’ll containerize our services for web developers into the Azure Maps Web SDK 2.0. Note that the Azure Map Control 1.x will continue to be operational. However, we will innovate on 2.0 moving forward. The upgrade path for 1.x to 2.0 is quite simple by changing the version number! As a part of the new Azure Maps Web SDK 2.0 we’re also including some new client features:

Azure Active Directory (AAD). Azure Maps now natively supports Azure Active Directory to keep your access to Azure Maps secure. With native AAD integration, ensure your access is protected when your applications call Azure Maps.

Services Module. The new Services Module adds support for AAD and a much cleaner API interface for accessing Azure Maps services. It works with both the Web SDK and also in NodeJS. Being part of the Azure family of products, the Azure Maps Services Module was designed to align with an initiative to unify Azure SDKs and was required in order to add support for AAD.

Stroke gradients. There are times in location services development when you’d want to have gradient colors throughout the stroke of a line. Azure Maps Web SDK 2.0 now natively supports the ability to fill a line with a gradient of colors to show transition from one segment of a line to the next. As an example, these gradient lines can represent changes over time and distance, or different temperatures across a connected line of objects.

Shaded Relief map style. Below you’ll read about the new Shaded Relief Map Style. This beautiful, new map style is available immediately in the Azure Maps Web SDK 2.0.

Polygon Fill Patterns. Representations of polygons on a map can be done in a plethora of ways. In many scenarios, there will be a need to create polygons on the map. With the Azure Maps Web SDK 2.0 there is native control for shapes, borders, and fills. Now, we natively support patterns as a fill in addition to a single color fill. Patterns provide a unique way to highlight a specific area to really make it standout, especially if that area is surrounded by other color-shaded polygons. As an example, patterns can be used to show an area in transition, an area significantly different from other areas (such as financially, population, or land use), or areas highlighting facets of mobility such as no-fly zones or areas where permits are required.

Shaded Relief map style

Azure Maps comes complete with a few map styles including the road, dark gray, night, and satellite/hybrid. We’re adding a new map style – Shaded Relief – to complement the existing map styles. Shaded Relief is just that – an elegantly designed map canvas complete with the contours of the Earth. The Azure Maps Web SDK comes complete with the Shaded Relief canvas and all functions work seamlessly atop of it.

Image composition

Azure Maps is introducing a new image compositor that allows customers to render raster map images annotated with points, lines, and polygons. In many circumstances you can submit your request along with your respective point data to render a map image. For more complex implementations, you’ll want to use the map image compositor in conjunction with data stored in the aforementioned Azure Maps Data Service.

We always appreciate feedback from our customers. Feel free to comment below or post questions to Stack Overflow or our Azure Maps Feedback Forums.
Quelle: Azure

Get started quickly using templates in Azure Data Factory

Cloud data integration helps organizations integrate data of various forms and unify complex processes in a hybrid data environment. A number of times different organizations have similar data integration needs and require repeat business processes. Data Engineers or data developers in these organizations want to quickly get started with building data integration solutions and avoid building same workflows repeatedly. Today, we are announcing the support for templates in Azure Data Factory (ADF) to get started quickly with building data factory pipelines and improve developer productivity along with reducing development time for repeat processes. The template feature enables a ‘Template gallery’ for our customers that contains use-case based templates, data movement templates, SSIS templates or transformation templates that you can use to get hands-on with building your data factory pipelines.

Simply click Create pipeline from template on the Overview page or click +-> Pipeline from template on the Author page in your data factory UX to get started.

Select any template from the gallery and provide the necessary inputs to use the template. You can also read detailed description about the template or visualize the end to end data factory pipeline.

You can also create new connections to your data store or compute while providing the template inputs.

Once you click Use this template, you are taken to the template validation output. This guides you to fill in the required properties needed to publish and run the pipeline created from the template.

In addition to using out of box templates from the Template gallery, you might want to save your existing pipelines as templates as well. This might be required if different business units within your organization want to use the same pipeline but with different inputs. The templates feature in data factory allows you to save your existing pipelines as templates as well.

Ability to save your pipeline as a template requires you to enable GIT integration (Azure Dev Ops GIT or GitHub) in your data factory.

The template is then saved in your GIT repo under the templates folder.

The template is now visible to anyone who has access to your GIT repo. This template can be seen in the Templates section of the resource explorer.

You can also see the template under the My templates section in the template gallery.

Saving the template to the Git repository generates two files. It generates an ARM template along with a manifest file that is saved in your Git repo. The ARM template contains all information about your data factory pipeline, including pipeline activities, linked services, datasets etc. The manifest file contains information about the template description, template author, template tile icons and other metadata about the template.

All the ARM template and manifest files for the out of box official templates provided in the Template gallery can be seen in the official data factory GitHub location. In future, we will be working with our partners to come up with a certification process wherein anyone can submit a template that they want to enable in the Template gallery. The data factory team will certify the pull request corresponding to the submitted template and make the submitted template available in the Template gallery.

Find more information about the templates feature in data factory.

Our goal is to continue adding features to improve the usability of Data Factory tools. Get started building pipelines easily and quickly using Azure Data Factory. If you have any feature requests or want to provide feedback, please visit the Azure Data Factory forum.
Quelle: Azure

Azure.Source – Volume 69

Now in preview

Account failover now in public preview for Azure Storage

If you want to control storage account failover so that you can determine when storage account write access is required and the secondary replication state is understood, account failover for geo-redundant storage (GRS) enabled storage accounts is now available in preview. If the primary region for your geo-redundant storage account becomes unavailable for an extended period of time, you can force an account failover. As is the case with most previews, account failover should not be used with production workloads. There is no production SLA until the feature becomes generally available.

Azure Stream Analytics now supports Azure SQL Database as reference data input

Reference data is a dataset that is static or slow changing in nature that you can correlate with real-time data streams to augment the data. Stream Analytics leverages versioning of reference data to augment streaming data by the reference data that was valid at the time the event was generated. You can try using Azure SQL Database as a source of reference data input to your Stream Analytics job today. This feature is available for public preview in all Azure regions. This feature is also available in the latest release of Stream Analytics tools for Visual Studio.

Also in preview

Public preview: Azure Service Bus for Go
Public preview: Azure Service Bus for Python
Azure Database for PostgreSQL read replica preview
MongoDB to Azure Cosmos DB online migration is in preview

Now generally available

Individually great, collectively unmatched: Announcing updates to 3 great Azure Data Services

Azure Data Lake Storage Gen2 and Azure Data Explorer are now generally available, while Azure Data Factory Mapping Data Flow is available in preview. Azure Data Lake Storage (ADLS) combines the scalability, cost effectiveness, security model, and rich capabilities of Azure Blob Storage with a high-performance file system that is built for analytics and is compatible with the Hadoop Distributed File System. Azure Data Explorer (ADX) is a fast, fully managed data analytics service for real-time analysis on large volumes of streaming data. ADX is capable of querying 1 billion records in under a second with no modification of the data or metadata required. ADX also includes native connectors to Azure Data Lake Storage, Azure SQL Data Warehouse, and Power BI and comes with an intuitive query language so that customers can get insights in minutes. With Mapping Data Flow in Azure Data Factory, customers can visually design, build, and manage data transformation processes without learning Spark or having a deep understanding of their distributed infrastructure.

Azure Data Lake Storage Gen2 is now generally available
General Availability: Azure Data Explorer

An overview of Azure Data Explorer (ADX) | Azure Friday

Manoj Raheja joins Lara Rubbelke to demonstrate Azure Data Explorer (ADX) and provide an overview of the service from provisioning to querying. ADX is a fast, fully managed data analytics service for real-time analysis on large volumes of streaming data. It brings together a highly performant and scalable cloud analytics service with an intuitive query language to deliver near-instant insights.

Code-free data transformation at scale using Azure Data Factory | Azure Friday

Learn about the new code-free visual data transformation capabilities in Azure Data Factory as Gaurav Malhotra joins Lara Rubbelke to demonstrate how you can visually design, build, and manage data transformation processes without learning Spark or having a deep understanding of the distributed infrastructure.

Azure Cost Management now general availability for enterprise agreements and more!

Azure Cost Management is now generally available for all our Enterprise Agreement (EA) customers from within the Azure Portal. Azure Cost Management enables you to monitor all you spend through easy to use dashboards, create budgets, and optimize your cloud spend. This post also announces the public preview for web direct Pay-As-You-Go customers and Azure Government cloud. Azure Cost Management is available for free to all customers and partners to manage Azure costs. The Cloudyn portal will continue to be available to customers while we integrate all relevant functionality into native Azure Cost Management.

Azure Cost Management preview for Azure Government
Azure Cost Management preview for pay-as-you-go

Microsoft Healthcare Bot brings conversational AI to healthcare

The Microsoft Healthcare Bot is a white-labeled cloud service that powers conversational AI for healthcare. It’s designed to empower healthcare organizations to build and deploy compliant, AI-powered virtual health assistants and chatbots that help them put more information in the hands of their users, enable self-service, drive better outcomes, and reduce costs. The Microsoft Healthcare Bot is now available in the Azure Marketplace.

Also generally available

QnA Maker simplifies knowledge base management for your Q&A bot
Azure Automation: Update Azure Modules runbook is open source
Language improvements in Azure Stream Analytics are now generally available
Blob output partitioning in Azure Stream Analytics
TomTom is expanding its partnership with Microsoft
New Lsv2 Azure virtual machines (VMs) for big data, databases, and data warehousing are now generally available
Logic Apps is now available in US Gov Arizona region
SQL Server Migration Assistant support for Azure SQL Database Managed Instance
Support for Amazon RDS SQL Server to Azure SQL Database Managed Instance online migrations
M-series virtual machines (VMs) are now available in Australia Central 2 region
Azure DevTest Labs: Configure resource group control for your lab
Azure HDInsight available in US DoD East, US DoD Central, US Gov Texas
Azure Log Analytics is available in West US 2

News and updates

Analytics in Azure is up to 14x faster and costs 94% less than other cloud providers. Why go anywhere else?

Julia White, Corporate Vice President, Microsoft Azure covers how Azure provides the most comprehensive set of analytics services from data ingestion to storage to data warehousing to machine learning and BI. Each of these services have been finely tuned to provide industry leading performance, security and ease of use, at unmatched value. In a recent study by GigaOm, they found that Azure SQL Data Warehouse is now outperforming the competition up to 14x and up to 94% cheaper when compared with our competitors.

Configure resource group control for your Azure DevTest Lab

You now have the option to configure all your lab virtual machines (VMs) to be created in a single resource group. Learn how you can improve governance of your development and test environments by using Azure polices that you can apply at the resource group level. This enables you to use a script to either specify a new or existing resource group within your Azure subscription in which to create all your lab VMs. ARM environments created in your lab will continue to remain in their own resource groups and will not be affected by any option you select while working with this API.

Reserved instances now applicable to classic VMs, cloud services, and Dev/Test subscriptions

Two new Azure Reserved VM Instances’ (RI) features are now available that can provide you with additional savings and purchase controls. Classic VMs and Cloud Services users can now benefit from the RI discounts. In addition, Enterprise Dev/Test and Pay-As-You-Go Dev/Test subscriptions can now benefit from the RI discounts.

New connectors added to Azure Data Factory empowering richer insights

Azure Data Factory (ADF) is a fully-managed data integration service for analytic workloads in Azure, that empowers you to copy data from 80 plus data sources with a simple drag-and-drop experience. Also, with its flexible control flow, rich monitoring, and CI/CD capabilities you can operationalize and manage the ETL/ELT flows to meet your SLAs. A set of eight new Azure Data Factory connectors are now available that enable more scenarios and possibilities for your analytic workloads, including the ability to ingest data from Google Cloud Storage and Amazon S3.

Intelligent Edge support grows – Azure IoT Edge now available on virtual machines

Azure IoT Edge enables you to bring cloud intelligence to the edge and act immediately on real-time data. Azure IoT Edge already supports a variety of Linux and Windows operating systems as well as a spectrum of hardware from devices smaller than a Raspberry Pi to servers. Supporting IoT Edge in VMware vSphere offers you even more choice if you want to run AI on infrastructure you already own. VMware simplified the deployment process of Azure IoT Edge to VMs using VMware vSphere. Additionally, vSphere 6.7 and later provide passthrough support for Trusted Platform Module (TPM), allowing Azure IoT Edge to maintain its industry leading security framework by leveraging the hardware root of trust.

Completers in Azure PowerShell

Since version 3.0, PowerShell has supported applying argument completers to cmdlet parameters. We added argument completers to the Azure PowerShell modules that allow you to select valid parameter values without needing to make additional calls to Azure. These completers make the required calls to Azure to obtain the valid parameter values. Argument completers added include: Location, Resource Group Name, Resource Name, and Resource Id.

Simplify Always On availability group deployments on Azure VM with SQL VM CLI

Always On availability groups (AG) provide high availability and disaster recovery capabilities to your SQL Server database, whether on-premises, in the cloud, or a combination of both. Deploying an Always On Availability Group configuration for SQL Server on Azure VM is now possible with a few simple steps using the expanded capabilities enabled by SQL VM resource provider and Azure SQL VM CLI.

Help us shape new Azure migration capabilities: Sign up for early access!

We are enhancing Azure Migrate to deliver a unified and extensible migration experience with a goal of enabling customers and partners to plan, execute, and track their end to end migration journey using Azure Migrate. Delivering an integrated end-to-end migration experience that enables you to discover, assess, and migrate servers to Azure is the goal. Sign up for the private preview to try enhanced assessment and migration capabilities.

Microsoft Azure portal February 2019 update

In February, the Azure portal will bring you updates to several compute (IaaS) resources, the ability to export contents of lists of resources and resource groups as CSV files, an improvement to the layout of essential properties on overview pages, enhancements to the experience on recovery services pages, and expansions of setting options in Microsoft Intune.

Azure Monitor January 2019 updates

Azure Monitor now integrates the capabilities of Log Analytics and Application Insights for powerful, end-to-end monitoring of your applications. Learn what was added throughout the month of January to Application Insights, Log Analytics, Azure Monitor Workbooks, and Azure Metrics. In addition, Workbooks are now available in Azure Monitor for VMs.

Lighting up healthcare data with FHIR: Announcing the Azure API for FHIR

The healthcare industry is rapidly adopting the emerging standard HL7 FHIR®, or Fast Healthcare Interoperability Resources. This robust, extensible data model standardizes semantics and data exchange so all systems using FHIR can work together. Azure API for FHIR® enables rapid exchange of data in the FHIR format and is backed by a managed Platform-as-a Service (PaaS) offering in the cloud. Simplify data management with a single, consistent solution for protected health information.

Azure DevOps Projects supporting Azure Cosmos DB and Azure Functions

In the latest deployment of Azure DevOps Projects now available to all customers, we have added support for Azure Cosmos DB and Azure Functions as target destinations for your application. This builds on the existing Azure App Service, Azure SQL Database, and Azure Kubernetes Service (AKS) support.

Find out when your virtual machine hardware is degraded with Scheduled Events

Scheduled Events will now be triggered when Azure predicts that hardware issues will require a redeployment to healthy hardware in the near future, and provide a time window when Azure will redeploy the VMs to healthy hardware if a live migration was not possible. You can initiate the redeployment of your VMs ahead of Azure automatically doing it.

Additional updates

Data Migration Assistant support for target readiness assessment for Azure SQL DB Managed Instance
NuGet, npm, and other Artifacts tasks support proxies – Sprint 147 Update
Ev3 and ESv3 series VMs are available in Azure HDInsight
Azure StorSimple 5000/7000 series will no longer be supported starting July 9, 2019

Technical content

Processing trillions of events per day with Apache Kafka on Azure

The Siphon team shares their experiences and learnings from running one of world’s largest Kafka deployments. Besides underlying infrastructure considerations, they discuss several tunable Kafka broker and client configurations that affect message throughput, latency and durability. After running hundreds of experiments, they standardized the Kafka configurations required to achieve maximum utilization for various production use cases. They also explain how to tune a Kafka cluster to configure producers, brokers and consumers for the best possible performance.

Best practices to consider before deploying a network virtual appliance

A network virtual appliance (NVA) is a virtual appliance primarily focused on network functions virtualization. A typical network virtual appliance involves various layers four to seven functions like firewall, WAN optimizer, application delivery controllers, routers, load balancers, IDS/IPS, proxies, SD-WAN edge, and more. Common best practices include: Azure-accelerated networking support, multi-NIC support, using Azure Load Balancer for a high availability (HA) port load balancing rule, and support for Virtual Machine Scale Sets (VMSS).

Build your own deep learning models on Azure Data Science Virtual Machines

The Practical Deep Learning for Coders 2019 course from fast.ai helps software developers start building their own state-of-the-art deep learning models. Developers who complete this course will become proficient in deep learning techniques in multiple domains including computer vision, natural language processing, recommender algorithms, and tabular data. Learn how you can run this course using the Azure Data Science Virtual Machines (DSVM).

Performance best practices for using Azure Database for PostgreSQL – Connection Pooling

This blog is a continuation of a series of blog posts to share best practices for improving performance and scale when using Azure Database for PostgreSQL service. This post focuses on the benefits of using connection pooling and provides recommendations to improve connection resiliency, performance, and scalability of applications running on Azure Database for PostgreSQL.

Azure Event Grid: The Whole Story

As promised, Jeremy Likness, a Microsoft Cloud Advocate, takes a thorough look at the serverless backbone for all your event-driven computing needs: Azure Event Grid, a single service for managing routing of all events from any source to any destination.

Pentesting Azure — Thoughts on Security in Cloud Computing

Tanya Janca, a Microsoft Cloud Advocate, shares a list of her thoughts on penetration testing (pentesting) Azure applications as she sets out to read Pentesting Azure Applications by Matt Burrough. She promises a future post once she finishes reading the book.

Azure shows

Episode 265 – Azure DevOps Server | The Azure Podcast

Cynthia and Evan talk to Jamie Cool, Director of Program Management at Microsoft, who gives us all the details and potential use-cases for the Azure DevOps Server in your organization.

HTML5 audio not supported

An overview of Azure Blueprints | Azure Friday

Alex Frankel joins Scott Hanselman to discuss Azure Blueprints. Environment creation can be a long and error prone process. Azure Blueprints helps you deploy and update cloud environments in a repeatable manner using composable artifacts such as policies, role-based access control, and Azure Resource Manager templates.

Enhanced monitoring capabilities and tags/annotations in Azure Data Factory | Azure Friday

Gaurav Malhotra and Scott Hanselman explore tagging support and enhanced monitoring capabilities, including dashboards and improved debugging support in Azure Data Factory. Data integration is complex and the ability to monitor your data factory pipelines is a key requirement for dev ops personnel inside an enterprise. Now, you can tag/annotate your data factory pipelines to monitor all the pipeline executions with that particular tag. In addition, Data Factory visual tools provide dashboards to monitor your pipelines and an ability to monitor your pipeline execution by the Integration Runtime (IR) upon which the activities execute.

Logic Apps Connector to Ethereum Blockchain Networks | Block Talk

This episode provides an overview of how to use our serverless Ethereum Connector to transform smart contracts into an automated, visual workflow using the rich Azure Logic Apps Connectors ecosystem. We introduce the core concepts of Logic Apps and demonstrate a sample workflow triggered by a Solidity event, including how to read smart contract properties and write them to Azure Blob storage.

Azure IoT Device Agent for Windows | Internet of Things Show

Customers across industries, whether in an industrial setting or retail environment, are looking for ways to remotely provision and manage their IoT devices. Direct device access may not always be feasible when IoT devices are out in the field or on the factory floor. Microsoft Azure IoT Device Agent enables operators to configure, monitor and manage their devices remotely from their Azure dashboard. In this episode of the #IoTShow you will get an overview of Microsoft Azure IoT Device Agent with a demo.

Visual Studio for Mac: Publish to Azure | Visual Studio Toolbox

In this video, Cody Beyer will demonstrate how to log in and publish a web project to Azure. Join him and learn how to get the most out of Visual Studio for Mac by combining it with the power of Azure.

How to manage your Kubernetes clusters | Kubernetes Best Practices Series

Learn best practices on how to manage your Kubernetes clusters from field experts in this episode of the Kubernetes Best Practices Series. In this intermediate-level deep dive, you will learn about cluster management and multi-tenancy in Kubernetes.

How to add the Azure Cloud Shell to Visual Studio Code | Azure Tips and Tricks

In this edition of Azure Tips and Tricks, learn how to add the Azure Cloud Shell to Visual Studio Code. To add the Azure Cloud Shell, make sure you have the “Azure Account” extension installed in Visual Studio Code.

Overview of VS Code Extensions for Azure Government

In this episode of the Azure Government video series, Steve Michelotti sits down with Yujin Hong, Program Manager on the Azure Government Engineering team, to discuss many of the incredible VS Code Extensions for Azure. VS Code has quickly become the most popular editor in the world and there are numerous reasons for this, but one of the key reasons is VS Code’s extensibility. There are numerous VS Code Extensions available for Azure, and now these same extensions can be utilized for Azure Government! In this demo-heavy video, Yujin will show the unified authentication experience that enable all these cool extensions to seamlessly authenticate to Azure Government. She then walks through several demos that show how easy these extensions make it for developers to work with Storage, App Service, Cosmos DB, and Azure Functions in Azure Government. If you’re a developer who works with Azure Government, this video is for you!

Modern Data Warehouse overview | Azure SQL Data Warehouse

How do you think about building out your data pipeline in Azure? Discover how the Modern Data Warehouse solution pattern can modernize your data infrastructure in the cloud and enable new business scenarios. This is the first episode of an 8-part series on Azure SQL Data Warehouse.

Paul Stovell on Octopus Deploy – Episode 22 | The Azure DevOps Podcast

Paul Stovell, the founder and CEO of Octopus Deploy, joins the podcast today. Paul is an expert on all things automated deployment and Cloud operations. He started Octopus Deploy back in 2011, but prior to that, he worked as a consultant for about five years. Octopus Deploy is a pretty major player in the market. Their mission? To do automated deployments really, really well. Today, it helps over 20,000 customers automate their deployments, and employs 40 brilliant people. It can be integrated with Azure DevOps services and many other build services. On this week’s episode, Paul talks about his career journey and what led him to create Octopus Deploy; his accomplishments, goals, and visions for Octopus Deploy; which build servers integrate best with Octopus Deploy; his tips and tricks for how to best utilize it; and his vision for the future of DevOps.

HTML5 audio not supported

Events

Cloud Commercial Communities webinar and podcast newsletter–February 2019

In this Cloud Commercial Communities monthly webinar and podcast update, get links to both the upcoming links for February and links to webinars and podcasts from January. Each month the team focuses on core programs, updates, trends, and technologies that Microsoft partners and customers need to know to increase success using Azure and Dynamics. While much of the content is available for on-demand consumption, attending live webinars enables you to participate in Q&A with the webinar hosts.

Customers, partners, and industries

Modernizing payment management for online merchants

Learn about Guru, which is Newgen's fully-integrated portal that enables merchants to have a complete view of their payments, generate reports, capture/void transactions, and perform refunds. Guru is a fully cloud-based solution hosted completely on Microsoft Azure. It is a fully-managed SaaS solution that comes as a value addition with Newgen's Payment Gateway—a cutting edge payment technology for merchants.

Azure IoT drives next-wave innovation in infrastructure and energy

Last week at the DistribuTECH conference in New Orleans, Azure IoT partners showcased new solutions that bring the next level of “smart” to our grids. We invited eight partners to the Microsoft booth to demonstrate their approach to modernizing infrastructure, and how Azure IoT dramatically accelerates time to results. Learn how each partner showed new use cases for utilities, infrastructure, and cities that take advantage of cloud, AI, and IoT. With solutions that take full advantage of the intelligent cloud and intelligent edge, we continue to demonstrate how cloud, IoT, and AI have the power to drastically transform every industry. Smart grids will drive efficiencies to power and utility companies, grid operators, and energy prosumers.

Advancing tactical edge scenarios with Dell EMC Tactical Microsoft Azure Stack and Azure Data Box family

Microsoft, working with partners like Dell EMC, shared new capabilities that continue to deliver the power of the intelligent cloud and intelligent edge to government customers and their partners. Last year, we announced Azure Stack availability for Azure Government customers. With Azure Stack for Azure Government, agencies can efficiently modernize their on-premises legacy applications that are not ready or a fit for the public cloud due to cyber defense concerns, regulations, or other requirements. Data Box products help agencies to migrate large amounts of data, for example backup, archive or big data analytics, to Azure when they are limited by time, network availability, or costs.

Investing in our partners’ success

While Microsoft has long been a partner-oriented organization, some things are different with cloud. Specifically, partners need Microsoft to be more than just a great technology provider, you need us to be a trusted business partner. This requires long-term commitment and the ability to continually adapt and innovate as the market shifts. This has been, and continues to be, our commitment. Our partnership philosophy is grounded in the foundation that we can only deliver on our mission if there is a strong and successful ecosystem around us. Julia White, Corporate Vice President, Microsoft Azure,  highlights our key partner-oriented investments and some of the resources to help our partners successfully grow their businesses.

Azure Marketplace new offers – Volume 31

The Azure Marketplace is the premier destination for all your software needs – certified and optimized to run on Azure. Find, try, purchase, and provision applications & services from hundreds of leading software providers. You can also connect with Gold and Silver Microsoft Cloud Competency partners to help your adoption of Azure. In the first half of January we published 67 new offers.

Colfax amplifies the power of its ESAB product portfolio with IoT

With the evolution of the Internet of Things (IoT), Colfax saw an opportunity to transform its businesses. What was unique about Colfax’s IoT initiative – named Data Driven Advantage (DDA) –  was their vision of enabling customers to leverage the extensive ESAB portfolio. They selected PTC Thingworx for Azure and the Microsoft Azure IoT platform. With ESAB Digital Solutions, customers now have data to understand how processes, labor, and material contribute to the cost of each part.

This Week in Azure – 8 February 2019 | A Cloud Guru – This Week in Azure

This time on Azure This Week, Lars talks about Lsv2-series VMs are now in general availability, there’s a new version of the Microsoft Threat Modelling Tool out, Azure trivia is back every Monday, and come meet the team from A Cloud Guru next week at the Ignite Tour in Sydney.

Quelle: Azure

Colfax amplifies the power of its ESAB product portfolio with IoT

If you’ve welded two pieces of metal together, chances are you’ve used equipment from ESAB, a business unit of Colfax Corporation. Started in Sweden in 1904, ESAB was acquired by Colfax in 2012, and offers a broad portfolio of welding, cutting, and gas management equipment to customers in virtually every industry across the globe including agriculture, building construction, energy, light and heavy manufacturing, transportation, and even medical and hospitals. Wherever things are made, ESAB is there.

With the evolution of the Internet of Things (IoT), Colfax saw an opportunity to transform its businesses. What was unique about Colfax’s IoT initiative – named Data Driven Advantage (DDA) –  was their vision of enabling customers to leverage the extensive ESAB portfolio.

Leveraging synergies of a broad portfolio

Many of ESAB’s customers manufacture highly configurable products like tractors, mining equipment, wind towers, and agricultural feed tanks. Imagine the complex process steps involved. First, hundreds of metal pieces are cut and placed on shelves. Then the parts move through the factory. At each step, there is labor required to set up a machine, fuse or cut parts, check for quality, and prepare for the next job. Workers grind metal, weld parts, and refill filler metal. Welds are documented extensively, often manually, for traceability in critical applications like shipbuilding where the paperwork can literally fill a shipping container.

“If customers can leverage the broad ESAB portfolio of welding equipment, power supplies, filler metals, welding tips, mechanized cutters – even helmets, gloves, and protective gear – in a connected way, they can gain incredible insights across their entire manufacturing processes,” explained Ryan Cahalane, Vice President of Digital Growth, Colfax.

Challenges on the IoT journey

Similarly to other companies, Colfax ran into challenges on its IoT journey, including educating executives, dealing with legacy products, ensuring their IoT solution integrated with existing distribution channels and sales teams, and a lack of coordination and inconsistent technology choices across business units.

After years in pilot purgatory, the company accelerated their digital growth success when they refocused on their core differentiation, gaining efficiencies from having a common technology platform across businesses, and leveraging ESAB’s deep application expertise and wide portfolio to unlock value for their customers. As part of this refocus, the company exited the business of developing a custom IoT platform. They selected PTC Thingworx for Azure and the Microsoft Azure IoT platform. “Deploying ThingWorx for Azure provides businesses like Colfax with a much more holistic offering around digital transformation,” said Ron Salvador, Senior Director at PTC.

The re-platforming started in December 2018, and by the HMI 2018 event, ESAB had a field-ready prototype of WeldCloud™ based on Thingworx and Azure that they began testing with customers. By FabTech in the fall of 2018, they had expanded the portfolio to include cutting applications with CutCloud™ and began rolling out the common Thingworx/Azure reference architecture to other business units’ products like orbital welding and gas monitoring.

Customers gain immediate benefits

Customers have seen immediate benefits. “With insights from WeldCloud™, one of our customers realized that they were using one station’s equipment less than 10 percent of the available time, yet it bottlenecked the overall process,” said Ludvig Enlund, GM and head of ESAB’s DDA initiative. “Plus, they were staffing it with expensive labor where the work didn’t require that level of expertise, meanwhile taking away from other more critical processes where higher skill was required. With this awareness, they rebalanced their processes, adjusted labor, and helped gain significant throughput on a line where every product made can be sold.”

With ESAB Digital Solutions, customers now have data to understand how processes, labor, and material contribute to the cost of each part. “In a job shop environment, customers are able to improve their quoting process, increase profit, and win even more business,” said Enlund. Not only do ESAB Digital Solutions like WeldCloud™ and CutCloud™ make otherwise impossible productivity improvements possible, they also give everyone involved in the operation the chance to elevate their performance with data. Some examples include:

•    Operations managers can identify new productivity drivers
•    Quality engineers can trace a defective weld and determine the correct repairs and how to prevent future issues
•    Service technicians can be proactively alerted if equipment has an anomaly or breaks down, allowing for preventative and even predictive maintenance
•    Welding engineers can use data to more quickly set up new test welding processes for new applications

Additionally, ESAB’s approach to digital transformation parallels their collaborative approach to customer preferences, where their broad portfolio provides value even where competitive equipment may be preferred. “Welding may be a science, but its practitioners consider good welding an art and become very attached to the tools they know,” said Enlund. “With the Universal Connector interface and Weld Quality (HKS), WeldCloud™ can easily work with other brands, and deliver most of the same value.”

Partnering to deliver customer value

As ESAB moves forward, it plans to continue focusing on partnerships that deliver customer value, having seen success from close relationships with Microsoft and PTC. “We are taking a page out of the Microsoft playbook,” said Cahalane. “The world is a different place. It’s moving fast. Competitors can become collaborators, and collaboration is key.” The company will continue to expand its Data Driven Advantage initiative across its large portfolio and is now piloting the same common technology platform in its own operations. They expect that the power of standard tools, consistent data models, and modern analytics will increase their own productivity, improve quality, and potentially enable new insights by closing the loop between products in the field and the operations that make them.
Quelle: Azure

Azure Marketplace new offers – Volume 31

We continue to expand the Azure Marketplace ecosystem. From January 1 to January 15, 2019, 67 new offers successfully met the onboarding criteria and went live. See details of the new offers below:

Virtual machines

Akumo Software: Akumo Software's platform extends datacenter environments between virtualized or cloud-based infrastructure. It provides a consistent and simple way to cost-effectively manage an on-demand datacenter.

BlogEngine.NET on Windows Server 2016: BlogEngine.NET is a lightweight, simple, user-friendly blog engine that can be an excellent alternative to WordPress. Easy to modify and extend, it is specifically designed for .NET developers.

BlogEngine.NET on Windows Server 2019: BlogEngine.NET is a lightweight, simple, user-friendly blog engine that can be an excellent alternative to WordPress. Easy to modify and extend, it is specifically designed for .NET developers.

Conductor4SQL Central Server: This virtual machine comes with all the components required for using Conductor4SQL, including Windows Server 2016, Microsoft SQL Server 2017, Microsoft Power BI Desktop, and Conductor4SQL.

Dell EMC NetWorker Virtual Edition 18.2: Dell EMC NetWorker software provides fast, efficient backup and recovery for enterprise applications and databases.

F5 BIG-IP Cloud Edition: This edition is comprised of per-app VEs and BIG-IQ centralized management. The former provides intelligent traffic management and web application firewall security, while the latter delivers deployment automation, management, and visibility.

GnuCash on Windows Server 2016: GnuCash helps you to track your bank accounts, income, stocks, expenditures, and more. Users have the freedom to run, copy, distribute, study, change, and improve the software. It also works on mobile operating systems.

GnuCash on Windows Server 2019: GnuCash helps you to track your bank accounts, income, stocks, expenditures, and more. Users have the freedom to run, copy, distribute, study, change, and improve the software. It also works on mobile operating systems.

IIS on Windows Server 2019: Key features of IIS on Windows Server 2019 include wildcard host headers, IIS administration PowerShell cmdlets, and improved coalescing of connections to deliver an uninterrupted and properly encrypted browsing experience.

Incorta Free Trial: With Incorta’s Direct Data Mapping engine, you get real-time aggregation of large, complex business data without needing a data warehouse.

InterSystems IRIS Community Edition: InterSystems IRIS is a complete data platform that provides developers the freedom to choose the language and data model best suited to rapidly develop their applications.

InterSystems IRIS Express Edition: InterSystems IRIS is a complete data platform that provides developers the freedom to choose the language and data model best suited to rapidly develop their applications. See additional pricing options for this edition.

Neo4j Enterprise VM Version 3.5: Neo4j's graph database platform helps organizations make sense of their data by revealing how people, processes, locations, and systems are interrelated. This approach powers apps tackling AI, fraud detection, master data, and recommendations.

NetScaler MA Service Agent 13.0: The NetScaler MA Service agent software works as an intermediary between the NetScaler Management and Analytics Service and the NetScaler instances within Microsoft Azure.

Nginx on Windows Server 2016: Features of Nginx on Windows Server 2016 include reverse proxy with caching, IPv6, load balancing, FastCGI support with caching, WebSockets, TLS/SSL with SNI, and the handling of static files, index files, and automatic indexing.

OpenCart on Windows Server 2016: Written in PHP, OpenCart is a free, open-source e-commerce platform available under the GNU General Public License, which allows end users to modify the software.

OpenCart on Windows Server 2019: Written in PHP, OpenCart is a free, open-source e-commerce platform available under the GNU General Public License, which allows end users to modify the software.

Puppet Enterprise: Puppet Enterprise lets you automate the entire lifecycle of your Azure infrastructure simply and securely, from initial provisioning through application deployment.

Pyramid 2018 – Windows Server: Pyramid 2018 lets business users do high-end analytics and data science on any browser or device without needing IT help. It's the next generation of self-service analytics with governance.

SQL 2017 Enterprise Edition w/ ER/Builder: With the ER/Builder data modeler for SQL 2017 on Windows Server 2016, you can manage an unlimited number of tables. You can also create an index, triggers, keys, stored procedures, views, generators, and domains.

SQL Server 2017 Standard Edition w/ ER/Builder: With the ER/Builder data modeler for SQL 2017 on Windows Server 2016, you can manage an unlimited number of tables. You can also create an index, triggers, keys, stored procedures, views, generators, and domains.

SQL Server 2017 Web Edition w/ ER/Builder: With the ER/Builder data modeler for SQL 2017 on Windows Server 2016, you can manage an unlimited number of tables. You can also create an index, triggers, keys, stored procedures, views, generators, and domains.

Strokk Webservices Demo: Wherever a password is used in a web form or an internal application, that piece of knowledge-based authentication can be hardened almost transparently with a behavioral biometrics second factor called keystroke dynamics.

Untangle NG Firewall: Use NG Firewall to connect remote locations and ensure safety, reliability, and performance while providing protection for your data, applications, and users.

Varnish Enterprise 6: Varnish Enterprise (VE), previously known as Varnish Plus, is our commercial/enterprise version of the popular open-source HTTP engine/reverse HTTP proxy Varnish Cache (VC).

VyOS 1.2 LTS: VyOS is a Linux-based open-source network operating system for routers and firewalls.

Windows Virtual Desktop: With Windows Virtual Desktop, Microsoft Office and Windows can be deployed and scaled on Azure in a few moments, including compliance and built-in security.

WordPress on Windows Server 2016: Quickly deploy WordPress on Windows 2016 with built-in MySql and phpMyAdmin. Host as many websites or applications as you need.

Xeams on CentOS: Get this secure and powerful mail server with a strong junk-filtering engine on CentOS. Xeams Community Edition is available as a free software supporting multiple platforms and all mail servers with smart-host functionality.

Xeams on Ubuntu: Get this secure and powerful mail server with a strong junk-filtering engine on Ubuntu. Xeams Community Edition is available as a free software supporting multiple platforms and all mail servers with smart-host functionality.

Xeams on Windows Server 2016: Get this secure and powerful mail server with a strong junk-filtering engine on Windows Server 2016. Xeams Community Edition is available as a free software supporting multiple platforms and all mail servers with smart-host functionality.

Xeams on Windows Server 2019: Get this secure and powerful mail server with a strong junk-filtering engine on Windows Server 2019. Xeams Community Edition is available as a free software supporting multiple platforms and all mail servers with smart-host functionality.

Web applications

Aggregion Blockchain Node: Aggregion operates a blockchain ecosystem enabling major copyright holders to fully control their global end-to-end distribution networks and licensing of digital content. Microsoft Azure products enhance the Aggregion blockchain platform.

Archive One: Archive One is a document management system designed to help document administrators classify, store, secure, search for, and retrieve essential company records. Make compliance and audits easy with Archive One.

Drupal with Azure Database for MariaDB: This solution uses a virtual machine for the application front end and the Azure Database for MariaDB service for the application data. Drupal is an open-source content management system used to create websites and apps.

Lavelle Networks ScaleAon SD-WAN: Lavelle Networks ScaleAon SD-WAN is a hybrid WAN. ScaleAon SD-WAN accelerates cloud adoption for enterprises by seamlessly extending the wide area network (WAN) across physical and virtual resources.

Lightning Network for Azure: This distribution provides a virtual machine instance that runs Bitcoin (btcd), Litecoin (ltcd), and Lightning Network (lnd or c-lightning). You can also run BTCPayServer as a sample application for the node network.

MariaDB Galera Cluster: MariaDB Galera is a multi-master database cluster solution for synchronous replication and high availability. This solution uses multiple virtual machines to replicate your data in a configurable number of nodes.

Mediant VE Session Border Controller (SBC): Enable Microsoft Teams Direct Routing or connect SIP trunks to Skype for Business Server. AudioCodes’ Mediant session border controllers make deployment easier and help users set up multi-SBC network interfaces.

Spanning Backup For Office 365: Spanning Backup for Office 365 provides automatic daily backup and recovery for Office 365 mail, calendars, OneDrive, and SharePoint.

Surge Identity (SaaS): Surge Identity is a cloud-based identity solution that enables secure sign-in using trusted identity and social providers, and it secures app-to-app communication using the latest industry security standards.

Tidal Migrations -Premium Insights for Source Code: Tidal Migrations provides your team with a simple, fast, and cost-effective cloud migration management solution. This add-on empowers your team with actionable insights on the apps you plan to refactor or re-platform to Azure.

TimeXtender Discovery Hub: This virtual machine runs Windows Server 2016 and the TimeXtender Discovery Hub. The Discovery Hub application server for Azure allows customers to build, deploy, and manage an enterprise-grade analytical architecture.

Vnomic Management for SAP Workloads: Select your SAP HANA workload requirements without worrying about underlying technical details. Vnomic will automatically compute and provision a complete and validated SAP HANA workload and deliver it on Azure in minutes.

Consulting services

2008 Windows/SQL End of Support Workshop – 2 days: End of support is looming for Windows 2008 and SQL Server 2008, and Piksel Group's workshop is here to help you understand your options and create an action plan.

2-Hour Azure Migration Briefing: This briefing by Flat Rock Technology will provide high-level information on what it takes to migrate to the cloud and to Microsoft Azure in particular.

Airlines ChatBot: 3 week implementation: This is a conversational AI implementation for airlines over Amadeus/Sabre supporting multiple channels and covering flight booking, status, disruption notification, check-in, boarding passes, FAQs, and more.

Azure Analytics 5-Day Readiness Assessment: Pythian Kick Analytics-as-a-Service puts the power of data analytics in the hands of your business users and solves the data silo problem. This five-day assessment is for customers in Canada.

Azure Analytics 5-Day Readiness Assessment (UK): Pythian Kick Analytics-as-a-Service puts the power of data analytics in the hands of your business users and solves the data silo problem. This five-day assessment is for U.K. customers.

Azure Analytics 5-Day Readiness Assessment (USA): Pythian Kick Analytics-as-a-Service puts the power of data analytics in the hands of your business users and solves the data silo problem. This five-day assessment is for U.S. customers.

Azure Design Assessment: 4-Day Assessment: CDW will review your Microsoft Azure environment to verify configuration and provide recommendations according to best practices.

Azure Governance Workshop: 5-Day Workshop: CDW will provide an in-depth look at the people, processes, and technology currently in place and document a governance plan that enables IT professionals to effectively support business needs.

Azure Jumpstart: 3-Day Implementation: CDW will help your organization choose and implement virtual networking technology. Learn the best approaches to deploying virtual machines, including the associated cloud services and storage accounts.

Azure migration & transformation two-day workshop: In this Azure migration and transformation workshop, Piksel Group will identify candidate cloud services and the benefits of moving to Azure.

Azure migration and transformation briefing (3h): Select from a range of Azure migration and transformation consultancy, implementation, and managed services, starting with a cloud briefing and initial cloud readiness assessment.

Azure Migration Assessment: 2-Day Assessment: CDW will work with you to deploy an assessment tool in your environment, ensure the tool is configured properly, run the tool, and help review and interpret the results.

Azure transformation five-day proof of concept: This Azure migration and transformation briefing, assessment, planning, and proof-of-concept activity by Piksel Group will identify and validate candidate services and benefits.

Cloud Operations & Monitoring: 3-day Assessment: Objektkultur Software will plot a strategy for your migration to the cloud and will support you in your change management process, enabling a conversion that integrates into your system landscape.

Connecting with ExpressRoute: 2 Day Implementation: Your organization will first need to engage with a WAN provider that supports ExpressRoute connectivity. CDW will configure the virtual network and gateway and will assist in establishing an ExpressRoute connection.

Cyber Security PEN Testing: 4 Week Assessment: Networks come under attack every day, and these attacks can disrupt business, create chaos, and cause reputational damage. A penetration test helps you understand how threat actors might penetrate your network.

Envisioning AI for IoT data: 2-day Workshop: This offer by TheDataTeam is for an AI envisioning workshop conducted at the client's site for discovering use cases that are of immense business value and solvable using Azure and TheDataTeam's Intellegion platform.

EOS Migration Pilot: 8-Wk Implementation: This is a pilot migration of legacy Windows Server workloads to Azure using both discovery and containerization tooling.

Leadership Development Solution: The Leadership Development Solution helps K-12 organizations make informed decisions. It provides strategic education services along with an Azure data warehouse and visualizations to improve the leadership placement process.

Migrate to Azure at 20% of Consumption 8-wk Impl: Migrate your workloads and apps to Azure at just 20 percent of your Azure consumption for the first year. This package by NetEnrich combines our tools expertise with our knowledge of datacenter and app migrations.

SAP on Azure / QAS Migration Service: 2-Week POC: This proof of concept involves free Azure consumption credits and a migration service offered by MSR IT Services covering SAP QAS/Sandbox landscapes.

SpotLITE Discovery for Azure MSP: 2 WKS Assessment: Green House Data’s SpotLITE discovery process is designed to determine the overall health of your IT systems and outline a plan to improve operational performance and include Azure.

SSO Using ADFS and ADConnect: 3-Day Implementation: Integrate your on-premises directories with Azure Active Directory to provide a common identity for access to both cloud and on-premises resources, simplifying things for your end users.

Quelle: Azure

Performance best practices for using Azure Database for PostgreSQL – Connection Pooling

This blog is a continuation of a series of blog posts to share best practices for improving performance and scale when using Azure Database for PostgreSQL service. In this post, we will focus on the benefits of using connection pooling and share our recommendations to improve connection resiliency, performance, and scalability of applications running on Azure Database for PostgreSQL. If you have not read the previous performance best practice blogs in the series, we would highly recommend reading the following blog posts to learn, understand, and adopt the recommended best practices for using Azure Database for PostgreSQL service.

Performance best practices for using Azure Database for PostgreSQL
Performance updates and tuning best practices for using Azure Database for PostgreSQL
Performance troubleshooting best practices using Azure Database for PostgreSQL features

In PostgreSQL, establishing a connection is an expensive operation. This is attributed to the fact that each new connection to the PostgreSQL requires forking of the OS process and a new memory allocation for the connection. As a result, transactional applications frequently opening and closing the connections at the end of transactions can experience higher connection latency, resulting in lower database throughput (transactions per second) and overall higher application latency. It is therefore recommended to leverage connection pooling when designing applications using Azure Database for PostgreSQL. This significantly reduces connection latency by reusing existing connections and enables higher database throughput (transactions per second) on the server. With connection pooling, a fixed set of connections are established at the startup time and maintained. This also helps reduce the memory fragmentation on the server that is caused by the dynamic new connections established on the database server.

The connection pooling can be configured on the application side if the app framework or database driver supports it. If that is not supported, the other recommended option is to leverage a proxy connection pooler service like PgBouncer or Pgpool running outside the application and connecting to the database server. Both PgBouncer and Pgpool are developed by the community and can be used with Azure Database for PostgreSQL. As we continue on, we will focus our conversation on PgBouncer in the context of real user experiences.

PgBouncer is a lightweight connection pooler that can be installed on the virtual machine (VM) running the application. The application connects to the PgBouncer proxy service running locally on the VM while PgBouncer service in-turn connects to the Azure Database for PostgreSQL service using the credentials and configuration settings specified in the pgbouncer.ini file. The maximum number of connections and default pool size can be defined in the configuration settings in pgbouncer.ini.

If your application is containerized and running on Azure Kubernetes Service (AKS), you can run PgBouncer as a sidecar proxy. As part of our commitment to provide native integration of best in class OSS databases with Azure’s industry leading ecosystem, we have published a PgBouncer sidecar proxy image in Microsoft container registry. PgBouncer sidecar runs with the application container in the same pod in AKS and provides connection pooling proxy to Azure Database for PostgreSQL. If the application container fails over or restarts, the sidecar container will follow thereby providing high availability with connection resiliency and predictable performance. Visit the docker hub page to learn more on how to access and use this image. For best practices around development with Azure Kubernetes Services, we would recommend to follow the documentation, “Connecting Azure Kubernetes Service and Azure Database for PostgreSQL.”

To give some estimates of the performance improvement when using PgBouncer for connection pooling with Azure Database for PostgreSQL, we ran a simple performance benchmark test with pgbench. pgbench provides a configuration setting to create new connection for every transaction so we leveraged that to measure the impact of connection latency on throughput of the application. The following are the observations with A/B testing comparing throughput with standard pgbench benchmark testing with and without PgBouncer. In the test, we ran pgbench with scale factor of 5 against Azure Database for PostgreSQL running on general purpose tier with 2 vCores (GP_Gen5_2). The only variable during the tests was PgBouncer. With PgBouncer, the throughput improved 4x times as shown below while connection latency was reduced by 40 percent.

PgBouncer, with its built-in retry logic can further ensure connection resiliency, high availability, and transparent application failover during the planned (scheduled/scale-up/scale-down) or unplanned failover of the database server. The retry logic is found to be very useful for OSS applications like CKAN or Apache Airflow using SQLAlchemy. Without the use of PgBouncer, the database failover events require the application service to be restarted for connections to be re-established following a connection failure. In this scenario, it is also important to set connection timeout sufficiently higher than the retry interval to allow retry attempts to proceed before timing out.

To summarize, as new connections are an expensive operation with PostgreSQL, especially for applications which opens new connections frequently, we highly recommend using connection pooling while running applications against Azure Database for PostgreSQL. If the application is not designed to leverage connection pooling out of the box you can leverage PgBouncer as a connection pooling proxy. The benefits of running application with PgBouncer proxy are:

Improved throughput and performance
No connection leaks by defining the maximum number of connections to the database server
Improved connection resiliency against restarts
Reduced memory fragmentation

We hope that you are taking advantage of Azure Database for PostgreSQL. Please continue to provide feedback on the features and functionality that you want to see next. If you need any help or have questions, please check out the “Azure Database for PostgreSQL Documentation.” You can also reach out to us by emailing the Ask Azure DB for PostgreSQL alias, and be sure to follow us on Twitter @AzureDBPostgres and #postgresql for the latest news and announcements.

Acknowledgements

Special thanks to Diana Putnam, Rachel Agyemang, Sudhakar Sannakkayala, Sunil Agrawal, Sunil Kamath, Bhavin Gandhi, Anitah Cantele, and Prabhat Tripathi for their contributions to this posting.
Quelle: Azure