Monat: Juli 2019

How HSBC built its PayMe for Business app on Microsoft Azure

Bank-grade security, super-fast transactions, and analytics 

If you live in Asia or have ever traveled there, you’ve probably witnessed the dramatic impact that mobile technology has had on all aspects of day to day life. In Hong Kong in particular, most consumers now use a smart phone daily, presenting new opportunities for organizations to deliver content and services directly to their mobile devices.

As one of the world’s largest international banks, HSBC is building new services on the cloud to enable them to organize their data more efficiently, analyze it to understand their customers better, and make more core customer journeys and features available on mobile first.

HSBC’s retail and business banking teams in Hong Kong have combined the convenience afforded by smart phones with cloud services to allow “cashless” transactions where people can use their smart phone to perform payments digitally. Today, over one and a half million people use HSBC’s PayMe app to exchange money with people in their personal network for free. And businesses are using HSBC’s new PayMe for Business app, built natively on Azure, to collect payments instantly, with 98 percent of all transactions completed in 500 milliseconds or less. Additionally, the businesses can leverage powerful built-in intelligence on the app to improve their sales and operations.

On today’s Microsoft Mechanics episode of “How We Built it,” Alessio Basso, Chief Architect of PayMe from HSBC, explains the approach they took and why.

Bank-grade security, faster time to delivery, dynamic scale and resiliency

The first decision Alessio and team made was to use fully managed services to allow them to go from ideation to a fully operational service in just a few months. Critical to their approach was adopting a microservices-based architecture with Azure Kubernetes Service and Azure Database for MySQL.

They designed each microservice to be independent, with their own instance of Azure managed services, including Azure Database for MySQL, Azure Event Hub, Azure Storage, Azure Key Vault for credentials and secrets management, and more. They architected for this level of isolation to strengthen security and overall application uptime, as shared dependencies are eliminated.

Each microservice can rapidly scale compute and database resources elastically and independently, based on demand. What’s more, Azure Database for MySQL, allows for the creation of read replicas to offload read-only and analytical queries without impacting payment transaction response times.

Also, from a security perspective, because each microservice runs within its own subnet inside of Azure Virtual Network, the team is able to isolate network and communications back and forth between Azure resources with service principals via Virtual Network service endpoints.

Fast and responsive analytics platform

At its core, HSBC’s PayMe is a social app that allows consumers to establish their personal networks, while facilitating the interactions and transactions with the people in their circle and business entities. In order to create more value for both businesses and consumers, Azure Cosmos DB is used for graph data modelled to store customer-merchant-transaction relationships.

Massive amounts of structured and unstructured data from Azure Database for MySQL, Event Hubs, and Storage are streamed and transformed. The team designed an internally developed data ingestion process, feeding an analytical model called S.L.I.M (simple, lightly, integrated model), optimized for analytics queries performance, as well as making data virtually available to the analytics platform, using Azure Databricks Delta’s unmanaged table capability.

Then machine learning within their analytics platform built on Azure Databricks allows for the quick determination of patterns and relationships, as well as for the detection of anomalous activity.

With Azure, organizations can immediately take advantage of new opportunities to deliver content and services directly to mobile devices, including a next-level digital payment platform.

To learn more about how HSBC architected their cashless digital transaction platform, please watch the full episode.
Learn more about achieving microservice independence with your own instance of a Azure managed service like Azure Database for MySQL.

Quelle: Azure

New ways to train custom language models – effortlessly!

Video Indexer (VI), the AI service for Azure Media Services enables the customization of language models by allowing customers to upload examples of sentences or words belonging to the vocabulary of their specific use case. Since speech recognition can sometimes be tricky, VI enables you to train and adapt the models for your specific domain. Harnessing this capability allows organizations to improve the accuracy of the Video Indexer generated transcriptions in their accounts.

Over the past few months, we have worked on a series of enhancements to make this customization process even more effective and easy to accomplish. Enhancements include automatically capturing any transcript edits done manually or via API as well as allowing customers to add closed caption files to further train their custom language models.

The idea behind these additions is to create a feedback loop where organizations begin with a base out-of-the-box language model and improve its accuracy gradually through manual edits and other resources over a period of time, resulting with a model that is fine-tuned to their needs with minimal effort.

Accounts’ custom language models and all the enhancements this blog shares are private and are not shared between accounts.

In the following sections I will drill down on the different ways that this can be done.

Improving your custom language model using transcript updates

Once a video is indexed in VI, customers can use the Video Indexer portal to introduce manual edits and fixes to the automatic transcription of the video. This can be done by clicking on the Edit button at the top right corner of the Timeline pane of a video to move to edit mode, and then simply update the text, as seen in the image below.

 

The changes are reflected in the transcript, captured in a text file From transcript edits, and automatically inserted to the language model used to index the video. If you were not already using a customer language model, the updates will be added to a new Account Adaptations language model created in the account.

You can manage the language models in your account and see the From transcript edits files by going to the Language tab in the content model customization page of the VI website.

Once one of the From transcript edits files is opened, you can review the old and new sentences created by the manual updates, and the differences between them as shown below.

All that is left is to do is click on Train to update the language model with the latest changes. From that point on, these changes will be reflected in all future videos indexed using that model. Of course, you do not have to use the portal to train the model, the same can be done via the Video Indexer train language model API. Using the API can open new possibilities such as allowing you to automate a recurring training process to leverage ongoing updates.

There is also an update video transcript API that allows customers to update the entire transcript of a video in their account by uploading a VTT file that includes the updates. As a part of the new enhancements, when a customer uses this API, Video Indexer also adds the transcript that the customers uploaded to the relevant custom model automatically in order to leverage the content as training material. For example, calling update video transcript for a video titled "Godfather" will result with a new transcript file named “Godfather” in the custom language model that was used to index that video.

Improving your custom language model using closed caption files

Another quick and effective way to train your custom language model is to leverage existing closed captions files as training material. This can be done manually, by uploading a new closed caption file to an existing model in the portal, as shown in the image below, or by using the create language model and update language model APIs to upload a VTT, SRT or TTML files (similarly to what was done until now with TXT files.)

 

Once uploaded, VI cleans up all the metadata in the file and strip it down to the text itself. You can see the before and after results in the following table.

 

Type
Before
After

VTT

NOTE Confidence: 0.891635
00:00:02.620 –> 00:00:05.080
but you don't like meetings before 10 AM.

but you don’t like meetings before 10 AM.

SRT

2
00:00:02,620 –> 00:00:05,080
but you don't like meetings before 10 AM.

but you don’t like meetings before 10 AM.

TTML

<!– Confidence: 0.891635 –>
<p begin="00:00:02.620" end="00:00:05.080">but you don't like meetings before 10 AM.</p>

but you don’t like meetings before 10 AM.

From that point on, all that is left to do is review the additions to the model and click Train or use the train language model API to update the model.

Next Steps

The new additions to the custom language models training flow make it easy for you and your organization to get more accurate transcription results easily and effortlessly. Now, it is up to you to add data to your custom language models, using any of the ways we have just discussed, to get more accurate results for your specific content next time you index your videos.

Have questions or feedback? We would love to hear from you! Use our UserVoice page to help us prioritize features, or email VISupport@Microsoft.com for any questions.
Quelle: Azure

Silo busting 2.0—Multi-protocol access for Azure Data Lake Storage

Cloud data lakes solve a foundational problem for big data analytics—providing secure, scalable storage for data that traditionally lives in separate data silos. Data lakes were designed from the start to break down data barriers and jump start big data analytics efforts. However, a final “silo busting” frontier remained, enabling multiple data access methods for all data—structured, semi-structured, and unstructured—that lives in the data lake.

Providing multiple data access points to shared data sets allow tools and data applications to interact with the data in their most natural way. Additionally, this allows your data lake to benefit from the tools and frameworks built for a wide variety of ecosystems. For example, you may ingest your data via an object storage API, process the data using the Hadoop Distributed File System (HDFS) API, and then ingest the transformed data using an object storage API into a data warehouse.

Single storage solution for every scenario

We are very excited to announce the preview of multi-protocol access for Azure Data Lake Storage! Azure Data Lake Storage is a unique cloud storage solution for analytics that offers multi-protocol access to the same data. Multi-protocol access to the same data, via Azure Blob storage API and Azure Data Lake Storage API, allows you to leverage existing object storage capabilities on Data Lake Storage accounts, which are hierarchical namespace-enabled storage accounts built on top of Blob storage. This gives you the flexibility to put all your different types of data in your cloud data lake knowing that you can make the best use of your data as your use case evolves.

Single storage solution

Expanded feature set, ecosystem, and applications

Existing blob features such as access tiers and lifecycle management policies are now unlocked for your Data Lake Storage accounts. This is paradigm-shifting because your blob data can now be used for analytics. Additionally, services such as Azure Stream Analytics, IoT Hub, Azure Event Hubs capture, Azure Data Box, Azure Search, and many others integrate seamlessly with Data Lake Storage. Important scenarios like on-premises migration to the cloud can now easily move PB-sized datasets to Data Lake Storage using Data Box.

Multi-protocol access for Data Lake Storage also enables the partner ecosystem to use their existing Blob storage connector with Data Lake Storage.  Here is what our ecosystem partners are saying:

“Multi-protocol access for Azure Data Lake Storage is a game changer for our customers. Informatica is committed to Azure Data Lake Storage native support, and Multi-protocol access will help customers accelerate their analytics and data lake modernization initiatives with a minimum of disruption.”

– Ronen Schwartz, Senior Vice President and General Manager of Data Integration, Big Data, and Cloud, Informatica

You will not need to update existing applications to gain access to your data stored in Data Lake Storage. Furthermore, you can leverage the power of both your analytics and object storage applications to use your data most effectively.

Multi-protocol access enables features and ecosystem

Multiple API endpoints—Same data, shared features

This capability is unprecedented for cloud analytics services because not only does this support multiple protocols, this supports multiple storage paradigms. We now bring you this powerful capability to your storage in the cloud. Existing tools and applications that use the Blob storage API gain these benefits without any modification. Directory and file-level access control lists (ACL) are consistently enforced regardless of whether an Azure Data Lake Storage API or Blob storage API is used to access the data.  

Multi-protocol access on Azure Data Lake Storage

Features and expanded ecosystem now available on Data Lake Storage

Multi-protocol access for Data Lake Storage brings together the best features of Data Lake Storage and Blob storage into one holistic package. It enables many Blob storage features and ecosystem support for your data lake storage.

Features
More information

Access tiers
Cool and Archive tiers are now available for Data Lake Storage. To learn more, see the documentation “Azure Blob storage: hot, cool, and archive access tiers.”

Lifecycle management policies
You can now set policies to a tier or delete data in Data Lake Storage. To learn more, see the documentation “Manage the Azure Blob storage lifecycle.”

Diagnostics logs
Logs for the Blob storage API and Azure Data Lake Storage API are now available in v1.0 and v2.0 formats. To learn more, see the documentation "Azure Storage analytics logging."

SDKs
Existing blob SDKs can now be used with Data Lake Storage. To learn more, see the below documentation:

Azure Blob storage client library for .NET
Azure Blob storage client library for Java
Azure Blob storage client library for Python

PowerShell
PowerShell for data plane operations is now available for Data Lake Storage. To learn more, see the Azure PowerShell quickstart.

CLI
Azure CLI for data plane operations is now available for Data Lake Storage. To learn more, see the Azure CLI quickstart.

Notifications via Azure Event Grid
You can now get Blob notifications through Event Grid. To learn more, see the documentation “Reacting to Blob storage events.” Azure Data Lake Storage Gen2 notifications are currently available.

 

Ecosystem partner
More information

Azure Stream Analytics
Azure Stream Analytics now writes to, as well as reads from, Data Lake Storage.

Azure Event Hubs capture
The capture feature within Azure Event Hubs now lets you pick Data Lake Storage as one of its destinations.

IoT Hub
IoT Hub message routing now allows routing to Azure Data Lake Storage Gen 2.

Azure Search
You can now index and apply machine learning models to your Data Lake Storage content using Azure Search.

Azure Data Box
You can now ingest huge amounts of data from on-premises to Data Lake Storage using Data Box.

Please stay tuned as we enable more Blob storage features using this amazing capability.

Next steps

All these new capabilities are available today in West US 2 and West Central US. Sign up for the preview today. For more information, please see our documentation on multi-protocol access for Azure Data Lake Storage.
Quelle: Azure

Making it easier to bring your Linux based web apps to Azure App Service

Application development has radically changed over the years. From having to host all the physical hardware hosting the app and its dependences on-premises, to moving to a model where the hardware is hosted by external companies yet still managed by the users on to hosting your apps on a fully managed platform where all hardware and software management is done by the hosting provider. And then finally over to a full serverless solution where no resources need to be set up to run applications.

The perception of complexity in running smaller solutions in the cloud are slowly being eradicated due to moving solutions to a managed platform, where even non-technical audiences can manage their application in the cloud.

A great example in the managed platform realm is Azure App Service. Azure App Service provides an easy way to bring source code or containers and deploy full web apps in minutes, with the ease of configuration settings at the hands of the app owner. Built in features such as secure sockets layer (SSL) certificates, custom domains, auto-scaling, setting up a continuous integration and deployment (CI/CD) pipeline, diagnostics, troubleshooting, and much more, provides a powerful platform for full cycle build and management of the applications. Azure App Service also abstracts all of the infrastructure and its management overhead away from the users, maintaining the physical hardware running the service, patching security vulnerabilities, and continuously updating the underlying operating system.

Even in the managed platform world where customers shouldn’t care about the underlying platform they are physically running on, the reality is that some applications, depending on their framework, perform better on a specific operating system. This is the reason the team is putting a lot of work into the Linux hosting offering and making it easier to try it out. This includes our recent announcement about the free tier for Linux web apps, making it quick and simple to try out the platform with no commitments.

We’re excited to introduce a promotional price on the Basic app service plan for Linux, which depending on regional meters in your datacenter of choice, leads to a 66 percent price drop!

You can use the free tier to test the platform out, and then move up to the Basic tier and enjoy more of the platform’s capabilities. You can host many frameworks on this tier, including WordPress sites, Node.js, Python, Java, and PHP sites, and one of the most popular options that we’ve seen on the Linux offering – custom docker containers. Running a container hosted in Azure App Service provides an easy on-ramp for customers wanting to enjoy a fully managed platform, but also want a single deployable artifact containing an app and all of its dependencies, or want to work with a custom framework or version beyond the defaults built into the Azure App Service platform.

You can even use the Linux offering with networking solutions to secure your app using the preview feature of Azure virtual networks (VNet) integration to connect to an on-premise database, or to call into an Azure virtual network of your choice. You may also use access restrictions to control where your app may receive traffic from and place additional safeguards on the platform level.

What now? If you have a web workload you’re thinking of taking to the next level, try out Azure App Service now! Explore all of the possibilities waiting for you as you host your code or container on a managed platform that currently hosts more than two million sites!

Create your free Azure trial today.

Post on the Microsoft Developer Network forum for questions about Azure App Service.

If you have a feature suggestion for the product, please enter it in the feedback forum.
Quelle: Azure

Conversational AI updates for July 2019

At Build, we highlighted a few customers who are building conversational experiences using the Bot Framework to transform their customer experiences. For example, BMW discussed its work on the BMW Intelligent Personal Assistant to deliver conversational experiences across multiple canvases by leveraging the Bot Framework and Cognitive Services. LaLiga built their own virtual assistant which allows fans to experience and interact with LaLiga across multiple platforms.

With the Bot Framework release in July, we are happy to share new releases of Bot Framework SDK 4.5 and preview of 4.6, updates to our developer tools, and new channels in Azure Bot Service. We’ll use the opportunity to provide additional updates for the Conversational AI releases from Microsoft.

Bot Framework channels

We continue to expend channels support and functionality for Bot Framework and Azure Bot Service.

Voice-first bot applications: Direct Line Speech preview

The Microsoft Bot Framework lets you connect with your users wherever your users are. We offer thirteen supported channels, including popular messaging apps like Skype, Microsoft Teams, Slack, Facebook Messenger, Telegram, Kik, as well as a growing number of community adapters.

Today, we are happy to share the preview of Direct Line Speech channel. This is a new channel designed for voice-first experiences for your Bot Framework utilizing Microsoft’s Speech Services technologies.  he Direct Line Speech channel is a native implementation of speech for mobile applications and IoT devices, with support for Text-to-speech, Speech-to-text, and custom wake words.  We’re happy to share that we’re now opening the preview to all Bot Framework customers.

Getting started with voice support to your bot is easy. Simply update to the latest Bot Framework SDK, configure the Direct Line Speech channel for your bot, and use the Speech SDK to embed voice into your mobile application or device today.

Better isolation for your bot: Direct Line App Service Extension

Direct Line and Webchat are used broadly by Bot Framework customers to provide chat experiences on their web pages, mobile apps, and devices. For some scenarios, customers have given us the feedback that they’d like a version of Direct Line that can be deployed in isolation, such as in a Virtual Network (VNET). A VNET lets you create your own private space in Azure and is crucial to your cloud network as it offers isolation, segmentation, and other key benefits. The Direct Line App Service Extension can be deployed as part of a VNET, allowing IT administrators to have more control over conversation traffic and improve latency in conversations due to reduction in the number of hops. Feel free to get started with Direct Line App Service Extension.

Bot Framework SDK

As part of the Bot Framework SDK 4.6 preview we updated Adaptive Dialog, which allows developers to dynamically update conversation flow based on context and events. This is especially handy when dealing with conversation context switches and interruptions in the middle of a conversation. Learn more by reading the documentation and reviewing the samples.

Continuing our commitment to the Open Source community and following on our promise to allow developers to use their favorite programing language, we updated Bot Framework Python SDK. The Python SDK now supports OAuth, Prompts, CosmosDB, and includes all major functionality in SDK 4.5. In addition we got new samples.

Addressing customers’ and developers’ ask for better testing tools, the July version of the SDK introduces a new unit testing capability. The Microsoft.Bot.Builder.testing package simplifies the process of unit testing dialogs in your bot. Check out the documentation and samples.

Introduced at Microsoft Build 2019, the Bot Inspector is a new feature in the Bot Framework Emulator which lets you debug and test bots on channels like Microsoft Teams, Slack, Cortana, and more. As you use the bot on specific channels, messages will be mirrored to the Bot Framework Emulator where you can inspect the message data that the bot received. Additionally, a snapshot of the bot memory state for any given turn between the channel and the bot is rendered as well.

Following enterprise customers asks, we put together a web chat sample for a single sign-on to enterprise apps using OAuth. In this sample, we show how to authorize a user to access resources on an enterprise app with a bot. Two types of resources are used to demonstrate the interoperability of OAuth, Microsoft Graph and GitHub API.

Solutions

Virtual agent solution accelerator

We updated the Virtual Assistant and associated skills to enable out-of-box support for Direct Line Speech opening voice assistant experiences with no additional steps. This includes middleware to enable control of the voice being used. Once a new Virtual Assistant has been deployed, you can follow instructions for configuring Virtual Assistant with the Direct Line Speech channel. The example test harness application is also provided to enable you to quickly and easily test Speech scenarios.

An Android app client for Virtual Assistant is also available which integrates with Direct Line Speech and Virtual Assistant, demonstrating how a device client can interact with your Virtual Assistant and render Adaptive Cards.

In addition, we have added out-of-box support for Microsoft Teams ensuring that your Virtual Assistant and skills work including authentication and adaptive cards. You can follow steps for creating the associated application manifest.

The Virtual Assistant Solution Accelerator provides a set of templates, solution accelerators, and skills to help build sophisticated conversational experiences. A new Android app client for Virtual Assistant that integrates with Direct Line Speech and Virtual Assistant demonstrates how a device client can interact with your Virtual Assistant and render adaptive cards. Updates also include support for Direct-Line Speech and Microsoft Teams.

The Dynamics 365 Virtual Agent for Customer Service preview provides exceptional customer service with intelligent, adaptable virtual agents. Customer service experts can easily create and enhance bots with AI-driven insights. The Dynamic 365 Virtual Agent is built on top of the Bot Framework and Azure.
Quelle: Azure

Azure Monitor for containers with Prometheus now in preview

Prometheus is a popular open source metric monitoring solution and is a part of Cloud Native Compute Foundation. We have many customers who like the extensive metrics which Prometheus provides on Kubernetes. However, they also like how easy it is to use Azure Monitor for containers which provides fully managed, out of the box monitoring for Azure Kubernetes Service (AKS) clusters. We have been receiving requests to funnel the Prometheus data into Azure Monitor and today, we are excited to share Prometheus integration with Azure Monitor for containers is now in preview and brings together the best of two worlds.

Typically, to use Prometheus you need to setup and manage a Prometheus server with a database. With the Azure Monitor integration, no Prometheus server is needed. You just need to expose the Prometheus end-point through your exporters or pods (application), and the containerized agent for Azure Monitor for containers can scrape the metrics for you. We have provided a seamless onboarding experience to collect Prometheus metrics with Azure Monitor. The example below shows how the coredns metrics, which is part of the kube-dns-metric, is collected into Azure Monitor for logs. 

You can also collect workload metrics from your containers by instrumenting Prometheus SDK into your application. The example below shows the collection of the prommetrics_demo_requests_counter. You can collect workload metrics through URL, endpoints, or pod annotation as well.

Full stack monitoring with Azure Monitor for containers

So how does Prometheus metrics fit in with the rest of the metrics including the recently added storage and network performance metrics that Azure Monitor for containers already provides. You can see how the metrics all fit together below. Azure Monitor for containers provides out of the box telemetry at the platform, container, orchestrator level, and to an extent the workload level. With the additional workload metrics from Prometheus you now get full stack, end to end monitoring view for your Azure Kubernetes Services (AKS) in Azure Monitor for containers.

Visualizing Prometheus metrics on Azure dashboard and alerting

Once the metrics are stored in Azure Monitor logs, you can query against the metrics using Log Analytics with Kusto Query Language (KQL). Here’s a sample query that instruments the Prometheus SDK.  You can quickly plot the result using queries in the Azure portal.

<Queries>
InsightsMetrics
| where Name == "prommetrics_demo_requests_counter_total"
| extend dimensions=parse_json(Tags)
| extend request_status = tostring(dimensions.request_status)
| where request_status == "bad"
| where TimeGenerated > todatetime('2019-07-02T09:40:00.000')
| where TimeGenerated < todatetime('2019-07-02T09:54:00.000')
| project request_status, Val, TimeGenerated | render timechart

You can pin the chart to your Azure dashboard and create your own customized dashboard. You can also pin your current pod and node charts to the dashboard from the Azure Monitor for container cluster view.

If you would like to alert against the Prometheus metrics, you can do so using alerts in Azure. 

This has been an exciting integration for us, and we are looking to continue our effort to help our customers on monitoring Kubernetes. For more information on configuring the agent to collect Prometheus data, querying, and using the data on Azure Monitor for containers, visit our documentation. Prometheus provides rich and extensive telemetry, if you need to understand the cost implications here’s a query which will show you the data ingested from Prometheus into Azure Monitor logs.

For available metrics on Prometheus, please go to Prometheus website.

For any feedback or suggestions, please reach out to us through the techforum or stackoverflow.
Quelle: Azure