Kategorie: Microsoft Azure

Microsoft Research: How we operate Deep Neural Network with Log Analytics

Microsoft Research is at the forefront of solving and tackling cutting edge problems with technologies such as Machine Learning and Deep Neural Networks (DNN). These technologies employ next generation server infrastructure that span immense Windows and Linux cluster environments. Additionally, for DNNs, these application stacks don’t only involve traditional system resources (CPUs, Memory), but also graphic processing units (GPUs).

With a nontraditional infrastructure environment, the Microsoft Research Operations team needed a highly flexible, scalable, and Windows and Linux compatible service to troubleshoot and determine root causes across the full stack.

Enter Azure Log Analytics

Azure Log Analytics, a component of Microsoft Operations Management Suite, natively supports log search through billions of records, real time metric collection, and rich custom visualizations across numerous sources. These out of the box features paired with the flexibility of available data sources made Log Analytics a great option to produce visibility & insights by correlating across DNN clusters & components.

The following diagram illustrates how Log Analytics offers the flexibility for different hardware and software components to send real time data within a single Deep Neural Network cluster node.

1. Linux Server System Resource Monitoring

Deep Neural Networks traditionally run on Linux, and Log Analytics supports major Linux distributions as first class citizens. The OMS Agent for Linux was also recently made generally available, built on the open source log collector FluentD.  By leveraging the Linux agent, we were able to easily collect system metrics at 10 second interval and all of our Linux logs without any customization effort.

2. NVIDIA GPU Information

The Log Analytics platform is also extremely flexible, allowing users to send data via a recently released HTTP POST API. We were able to write a custom Python application to retrieve data from their NVIDIA GPUs and unlock the ability to alert based off of metrics such as GPU Temperature. Additionally, these metrics can be visualized with Custom Views to create rich performance graphs for the team to further monitor.

Whoa, I’d love to learn more

We wrote this post to showcase the flexibility Log Analytics offers customers in the type of data sources that can onboard. Additionally, check out the full walkthrough on the MSOMS blog that includes Python code examples if you are interested in replicating this type of insight.
Finally, if you are completely new to Log Analytics be sure to try our fully hydrated demo environment located here, or sign up for a free Microsoft Operations Management Suite subscription so you can test out all these capabilities with your own environment.
Quelle: Azure

Data Factory supports multiple web service inputs for Azure ML Batch Execution

For orchestrating workloads on Azure ML (Machine Learning) batch execution web services, Azure Data Factory supports a built-in activity, namely Azure ML Batch Execution activity. Customers can leverage this activity to operationalize their ML models at scale.

Little while ago, Azure ML added support to allow multiple Web Service Inputs for a given experiment. Consequently, customers have been looking to leverage this capability through Azure Data Factory. Data Factory now supports configuring the ML Batch Execution Activity to pass multiple Web Service Inputs to the ML web service.

Suppose you have an Azure ML experiment which accepts more than one Web Service Input.

Note the names of the created Web service inputs, as you must use these names when specifying the endpoints in your Data Factory Pipeline. The name can be found in the Properties pane of the module. By default the first Web Service Input module you create will be named “input1,” the next one “input2,” and so on. If you rename the modules, be sure to update the names in the webServiceInputs property in your Data Factory pipeline accordingly.

In your Azure Data Factory Pipeline, you can use the new WebServiceInputs property instead of the existing WebServiceInput property to specify the inputs into your experiment. 

"typeproperties":
{
"webServiceInputs":
{
"trainingData": "NameOfInputDataset1",
"scoringData": "NameOfInputDataset2"
},
"webServiceOutputs":
{
"output1": "NameOfOutputDataset"
},
"globalParameters": {}
}

For more information on the Azure ML Batch Execution activity in Azure Data Factory, refer to this documentation page.

If you have any feedback on the above capabilities please visit Azure Data Factory User Voice and/or MSDN Forums to reach out. We are eager to hear from you!
Quelle: Azure

Project Bletchley – Blockchain infrastructure made easy

We are very excited to announce the next phase of our support of blockchain on Microsoft Azure with the launch of an early version of the Bletchley v1 infrastructural substrate.  We expand on the work we have done to create a diverse distributed ledger ecosystem for private dev/test to focus on the requirements of private multi-node consortium networks.

By leveraging the capabilities of Microsoft Azure Resource Manager (ARM), we have shipped an Azure Quickstart Template to make it sufficiently easier and quicker to deploy and configure a consortium Ethereum network with minimal Azure and Ethereum knowledge.  With a handful of user inputs and a simple single-click deployment, you can provision a fully configured blockchain network topology consisting of many nodes in minutes, using Microsoft Azure compute, networking, and storage services across the globe.

Rather than spending hours building out and configuring the infrastructure, we have automated these time-consuming pieces to allow you to focus on your core business – re-envisioning and reinventing business processes to come up with the new stories of tomorrow.

An illustration of the deployed network topology is shown below.  At a high level, the template provisions and configures a subnet of mining nodes for each consortium member and a set of load-balanced transaction nodes that members can share to communicate with the network.  Through the administrator web page, you can configure additional Ethereum accounts to get started with smart contract and eventually application development.  For additional information and a detailed walkthrough of the template, we have published a detailed walkthrough.

However, this is just the beginning!  We are “releasing early and releasing often” to provide you with the latest updates quickly and to get your feedback throughout the development of the service.  Keep an eye out for further updates including support for additional Microsoft services, like Azure Active Directory and Key Vault, and other blockchain protocols.  In the meantime, we genuinely value your feedback.  Do not hesitate to leave a comment or send me an email with questions, feedback, or additional requests as you begin.  We are excited to be embarking on this journey with you.
Quelle: Azure

Build cloud apps at warp speed

One of your best customers just tweeted about a problem with your product and you want to respond to them ASAP. It would be great if you could automatically catch this type of communications and automagically respond with either the right documentation or escalate this to your support team. But the thought of writing an application to handle this event, with all that entails – allocating VMs, assigning staff to manage either the IaaS instances or the cloud service, not to mention the cost of development, (which might include software licenses) all that seems like a lot just to recognize and handle a tweet.

What if you could catch the tweet, direct it to the right person and respond to the customer quickly with no code and no infrastructure hassles: no systems-level programming, no server configuration step, not even code required – just the workflow. Just the business process.

It’s possible in the new era of #cloud computing.  With newly introduced capabilities in the Microsoft Cloud – Microsoft Flow, Microsoft PowerApps, and Azure Functions, you can design your workflow in a visual designer and just deploy it.

Now in preview, these new cloud offerings foreshadow the future of cloud applications.

Intrigued? Read on.

Take a look to the left. There’s the Microsoft Flow designer being set up to tell your Slack channel any time somebody complains about your product. 

That’s it. One click and voila: your workflow is running!

(And there’s the result in Slack!)

But perhaps your smart support representative contacts the unhappy customer – who it turns out has a valid issue. Your rep takes down the relevant information and starts a new workflow to have the issue looked at.

Need a server for that? No! With Microsoft Power Apps, you can visually design a form for your rep: and it can kick off a Flow.  Want that app mobile-enabled on any smartphone? No problem, as you see below. And as it shows you use the Common Data Model available in PowerApps enabling a lingua franca between applications.

If you need more sophisticated, or custom processing, your developers can create Azure Functions on the event, say, updating an on-premises or cloud-based sentiment analysis engine with the tweet, or invoking a marketing application to offer an incentive. Again: no server. (In fact, no IDE either: your devs write their business logic code directly on the Azure portal and deploy from there.)

So why do I say Microsoft Flow, PowerApps and Functions presage a new model of cloud applications? Because increasingly, cloud apps are evolving toward a lego-block model of “serverless” computing: where you create and pay only for your business logic, where chunks of processing logic are connected together to create an entire business application.

Infrastructure? Of course it’s there (“serverless” may not be the best term), but it’s under the covers: Azure manages the servers, configures them, updates them and ensures their availability. Your concern is what it should be: your business logic.

This is potentially a seismic shift in how we think about enterprise computing.

Think about it: with PowerApps your business users can quickly create apps, and with Microsoft Flow, create business processes with a few clicks. With Flow’s bigger cousin, Azure Logic Apps, you can quickly connect to any industry-standard enterprise data source such as your local ERP system, a data warehouse, support tools and many others via open protocols and interfaces such as EDIFACT/X.12, AS2, or XML. And you can easily connect to a wide variety of social media and internet assets, like Twitter, Dropbox, Slack, Facebook and many others. With Functions you can catch events generated by Logic Apps and make decisions in real time.

And you haven’t deployed a single server. What code you’ve written is business logic only; not administration scripts or other code with no business value. Your developers have focused on growing your business. And, most importantly, you’ve created a rich, intelligent end-to-end application –by simply attaching together existing blocks of logic.

Like Lego blocks. Other cloud platforms offer serverless options, but none as deep and as varied as Microsoft’s, empowering everyone in your organization, from business analyst to developer, with tools appropriate to their skills. For enterprises, the implications could not be more profound.

Maybe it’s appropriate, on this fiftieth anniversary of Star Trek, that with tools on the Microsoft Cloud, you can run your business at warp speed using Azure.
Quelle: Azure