Kategorie: Microsoft Azure

Run massive parallel R Jobs in Azure, now at a fraction of the price

We continue to add new capabilities to our lightweight R package, doAzureParallel, built on top of Azure Batch that allows you to easily use Azure's flexible compute resource right from your R session. Combined with the recently announced low-priority VMs on Azure Batch, you can now run your parallel R jobs at a fraction of the price. We also included other commonly requested capabilities to enable you to do more, and to do it more easily, with doAzureParallel.

Using R with low priority VMs to reduce cost

Our second major release comes with full support for low-priority VMs, letting R users run their jobs on Azure’s surplus compute capacity at up to an 80% discount.

For data scientists, low-priority is great way to save costs when experimenting and testing their algorithms, such as parameter tuning (or parameter sweeps) or comparing different models entirely. And Batch takes care of any pre-empted low-priority nodes by automatically rescheduling the job to another node.

You can also mix both on-demand nodes and low-priority nodes. Supplementing your regular nodes with low-priority nodes gives you a guaranteed baseline capacity and more compute power to finish your jobs faster. You can also spin up regular nodes using autoscale to replace any pre-empted low-priority nodes to maintain your capacity and to ensure that your job completes when you need it.

​Other new features

Aside from the scenarios that low-priority VMs enable, this new release includes additional tools and common feature asks to help you do the following:

Parameter tuning & cross validation with Caret
Job management and monitoring to make it easier to run long-running R jobs
Leverage resource files to preload data to your cluster
Additional utility to help you read from and write to Azure Blob storage
ETL and data prep with Hadley Wickham’s plyr

​Getting started with doAzureParallel

doAzureParallel is extremely easy to use. With just a few lines of code, you can register Azure as your parallel backend which can be used by foreach, caret, plyr and many other popular open source packages.

Once you install the package, getting started is as simple as few lines of code:

# 1. Generate your credentials config and fill it out with your Azure information
generateCredentialsConfig(“credentials.json”)

# 2. Set your credentials
setCredentials(“credentials.json”)

# 3. Generate your cluster config to customize your cluster
generateClusterConfig(“cluster.json”)

# 4. Create your cluster in Azure passing, it your cluster config file.
cluster <- makeCluster(“cluster.json”)

# 5. Register the cluster as your parallel backend
registerDoAzureParallel(cluster)

# Now you are ready to use Azure as your parallel backend for foreach, caret, plyr, and many more

For more information, visit the doAzureParallel Github page for a full getting started guide, samples and documentation.

We look forward to you using these capabilities and hearing your feedback. Please contact us at razurebatch@microsoft.com for feedback or feel free to contribute to our Github repository.

Additional information:

Download and get started with doAzureParallel
For questions related to using the doAzureParallel package, please see our docs, or feel free to reach out to razurebatch@microsoft.com
Please submit issues via Github

Additional resources:

See Azure Batch, the underlying Azure service used by the doAzureParallel package
More general purpose HPC on Azure
Learn more about low-priority VMs
Visit our previous blog post on doAzureParallel

Quelle: Azure

Announcing public preview of disaster recovery for Azure IaaS virtual machines

I am excited to announce the public preview of disaster recovery for Azure IaaS virtual machines (VMs) using Azure Site Recovery (ASR). You can now easily replicate and protect IaaS based applications running on Azure to a different Azure region of your choice without deploying any additional infrastructure components or software appliances in your subscription. This new capability, along with Azure Backup for IaaS virtual machines, allows you to create a comprehensive business continuity and disaster recovery strategy for all your IaaS based applications running on Azure. As you move production applications to the cloud, Azure natively provides you the high availability and reliability that your mission critical workloads need. However, compliance requirements such as ISO 27001 still require that you have a provable disaster recovery solution in place as part of a business continuity plan (BCP). The set of features that we are announcing today for Azure IaaS fill this important need. Disaster Recovery for Azure IaaS applications extends ASR’s existing functionality, and further simplifies the onboarding experience for customers: Offered “as-a-Service” – You do not need any additional software infrastructure (VMs or appliances) in your Azure subscription to enable this functionality. You avoid all the complexity and cost associated with deploying, monitoring, patching and maintaining any DR infrastructure. Simplified experience – Enabling cross-region DR for an application is so simple that all you need to do is select the VMs you want to protect, choose a target Azure region, select replication settings and you are good to go. Application-aware recovery – Whether you are an application owner, disaster recovery admin or a managed service provider, ASR lets you stay in control at all times – you can decide when and how to orchestrate a failover. With support for best-in-class recovery point objective (RPO), recovery time objective (RTO) and ASR’s powerful Recovery Plans, your applications can meet the recovery requirements that your business demands. Non-disruptive DR drills – With ASR’s test failover capability, you can easily perform a DR drill anytime without any impact to the primary production workload or to ongoing replication, giving you the confidence that your DR solution will work when you need it. Cross-region DR feature is now available in all Azure public regions where ASR is available. Click here to get started.
Quelle: Azure

Streamlining Kubernetes development with Draft

I’m Gabe Monroy, the lead PM for containers on Microsoft Azure. I joined Microsoft in April in the acquisition of Deis. At Deis, our team was always laser-focused on making containers easier to use – and easier to deploy on Kubernetes specifically. Now as part of Microsoft, I’m thrilled to continue that mission and share the first piece of open-source software we’re releasing as part of the Azure Container Service team.

Application containers have skyrocketed in popularity over the last few years. In recent months, Kubernetes has emerged as a popular solution for orchestrating these containers. While many turn to Kubernetes for its extensible architecture and vibrant open-source community, some still view Kubernetes as too difficult to use.

Today, my team is proud to announce Draft, a tool that streamlines application development and deployment into any Kubernetes cluster. Using two simple commands, developers can now begin hacking on container-based applications without requiring Docker or even installing Kubernetes themselves.

Draft in action

Draft targets the “inner loop” of a developer’s workflow – while developers write code and iterate, but before they commit changes to version control. Let’s see it in action.

Setting sail for Kubernetes

When developers run “draft create” the tool detects the application language and writes out a simple Dockerfile and a Kubernetes Helm chart into the source tree. Language detection uses configurable Draft “packs” that can support any language, framework, or runtime environment. By default, Draft ships with support for languages including Node.js, Go, Java, Python, PHP, and Ruby.

You can customize Draft to streamline the development of any application or service that can run on Kubernetes. Draft packs are just a simple detection script, a Dockerfile, and a Helm chart.

This simple yet flexible user experience for developers is inspired by PaaS systems like Deis and Cloud Foundry which support the concept of buildpacks. However, Draft differs from buildpack-oriented PaaS systems because it writes out build and deployment configuration into the source tree, making it trivial to construct continuous integration (CI) pipelines that can bring these containers all the way to production.

Edit locally, dev remotely

Once developers run “draft create”, hacking on the application is as simple as typing “draft up”. This ships source code to any Kubernetes cluster (running either locally or in the cloud), builds it in that cluster using the Dockerfile, and deploys it into a dev environment using the Helm Chart. Developers can then test their app live, and any changes in their editor or IDE will be made available in seconds. With a Jenkins registry and a Kubernetes cluster, developers can build on a tight iteration loop from code through deployment, either in concert with an operations team or on their own laptop.

While some developers often start alone, pointing Draft at a Kubernetes cluster running on a laptop, Draft works equally well on a remote Kubernetes cluster. This allows developers to edit code locally, but have their dev environment running in the cloud where they can access all their app’s production dependencies. Three cheers for “dev” and “prod” parity!

Get started today

ACS has a long history of openness including support for multiple orchestrators and the open source ACS-Engine project, which helps customers customize their own ACS cluster. We are proud to extend that open spirit to new projects that help developers succeed with container technology. As of today, Draft is open source and available at https://aka.ms/draft. Spin up a Kubernetes Cluster on ACS and take Draft for a test drive today.
Quelle: Azure

Getting Started with the Video Indexer API

Earlier this month at BUILD 2017, we announced the public preview of Video Indexer as part of Microsoft Cognitive Services. Video Indexer enables customers with digital and audio content to automatically extract metadata and use it to build intelligent innovative applications. You can quickly sign up for Video Indexer from https://vi.microsoft.com/ and try the service out for free during our preview period.

On top of using the portal, developers can easily build custom apps using the Video Indexer API. In this blog, I will walk you through an example of using the Video Indexer API to do a search on a keyword, phrase, or detected person’s name across all public videos in your account as well as sample videos and then to get the deep insights from one of the videos in the search results.

Getting Access to the Video Indexer API

To get started with the Video Indexer API, you must sign in using a Microsoft, Google, or Azure Active Directory account. Once signed in with your preferred account, you can easily subscribe to our free preview of the Video Indexer API. The following steps will walk you through the process of registering for access.

To subscribe to the API, go to the Products tab and click Free Preview. On the next page, click the Subscribe button. You should now have access to the API. If you find that you do not have access, contact visupport@microsoft.com.

After getting access, you can then return to the Products tab, and go to the Video Indexer APIs – Production link.

 

You should now see the Video Indexer API documentation page. On the left side of the page, you will see a list of several action options. Each action page contains information about that request including which parameters are optional and which ones are required. You can test any of these by clicking Try it, setting the appropriate parameters, and then clicking Send.

 

To use an external tool like Postman to test the API, you will need to download the Video Indexer APIs – Production Swagger .json file. You can do this by selecting the API definition download button on the top right of the page and choosing Open API to get the Swagger .json file. Save the file somewhere locally on your machine to use in the next section.

Here, I will demonstrate how to use Postman to test the API. To follow along, you can download and install Postman here. Launch Postman and click Import in the top left.

 

Navigate to and choose the Video Indexer APIs – Production Swagger .json file that you previously downloaded and saved locally.

 

You should now see the API actions under Collections.

 

To submit calls to any of the actions, you need a key that is specific to your subscription. This can be found by going back to Video Indexer APIs – Production page and clicking Try it. Once it gets to the next page, you can scroll down to the Headers section and find where it says Ocp-Apim-Subscription-Key. You can see your key by clicking on the eye icon.

 

Copy both the name of the key (Ocp-Apim-Subscription-Key) and the key itself because you will need both for Postman.

Running a Search Call Across Videos

Going back to Postman, go to the action call that you want to test out. In this case, I will start with search, which is going to be a typical user interaction with the API. In particular, I’m showing a search on a keyword across all public videos in your account as well as sample videos. Search is an HTTP GET call with the request URL https://videobreakdown.azure-api.net/Breakdowns/Api/Partner/Breakdowns/Search

 

Go to Headers and enter in the name of the key (Ocp-Apim-Subscription-Key) where it says key and the key itself where it says value. You can set a Header Preset on Postman using the key and value to prevent having to type them in every time. It saves time and is easy to set up, so it is definitely worth doing!

 

To set the parameters of the action, click Params and set the values for the parameters you wish to set. Remove the unchanged parameters by hovering to the right corner of the parameter and selecting the x.

In this search example, I’m setting the privacy to “Public”, language to “English”, textScope to “Transcript”, and searchInPublicAccount to “true”. I am also clearing out all the parameters that I have not changed. For query, enter in the word that you would like to search for across the videos. In this example, let’s search for the keyword “Azure”.

 

Upon selecting Send, you will get a JSON response with the results of the search.

 

The JSON response of the search contains a results section that gives back the videos that contain your query term and the relevant time ranges from each resulting video. The section is an array in which each element is a resulting video along with its basic information, social likes, number of views, and search matches with start times.

Below is an outline of some of what to expect in the JSON response for search.

JSON Response for Search

Results (Array – each element has the following info)

Basic video and user information

accountId
id
name
description
userName
createTime
privacyMode
state

social

likes
views

searchMatches (Array – each element has the following info)

startTime
type (tells the user whether match is from audio based transcript or OCR)
text

You should also test this out by uploading and processing a few of your own videos in your account. You can do this on the Video Indexer Preview site if you are logged in, or you can use the Upload HTTP POST call from the Video Indexer API in Postman.  For your search request, set searchInPublicAccount to “false” to only search through the videos on your account. Set the query to a keyword that is more relevant to your videos and privacy to either “Public” or “Private” based on the settings of your video.

Next, I will show how to take the results of a search and get the expanded insights of the video.

Running a Breakdown Call on a Video

Take the id of the first result of your search.

 

Now go to the breakdown action. Breakdown is the HTTP GET call with the request URL https://videobreakdown.azure-api.net/Breakdowns/Api/Partner/Breakdowns/:id

You will need to put in your subscription key name and key again. If you have a preset set up with the key, you will just need to select it.

Click the Params button and enter in the id from the search result for the id parameter. Set the language of the breakdown to “English” and click Send.

You should now see the JSON response for the breakdown request.

 

The JSON response of the breakdown contains general information on the video and the account that uploaded it in addition to three sections called summarizedInsights, breakdowns, and social.

The summarizedInsights section holds information on the distinct faces, topics, and audio effects in the video as well as the different time ranges in which each appear. In addition, the section  provides information on positive, negative, and neutral sentiments throughout the video as well as the time ranges for each.

The breakdowns section serves as a more expansive version of the summarizedInsights. Here, you will find transcript blocks, categories of audio effects, and information to allow for content moderation. The breakdowns section also provides more details on topics, faces, and voice participants of the video.

The trasncriptBlocks section within breakdowns serves as a timeline of the video. You will find information on lines, OCRs, faces, etc. for each time block.  The social section provides data on likes and number of views.

Below is an outline of some of what to expect in the JSON response for breakdown.

JSON Response for Breakdown

Basic video and user information

This section has extensive information on the name, owner, id, etc. of the video.

summarizedInsights

faces
topics
sentiments
audio effects

breakdowns

general information on video

accountID
id
state
processingProgress
externalURL

insights

trancsriptBlocks
topics
faces
contentModeration
audioEffectsCategories

social

likes
viewsYou can get a more information on the JSON response for breakdown here.You have the data and are now well on your way to having more insights on your videos and the opportunity to further innovate. Try a few more examples with your own content!

For more details, please take a look at the Video Indexer Documentation. Follow us on Twitter @Video_Indexer to get the latest news on the Video Indexer.

If you have any questions or need help, contact us at visupport@microsoft.com.
Quelle: Azure

Building an Azure Analysis Services Model for Azure Blobs — Part 2

The May 2017 release of SSDT Tabular introduces support for named expressions in Tabular 1400. The second part of the article “Building an Azure Analysis Services Model on Top of Azure Blob Storage” on the Analysis Services team blog takes advantage of this exciting capability to build streamlined table queries for the more than 1,000 source files in a 1-terabyte TPC-DS data set. The article also outlines and implements an incremental data loading strategy to keep the generated load against Azure Blob storage below throttling thresholds. The initial deployment of the data model on an Azure Analysis Services server with approximately 10 GB of source data finished in roughly 15 minutes. Subsequent load cycles will increase the data volume up to the maximum server capacity in Azure Analysis Services.

For more details, see the blog article “Building an Azure Analysis Services Model on Top of Azure Blob Storage – Part 2” on the Analysis Services team blog.
Quelle: Azure

What’s brewing in Visual Studio Team Services: May 2017 Digest

This post series provides the latest updates and news for Visual Studio Team Services and is a great way for Azure users to keep up-to-date with new features being released every three weeks. Visual Studio Team Services offers the best DevOps tooling to create an efficient continuous integration and release pipeline to Azure. This sprint has our Build 2017 conference deliverables in it, so it’s a big one, especially in the CI/CD space. Here’s a recap with all of our conference presentations.

One of our goals is to keep lowering the barrier to entry for automating your application deployment. The ease with which teams can deploy and validate their application is a huge part of how quickly they are able to ship. While our CI/CD system is completely open, by doing deep integrations with Azure we can make setting up deployments extremely simple. It also unlocks many opportunities for richer workflows that span both development and operations. To that end, we are continuing to strive to make VSTS + Azure the best end-to-end DevOps experience.

This month brings a bunch of new capabilities toward realizing that goal. We have significantly expanded the breadth of app type we support:

We now support using the automation agent on the VMs to which you deploy and using it to drive your application deployment. This has easily been our most requested feature for Release Management and we’re excited for it go live.
We continue to give more and more focus to containers. This sprint, we introduce native support for Kubernetes and Service Fabric, the latter being a great option for Windows containers.
We already have great support for deploying to Azure Web Apps, but we’ve expanded the app types we support with our native task to include Node, PHP, and Linux Web Apps with containers. We’ve also expanded the entry point for setting up CI/CD with more options in the Azure portal configuration UI and introduced the ability to set up CI/CD for Azure Web Apps from the AZ CLI.

Let’s dive in!

VM Deployment (Public Preview)

Release Management now supports robust out-of-the-box multi-machine deployment. You can now orchestrate deployments across multiple machines and perform rolling updates while ensuring high availability of the application throughout.

Agent-based deployment capability relies on the same build and deployment agents. However, unlike the current approach where you install the build and deployment agents on a set of proxy servers in an agent pool and drive deployments to remote target servers, you install the agent on each of your target servers directly and drive rolling deployment to those servers. You can use the full task catalog on your target machines.

A deployment group is a logical group of targets (machines) with agents installed on each of them. Deployment groups represent your physical environments, such as single-box Dev, multi-machine QA, and a farm of machines for UAT/Prod. They also specify the security context for your physical environments.

You can use this against any VM that you register our agent with. We’ve also made it very easy to register with Azure with support for a Azure VM extension that auto-installs the agent when the VM spins up. We will automatically inherit the tags on the Azure VM when it’s registered in VSTS.

Once you have a deployment group, you simply configure what you want us to execute on that deployment group. You can control what gets run on which machines using tags and control how fast or slow the rollout happens.

When the deployment is run, the logs show the progression across the entire group of machines you are targeting.

This feature is now an integrated part of Release Management. There are no additional licenses required to use it.

While we’re on the topic of deploying to different environments, check out our post on configuring your release pipelines for safe deployments.

Azure virtual machine scale set deployment

Another common pattern being use for deployment is to create a full machine image for each version of the application and then deploy that. To make that easier we have a new Build immutable machine image task that uses Packer to generate a machine image after deploying applications and all the required prerequisites. The tasks takes either deployment script or packer configuration template to create the machine image and stores it in an Azure Storage account. This image can than be used for Azure Virtual Machine Scale Set deployments that work well for this type of immutable image deployment. You can learn more in our post on deploying applications to VM scale sets.

Built-in tasks for building and deploying container based applications

With this release we have pulled most of the tasks in our Docker extension into the product by default, improved them, and introduced a set of new tasks and templates for making a set of container scenarios easier.

Docker: Build, push, or run Docker images, or run a Docker command. This task can be used with Docker or Azure Container registry. You can now use our built-in service principal authentication with ACR to make it even easier to use.
Docker-Compose: Build, push, or run multi-container Docker applications. This task can be used with Docker or Azure Container registry.
Kubernetes: Deploy, configure, or update your Kubernetes cluster in Azure Container Service by running kubectl commands.
Service Fabric: Deploy containers to a Service Fabric Cluster. Service Fabric is the best choice today for running Windows Containers in the cloud. In fact, this is where more and more of VSTS itself is running each sprint.

Azure Web App deployment updates

We have made many enhancements for Azure Web Applications:

Azure App Service deployment task supports Node.js, Python applications to be deployed.
Azure App Service deployment task supports deploying to Azure Web App for Linux using containers.
Azure portal Continuous Delivery is expanded to now support Node applications.

We have also introduced CI/CD support into the latest version of the Azure CLI for configuring CI/CD. Here is an example:

az appservice web source-control config –name mywebapp –resource-group mywebapp_rg –repo-url https://myaccount.visualstudio.com/myproject/_git/myrepo –cd-provider vsts –cd-app-type AspNetCore

Deploy to Azure Government Cloud

Customers with Azure subscriptions in Government clouds can now configure Azure Resource Manager service endpoint to target national clouds.

With this, you can now use Release Management to deploy any application to Azure resources hosted in government clouds, using the same deployment tasks. Read more about this in our on setting up continuous delivery to Microsoft Azure Government.

Automatic linking from work items to builds

With this new setting in the build definition, users can track the builds that have incorporated their work without having to search through a large set of builds manually. Each successful build associated with the work item automatically appears in the development section of the work item form.

To enable this feature, toggle the setting under Options in your build definition.

Note: The feature is only available for definitions building Team Services Git or TFVC repos, and only through the new build definition editor.

Using Jenkins for Continuous Integration with Team Services

Jenkins is a popular continuous integration build server, and there are multiple ways to use Jenkins as a CI server with Team Services. Jenkins’ built-in Git Plugin or Team Foundation Server Plugin can poll a Team Services repository every few minutes and queue a job when changes are detected. For those who need tighter integration, Team Services provides two additional ways to achieve it: 1) the Jenkins Service Hook, and 2) Jenkins build and release tasks.

Team Services adds capabilities over the Jenkins Service Hook by including connectors that allow its build and release systems to integrate with Jenkins. These connectors can be chosen from the list of tasks to execute as steps in a Team Services build or release definition.

A Team Services build or release will queue a Jenkins job and download resulting artifacts. Since these tasks execute in a light-weight, long-polling agent that can be installed in your data center, there is no need to modify inbound firewall rules for Team Services to access your Jenkins server from the cloud.

You can learn more in our blog post on integrating with Jenkins.

Maven for Package Management (Public Preview)

Java developers share components by packaging up their code in Maven artifacts, the Java equivalent of a NuGet package. Team Services customers needing a place to host Maven artifacts used to have to use third-party services, like Nexus or Artifactory, to meet their needs. We’re proud to announce that Team Services Package Management now supports hosting Maven artifacts! Check out our getting started guide.

You’ll also want to check out our recent blog post on the extensive support for Java development with Team Services.

New Git branch policies configuration experience

Branch policies are a great way to ensure quality in your branches by requiring code reviews, automatically running a build and tests for each PR, and more. We’ve redesigned the branch policies configuration experience and added some great new capabilities. One of the most powerful features is the ability to configure policies for branch folders. You can do this from the Branches view by selecting a branch folder and choosing Branch policies from the context menu.

This will open the new policies configuration UX, where you can configure policies that apply to all of the branches in the branch folder.

If you’re using the build policy, you can now configure multiple builds for a single branch. There are also new options to specify the type of trigger, automatic or manual. Manual triggers are useful for things like automated test runs that might take a long time to run, and you only really need to run once before completing the pull request. The build policy also has a display name that is useful if you’re configuring multiple builds.

Share Git pull requests with teams

The Share Pull Request action is a handy way to notify reviewers. In this release, we’ve added support for teams and groups, so you can notify everyone involved the pull request in a single step.

Visualize your Git repository

Team Services now supports showing a graph while showing commit history for repositories or files. Now you can easily create a mental model of all your branches and commits for your git repositories using git graph. The graph shows all your commits in topological order.

The key elements of the git graph include:

The git graph is right-aligned, so commits associated with the default branch or the selected branch appear on the right while the rest of the graph grows on the left.
Merge commits are represented by grey dots connected to their first parent and second parent.
Normal commits are represented by blue dots.
If the parent commit of a commit is not visible in the view port on the next 50 commits, then we excise the commit connection. Once you click the arrow, the commit is connected to its parent commit.

You can read more in our post on the Git graph and advanced filters.

Delivery Plans general availability

We are excited to announce that Delivery Plans is out of preview and is now included in the basic access level of VSTS. Delivery Plans is an organizational tool that helps users drive cross-team visibility and alignment by tracking work status on an iteration-based calendar. Users can tailor their plan to include any team or backlog level from across projects in the account. Furthermore, Field Criteria on Plans enables users to further customize their view, while Markers highlight important dates.

Delivery Plans is currently only available for VSTS; however, it will be included in the upcoming TFS 2017 Update 2 release.

Check out the marketplace page for Delivery Plans to learn more and install the extension.

Delivery timeline markers

Have you been looking for a way to highlight key dates on your Delivery Plan? Now you can with plan markers. Plan markers let you visualize key dates for teams directly on your deliver plan. Markers have an associated color and label. The label shows up when you click the marker dot.

Work item search general availability

Thank you all for installing and using the Work Item Search preview from the marketplace. It has been one of our most highly rated extensions. With this release, we are making it easier for you to use work item search by making it a built-in feature of VSTS.

You can get started with work item search using the search box:

Updated process customization experience

We have modernized our pages when customizing your process. The page now includes a breadcrumb in the top to clearly show the context you are in when editing the process or the work item types inside the process.

Also, it’s much easier to start customizing your work item form. When you select Customize from the context menu in a work item, we automatically create an inherited process for you, if you are not already using one, and bring you into the layout editor.

Insight into your projects with Analytics

Our new Analytics service brings you and your team new insights into the health and status of your work. Analytics is currently in preview, and at this early stage includes three new dashboard widgets, Lead Time, Cycle Time, and Cumulative Flow Diagram (CFD). You can install Analytics from the VS Team Services Marketplace.

We released a long list of new features the last couple of sprints. Be sure to read the release notes for May 11th and April 19th for a full list.

Happy coding!
Quelle: Azure