Kategorie: Microsoft Azure

Provision on-demand Spark clusters on Docker using Azure Batch's infrastructure

Since its release 3 years ago, Apache Spark has soared in popularity amongst Big Data users, but is also increasingly common in the HPC space. However, spinning up a Spark cluster, on-demand, can often be complicated and slow. Instead, Spark developers often share pre-existing clusters managed by their company’s IT team. In these scenarios, Spark developers run their Spark applications on static clusters that are in constant flux between under-utilization and insufficient capacity. You’re either out of capacity, or you’re burning dollars on idle nodes.

I’m excited to announce our beta release of the Azure Distributed Data Engineering Toolkit – an open source python CLI tool that allows you to provision on-demand Spark clusters and submit Spark jobs directly from your CLI.

After closely studying how Spark users interact with their clusters, we designed the Azure Distributed Data Engineering Toolkit to extend the native Spark experience, allowing you to provision clusters, and giving you the full end-to-end experience of running Spark at scale. Furthermore, this toolkit inherits Azure Batch’s fast provision time, taking only 3-5 minutes to provision your Spark cluster. With a Spark native experience and fast spin-up time, this toolkit allows you to easily run your Spark experiments, enabling you to do more, easily and in less time.

For those of you who need specific software pre-installed for your Spark application, this toolkit also gives you the ability to bring your own Docker image, making setup simple and reproducible.

The Azure Distributed Data Engineering Toolkit is free to use – you only pay for the cores you consume. Because it is built on Azure Batch, it has the ability to provision low-priority VMs, letting you run your Spark jobs at an 80% discount, making it a great tool for experimentation, testing, and other low priority work.

Today, this toolkit only supports Spark, however, we plan to support other distributed data engineering frameworks in a similar vein.

Create your Spark cluster

Once you have the Azure Distributed Data Engineering Toolkit installed you can start by creating a Spark cluster with this simple CLI command:

$ aztk spark cluster create
–id <my_spark_cluster_id>          –size <number_of_nodes>          –vm-size <vm_size>

Instead of using –size, you can optionally use –size-low-priority to provision low priority VMs.

You can also add the –username and –ssh-key parameters to create an user for the cluster. This can optionally be done in a separate aztk spark cluster add-user command.

Submit a job

Once your Spark cluster is ready, submit jobs against your cluster(s) with the standard spark-submit command using the aztk spark cluster submit command.

$ aztk spark cluster submit        –id <my_spark_cluster>        –name <my_spark_job_name>
[options]        <app jar | python file>        [app parameters]

And just like with the standard spark-submit command, the output will be streamed to the console.

Quick start demo

Here’s an example of how you can get a 2TB Spark cluster with low-priority VMs (80% discounted) in about 5 minutes:

$ aztk spark cluster create –id my_cluster –size-low-pri 16 –vm-size Standard_E16_v3

# Grab some coffee…

# BOOM! Your 2TB Spark cluster is up. Start submitting jobs!
$ aztk spark cluster submit –id my_cluster –name my_massive_job my_pyspark_app.py

Interactive mode

To get started, most users will want to work interactively with their Spark clusters. The Azure Distributed Data Engineering Toolkit supports working interactively with the aztk spark cluster ssh command that helps you ssh into the cluster’s master node, but also helps you port-forward your Spark Web UI and Spark Jobs UI to your local machine:

$ aztk spark cluster ssh –id <my_spark_cluster_id>

By default, we port forward the Spark Web UI to localhost:8080, Spark Jobs UI to localhost:4040, and Jupyter to localhost:8888. These defaults can be configured in the .thunderbolt/ssh.yaml file, if needed.

Once you run the command, you should be able to interact with your Spark Web UI by going to localhost:8080 on your local machine:

Managing your Spark cluster(s)Checkout some of the other commands you can use to manage your Spark cluster(s):

# Get a summary of all the Spark clusters you have created with Azure Thunderbolt
$ aztk spark cluster list

# Get a summary on a specific Spark cluster
$ aztk spark cluster get –id <my_spark_cluster_id>

# Delete a specific Spark cluster
$ aztk spark cluster delete –id <my_spark_cluster_id>

We look forward to you using these capabilities and hearing your feedback. Please contact us at askaztk@microsoft.com for feedback or feel free to contribute to our Github repository.

Additional information

Download and get started with the Azure Distributed Data Engineering Toolkit
Please feel free to submit issues via Github

Additional resources

See Azure Batch, the underlying Azure service used by the Azure Distributed Data Engineering Toolkit
More general purpose HPC on Azure

Quelle: Azure

Announcing the general availability of Azure Application Insights SDK for Node.js 1.0

At Microsoft, we are committed to enabling great experiences for developers building and running Node.js apps on Azure. Today, we are announcing the general availability of the Azure Application Insights SDK for Node.js 1.0, the culmination of many successful iterations and built on listening to customer feedback.

Application Insights is an Application Performance Management tool that monitors your apps, services and components in production, after you deploy them, to help you discover and rapidly diagnose performance bottlenecks and other issues. The 1.0 release primarily brings performance, reliability and stability improvements to the Application Insights SDK for Node.js.

The Application Insights SDK for Node.js is an open source project hosted on GitHub.

Getting started

If you are new to Azure Applications Insights, check out the detailed getting started guide that walks you through a step-by-step process of setting up a monitoring for your Node.js application or service. You can use Application Insights regardless of where your Node.js apps is running: Virtual Machines on Azure or on-premises, containers, Web Apps on Linux, and even on Internet-of-Things devices and Electron desktop apps. All you need to do is insert a few lines of code in your application to start getting usage telemetry and data to help diagnose issues in production.

Application Map

Application Map is a visual layout of the dependency relationships of your application components. Each component shows KPIs such as load, performance, failures, and alerts, to help you discover any component causing a performance issue or failure. You can navigate through from any component to more detailed diagnostics, such as Application Insights events. If your app uses Azure services, you can also click through to Azure diagnostics and recommendations, such as those provided by SQL Database Advisor. The 1.0 release of the Application Insights SDK for Node.js brings some enhancements in this space, helping you to easily identify dependencies by showing the icons for third-party products and services, including Redis, MongoDB, PostgreSQL, and more.

Try out this feature and let us know what else you would like to see in the Application Map to better serve your needs.

Improved NPM module support

The popularity of Node.js stems from its third-party module ecosystem, with hundreds of thousands of packages on NPM. Today, Node.js applications are using NPM packages more than ever, and it is often necessary to get insights from these modules to be able to truly diagnose complex problems. Application Insights aims to offer a great diagnostic experience out of the box, providing deep and actionable diagnostics information from your entire application, including its dependencies. With the 1.0 release of the Application Insights SDK for Node.js, we have added automatic instrumentation to two additional modules. PostgreSQL and the winston logging framework have joined the growing list of NPM packages that we support out-of-the-box, along with bunyan, console, mongodb, mongodb-core, mysql, redis, and pg-pool.

Breaking changes in this release

Breaking changes are always a hard decision, and we don’t take it lightly. But to bring consistency with other Application Insights SDKs and to allow future extensibility, this release also includes some breaking changes across the APIs. Please review the README on the GitHub project page for new method and property names, as well as for migration instructions. Additionally, the default configuration in this SDK has been changed by turning on, by default, storage of telemetry to disk when your app is offline.

Try it out

We invite you all to try out Application Insights for Node.js and start monitoring your Node.js apps and services today.

We will continue to enhance our customers’ experience by automatically instrumenting more third-party modules and improving Application Map integration and context tracking, based on customer feedback.

We are eager to hear what you think: if you have any question or are experiencing problems with the Application Insights SDK for Node.js, please open an issue on the project page on GitHub.
Quelle: Azure

Microsoft announces next Azure OpenDev: October 25, 2017

I’m so excited to announce that the next episode of Azure OpenDev, our live technical series featuring open source community leaders, will be streaming live on October 25 at 9:00 AM (Pacific Time). This time we’re featuring DevOps, which increases business agility by delivering business value to customers faster. You’ll learn how to combine leading OSS projects in a DevOps toolchain with the power of Azure, benefiting both legacy and cloud-native apps.

Many opportunities to learn, interact, and explore

The event will include open source developers and advocates from great OSS DevOps projects and companies, such as HashiCorp’s Nic Jackson, the Jenkins Project’s Tyler Croy and other community advocates from Chef, GitHub, and Elastic. They’ll be demoing some great examples of deploying and using these open source technologies with Azure, plus answering your questions live on Twitter and Gitter. I hope you’ll save the date to participate in this community event.

Save the date – and spread the word

Visit Azure OpenDev to learn more about the event and save the date to your calendar. Please help us spread the word to your friends and colleagues. The more people who watch it and interact with the speakers, the more fun it is! As we want to make the OpenDev experience more social, we will also host viewing parties at Microsoft offices and other locations around the world.

If you are unable to watch Azure OpenDev on October 25th at 9:00 AM PT and enjoy the full live experience, you still have a chance! After the event, all the sessions will be available on-demand on the event webpage.

I’m super excited to be hosting, and I’ll also be talking about how anyone can contribute to OSS, beyond just writing code. I look forward to engaging with you all live on Twitter and Gitter during this community event.

See y’all online!
Quelle: Azure

Manage your Azure VM backups with Azure CLI 2.0 – Preview

 

We are excited to announce that you can now use Azure CLI to easily manage your Windows and Linux Azure VM backups using Azure Backup. With Azure Backup, you can backup Azure Windows and Linux VMs, VMs running on Premium storage, and on Managed Disks as well with application-consistency. Now with CLI support, take advantage of CLI features such as smart defaults for most common operations, tab completion and pipe-able outputs to simply and quickly manage your VM backup and recovery operations. Azure Backup is supported with Azure CLI 2.0 only and is currently in preview.

How CLI will help

Cross-platform support: With Azure CLI, start managing your Azure VM backups from anywhere, irrespective of the platform you use.
Managing at scale: Easily handle operations at scale by including CLI commands in scripts and programs.
Automation: In conjunction with the command line tools, query on command outputs to identify important triggers and program subsequent actions. For example, backup health information can be programmatically retrieved, and subsequent operations can be initiated based on the health status.

Sample Scenarios:

Enabling Backup on a VM:

To protect a previously unprotected VMto a Recovery Services vault (within the same location and Resource group) with a default policy, use this command:  

az backup protection enable-for-vm
–resource-group myResourceGroup
–vault-name myRecoveryServicesVault
–vm myVM
–policy-name DefaultPolicy

For more information, refer to this quick start article to start backing up your VMs.

Restore:

You can restore files from VM backups without setting up any additional infrastructure and with a single command.

Use az backup recoverypoint list command to get recovery point name and then

az backup restore files mount-rp
–resource-group myResourceGroup
–vault-name myRecoveryServicesVault
–container-name myVM
–item-name myVM
–rp-name myRecoveryPointName

This will download a script to mount the recovery point. Run the script on the machine where you want to copy the files.

You can also restore disks from your VM backups and create your VMs from restored disks.

Management:

View your backup items’ health status using az backup item list command.

Azure Backup looks out for any configuration issues that might adversely impact VM backups and provides corresponding corrective measures. This is known as Backup pre-check. In CLI, this information is provided via “healthStatus” and “healthDetails” in az backup item list command’s JSON output. For example, in the image below, the health details section points out that the next backup might not be successful since the agent on the Linux machine is not up to date and it is recommended to have the agent upgraded.

You can use “–query” option along with az backup item list to extract health status and health details, and perform the recommended action, as suggested in the health details section, to ensure continuity of backups.

az backup item list
–resource-group myResourceGroup
–vault-name myRecoveryServicesVault
–container-name myVM
–query “[].properties.healthDetails

Getting Started

Install or upgrade to Azure CLI 2.0 to start using backup CLI commands.

Just type “az backup -h” to get more information related to backup CLI commands and start to manage your Azure VM backups

Related links and additional content

Azure backup CLI commands reference documentation.
New to Azure Backup? Sign up for a free Azure trial subscription.
Need help? Reach out to Azure Backup forum for support.
Tell us how we can improve Azure Backup by contributing new ideas and voting up existing ones.
Follow us on Twitter @AzureBackup for latest news and updates.

Quelle: Azure

Achieve better savings with best-in-class cost management on Azure

Last week at Ignite, our customers and partners like UPS, HSBC, Ernest & Young, and more shared incredible stories of how they are transforming their IT and businesses with Azure – revamping dev/test processes, modernizing applications, running mission critical workloads such as SAP, and taking their cloud investments to the next level with machine learning and AI. We also had an opportunity to share the latest Azure innovations and roadmap.

As customers look to do more with Azure, they want to know how they can lower, optimize and manage their cloud spend. To address this need, we are working hard to save you money on Azure with our existing billing model, new free cost management capabilities, and compelling offers.

Better than per-second billing for VMs

In Azure, we bill all virtual machines (VMs) running Linux and Windows on a per-minute basis, rounded down to the nearest minute. We only charge for the full minutes you use. This saves you money and simplifies your bill. Imagine a Linux VM or Windows VM that ran for 10 minutes and 59 seconds (659 seconds). On Azure, we would bill you only for 600 seconds while other cloud providers would charge for 659 seconds. In fact, we have been doing this for years.

In addition, Azure led the industry with a new service called Azure Container Instances (ACIs) that realizes the true value of a per-second billing model. ACI offers containers that start in seconds and are billed per-second without any infrastructure management. Customer interest of this innovative service is incredibly high, especially due to its ability to extend container orchestration offerings such as Kubernetes.

Free cost management solution

Beyond how we bill, we enable you to easily manage and optimize your cloud spend. Azure is the only cloud provider that offers robust cloud cost management and optimization without any additional charge. Azure Cost Management, also known as Cloudyn, empowers you to monitor, allocate, and optimize cloud spend so you maximize resource utilization and reinvest cost savings to further accelerate your business priorities in the cloud. For example, when you first deploy cloud projects you may not necessarily understand the capacity requirements, and as a result you may provision more resources than you need. Azure Cost Management is available for free for managing Azure spend.

Best savings for Windows Server workloads

Finally, if you are using Windows Server, Azure offers cost savings of up to 82%. This cost saving comes from the combination of Azure Hybrid Benefits, that enable customers with software assurance to use their existing Windows Server licenses in Azure, and the newly announced Azure Reserved VM Instances (RIs). We are also working to bring similar Azure Hybrid Benefits to SQL Server and expand RIs to additional Azure services. 

Other cost savings in Azure

The above are just a few examples of the cost savings features in Azure. We also recently announced the general availability of Azure Batch (low priority VMs), Azure RIs for Windows and Linux VMs, Partial Core Sizes for Oracle and SQL Licensing, and Burstable VMs. These offers will make your deployments on Azure more cost effective without compromising your cloud experience.

We are committed to helping you save money with a great billing and cost management experience so you can continue to do even more with Azure. For more information on Azure, please visit http://www.azure.com.
Quelle: Azure

Microsoft Azure allies with nearForm for Node.js Developer Migration and Support Services

At Node.js Interactive 2017, starting tomorrow in Vancouver, BC, nearForm and Microsoft are announcing a new alliance to help customers migrate Node.js apps to Azure, and provide enterprise-grade support for them. By partnering with nearForm we are now bringing their multi-year expertise in architecting, designing and supporting Node.js apps to developers as they adopt Azure and build on top of our cloud.

Every day, more and more Node.js developers choose to build and deploy on top of Azure, and Microsoft continues in its commitment to make investments in the area. The recent announcements of Azure App Service on Linux and Azure Web App for Containers, combined with support for Node.js across Azure Functions, Azure Container Service and pre-provisioned VM images, provide developers a great range of choices to host their Node.js apps and services across several cloud compute models offered by Azure.

nearForm Migration and Developer Support Services

nearForm is recognized as a leader in architecting and developing full-stack JavaScript solutions. With world-class teams of software architects, designers, developers, DevOps engineers and open source tooling experts, nearForm takes its clients worldwide through a current-architecture review workshop to a fully deployed high-performance new stack on Azure in a short amount of time.

As part of the alliance, Microsoft and nearForm are announcing today the nearForm Migration and Developer Support services, which enables customers to confidently move to the cloud with Node.js on Azure. Beyond the migration of technology, nearForm can help businesses migrate to offering new Software-as-a-Service (SaaS) capabilities: the nearForm design sprint is an intensive engagement which enables customers to generate a prototype of new business ideas in just five days. Customers can then work with nearForm's Solutions teams to turn the prototype into a production system.

nearForm's Developer Support service ensures that teams have direct access to the most knowledgeable Node.js developers in the industry, including Node.js Core contributors. This level of expertise can guide customers step-by-step through the process.

Customers interested in the Migration and Developer Services can request a quote from nearForm directly.

Free Node Clinic at Node Interactive 2017

Attendees at Node.js Interactive, in Vancouver from October 4-6, can request a Node Clinic at the event with nearForm’s Node.js experts and Microsoft’s Azure experts, including members of the Node.js Technical Steering Commitee. At each Node Clinic, clients can have a free, one-to-one, twenty-minute session to discuss any issues with their Node.js applications and services.
Quelle: Azure

Securing the Intelligent Edge

The Intelligent Edge brings the power of the cloud to mobile and Internet of Things (IoT) devices and demands security for trust. Azure IoT Edge is an implementation of the Intelligent Edge. Cloud-enabled computing at the edge means concentrating data, and therefore inherent value even if only momentarily. It also means moving tremendous value from the cloud to the edge in the form of intellectual property, algorithms, curated parameters, and value operations like policy enforcements, metering, and monetization. The Intelligent Edge is without a doubt a high-value bullseye to nefarious hacking and demands a high bar for security. Securing it requires a community effort and Microsoft, in alliance with many companies, is making great strides in this direction.

Why should an Intelligent Edge device demand more security than the cloud? Unlike the cloud that resides in remotely secluded and protected datacenters, the Intelligent Edge device is physically accessible to potentially malicious actors, and therefore exposed to greater threats. These range from physical tampering to exploits from repurposing tools and knowledge from other disciplines.

Figure 1: Additional threats to consider with cloud-enabled computing in Mobile and Internet of Things devices.

Knowing these threats exist creates awareness to foster the right solution. A careful security evaluation will reveal an optimal solution path that sometimes may need no more than diligence. This evaluation can happen in-house with sufficient expertise or farmed out to security auditors. Microsoft curates and maintains a list of security auditors in the Security Program for Azure IoT as a one stop shop evaluator. 

With dependence on applications, deployments, and risk profiles, it stands to reason that the optimal security for the Intelligent Edge device should fall on a spectrum of protection rigor.

On one end of this spectrum Intelligent Edge devices are deployed in physically secured and access controlled facilities, utilizing the so-called guards, dogs, and guns protection model. Depending on security audit and risk assessment results, due diligence on security best practices might be sufficient for this deployment. The Internet of Things from the ground up offers some guidance on areas for diligence.

On the other end of the spectrum are Intelligent Edge devices deployed in physically hostile environments. These include devices installed where they may be physically accessed without accountability. Examples may include smart building controllers for public buildings. Accessibility does not need to be localized to the installed device as a clever hacker may purchase a similar device for offline analysis and use the knowledge to attack installed devices. Intelligent Edge devices deployed in hostile environments therefore require security hardened hardware as the device itself is the last defense. Security hardened hardware offers resistance against attacks on keys and some offer trusted execution environments for higher trust.

There are several approaches to building security hardened hardware. For most, security hardening starts with appropriate processor architectures like those from ARM and Intel that add security features like secure enclaves. Silicon manufacturers then build on these processors to deliver hardened silicon chips that end up as the processing core of hardened Intelligent Edge devices. The Seven Properties of Highly Secure Devices offers a glimpse of necessary considerations for building hardened IoT devices. The hardening effort continues with firmware and software development best practices to adequately utilize the hardware security features. An ideal hardened Intelligent Edge device should lend itself to seamless secure provisioning, key management, authentication, authorization, and integrity protection of all content in memory and storage at all times.

Regardless of the approach, building secure hardware for hardened devices is difficult. It is for this reason Microsoft allies with domain masters of hardened hardware. Today we share progress with two of our secured silicon allies: NXP for LS1012 and Microchip for ATSAMA5D2, with both product families built using the ARM processor architecture with TrustZone® technology.

Firmware technology that provides the hardened hardware benefits to operating systems and applications is specialized and requires high expertise to develop. Today we share progress from one of our allies, Sequitur Labs, that has developed such firmware for both the LS1012 and ATSAMA5D2 secured hardware from NXP and Microchip respectively. Information on Sequitur’s solutions for hardened Azure IoT Edge devices is available from the Sequitur Labs website.

Microsoft is committed to making cloud-enabled edge computing a reality and regards security as a fundamental requirement for success. Microsoft is forging alliances to make Azure IoT Edge a secure Intelligent Edge platform that is operating system, processor architecture, and hardware agnostic.

Learn more about the Azure IoT Edge Intelligent Edge.
Quelle: Azure