Enhanced loading, monitoring, and troubleshooting experience for Azure SQL Data Warehouse

We are excited to share that Azure SQL Data Warehouse has introduced updates to the Azure portal and SQL Server Management Studio (SSMS) to provide a seamless experience when loading, monitoring, and developing your SQL Data Warehouse. The updates include integrated support for loading from 20+ data stores on premise and in the cloud, a simple process to troubleshoot common issues, and highly requested functionality within SSMS.

SQL Data Warehouse is your go-to SQL based view across data, offering a fast, fully managed, petabyte-scale cloud solution. It is highly elastic, enabling you to provision and scale up to 60 times larger in seconds. You can scale compute and storage independently, allowing you to range from burst to archival scenarios, and pay based off what you&;re using instead of being locked into a confined bundle.

Azure Portal

Quick Load

You can now quickly integrate your SQL Data Warehouse with Azure Data Factory (ADF) with the new task panel which consists of common commands to execute against your data warehouse.

Azure Data Factory is a fully managed cloud-based data integration service which can be used to populate a SQL Data Warehouse with data from your existing system. It saves you valuable time enabling you to focus on evaluating and building solutions and deriving faster insights. Here are the key benefits of loading data using Azure Data Factory with SQL Data Warehouse:

Rich data store support: built-in support for 20+ data stores on-premises and in the cloud.
Easy to set up: intuitive wizard with no scripting required.
Secure and compliant: data is transferred over HTTPS or ExpressRoute and the global service presence ensures your data never leaves the geographical boundary.
Unparalleled performance by using PolyBase: using PolyBase is the most efficient way to move data into Azure SQL Data Warehouse. To load from Azure Blob Storage, ADF uses PolyBase directly. To load from data stores other than Azure Blob Storage, ADF uses the staging blob feature to convert your source data into PolyBase compatible format, and then use PolyBase to load into SQL DW.

Troubleshooting

You can simply troubleshoot issues related to your SQL Data Warehouse with the new version of the troubleshoot blade. This provides instructions on how to self diagnose your SQL Data Warehouse for common issues.

Monitoring

In the query drill-down blade, you can now conveniently view the number of concurrency slots that your queries are consuming along with its resource class. This enables you to manage your workload.

SQL Server Management Studio

We have addressed issues across SQL Server Management Studio (SSMS) allowing you to:

Execute the Generate Scripts wizard for database users and user defined functions
View your logical server name within the table properties of your database
Drop your database simply through the Object Explorer
Leverage templates for stored procedures and scalar-valued functions

Learn More

Create a data warehouse today or check out the many resources for learning more about SQL Data Warehouse, including:

What is Azure SQL Data Warehouse?
SQL Data Warehouse best practices
Video library
MSDN forum
CAT team blogs
Stack Overflow forum

Quelle: Azure

AzCopy 5.1.1 Release

We are pleased to announce the release of AzCopy 5.1.1, and have made the following updates in this release:

Integration of Data Movement Library into AzCopy

A request we get often is "how does AzCopy work?" With this release, we have now integrated DMLib into AzCopy, which allows you to see exactly how our transfer logic works. Please refer to our Data Movement Library page to learn more.

We&;ve also made the following improvements:

Perform Blob and File uploads using a Write-Only SAS. When you use a Write-Only destination SAS, you must add the /Y parameter and remove the /XO and /XN parameters.
Create a destination container, share, or table with Account SAS.
During copy operation, show total amount of data transferred and average transfer speed.
Use full path in verbose log instead of relative path.
Journal file name has changed from AzCopyEntires.jnl to AzCopyCheckpoint.jnl.

Download the latest release version and learn more about how to use AzCopy by checking out the Getting Started with AzCopy documentation.

As always, we look forward to your feedback.

Microsoft Azure Storage Team
Quelle: Azure

Announcing Exclude Disk Support for Hyper-V VMs with ASR

A key ask that Azure Site Recovery (ASR) customers often have is the ability to exclude disks from replication, either to optimize the replication bandwidth consumed or to optimize the target-side resources utilized by such disks. ASR&;s VMware to Azure scenario has had this capability since earlier this year. Today, we are announcing the availability of this feature for ASR&039;s Hyper-V to Azure scenario as well.

Why do customers exclude disks from replication?

Excluding disks from replication is often necessary because:

The data churned on the excluded disk is not important or doesn’t need to be replicated (or / and)
Storage and network resources can be saved by not replicating this churn

Let&039;s elaborate on what data isn&039;t “important”? The importance of replicated data is determined by its usefulness at the time of Failover. Data that is not replicated must also not be needed at the time of failover. The absence of this data should not impact the Recovery Point Objective (RPO) in any way.

What are the typical scenarios?

There are some specific examples of data churn that can be easily identified and are great candidates for exclusion – for example any page file writes, Microsoft SQL server tempdb writes etc. Depending on the workload and the storage subsystem, the page file can register a significant amount churn. However, replicating this data from the primary site to Azure would be resource intensive and completely worthless. Thus the replication of a VM with a single virtual disk having both the OS and the page file can be optimized by:

Splitting the single virtual disk into two virtual disks – one with the OS and one with the page file
Excluding the page file disk from replication

Similarly, for Microsoft SQL Server with tempdb and system database file on the same disk can be optimized by:

Keeping the system database and tempdb on two different disks
Excluding the tempdb disk from replication.

How to Exclude disk from replication?

Follow the normal Enable replication workflow to protect a VM from ASR portal. In the 4th step of Enable replication there is a new column named DISK TO REPLICATE to exclude disk from the replication. By default all the disks are selected for the replication. Unselect the VHD that you want to exclude from replication and complete the steps to enable the replication. Learn more about it in the Hyper-V to Azure(with VMM) or Hyper-V to Azure(no VMM) documentation. You can also view this video to see the feature in action.

Excluding the SQL Server tempdb disk

Let&039;s consider the Hyper-V to Azure scenario for a SQL Server virtual machine that has a tempdb which can be excluded.

Name of the Hyper-V VM: SalesDB

Disks on the source Hyper-V VM:

VHD name
Guest OS disk#
Drive letter
Data type on the disk

DB-Disk0-OS
DISK0
C:
OS disk

DB-Disk1
Disk1
D:
SQL system database and User Database1

DB-Disk2 (Excluded the disk from the protection)
Disk2
E:
Temp files

DB-Disk3 (Excluded the disk from the protection)
Disk3
F:
SQL tempdb database (folder path(F:MSSQLData) –> note down the folder path before failover

DB-Disk4
Disk4
G:
User Database2

Since data churn on two disks of the VM are temporary in nature, while protecting SalesDB VM, exclude Disk2 and Disk3 from the replication. ASR will not replicate those disks and on failover those disks will not be present on the failover VM on Azure

Disks on the Azure VM after failover:

Guest OS disk#
Drive letter
Data type on the disk

DISK0
C:
OS disk

Disk1
E:
Temporary storage [Azure adds this disk and assigns the first available drive letter]

Disk2
D:
SQL system database and User Database1 

Disk3
G:
User Database2

Since Disk2 and Disk3 were excluded from SalesDB VM, E: is the first drive letter from the available list. Azure assigns E: to temporary storage volume. For all the replicated disks, drive letter remains the same.

Disk3 which was SQL tempdb disk (tempdb folder path F:MSSQLData) and excluded from replication, the disk is not available on the failover VM. As a result, the SQL service is in stopped state and it needs the F:MSSQLData path.

There are two ways in which you can create this path.

Add a new disk and assign tempdb folder path or
Use existing temporary storage disk for tempdb folder path

Add a new disk:

Note down the SQL tempdb.mdf and tempdb.ldf path before failover.
From the Azure portal, add a new disk to the failover VM with the same or more size as that of source SQL tempdb disk (Disk3).
Login to the Azure VM. From the disk management(diskmgmt.msc) console initialize and format the newly added disk.
Assign the same drive letter that was used by SQL tempdb disk (F:).
Create tempdb folder on F: volume (F:MSSQLData).
Start SQL service from service console.

Use existing temporary storage disk for SQL tempdb folder path:

      1.    Open a command line console

      2.    Run SQL server in recovery mode from command line console

Net start MSSQLSERVER /f / T3608

       3.   Run the following sqlcmd to change the tempdb path to new path

sqlcmd -A -S <SQL server instance name>

sqlcmd -A -S SalesDB

USE master;

GO

ALTER DATABASE tempdb

MODIFY FILE (NAME = tempdev, FILENAME = &039;E:MSSQLtempdatatempdb.mdf&039;);

GO

ALTER DATABASE tempdb

MODIFY FILE (NAME = templog, FILENAME = &039;E:MSSQLtempdatatemplog.ldf&039;);

GO

      4.   Stop Microsoft SQL server service.

Net stop MSSQLSERVER

       5.   Start Microsoft SQL server service.

Net start SSQLSERVER

Refer to the following Azure guideline for temporary storage disk

Using SSDs in Azure VMs to store SQL Server TempDB and Buffer Pool Extensions
Performance best practices for SQL Server in Azure Virtual Machines

Failback (from Azure to on-premises)

Now let&039;s understand what all disks will be replicated when you do failover from Azure to your on-premises Hyper-V host. Disks that you create manually in Azure will be not be replicated. For example, if you fail over three disks and create two directly in Azure VM, only three disks which were failed over will be failed back. You can&039;t include disks created manually in failback or in re-protect from on-premises to Azure. It also does not replicate temporary storage disk to on-premises.

Failback to OLR

When failback is done to the original location, failback VM disk configuration remains the same as that of original VM disk configuration. That means the disks which were excluded from Hyper-V site to Azure, will be available on the failback VM.

In the above example, Azure VM disk configuration:

Guest OS disk#
Drive letter
Data type on the disk

DISK0
C:
OS disk

Disk1
E:
Temporary storage [Azure adds this disk and assigns the first available drive letter]

Disk2
D:
SQL system database and User Database1

Disk3
G:
User Database2

After planned failover from Azure to on-premises Hyper-V, disks on the Hyper-V VM(Original Location Replication):

VHD name
Guest OS disk#
Drive letter
Data type on the disk

DB-Disk0-OS
DISK0
C:
OS disk

DB-Disk1
Disk1
D:
SQL system database and User Database1

DB-Disk2 (Excluded disk)
Disk2
E:
Temp files

DB-Disk3
(Excluded disk)
Disk3
F:
SQL tempdb database (folder path(F:MSSQLData)

DB-Disk4
Disk4
G:
User Database2

Exclude Paging file disk

Let&039;s consider the Hyper-V to Azure scenario for a virtual machine which has a pagefile disk that can be excluded.

There are two cases:

Case1: Pagefile is configured on the D: drive

Hyper-V VM disk configuration:

VHD name
Guest OS disk#
Drive letter
Data type on the disk

DB-Disk0-OS
DISK0
C:
OS disk

DB-Disk1
(Excluded the disk from the protection)
Disk1
D:
pagefile.sys

DB-Disk2
Disk2
E:
User data 1

DB-Disk3
Disk3
F:
User data 2

Pagefile settings on the Hyper-V VM:

After you failover the VM from Hyper-V to Azure, disks on Azure VM:

VHD name
Guest OS disk#
Drive letter
Data type on the disk

DB-Disk0-OS
DISK0
C:
OS disk

DB-Disk1
Disk1
D:
Temporary storage –> pagefile.sys

DB-Disk2
Disk2
E:
User data 1

DB-Disk3
Disk3
F:
User data 2

Since Disk1 (D:) was excluded, D: is the first drive letter from the available list, Azure assigns D: letter to temporary storage volume.  Since D: is available on the Azure VM, pagefile setting of the VM remains the same.

Pagefile settings on Azure VM:

 

Case2: Pagefile file is configured on any other drive(other than D: drive)

Hyper-V VM disk configuration:

VHD name
Guest OS disk#
Drive letter
Data type on the disk

DB-Disk0-OS
DISK0
C:
OS disk

DB-Disk1 (Excluded the disk from the protection)
Disk1
G:
pagefile.sys

DB-Disk2
Disk2
E:
User data 1

DB-Disk3
Disk3
F:
User data 2

Pagefile settings on the Hyper-V VM:

After you failover the VM from Hyper-V to Azure, disks on Azure VM:

VHD name
Guest OS disk#
Drive letter
Data type on the disk

DB-Disk0-OS
DISK0
C:
OS disk

DB-Disk1
Disk1
D:
Temporary storage –> pagefile.sys

DB-Disk2
Disk2
E:
User data 1

DB-Disk3
Disk3
F:
User data 2

Since D: is the first drive letter available from the list, Azure assigns D: to temporary storage volume. For all the replicated disks, drive letter remains the same. Since G: disk is not available system will use C: drive for pagefile.

Pagefile settings on Azure VM:

You can check out additional product information, and start replicating your workloads to Microsoft Azure using Azure Site Recovery today. You can use the powerful replication capabilities of Site Recovery for 31 days at no charge for every new physical server or virtual machine that you replicate. Visit the Azure Site Recovery forum on MSDN for additional information and to engage with other customers, or use the ASR User Voice to let us know what features you want us to enable next.

Azure Site Recovery, as part of Microsoft Operations Management Suite, enables you to gain control and manage your workloads no matter where they run (Azure, AWS, Windows Server, Linux, VMware or OpenStack) with a cost-effective, all-in-one cloud IT management solution. Existing System Center customers can take advantage of the Microsoft Operations Management Suite add-on, empowering them to do more by leveraging their current investments. Get access to all the new services that OMS offers, with a convenient step-up price for all existing System Center customers. You can also access only the IT management services that you need, enabling you to on-board quickly and have immediate value, paying only for the features that you use.

Pagefile is configured on D: drive
Pagefile file is configured on any other drive(other than D: drive)

Quelle: Azure

New Azure Logic Apps innovation – general availability of cloud-based Enterprise Integration Pack

Businesses are looking for more ways to reduce infrastructure costs without compromising service availability. This results in companies looking for newer cloud development architectures like serverless, giving rise to the need for event triggered integration across multiple third party services.  Developers are turning to serverless solutions like Azure Logic Apps and Azure Functions to automate workflows and integrate systems, thereby accelerating application delivery and reducing costs. Logic Apps enables customers to quickly and easily build powerful integration solutions using a visual designer and a wide set of out-of-the-box connectors such as Dynamics CRM, Salesforce, Office 365 and many more.

Today I am excited to announce another important milestone in integration- the general availability of Enterprise Integration Pack within Logic Apps, which further simplifies business-to-business (B2B) communications in the cloud. It enables you to more easily process business transactions reliably, track and troubleshoot B2B events and leverage additional out-of-the-box connectors.

Electronic Data Interchange (EDI) and Business-to-Business (B2B) transactions

With Enterprise Integration Pack, you can take advantage of a faster, more reliable and versatile B2B/EDI solution than traditional integration solutions. Integration accounts within Enterprise Integration Pack quickly create and manage cloud based B2B related artifacts such as maps, schemas, trading partners, agreements and certificates. With this release, electronic data interchange (EDI) has never been easier. You can send, receive and troubleshoot B2B transactions across a wide variety of protocols including  AS2, EDIFACT and X12. Customers like Mission Linen Supply are already realizing the benefits of EDI capabilities in Logic Apps: “Today, with our Azure Logic Apps solution, we can get suppliers onboarded within two weeks versus the two months or longer that the [Electronic Data Interchange] provider required. The faster we can integrate partners, the faster we can grow our business.” – Dave Pattison, Director of IT, Mission Linen Supply

Below is a view of the new integration account within Enterprise Integration Pack:

Management capabilities

The ability to view and troubleshoot B2B events via system management solutions is as important as enabling comprehensive EDI capabilities. With the Enterprise Integration Pack, you can track B2B events in a number of flexible ways like built-in tracking which can be routed to Microsoft Operations Management Suite (OMS) using the out-of-the-box tracking portal. You can easily view and troubleshoot B2B transactions over AS2 and X12 formats (with EDIFACT coming in the next few weeks). Additionally, a new RESTful tracking API enables you to send tracking events from both Logic App executions as well as other applications for end to end visibility. You can also add and correlate tracking data across your entire business process in Operations Management Suite.

Here is a view of tracking B2B events through the Operations Management Suite portal:

Enterprise connectors

We realize that many of our customers are often dealing with mission-critical applications and business processes that can lead to complex, time consuming connection configuration steps. With Logic Apps, we have added more enterprise connectors that make it simple and fast to establish connections with business applications. For instance, with the SAP connector, you can easily connect your on-premises SAP systems to cloud applications using Logic Apps without the need for complex coding required by other serverless products in the market. Today, the MQ Series and SAP ECC connectors are in preview, with more connectors coming in the next few months. For a full list of all currently available connectors, please visit the Logic Apps connectors reference.

Get started today!

With the general availability of Enterprise Integration Pack, you can now start using these services in production with full SLA and support. We are committed to continuous delivery of serverless compute and integration capabilities and will continue to share updates about investments and new releases. In the meanwhile, learn more and try our serverless offerings – Azure Logic Apps and Azure Functions. As always, please share your comments and continue to engage with our Logic Apps and Functions teams.

Also, be sure to check out Logic Apps pricing page and the documentation page on how you can start consuming these features today.
Quelle: Azure

Introducing Change Feed support in Azure DocumentDB

 We’re excited to announce the availability of Change Feed support in Azure DocumentDB! With Change Feed support, DocumentDB provides a sorted list of documents within a DocumentDB collection in the order in which they were modified. This feed can be used to listen for modifications to data within the collection and perform actions such as:

Trigger a call to an API when a document is inserted or modified
Perform real-time (stream) processing on updates
Synchronize data with a cache, search engine, or data warehouse

DocumentDB&;s Change Feed is enabled by default for all accounts, and does not incur any additional costs on your account. You can use your provisioned throughput in your write region or any read region to read from the change feed, just like any other operation from DocumentDB.

In this blog, we look at the new Change Feed support, and how you can build responsive, scalable and robust applications using Azure DocumentDB.

Change Feed support in Azure DocumentDB

Azure DocumentDB is a fast and flexible NoSQL database service that is used for storing high-volume transactional and operational data with predictable single-digit millisecond latency for reads and writes. This makes it well-suited for IoT, gaming, retail, and operational logging applications. These applications often need to track changes made to DocumentDB data and perform various actions like update materialized views, perform real-time analytics, or trigger notifications based on these changes. Change Feed support allows you to build efficient and scalable solutions for these patterns.

Many modern application architectures, especially in IoT and retail, process streaming data in real-time to produce analytic computations. These application architectures (“lambda pipelines”) have traditionally relied on a write-optimized storage solution for rapid ingestion, and a separate read-optimized database for real-time query. With support for Change Feed, DocumentDB can be utilized as a single system for both ingestion and query, allowing you to build simpler and more cost effective lambda pipelines. For more details, read the paper on DocumentDB TCO.

 

Stream processing: Stream-based processing offers a “speedy” alternative to querying entire datasets to identify what has changed. For example, a game built on DocumentDB can use Change Feed to implement real-time leaderboards based on scores from completed games. You can use DocumentDB to receive and store event data from devices, sensors, infrastructure, and applications, and process these events in real-time with Azure Stream Analytics, Apache Storm, or Apache Spark using Change Feed support.

Triggers/event computing: You can now perform additional actions like calling an API when a document is inserted or modified. For example, within web and mobile apps, you can track events such as changes to your customer&039;s profile, preferences, or location to trigger certain actions like sending push notifications to their devices using Azure Functions or App Services.

Data Synchronization: If you need to keep data stored in DocumentDB in sync with a cache, search index, or a data lake, then Change Feed provides a robust API for building your data pipeline. Change feed allows you to replicate updates as they happen on the database, recover and resume syncing when workers fail, and distribute processing across multiple workers for scalability.

 

Working with the Change Feed API

Change Feed is available as part of REST API 2016-07-11 and SDK versions 1.11.0 and above. See Change Feed API for how to get started with code.

 

 

The change feed has the following properties:

Changes are persistent in DocumentDB and can be processed asynchronously.
Changes to documents within a collection are available immediately in the change feed.
Each change to a document appears only once in the change feed. Only the most recent change for a given document is included in the change log. Intermediate changes may not be available.
The change feed is sorted by order of modification within each partition key value. There is no guaranteed order across partition-key values.
Changes can be synchronized from any point-in-time, that is, there is no fixed data retention period for which changes are available.
Changes are available in chunks of partition key ranges. This capability allows changes from large collections to be processed in parallel by multiple consumers/servers.
Applications can request for multiple Change Feeds simultaneously on the same collection.

Next Steps

In this blog post, we looked the new Change Feed support in Azure DocumentDB.

Learn more about Change Feed support in Azure DocumentDB
Upgrade to .NET SDK 1.11.0 with Change Feed support
Create a new DocumentDB account from the Azure Portal or download the DocumentDB Emulator
Stay up-to-date on the latest DocumentDB news and features by following us on Twitter @DocumentDB or reach out to us on the developer forums on Stack Overflow

Quelle: Azure

Use the Cloud to help people in need

In this holiday time, peoples’ thoughts turn to helping those less fortunate. If you’re a charity, helping people is your business all year round, and every dollar spent on computing infrastructure is a dollar taken away from those you’re trying to help. That’s why, increasingly, charities and nonprofits are turning away from expensive on-premises solutions and moving to the cloud.

Take Minnesota-based RREAL, the Rural Renewable Energy Alliance, which helps low-income households receive the advantages of solar power.

As their business grew, RREAL needed ways to track new customers, to manage their manufacturing, and to collaborate without incurring the costs and overhead of expensive infrastructure. Azure-based Dynamics GP and Dynamics CRM along with Office 365 met these needs. And with Dynamics’ Business Analyzer they track KPI’s, and with PowerBI they have a have real-time dashboard.

The result: they have cut installation time by over 50 percent: more people get cheap, renewable power faster.

World Animal Protection is a global organization focusing on animal welfare. Like all charities, transparency is a key goal: donors need to know how their contributions are being used. And, because their teams can be anywhere in the world, their applications must be accessible globally. Finally, as with all nonprofits, keeping administrative costs down is essential.

With infrastructure and applications managed by the Microsoft Cloud, workers can devote more time to their core mission: improving the lot of the world’s animals. Their technology that best suited their needs: Office 365, Dynamics and Azure.

The mission of UK-based JustGiving is to “ensure no great cause goes unfunded.” With a large social network of donors, they found that an individual’s “social graph” is a good predictor of that person’s next donation: if your friends care about something, you probably do too.

But in considering a technology solution, the size of this dataset — with 361 million connections – was daunting. Enter HDInsight, Microsoft Azure’s big data service, based on Hadoop, Spark, and R. A relational database – based on rows and columns – was inappropriate for a scenario where one person could link to any number of others. Instead, their solution, called GiveGraph™, integrated Facebook’s OpenGraph technology with HDInsight to provide a robust, massively scalable database of people’s relationships to other people — and to their causes.

I hope that you too will consider giving during this holiday season. We’re proud that it’s an important part of our corporate culture: Since our giving program began, Microsoft employees have donated over $1 billion to worthy causes, and just this year our Philanthropies group announced a program to provide Azure, Office 365, Dynamics and other products free to eligible organizations.

* * *

Of course, the benefits reaped by non-profits and charitable organizations from the cloud – reduced costs, increased agility, new kinds of innovative solutions, global reach – are the same ones any enterprise will reap by using the cloud.

As I mentioned in my last post, we see the cloud providing nearly limitless opportunities to improve our lives; the cloud should benefit everyone. We believe governments and enterprises, along with vendors like ourselves, should work together to ensure that, jointly, the cloud we all build is trusted, responsible and inclusive.

Our ebook, A Cloud for Global Good, provides an initial set of 78 policy recommendations in 15 categories for governments and regulatory bodies to consider. The topic areas are thought-provoking, and far from simple: ranging from how we all should about personal privacy to government access to data, to cross-border data flows to supporting digital literacy around the world. As I said last time, we don’t have all the answers; we don’t even have all the questions. However, if with A Cloud for Global Good we can start a conversation, we’ll have achieved something.

So: let us know what you think!
Quelle: Azure

Latest update of Azure Analysis Services preview brings scale up and down

The latest update to the Azure Analysis Services preview brings the ability to increase or reduce the capacity of a server after it has been created. For example, if there is a particular time of year when you expect query volume to be higher, you can simply scale the service up to get more QPUs. After that period is over, you can reduce your capacity by simply scaling down. The ability to scale up and down and adapt to changing workload demand is a one of the benefits of using Azure Analysis Services to host your models

How to scale

Pricing tier scale up and down can be done manually in the Azure portal or automated through the Azure ARM APIs and PowerShell.

To use this feature in the Azure portal, simply locate your server.

On the left-hand side, you will see an option called scale.

Clicking on this will now let you choose a new pricing tier. Currently my server is set to S1. I will scale it up to S2 by selecting S2 Standard and clicking Select.

Note: When scaling down, ensure that you have enough cache capacity for the models which are already deployed to the server or all models may not be able to load until you scale back up. You can see your current memory usage by clicking on Metrics on the left-hand side.

Within a few moments, you will see that the server now has the capacity of an S2.

 

 

Learn more about Azure Analysis Services.
Quelle: Azure

Azure IoT Hub message routing dramatically simplifies IoT solution development

IoT solutions can be complex, and we’re always working on ways to simplify them.

As we work with customers on real-world, enterprise-grade IoT solutions built on Azure IoT, one pattern we’ve noticed is how businesses route inbound messages to different data processing systems.

Imagine millions of devices sending billions of messages to Azure IoT Hub. Some of those messages need to be processed immediately, like an alarm indicating a serious problem. Some messages are analyzed for anomalies. Some messages are sent to long term storage. In these cases, customers have to build routing logic to decide where to send each message:

While the routing logic is straightforward conceptually, it’s actually really complex when you consider all of the details you have to handle when you build a dispatching system: handling transient faults, dealing with lost messages, high reliability, and scaling out the routing logic.

To make all this easier, we’ve made a great new feature to IoT Hub generally available: message routing. This allows customers to setup automatic routing to different systems via Azure messaging services and routing logic in IoT Hub itself, and we take care of all of the difficult implementation architecture for you:

You can configure your IoT hub to route messages to your backend processing services via Service Bus queues, topics, and Event Hubs as custom endpoints for routing rules. Queuing and streaming services like Service Bus queues and Event Hubs are used in many if not all messaging applications. You can easily set up message routing in the Azure portal. Both endpoints and routes can be accessed from the left-hand info pane in your IoT Hub:

You can add routing endpoints from the Endpoints blade:

You can configure routes on your IoT Hub by specifying the data stream (device telemetry), the condition to match, and the endpoint to which matching messages are written.

Message routing conditions use the same query language as device twin queries and device jobs. IoT Hub evaluates the condition on the properties of the messages being sent to IoT Hub and uses the result to determine where to route messages. If messages don’t match any of your routes, the messages are written to the built-in messages/events endpoint just like they are today.

We have also enhanced our metrics and operations monitoring logs to make it easy for youto tell when an endpoint is misbehaving or whether a route was incorrectly configured. You can learn about the full set of metrics that IoT Hub provides in our documentation including how each metric is used.

Azure IoT is committed to offering our customers high availability message ingestion that is secure and easy to use. Message routing takes telemetry processing to the next step by offering customers a code-free way to dispatch messages based on message properties. Learn more about today&;s enhancements to Azure IoT Hub messaging by reading the developer guide. We firmly believe in customer feedback, so please continue to submit your suggestions through the Azure IoT User Voice forum or join the Azure IoT Advisors Yammer group.
Quelle: Azure

Scaling Up "Project Springfield" using Azure

This guest post is from the Project Springfield team in Microsoft’s Artificial Intelligence and Research group. Project Springfield delivers pioneering artificial intelligence for finding security issues as a cloud service. Learn how the team used Azure to meet and exceed scaling challenges on a tight timeline.

The Project Springfield engineering team, led by William Blum, had built the first release of “Project Springfield,” which helped customers find “million-dollar” security bugs by combining pioneering “whitebox fuzzing” technology from Microsoft Research with the elasticity of the cloud. Customers could upload their software to Project Springfield, which created a fuzzing lab in Azure. Each fuzzing lab tested with a portfolio of methods, looked for crashes from the test cases, and then picked the highest value issues to report. The power of Azure enabled this compute intensive process to scale up and scale down as customers’ demands changed, while simultaneously collecting data from every run to improve the service. To do this, Project Springfield had to dynamically create large numbers of virtual machine and network resources and manage them on behalf of the customer.

We had built the initial product on Azure, using the classic Azure management interface to dynamically provision virtual machines and networking resources. Now it was time to prepare for a new wave of customers – which meant scaling up the service by orders of magnitude. Scaling with the classic Azure would be a challenge. For example, each fuzzing lab used up a different cloud service on Project Springfield, yet there was limit of just 200 cloud services per subscription. That meant if the customers, in aggregate, ever needed to test more than 200 pieces of software at a time, Project Springfield would need to partition fuzzing labs across subscriptions even if each subscription otherwise had enough virtual machines available to serve customers. There had to be a better way.

We found that better way by re-architecting the service with the Azure management interface Azure Resource Manager as well as Service Fabric, Microsoft&;s micro-service-based application platform. With Azure Resource Manager, virtual machines, virtual networks, and load balancers are all treated as different resources. These resources can be combined in an Azure Resource Manager template, which is a JSON object defining what resources we needed and how they fit together. All the resources Project Springfield needs for a security testing lab are specified by a single template. When a customer needs a new lab, Azure can read the template and then dynamically create all the resources needed from the template. With Service Fabric, we could easily port our backend worker roles to micro services and dynamically scale up and scale down backend resources based on customer needs. The payoff was that instead of being locked into a single inflexible bundle, we could dynamically reshape the way resources were deployed.

Re-architecting the service around the new deployment concepts introduced by ARM required some work. The work paid off as we found that Azure’s infrastructure-as-a-service capabilities gave us better control and finer granularity over the configuration of our network and compute resources. Once adjusted to a new way of thinking, we could see how to make Project Springfield even more efficient and deliver value. For example, we realized that by using Azure Network Security Groups, we could enable each customer to set different IP address restrictions on who could access their Project Springfield resources – a key feature for enterprise users.

Even better, betting on Azure made us “future proof.” As Azure launched new features, such as support for Red Hat Linux, Windows Server Containers and more, we could see how they would let Project Springfield meet customer needs. With Azure Resource Manager, these are now just different kinds of resources in a resource manager template. That gave the team a single consistent way for managing fuzzing labs and laid the foundation for eventually offering different types of fuzzing labs for different customer needs.

By using the new capabilities of Azure, such as Azure Resource Manager, we achieved our scale goals in four months. That meant we could bring on customers and partners for trials of Project Springfield as fast as we could call them, without worrying that we would run out of capacity. What’s more, building on Azure set the team up for success as new capabilities came online. At Microsoft Ignite 2016, OSIsoft & Deschutes Brewery, EY, and Leviathan Security Group stood on stage and told the world about the value they saw in Project Springfield. Within a week over a thousand people signed up for trials! That’s a win by any standard.

Related content

Security testing in the cloud with F# and Project Springfield

Azure Resource Manager overview

Project Springfield: a cloud service built entirely in F#

More about Project Springfield

The Project Springfield team
Quelle: Azure

StorSimple as a Backup Target

We are pleased to announce that backup-to-disk workloads will be supported on the StorSimple 8000 Series devices running Update 3 or later. We are also making deployment guides available to help configure backup applications that use StorSimple as a disk-based backup target.

StorSimple combines the performance and compatibility of local storage with the scale, resiliency, and availability of cloud object storage, providing the best of both worlds.

StorSimple Updates?

With the release of Update 3 for the StorSimple 8000 series devices, we have introduced a new backup mode that tunes the device and enables it to be used as a backup-to-disk target. We have also included improvements to cloud throughput and enhancements around dealing with high frequency data churn. These additions enable the device to be configured as both primary and secondary backup target.

Why use StorSimple as a backup target?

Typically, >95% of restores are served from the last 7 days of backups. StorSimple’s cloud-centric architecture ensures that the most recently written backup will remain on the device’s local storage capacity, while the older, archival-class backups tier out to cloud storage.

This approach facilitates:

 

Faster backups using local storage
Faster restore from local storage
Azure as offsite storage
On-demand elastic expansion, cloud scale, and cloud economics for backup capacity
Built-in Disaster Recovery in Azure IaaS
Removal of offsite media management
No media migration between backup media types or formats
Enterprise grade device with dual storage controllers, both 1 GbE and 10 GbE connectivity, and seamless integration with Microsoft Azure.
Reuse data in Azure using StorSimple device in Azure

Once the backup data is in the cloud, StorSimple device in Azure and StorSimple Data Manager service can be used to transform the data into workable datasets for a myriad use cases.

Which backup software applications are supported today?

StorSimple is certified with Veritas Backup Exec, Veritas NetBackup, and Veeam

Please refer the below configuration guides for vendor-specific configurations.

Configure StorSimple with Veritas Backup Exec
Configure StorSimple with Veritas NetBackup
Configure StorSimple with Veeam

There are ongoing efforts to certify additional software vendors – watch this space for the latest news!

Typical deployment topology

The StorSimple 8000 series volumes are deployed as disk-based backup targets for backup applications. An example of such a deployment is as follows:

Quelle: Azure