Azure Stream Analytics support for IoT Hub Operations Monitoring

Azure Stream Analytics is a real-time, highly scalable, and fully managed stream analytics service that can process millions of events per second and perform aggregations and computations in near real-time.

Stream Analytics and Azure IoT Hub have worked really well together for some time, allowing you to easily gather insights over the data your IoT devices send to the IoT Hub. To get going with the IoT Hub all you need to do is simply configure an Input as described in the Create an IoT Hub data stream input documentation. We have many customers using this today with their IoT solutions and it works really well; however, a common ask from customers is how to monitor the status of operations on your IoT hub in real time. Well, today, we’re happy to announce that you can now do this by hooking up to the IoT Hub Operations Monitoring endpoint.

IoT Hub operations monitoring enables you to monitor the status of operations on your IoT hub in real time. IoT Hub tracks events across several categories of operations, and you can opt into sending events from one or more categories to an endpoint of your IoT hub for processing. You can monitor the data for errors or set up more complex processing based on data patterns.

IoT Hub monitors five categories of events:

Device identity operations
Device telemetry
Cloud-to-device commands
Connections
File uploads

For more information on IoT Hub Operations Monitoring please refer to the Introduction to operations monitoring documentation.

A common ask by customers is how to know in near real-time if a device disconnects from your IoT Hub and does not reconnect within a period of time, say one minute. When this occurs you want to send an email or kick-off a workflow in near real-time. For some devices it is crucial that these alerts go out as soon as possible and maintenance is carried out before there becomes a problem.

To demonstrate how easy it is to do this with Stream Analytics we’re going to use the IoT Hub Operations Monitoring capabilities and configure the “Connections” event monitoring.

 

As you can see from the image above, enabling starts logging events when devices connect and when they disconnect. To start using this, toggle the Connections switch to Verbose as shown:

 

Once this has been configured, events should start being captured when devices connect to your IoT hub. To verify this, use a tool like the Service Bus explorer and connect it to the "Event Hub – compatible name / endpoint" and view the messages.

A sample message collected could resemble something like this:

{
"durationMs": 1234,
"authType": "{"scope":"hub","type":"sas","issuer":"iothub"}",
"protocol": "Amqp",
"time": "2016-09-13T20:00Z",
"operationName": "deviceConnect",
"category": "Connections",
"level": "Error",
"statusCode": 4XX,
"statusType": 4XX001,
"statusDescription": "MessageDescription",
"deviceId": "device-ID"
}

Similarly, when a device disconnects from the IoT Hub, an event will be captured with the operationName == deviceDisconnect.

Now that we have confirmed these messages are arriving in our IoT Hub, using them in a Stream Analytics job is easy:

1. Create a new Stream Analytics job.

For assistance in creating a new Stream Analytics job, refer to How to create a data analytics processing job for Stream Analytics

2. Create a data stream Input pointed to your IoT Hub. Be sure to select “Operations monitoring” from Endpoint and not “Messaging”.

3. Create the following query:

WITH
Disconnected AS (
SELECT *
FROM input TIMESTAMP BY [Time]
WHERE OperationName = 'deviceDisconnect&039;
AND Category = &039;Connections&039;
),
Connected AS (
SELECT *
FROM input TIMESTAMP BY [Time]
WHERE OperationName = &039;deviceConnect&039;
AND Category = &039;Connections&039;
)

SELECT Disconnected.DeviceId, Disconnected.Time
INTO Output
FROM Disconnected
LEFT JOIN Connected
ON DATEDIFF(second, Disconnected, Connected) BETWEEN 0 AND 180
AND Connected.deviceId = Disconnected.deviceId
WHERE Connected.DeviceId IS NULL

This query has two steps: the first step that gets all disconnect events, and the second that gets all connect events.

We then join these two streams together using the Stream Analytics DATEDIFF operation on the LEFT JOIN, and then filter out any records where there was a match. This gives us devices that had a disconnect event, but no corresponding connect event within 180 seconds.

The output of this job can now be directed to any of the supported Stream Analytics outputs, including Service Bus queues. Once it lands in a Service Bus Queue, it is easy to create an Azure Function, or even an Azure Logic App, which will run as soon any message is published to the queue.

And just like that, with a very simple SQL-like query you can have real-time updates from your IoT Hub Operations Monitoring endpoint.

Sometimes I miss the good old days when coding complex scenarios like this was difficult and time consuming…no, wait, I don’t! Using the PaaS services and serverless computing capabilities of Azure is so much easier and powerful, allowing me to focus on building value add.

Related Services

Microsoft Azure IoT Hub – connect, monitor, and manage millions of IoT assets

Microsoft Azure Event Hubs – ingest data from websites, apps, and devices

Microsoft Azure Service Bus – Keep apps and devices connected across private and public clouds

Microsoft Azure Logic Apps – Quickly build powerful integration solutions

Microsoft Azure Functions – Process events with serverless code

Next Steps

We’re really excited about this close integration with IoT Hubs and hope it will unlock many new, exciting capabilities for you and your IoT applications.

We invite you to provide feedback on our User Voice page about what you want added next to the service!

If you are new to either Microsoft Azure or Stream Analytics, try it out by signing up for a free Azure trial account and create your first Stream Analytics job.

If you need help or have questions, please reach out to us through the MSDN or Stackoverflow forums, email the product team directly.
Quelle: Azure