Explore and analyze time stamped data with Time Series Insights

Lab Scenario

Your hard work implementing Azure IoT services and tools has paid off. Contoso has rolled out their “Asset Condition Tracking System” that monitors the environmental conditions of cheese containers during shipment.

Two weeks after launching the new system it has revealed a temperature spike in-transit for a specific shipment. Some of the cheese in the shipment was ruined, but the new system did ensure that the affected cheese wasn’t delivered to the customer. Since you know the Azure IoT side of the monitoring system better than anyone, you will lead the investigation.

Management has asked you to determine if the system can be improved, hopefully to the point where it will prevent the loss of product in the future. You correlate the IoT devices’ sensor data taken from the trucks and planes used during the shipment. It appears that the temperature in one of the trucks rose unexpectedly in a particular area of the vehicle and created the heat spike in one of the transport containers (which was equipped with an IoT device monitoring temperature and humidity).

Your team decides that further improvements to the monitoring system will require near real-time data exploration and root-cause analysis.

You propose adding Time Series Insights to the Azure IoT solution. This will enable Contoso to quickly store, visualize, and query the large amounts of time series data that are generated by the IoT devices in the trucks, planes, and containers, and to visualize changes over time.

The following resources will be created:

Lab 10 Architecture

In This Lab

In this lab, you will complete the following activities:

  • Configure Lab Prerequisites (required Azure resources)
  • Create an Azure Time Series Insights (TSI) environment
  • Connect to IoT Hub with Time Series Insights (TSI)
  • View time series data using the Time Series Insights (TSI) Explorer

Lab Instructions

Exercise 1: Configure Lab Prerequisites

This lab assumes the following Azure resources are available:

Resource Type Resource Name
Resource Group rg-az220
IoT Hub iot-az220-training-{your-id}
Device ID sensor-th-truck0001
Device ID sensor-th-airplane0001
Device ID sensor-th-container0001

To ensure these resources are available, complete the following steps.

  1. In the virtual machine environment, open a Microsoft Edge browser window, and then navigate to the following Web address:

    +++https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FMicrosoftLearning%2FAZ-220-Microsoft-Azure-IoT-Developer%2Fmaster%2FAllfiles%2FARM%2Flab10.json+++

    NOTE: Whenever you see the green “T” symbol, for example +++enter this text+++, you can click the associated text and the information will be typed into the current field within the virtual machine environment.

  2. If prompted to log in to the Azure portal, enter the Azure credentials that you are using for this course.

    The Custom deployment page will be displayed.

  3. Under Project details, in the Subscription dropdown, ensure that the Azure subscription that you intend to use for this course is selected.

  4. In the Resource group dropdown, select rg-az220.

    NOTE: If rg-az220 is not listed:

    1. Under the Resource group dropdown, click Create new.
    2. Under Name, enter rg-az220.
    3. Click OK.
  5. Under Instance details, in the Region dropdown, select the region closest to you.

    NOTE: If the rg-az220 group already exists, the Region field is set to the region used by the resource group and is read-only.

  6. In the Your ID field, enter the unique ID you created in Exercise 1.

  7. In the Course ID field, enter az220.

  8. To validate the template, click Review and create.

  9. If validation passes, click Create.

    The deployment will start.

  10. Once the deployment has completed, in the left navigation area, to review any output values from the template, click Outputs.

    Make a note of the outputs for use later:

    • connectionString
    • deviceIDs

    Important: The deviceIDs output contains a JSON array with the connection strings for each device. Use the text editor of your choice to extract the connection strings:

     [
         {
             "name":"createDevice-sensor-thl-truck0001",
             "connectionString":"HostName=iot-az220-training-dm072321.azure-devices.net;DeviceId=sensor-thl-truck0001;SharedAccessKey=yJV1FfKyVg5SBA5M9L3D3u1DqoiINYDaEuXv3KWHekA="
         },
         {
             "name":"createDevice-sensor-thl-airplane0001",
             "connectionString":"HostName=iot-az220-training-dm072321.azure-devices.net;DeviceId=sensor-thl-airplane0001;SharedAccessKey=GN+aU3P4oDAhLddLuCKKwTYkinGwl828KvGQTZwxYMc="
         },
         {
             "name":"createDevice-sensor-thl-container0001",
             "connectionString":"HostName=iot-az220-training-dm072321.azure-devices.net;DeviceId=sensor-thl-container0001;SharedAccessKey=lNZEp9xMYBYOidaDxHZcH9MWdOrDgyhmmUrZYBIhwXg="
         }
     ]
    

The resources have now been created.

Exercise 2: Setup Time Series Insights

Azure Time Series Insights (TSI) is an end-to-end platform-as-a-service offering used to collect, process, store, analyze, and query data from IoT solutions at scale. TSI is designed for ad hoc data exploration and operational analysis of data that’s highly contextualized and optimized for time series.

In this exercise, you will setup Time Series Insights integration with Azure IoT Hub.

  1. Login to portal.azure.com using your Azure account credentials.

    If you have more than one Azure account, be sure that you are logged in with the account that is tied to the subscription that you will be using for this course.

  2. On the Azure portal menu, click + Create a resource.

  3. On the New blade, in the Search the Marketplace textbox, enter time series insights

  4. In the search results, click Time Series Insights.

  5. On the Time Series Insights blade, click Create.

  6. In the Subscription dropdown, select the subscription that you are using for this course.

  7. In the Resource group dropdown, click rg-az220.

  8. On the Create Time Series Insights environment blade, in the Environment name field, enter tsi-az220-training

  9. In the Location dropdown, select the Azure region used by your resource group.

  10. In the Tier field, ensure that the Gen1 (S1) pricing tier is selected and that Capacity is set to 1.

  11. In the Environment name field, enter tsi-az220-training-{your-id}.

  12. At the bottom of the blade, click Next: Event Source.

  13. Under the EVENT SOURCE DETAILS section, ensure that Create an event source? is set to Yes.

  14. In the Source type dropdown, ensure that IoT Hub is selected.

  15. In the Name field, enter iot-az220-training-{your-id} to specify a unique name for this Event Source.

  16. In the Subscription dropdown, select the subscription that you are using for this course.

  17. In the IoT Hub name dropdown, select the iot-az220-training-{your-id} Azure IoT Hub service that’s already been provisioned.

  18. In the IoT Hub access policy name dropdown, click iothubowner.

    In a production environment, it’s best practice to create a new Access Policy within Azure IoT Hub to use for configuring Time Series Insights (TSI) access. This will enable the security of TSI to be managed independently of any other services connected to the same Azure IoT Hub. You are not doing that here for convenience reasons.

  19. Under the CONSUMER GROUP section, next to the IoT Hub consumer group dropdown, click New.

  20. In the IoT Hub consumer group box, enter tsievents and then click Add.

    This will add a new Consumer Group to use for this Event Source. The Consumer Group needs to be used exclusively for this Event Source, as there can only be a single active reader from a given Consumer Group at a time.

  21. Under the Start options section, in the STart time dropdown, select Beginning now (default).

  22. Under the TIMESTAMP section, leave the Property Name blank.

  23. At the bottom of the blade, click Review + create.

    Note: If you are returned immediately to the Event Source pane, double check that you clicked Add to the right of the IoT Hub consumer group field - you cannot create the TSI resource until you have created the consumer group.

  24. At the bottom of the blade, click Create.

    Note: Deployment of Time Series Insights (TSI) will take a couple minutes to complete.

  25. Once your Time Series Insights deployment is complete, navigate back to your dashboard.

  26. Refresh your resource group tile, and then click tsi-az220-training.

    You may need to resize your dashboard to see all of your resources.

    Note: You gave the Time Series Insights environment resource the name tsi-az220-training. You should also see the Time Series Insights event source that you created, but for now you want to have the TSI environment opened.

  27. On the Time Series Insights environment blade, on the left-side menu under Settings, click Event Sources.

  28. On the Event Sources pane, notice the iot-az220-training-{your-id} Event Source in the list.

    This is the event source that you configured when the TSI resource was created.

  29. To view the event source details, click iot-az220-training-{your-id}.

    Notice that the configuration of the event source matches what was set when the Time Series Insights resource was created.

Exercise 3: Run Simulated IoT Devices

In this exercise, you will run the simulated devices so they starts sending telemetry events to Azure IoT Hub.

  1. Open Visual Studio Code.

  2. On the File menu, click Open Folder.

  3. In the Open Folder dialog, navigate to the lab 10 Starter folder.

    In Lab 3: Setup the Development Environment, you cloned the GitHub repository containing lab resources by downloading a ZIP file and extracting the contents locally. The extracted folder structure includes the following folder path:

    • Allfiles
      • Labs
        • 10-Explore and analyze time stamped data with Time Series Insights
          • Starter
  4. In the Open Folder dialog, click ContainerSimulation, and then click Select Folder.

    If prompted, load the C# extension and/or perform a restore.

  5. In the EXPLORER pane, to open the Program.cs file, click Program.cs.

  6. Locate the variables used to assign the connections strings

     private readonly static string connectionStringTruck = "{Your Truck device connection string here}";
     private readonly static string connectionStringAirplane = "{Your Airplane device connection string here}";
     private readonly static string connectionStringContainer = "{Your Container device connection string here}";
    
  7. Update the variable assignments with the connection strings that you saved earlier in the lab.

    Be sure to replace the placeholder values with the Connection String for the corresponding IoT device.

  8. On the File menu, click Save.

  9. On the View menu, click Terminal.

  10. Within the Terminal pane, ensure that the command prompt specifies the path to the lab 10 /Starter/ContainerSimulation directory.

  11. At the command prompt, to build and run the ContainerSimulation app, enter the following command:

     dotnet run
    
  12. Notice the messages displayed in the Terminal pane.

    Once the ContainerSimulation app is running, it will begin outputting telemetry data to the terminal. This is the telemetry data that it is sending to Azure IoT Hub.

    When the ContainerSimulation app is running, the Terminal output will look similar to the following:

     12/27/2019 8:51:30 PM > Sending TRUCK message: {"temperature":35.15660452608195,"humidity":48.422323938240865}
     12/27/2019 8:51:31 PM > Sending AIRPLANE message: {"temperature":17.126545186374237,"humidity":36.46941012936869}
     12/27/2019 8:51:31 PM > Sending CONTAINER message: {"temperature":21.986403302500637,"humidity":47.847680384455096}
     12/27/2019 8:51:32 PM > Sending TRUCK message: {"temperature":36.10474464823629,"humidity":48.82029906486022}
     12/27/2019 8:51:32 PM > Sending AIRPLANE message: {"temperature":16.55005930170971,"humidity":36.49988437459935}
     12/27/2019 8:51:32 PM > Sending CONTAINER message: {"temperature":21.811727088543286,"humidity":50.0}
    
  13. Leave the ContainerSimulation app running for the remainder of this lab.

    This will ensure device telemetry from the three devices (Container, Truck, and Airplane) is being sent to Azure IoT Hub.

  14. After the ContainerSimulation app has been running for 30 seconds, you will see a message telling you that the Container device is changing transport methods.

    The transport method will change between Truck and Airplane every 30 seconds. The Terminal output will look like the following when this happens:

     12/27/2019 8:51:40 PM > CONTAINER transport changed to: TRUCK
    

    Note: In production the shipping container would only change transport methods during the normal course of shipping. For the simulated scenario in this lab, it’s performed every 30 seconds to give a short enough data duration that will fit during the course of performing the steps in this lab.

Exercise 4: View Time Series Insights Explorer

In this exercise, you will be introduced to working with time series data using the Time Series Insights (TSI) Explorer.

  1. If needed, log in to portal.azure.com using your Azure account credentials.

    If you have more than one Azure account, be sure that you are logged in with the account that is tied to the subscription that you will be using for this course.

  2. On your Resource group tile, click tsi-az220-training.

  3. On the Time Series Insights environment blade, at the top of the Overview pane, click Go to TSI Explorer.

    This will open the Time Series Insights Explorer in a new browser tab.

  4. On the left-side menu, ensure that Analyze is selected.

    You can expand the navigation menu to display the button names. The two options are “Analyze” and “Model”. Choose Analyze.

    Collapse the navigation menu to ensure that you can see the query edit area on the left side of the page.

  5. To begin to edit a query, click Add new query.

  6. In the query editor, under Query 1, open the MEASURE dropdown, and then click temperature.

  7. Open the SPLIT BY dropdown, and then click iothub-connection-device-id.

    When you run the query, this will split the graph to show the telemetry from each of the IoT Devices separately on the graph.

  8. At the top of the page, to have the display automatically refresh, click Auto Refresh.

    When Auto refresh is enabled, the display will be updated every 30 seconds to display the latest data. This only applies to the last 1 hour of available data.

  9. Notice the graph now displays the temperature sensor event data from the IoT Devices within Azure IoT Hub in a Line Chart.

  10. Notice the list of Device IDs to the left of the graph.

    Hovering the mouse over a specific Device ID will highlight it’s data on the graph display.

  11. Take a moment to examine the temperature data (graphs) for the telemetry streaming into the system from the three simulated devices.

  12. Notice that the spikes in temperature of the sensor-th-container0001 correlate with the temperature spikes of either the sensor-th-truck0001 or the sensor-th-airplane0001.

    This gives you an indication that the sensor-th-container0001 is being transported by Truck or Airplane at those times.

  13. To add a second query to the display, click Add new query.

  14. Under Query 2, set the MEASURE dropdown to humidity, set the SPLIT BY dropdown to iothub-connection-device-id, and then click Add.

    Notice that there are now two graphs displayed. The top graph shows temperature while the lower graph shows humidity, both using their own Y-axis scale.

  15. Position your mouse pointer over one of the graph lines.

    Notice that when you hover the mouse cursor over the graph, a popup will display the details for a point on the graph. The popup displays the minimum (min), average (avg), and maximum (max) values for the data points in the graph (over the short time represented by that point). The time range associated with the selected data point is displayed along the time axis at the bottom of the display.

  16. Above the vertical axis line, notice the options that you can use to control graph settings.

  17. Change the Interval setting to 15 seconds.

    Notice how the appearance of the data changes as you increase the interval.

Once you have completed exploring the data, don’t forget to stop the container simulator app by pressing CTRL+C in the terminal.