Explore and analyze time stamped data with Time Series Insights

Lab Scenario

Your hard work implementing Azure IoT services and tools has paid off. Contoso has rolled out their “Asset Condition Tracking System” that monitors the environmental conditions of cheese containers during shipment.

Two weeks after launching the new system it has revealed a temperature spike in-transit for a specific shipment. Some of the cheese in the shipment was ruined, but the new system did ensure that the affected cheese wasn’t delivered to the customer. Since you know the Azure IoT side of the monitoring system better than anyone, you will lead the investigation.

Management has asked you to determine if the system can be improved, hopefully to the point where it will prevent the loss of product in the future. You correlate the IoT devices’ sensor data taken from the trucks and planes used during the shipment. It appears that the temperature in one of the trucks rose unexpectedly in a particular area of the vehicle and created the heat spike in one of the transport containers (which was equipped with an IoT device monitoring temperature and humidity).

Your team decides that further improvements to the monitoring system will require near real-time data exploration and root-cause analysis.

You propose adding Time Series Insights to the Azure IoT solution. This will enable Contoso to quickly store, visualize, and query the large amounts of time series data that are generated by the IoT devices in the trucks, planes, and containers, and to visualize changes over time.

The following resources will be created:

Lab 10 Architecture

In This Lab

In this lab, you will complete the following activities:

  • Verify that the lab prerequisites are met (that you have the required Azure resources)
  • Create an Azure Time Series Insights (TSI) environment
  • Create IoT Hub and simulated device (using CLI)
  • Connect to IoT Hub with Time Series Insights (TSI)
  • Create and deploy TSI resources by using templates with Time Series Insights

Lab Instructions

Exercise 1: Verify Lab Prerequisites

This lab assumes the following Azure resources are available:

Resource Type Resource Name
Resource Group AZ-220-RG
IoT Hub AZ-220-HUB-{YOUR-ID}
Device ID TruckDevice
Device ID AirplaneDevice
Device ID ContainerDevice

If these resources are not available, you will need to run the lab10-setup.azcli script as instructed below before moving on to Exercise 2. The script file is included in the GitHub repository that you cloned locally as part of the dev environment configuration (lab 3).

The lab10-setup.azcli script is written to run in a bash shell environment - the easiest way to execute this is in the Azure Cloud Shell.

  1. Using a browser, open the Azure Cloud Shell and login with the Azure subscription you are using for this course.

  2. Verify that the Azure Cloud Shell is using Bash.

    The dropdown in the top-left corner of the Azure Cloud Shell page is used to select the environment. Verify that the selected dropdown value is Bash.

  3. On the Azure Shell toolbar, click Upload/Download files (fourth button from the right).

  4. In the dropdown, click Upload.

  5. In the file selection dialog, navigate to the folder location of the GitHub lab files that you downloaded when you configured your development environment.

    In Lab 3: Setup the Development Environment, you cloned the GitHub repository containing lab resources by downloading a ZIP file and extracting the contents locally. The extracted folder structure includes the following folder path:

    • Allfiles
      • Labs
        • 10-Explore and analyze time stamped data with Time Series Insights
          • Setup

    The lab10-setup.azcli script file is located in the Setup folder for lab 10.

  6. Select the lab10-setup.azcli file, and then click Open.

    A notification will appear when the file upload has completed.

  7. To verify that the correct file has uploaded in Azure Cloud Shell, enter the following command:

     ls
    

    The ls command lists the content of the current directory. You should see the lab10-setup.azcli file listed.

  8. To create a directory for this lab that contains the setup script and then move into that directory, enter the following Bash commands:

     mkdir lab10
     mv lab10-setup.azcli lab10
     cd lab10
    
  9. To ensure that the lab10-setup.azcli has the execute permission, enter the following command:

     chmod +x lab10-setup.azcli
    
  10. On the Cloud Shell toolbar, to edit the lab10-setup.azcli file, click Open Editor (second button from the right - { }).

  11. In the FILES list, to expand the lab10 folder and open the script file, click lab10, and then click lab10-setup.azcli.

    The editor will now show the contents of the lab10-setup.azcli file.

  12. In the editor, update the {YOUR-ID} and {YOUR-LOCATION} assigned values.

    Referencing the sample below as an example, you need to set {YOUR-ID} to the Unique ID that you created at the start of this course - i.e. CAH191211, and set {YOUR-LOCATION} to the location that makes sense for your resources.

     #!/bin/bash
    
     YourID="{YOUR-ID}"
     RGName="AZ-220-RG"
     IoTHubName="AZ-220-HUB-$YourID"
    
     Location="{YOUR-LOCATION}"
    

    Note: The Location variable should be set to the short name for the location. You can see a list of the available locations and their short-names (the Name column) by entering this command:

    az account list-locations -o Table
    
    DisplayName           Latitude    Longitude    Name
    --------------------  ----------  -----------  ------------------
    East Asia             22.267      114.188      eastasia
    Southeast Asia        1.283       103.833      southeastasia
    Central US            41.5908     -93.6208     centralus
    East US               37.3719     -79.8164     eastus
    East US 2             36.6681     -78.3889     eastus2
    
  13. In the top-right of the editor window, to save the changes made to the file and close the editor, click , and then click Close Editor.

    If prompted to save, click Save and the editor will close.

    Note: You can use CTRL+S to save at any time and CTRL+Q to close the editor.

  14. To create the resources required for this lab, enter the following command:

     ./lab10-setup.azcli
    

    This script can take a few minutes to run. You will see JSON output as each step completes.

    The script will first create a resource group named AZ-220-RG and an IoT Hub named AZ-220-HUB-{YourID}. If they already exist, a corresponding message will be displayed. The script will then add three devices to the IoT hub and display the device connection string. The device IDs are: TruckDevice, AirplaneDevice, and ContainerDevice.

  15. Notice that, once the script has completed, the connection string for each device is displayed.

    The connection string starts with “HostName=”

  16. Copy the connection string for each device into a text document.

    Once you have saved the connection strings to a location where you can find them easily, you will be ready to continue with the lab. The connection strings specify the device ID for the device that they are associated with.

Exercise 2: Setup Time Series Insights

Azure Time Series Insights (TSI) is an end-to-end platform-as-a-service offering used to collect, process, store, analyze, and query data from IoT solutions at scale. TSI is designed for ad hoc data exploration and operational analysis of data that’s highly contextualized and optimized for time series.

In this exercise, you will setup Time Series Insights integration with Azure IoT Hub.

  1. Login to portal.azure.com using your Azure account credentials.

    If you have more than one Azure account, be sure that you are logged in with the account that is tied to the subscription that you will be using for this course.

  2. In the Azure Portal, on the left side navigation menu, click + Create a resource.

  3. On the New blade, in the Search the Marketplace textbox, enter time series insights

  4. In the search results, click Time Series Insights.

  5. On the Time Series Insights blade, click Create.

  6. On the Create Time Series Insights environment blade, in the Environment name field, enter AZ-220-TSI

  7. In the Subscription dropdown, select the subscription that you are using for this course.

  8. In the Resource group dropdown, click AZ-220-RG.

  9. In the Location dropdown, select the Azure region used by your resource group.

  10. In the Tier field, ensure that the S1 pricing tier is selected and that Capacity is set to 1.

  11. At the bottom of the blade, click Next: Event Source.

  12. Under the EVENT SOURCE DETAILS section, ensure that Create an event source? is set to Yes.

  13. In the Name field, enter AZ-220-HUB-{YOUR-ID} to specify a unique name for this Event Source.

  14. In the Source type dropdown, ensure that IoT Hub is selected.

  15. In the Select a hub dropdown, ensure that Select existing is selected.

    This will allow you to select an existing IoT Hub that’s already been provisioned.

  16. In the Subscription dropdown, select the subscription that you are using for this course.

  17. In the IoT Hub name dropdown, select the AZ-220-HUB-_{YOUR_ID}_ Azure IoT Hub service that’s already been provisioned.

  18. In the IoT Hub access policy name dropdown, click iothubowner.

    In a production environment, it’s best practice to create a new Access Policy within Azure IoT Hub to use for configuring Time Series Insights (TSI) access. This will enable the security of TSI to be managed independently of any other services connected to the same Azure IoT Hub. We are not doing that here for convenience reasons.

  19. Under the CONSUMER GROUP section, next to the IoT Hub consumer group dropdown, click New.

  20. In the IoT Hub consumer group box, enter tsievents and then click Add.

    This will add a new Consumer Group to use for this Event Source. The Consumer Group needs to be used exclusively for this Event Source, as there can only be a single active reader from a given Consumer Group at a time.

  21. Under the TIMESTAMP section, leave the Property Name blank.

  22. At the bottom of the blade, click Review + create.

    Note: If you are returned immediately to the Event Source pane, double check that you clicked Add to the right of the IoT Hub consumer group field - you cannot create the TSI resource until you have created the consumer group.

  23. At the bottom of the blade, click Create.

    Note: Deployment of Time Series Insights (TSI) will take a couple minutes to complete.

  24. Once your Time Series Insights deployment is complete, navigate back to your dashboard.

  25. Refresh your resource group tile, and then click AZ-220-TSI.

    You may need to resize your dashboard to see all of your resources.

    Note: You gave the Time Series Insights environment resource the name AZ-220-TSI. You should also see the Time Series Insights event source that you created, but for now you want to have the TSI environment opened.

  26. On the Time Series Insights environment blade, in the left side navigation menu, under Settings, click Event Sources.

  27. On the Event Sources pane, notice the AZ-220-HUB-_{YOUR-ID}_ Event Source in the list.

    This is the event source that was configured when the TSI resource was created.

  28. To view the event source details, click AZ-220-HUB-_{YOUR-ID}_.

    Notice that the configuration of the event source matches what was set when the Time Series Insights resource was created.

Exercise 3: Run Simulated IoT Devices

In this exercise, you will run the simulated devices so they starts sending telemetry events to Azure IoT Hub.

  1. Open Visual Studio Code.

  2. On the File menu, click Open Folder.

  3. In the Open Folder dialog, navigate to the Starter folder for lab 10.

    In Lab 3: Setup the Development Environment, you cloned the GitHub repository containing lab resources by downloading a ZIP file and extracting the contents locally. The extracted folder structure includes the following folder path:

    • Allfiles
      • Labs
        • 10-Explore and analyze time stamped data with Time Series Insights
          • Starter
  4. In the Open Folder dialog, click Starter, and then click Select Folder.

    If prompted, load the C# extension and/or perform a restore.

  5. In the EXPLORER pane, to open the DeviceSimulation.cs file, click DeviceSimulation.cs.

  6. Locate the variables used to assign the connections strings

     private readonly static string connectionString_Truck = "{Your Truck device connection string here}";
     private readonly static string connectionString_Airplane = "{Your Airplane device connection string here}";
     private readonly static string connectionString_Container = "{Your Container device connection string here}";
    
  7. Update the variable assignments with the connection strings that you saved earlier in the lab.

    Be sure to replace the placeholder values with the Connection String for the corresponding IoT Device.

  8. On the File menu, click Save.

  9. On the View menu, click Terminal.

  10. Within the Terminal pane, ensure that the command prompt specifies the path to the lab 10 /Starter directory.

  11. At the command prompt, to build and run the DeviceSimulation app, enter the following command:

     dotnet run
    
  12. Notice the messages displayed in the Terminal pane.

    Once the DeviceSimulation app is running, it will begin outputting telemetry data to the terminal. This is the telemetry data that it is sending to Azure IoT Hub.

    When the DeviceSimulation app is running, the Terminal output will look similar to the following:

     12/27/2019 8:51:30 PM > Sending TRUCK message: {"temperature":35.15660452608195,"humidity":48.422323938240865}
     12/27/2019 8:51:31 PM > Sending AIRPLANE message: {"temperature":17.126545186374237,"humidity":36.46941012936869}
     12/27/2019 8:51:31 PM > Sending CONTAINER message: {"temperature":21.986403302500637,"humidity":47.847680384455096}
     12/27/2019 8:51:32 PM > Sending TRUCK message: {"temperature":36.10474464823629,"humidity":48.82029906486022}
     12/27/2019 8:51:32 PM > Sending AIRPLANE message: {"temperature":16.55005930170971,"humidity":36.49988437459935}
     12/27/2019 8:51:32 PM > Sending CONTAINER message: {"temperature":21.811727088543286,"humidity":50.0}
    
  13. Leave the DeviceSimulation app running for the remaining duration of this lab.

    This will ensure device telemetry from the three devices (Container, Truck, and Airplane) is being sent to Azure IoT Hub.

  14. After the DeviceSimulation app has been running for 30 seconds, you will see a message telling you that the Container device is changing transport methods.

    The transport method will change between Truck and Airplane every 30 seconds. The Terminal output will look like the following when this happens:

     12/27/2019 8:51:40 PM > CONTAINER transport changed to: TRUCK
    

    Note: In production the shipping container would only change transport methods during the normal course of shipping. For the simulated scenario in this lab, it’s performed every 30 seconds to give a short enough data duration that will fit during the course of performing the steps in this lab.

Exercise 4: View Time Series Insights Explorer

In this exercise, you will be introduced to working with time series data using the Time Series Insights (TSI) Explorer.

  1. If needed, log in to portal.azure.com using your Azure account credentials.

    If you have more than one Azure account, be sure that you are logged in with the account that is tied to the subscription that you will be using for this course.

  2. On your Resource group tile, click AZ-220-TSI.

  3. On the Time Series Insights environment blade, at the top of the Overview pane, click Go to Environment.

    This will open the Time Series Insights Explorer in a new browser tab.

  4. On the toolbar at the top of the page, if there is an option to enable the Preview, set Preview to On.

  5. On the left side navigation menu, ensure that Analyze is selected.

    You can expand the navigation menu to display the button names. The two options are “Analyze” and “Model”. Choose Analyze.

    Collapse the navigation menu to ensure that you can see the query edit area on the left side of the page.

  6. In the query editor, open the MEASURE dropdown, and then click temperature.

  7. Open the SPLIT BY dropdown, and then click iothub-connection-device-id.

    When you run the query, this will split the graph to show the telemetry from each of the IoT Devices separately on the graph.

  8. Click Add.

  9. At the top of the page, to have the display automatically refresh, click Auto Refresh.

    When Auto refresh is enabled, the display will be updated every 30 seconds to display the latest data. This only applies to the last 1 hour of available data.

  10. Notice the graph now displays the temperature sensor event data from the IoT Devices within Azure IoT Hub in a Line Chart.

  11. Notice the list of Device IDs to the left of the graph.

    Hovering the mouse over a specific Device ID will highlight it’s data on the graph display.

  12. Take a moment to examine the temperature data (graphs) for the telemetry streaming into the system from the three simulated devices.

  13. Notice that the spikes in temperature of the ContainerDevice correlate with the temperature spikes of either the TruckDevice or the AirplaneDevice.

    This gives you an indication that the ContainerDevice is being transported by Truck or Airplane at those times.

  14. To add a second query to the display, set the MEASURE dropdown to humidity, set the SPLIT BY dropdown to iothub-connection-device-id, and then click Add.

    Notice that there are now two graphs displayed. The top graph shows temperature while the lower graph shows humidity, both using their own Y-axis scale.

  15. Position your mouse pointer over one of the graph lines.

    Notice that when you hover the mouse cursor over the graph, a popup will display the details for a point on the graph. The popup displays the minimum (min), average (avg), and maximum (max) values for the data points in the graph (over the short time represented by that point). The time range associated with the selected data point is displayed along the time axis at the bottom of the display.

  16. Above the vertical axis line, notice the options that you can use to control graph settings.

  17. Change the Interval setting to 15 seconds.

    Notice how the appearance of the data changes as you increase the interval.

Once you have completed exploring the data, don’t forget to stop the device simulator app by pressing CTRL+C in the terminal.