Collecting, organizing, monitoring, and visualizing industrial data at scale using AWS IoT SiteWise (Part 2)

Post by Asim Kumar Sasmal, Senior Data Architect, and Mark Gilbert, Senior Consultant in the IoT Global Specialty Practice of AWS Professional Services

 

[Before reading this post, read Part 1 in the series.]

In Part 1 of this series, you saw how to ingest data from industrial sites in a secure, cost-effective, and reliable manner using AWS IoT SiteWise (under limited preview).

In Part 2, you learn how to:

  • Model your assets from the data ingested in Part 1 using AWS IoT SiteWise asset modeling.
  • Enable conditions monitoring and send notifications or alerts using AWS IoT Events.
  • Visualize dashboards and reports using Amazon QuickSight.

AWS IoT SiteWise is currently in limited preview. You must request an invitation to participate. To learn more, see the AWS IoT SiteWise Preview. While in limited preview, AWS IoT SiteWise service is subject to change.

Overview

You use multiple AWS services in this solution:

About AWS IoT Events

AWS IoT Events is a fully managed service that makes it easy to detect and respond to events from IoT sensors and applications. It lets you monitor your equipment or device fleets for failures or changes in operation. It can also trigger actions when such events occur. For more information, see Getting Started with the AWS IoT Events Console.

About Amazon QuickSight

Amazon QuickSight is a fast business-analytics service that you can use to:

  • Build visualizations.
  • Perform ad hoc analysis.
  • Get business insights quickly from your data in a self-serve fashion.

As a fully managed and hosted service, there is no client-server to manage. With Amazon QuickSight, you can easily create and publish interactive dashboards that include built-in ML Insights. Access the dashboards from any device and embed them into your custom applications, portals, and websites. With pay-per-session pricing, Amazon QuickSight lets you give everyone access to the data they need, while only paying for what you use.

The following diagram illustrates the high-level, end-to-end solution described in this post.

Walkthrough

There are five sections in this walkthrough:

  • Modeling your assets using AWS IoT SiteWise asset modeling.
  • Setting up an AWS IoT Events detector model for condition monitoring.
  • Setting up an AWS IoT Analytics dataset content delivery to the AWS IoT Events detector model.
  • Setting up an AWS IoT Analytics dataset to visualize in Amazon QuickSight.
  • Visualizing the AWS IoT Analytics dataset in Amazon QuickSight.

Prerequisites

Use the prerequisites from Part 1.

Modeling your assets using AWS IoT SiteWise asset modeling

In Part 1 of this post series, you set up your data ingestion from the OPC-UA server (KEPServerEX). Now, model your assets using AWS IoT SiteWise asset modeling.

Create three groups for your assets to represent two locations (East and West) in the state of Illinois. In Part 1 of this post series, you set up three wind turbines in KEPServerEX. Assume that one of the three wind turbines is in the East location and the remaining two are in the West location. These wind turbines are associated with the new asset groups.

  1. In the AWS IoT SiteWise console, choose View, Create new view, and create a view named “Wind Turbine Illinois View”.
  2. From the Library tools area on the lower right of the screen, drag and drop the New group icon three times to create three asset groups (Illinois, West, East). Add a few relevant attributes (such as Region and area) to identify the groups.
  3. After groups are created, right click on each group and select ‘Add to an existing group‘ from the list to create a hierarchy as shown in the screenshot:                        
  4. In the Library area on the right, choose Templates, Create template.
  5. On the Create new template screen, under Template information area, enter a template name for Name.
  6. In the Attribute definitions area, define the attributes shown in the following screenshot.        
  7. In the Measurement definitions area, define the measurements shown in the following screenshot.    
  8. In the Derived measurement definitions area, create a derived measurement named High_Temperature_Warning when the Equipment Temperature measurement is greater than 100ºF as shown in the following screenshot.    
    • You can select other mathematical functions available from the list, such as gte (greater than or equal), lte (lower than or equal), eq (equal), and pi, as per your use case.
  9. In the Metric definitions area, create two metrics named Max Generated Power and Max Equipment Temperature as shown in the following screenshot. Both use the max function on Generated Power and Equipment Temperature measurements. The Generated Power metric is calculated over a rolling 15-minute interval; the Equipment Temperature metric is calculated over a rolling 5-minute interval. To auto-populate the list of available functions, type the first letter of a mathematical function.    
  10. Using the template that you created earlier, create three assets: Turbine West 1, Turbine West 2, and Turbine East 1 (one for each wind turbine). Drag the template from the Library area to the View canvas and see the following Measurements screen. Choose the appropriate OPC-UA tag from your streaming data for the measurements that you defined earlier during the template creation.    
  11. After creating the three assets, right-click on the icon for each asset and from the Options list, select Add to an existing group. You get the following View canvas, showing the East location with one wind turbine and the West location with the remaining two wind turbines.    
  12. Edit the West group to create a roll-up metric named max_temperature_west as shown in the following screenshot. Repeat this step for the East group.    
  13. Choose Monitor and observe the measurements or metrics that you defined earlier. On the Monitor canvas, from the Layer display panel, select the little eye icons for Metrics or Measurements based on what you want to monitor.
    • In Metrics view, in the following diagram, the West group shows the maximum temperature from Turbine West 1 and Turbine West 2 (Max Equipment Temperature). Similarly, the East group shows the maximum temperature from Turbine East 1.    
    • The following diagram shows the groups and wind turbines in Measurements view.    

In addition to monitoring the metrics and measurements on the AWS IoT SiteWise console, you may also want to extract some metrics and measurements data programmatically to your internal custom apps. Use the AWS IoT SiteWise GetMetricData and GetMeasurementData APIs.

Setting up an AWS IoT Events detector model for condition monitoring

Now, to monitor the average equipment temperature during the last five minutes for the Unit 2 PLC of all three wind turbines, create an AWS IoT Events detector model named iote_equip_temp_detector_model.

The detector model has two states named Good and Critical. There may be a situation where the average equipment temperature climbs above a threshold of 70ºF (a manufacturer’s specification) three times in a row. In that case, the detector model instance for the corresponding wind turbine’s Unit 2 PLC switches its state from Good to Critical.

Similarly, the detector model instance switches its state from Critical to Good when the average equipment temperature drops below 70ºF three times in a row. Each state transition sends an SNS email notification to your operations team to perform the necessary actions.

Use the following steps to create the detector model.

(1) Create an AWS IoT Events input named iote_equip_temp:

aws iotevents create-input --cli-input-json file://iote_equip_temp_input.json --region 'us-west-2' --profile swblogpost

{

    "inputConfiguration": {

        "inputName": "iote_equip_temp",

        "inputDescription": "AWS IoT Events Equipment Temperature Monitoring",

        "inputArn": "arn:aws:iotevents:us-west-2:xxxxxxxxxxx:input/iote_equip_temp",

        "creationTime": 1561847255.99,

        "lastUpdateTime": 1561847255.99,

        "status": "ACTIVE"

    }

}

The file iote_equip_temp_input.json contains the following code:

{
    "inputName": "iote_equip_temp",
    "inputDescription": "IoT Events Equipment Temperature Monitoring",
    "inputDefinition": {
        "attributes": [
            {
                "jsonPath": "name"
            },
            {
                "jsonPath": "avg_value"
            }
        ]
    }
}

(2) Create an AWS IoT Events detector model named iote_equip_temp_detector_model:

aws iotevents create-detector-model --cli-input-json file://iote_equip_temp_detector_model.json --region 'us-west-2' --profile swblogpost

{

    "detectorModelConfiguration": {

        "detectorModelName": "iote_equip_temp_detector_model",

        "detectorModelVersion": "1",

        "detectorModelDescription": "AWS IoT Events Equipment Temperature Monitoring Detector Model",

        "detectorModelArn": "arn:aws:iotevents:us-west-2:xxxxxxxx:detectorModel/iote_equip_temp_detector_model",

        "roleArn": "arn:aws:iam::xxxxxxxx:role/iote_equip_temp_role",

        "creationTime": 1561847287.983,

        "lastUpdateTime": 1561847287.983,

        "status": "ACTIVATING",

        "key": "name"

    }

}

The file iote_equip_temp_detector_model.json contains the following code:

{
    "detectorModelName": "iote_equip_temp_detector_model",
    "detectorModelDescription": "AWS IoT Events Equipment Temperature Monitoring Detector Model",
    "detectorModelDefinition": {
        "states": [
            {
                "stateName": "Critical",
                "onInput": {
                    "events": [
                        {
                            "eventName": "DecrementcriticalCounter",
                            "condition": "convert(Decimal,$input.iote_equip_temp.avg_value) <= 70",
                            "actions": [
                                {
                                    "setVariable": {
                                        "variableName": "criticalCounter",
                                        "value": "$variable.criticalCounter - 1"
                                    }
                                }
                            ]
                        }
                    ],
                    "transitionEvents": [
                        {
                            "eventName": "to_Good",
                            "condition": "$variable.criticalCounter <= 1",
                            "actions": [
                                {
                                    "sns": {
                                        "targetArn": "arn:aws:sns:us-west-2:xxxxxxxx:iote_send_email_sns_topic"
                                    }
                                }
                            ],
                            "nextState": "Good"
                        }
                    ]
                },
                "onEnter": {
                    "events": []
                },
                "onExit": {
                    "events": []
                }
            },
            {
                "stateName": "Good",
                "onInput": {
                    "events": [
                        {
                            "eventName": "IncrementcriticalCounter",
                            "condition": "convert(Decimal,$input.iote_equip_temp.avg_value)  > 70",
                            "actions": [
                                {
                                    "setVariable": {
                                        "variableName": "criticalCounter",
                                        "value": "$variable.criticalCounter + 1"
                                    }
                                }
                            ]
                        },
                        {
                            "eventName": "ResetcriticalCounter",
                            "condition": "convert(Decimal,$input.iote_equip_temp.avg_value) <= 70",
                            "actions": [
                                {
                                    "setVariable": {
                                        "variableName": "criticalCounter",
                                        "value": "0"
                                    }
                                }
                            ]
                        }
                    ],
                    "transitionEvents": [
                        {
                            "eventName": "to_Critical",
                            "condition": "$variable.criticalCounter >= 2",
                            "actions": [
                                {
                                    "sns": {
                                        "targetArn": "arn:aws:sns:us-west-2:xxxxxxxx:iote_send_email_sns_topic"
                                    }
                                }
                            ],
                            "nextState": "Critical"
                        }
                    ]
                },
                "onEnter": {
                    "events": [
                        {
                            "eventName": "Initialization",
                            "condition": "true",
                            "actions": [
                                {
                                    "setVariable": {
                                        "variableName": "criticalCounter",
                                        "value": "0"
                                    }
                                }
                            ]
                        }
                    ]
                },
                "onExit": {
                    "events": []
                }
            }
        ],
        "initialStateName": "Good"
    },
    "roleArn": "arn:aws:iam::xxxxxxxx:role/iote_equip_temp_role",
    "key": "name"
}

Remember to replace the following that you noted earlier:

  • The IAM role (iote_equip_temp_role) ARN
  • The SNS topic (iote_send_email_sns_topic) ARN

In case you were wondering about the condition “$variable.criticalCounter >= 2” for transition event “to_Critical” that should have been “>=3”, it is due to a current limitation in AWS IoT Events.

(3) In the AWS IoT Events console, choose Detector models and select iote_equip_temp_detector_model. Choose Edit and verify the detector model that you just created in the AWS CLI.

(4) Create a new IAM policy or modify the existing policy of your IAM role iote_equip_temp_role with the following permissions on the AWS IoT Events input (iote_equip_temp) that you created earlier:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iotevents:BatchPutMessage",
            "Resource": [
                "arn:aws:iotevents:us-west-2:xxxxxxxx:input/iote_equip_temp"
            ]
        }
    ]
}

Remember to replace the IAM role ARN of your AWS IoT Events input iote_equip_temp noted earlier.

Setting up an AWS IoT Analytics dataset content delivery to the AWS IoT Events detector model

Now that you have created your AWS IoT Events detector model, create an AWS IoT Analytics dataset named iote_equip_temp_dataset. The content delivery target is the AWS IoT Events detector model input (iote_equip_temp).

aws iotanalytics create-dataset --cli-input-json file://mydataset.json --region 'us-west-2' --profile swblogpost

{

    "datasetName": "iote_equip_temp_dataset",

   "datasetArn": "arn:aws:iotanalytics:us-west-2:xxxxxxxx:dataset/iote_equip_temp_dataset"

}

The file mydataset.json contains the following code:

{
    "datasetName": "iote_equip_temp_dataset",
    "actions": [
        {
            "actionName": "myaction",
            "queryAction": {
                "sqlQuery": "SELECT replace(replace(name, '/', '-'),' ') name, avg(cast(value AS double)) AS avg_value FROM windturbine_gw where from_unixtime(timestamp/1000) > current_timestamp - interval '5' minute and name like '%Equipment%Temperature%' group by 1"
            }
        }
    ],
    "contentDeliveryRules": [
        {
            "destination": {
                "iotEventsDestinationConfiguration": {
                    "inputName": "iote_equip_temp",
                    "roleArn": "arn:aws:iam::xxxxxxxx:role/iote_equip_temp_role"
                }
            }
        }
    ],
    "triggers": [
        {
            "schedule": {
                "expression": "cron(0/5 * * * ? *)"
            }
        }
    ]
}

Remember to replace the IAM role ARN noted earlier. After the dataset is created, either wait five minutes for the dataset to run as scheduled or run it manually from AWS CLI as follows:

aws iotanalytics create-dataset-content --dataset-name "iote_equip_temp_dataset" --region 'us-west-2' --profile swblogpost

{

    "versionId": "9f2387a6-461e-448c-b5d9-51f7c8fb1d6b"

}

Wait for the content to be created by running the following command. The state should show as “SUCCEEDED” for your dataset content to be available on Amazon S3.

aws iotanalytics get-dataset-content --dataset-name "iote_equip_temp_dataset" --region 'us-west-2' --profile swblogpost

{

    "entries": [

        {

            "dataURI": "https://aws-iot-analytics-dataset-a1deff2c-5b74-4ac9-85e0-b48bee7bd9ab.s3.us-west-2.amazonaws.com/results/xxxxxxxxxxxxxxxxxxxx"

        }

    ],

    "timestamp": 1561849960.616,

    "status": {

        "state": "SUCCEEDED"

    }

}

Your AWS IoT Analytics dataset results are sent to the AWS IoT Events detector model.

In the AWS IoT Events console, choose Detector models, iote_equip_temp_detector_model to see the three detector model instances – one for each Unit 2 PLC of all three wind turbines.

After the AWS IoT Analytics dataset ran a few times as scheduled, Unit 2 PLC from Wind Turbine 1 transitioned from a Good to a Critical state. Meanwhile, Wind Turbine 2 and Wind Turbine 3 are in a Good state due to both having an acceptable average equipment temperature.

You should have received the following email indicating a Critical status for Wind Turbine 1 Unit 2 PLC.

Setting up an AWS IoT Analytics dataset to visualize in Amazon QuickSight

One of your requirements for an end-to-end site-to-cloud use case is to visualize the lowest-grain Equipment Temperature readings for all three wind turbines for the last hour in Amazon QuickSight.

Create an AWS IoT Analytics dataset named quicksight_equip_temp_dataset producing the lowest-grain equipment temperature readings for the last hour for all three wind turbines as follows:

aws iotanalytics create-dataset --cli-input-json file://mydataset2.json --region 'us-west-2' --profile swblogpost

{

    "datasetName": "quicksight_equip_temp_dataset",

   "datasetArn": "arn:aws:iotanalytics:us-west-2:806172930479:dataset/quicksight_equip_temp_dataset"

}

The file mydataset2.json contains the following code:

{
    "datasetName": "quicksight_equip_temp_dataset",
    "actions": [
        {
            "actionName": "myaction",
            "queryAction": {
                "sqlQuery": "SELECT from_unixtime(timestamp/1000) AS timestamp, cast(value AS double)         AS equip_temp    , replace(replace(name, '/', '-'),' ')             AS name FROM windturbine_gw WHERE   from_unixtime(timestamp/1000) > current_timestamp - interval '1' hour AND name like '%Equipment%Temperature%'"
            }
        }
    ],
    "triggers": [
        {
            "schedule": {
                "expression": "cron(0 * * * ? *)"
            }
        }
    ]
}

Remember to replace the IAM role ARN noted earlier. As explained earlier, after the AWS IoT Analytics dataset is created, either wait for the dataset to run as scheduled (which in this case is one hour) or run it manually from the AWS CLI.

Visualizing the AWS IoT Analytics dataset in Amazon QuickSight

AWS IoT Analytics provides direct integration with Amazon QuickSight. To visualize the AWS IoT Analytics dataset quicksight_equip_temp_dataset, see Visualizing AWS IoT Analytics Data with QuickSight.

After you create an Amazon QuickSight dataset for your AWS IoT Analytics dataset (quicksight_equip_temp_dataset), you can create a sample analysis.

Summary

In Part 2 of this two-part series, you saw how to model your assets from the data ingested in Part 1 using AWS IoT SiteWise asset modeling. You also learned how to enable conditions monitoring and send notifications and alerts using AWS IoT Events. Finally, you saw how to visualize dashboards and reports using Amazon QuickSight.

This series described a secure, cost-effective, and reliable end-to-end field-to-cloud solution. You learned how to:

  • Ingest large amounts of streaming data from hundreds of industrial sites having tens of thousands of PLCs/sensors.
  • React to that data by democratizing access in near-real time for unbounded analytics opportunities.
  • Visualize the analytical insights.