Integrate data into SiteWise Edge using an MQTT-enabled, V3 gateway

This tutorial guides you through integrating third-party devices and sensors that use MQTT messaging protocol with the AWS IoT SiteWise MQTT-enabled, V3 gateway. You will learn how to set up an AWS IoT SiteWise edge gateway to collect and monitor data from your MQTT-enabled devices. AWS IoT SiteWise enables you to collect, process, and monitor industrial equipment data. Use SiteWise Edge capabilities to optimize industrial IoT operations, and transform raw data into actionable insights.

In this tutorial, we use data from a wind farm demonstration to illustrate key concepts. After you become familiar with the process, you can repeat the tutorial with your own data.

After you complete this tutorial, you can do the following items:

Additionally, you can leverage your edge gateway capabilities by connecting to other AWS IoT services to perform the following tasks:

Prerequisites

To complete this tutorial, you need the following:

Step 1: Create an AWS IoT policy

This tutorial uses the AWS IoT policy you created in the Ingest data tutorial. This policy sets the security rules for your devices and creates a digital representation of your external devices and sensors in AWS IoT. The policy allows your third-party devices to send data to AWS IoT Core using MQTT (Message Queuing Telemetry Transport). For more information about MQTT messages, see What is MQTT?.

Console

Ensure completion of an AWS IoT policy. For detailed instructions, see Step 1 in the Ingest data tutorial.

To verify you have an active AWS IoT policy

1. Navigate to the AWS IoT console.

2. In the left navigation pane, choose Securities, then Policies.

3. Choose the policy you created. For example, SiteWiseTutorialDevicePolicy.

4. Confirm that the policy's status is listed as Active.

AWS CLI

Ensure completion of an AWS IoT policy. For detailed instruction, see Step 1 in the Ingest data tutorial.

Use the following AWS CLI get-policy command in the AWS CLI Command Reference to verify you have an active AWS IoT policy:

aws iot get-policy --policy-name "SiteWiseTutorialDevicePolicy"

This policy enables your AWS IoT devices to establish connections and to communicate with device shadows using MQTT messages. To interact with device shadows, your AWS IoT things publish and receive MQTT messages on topics that start with $aws/things/thing-name/shadow/. This policy incorporates a thing policy variable known as ${iot:Connection.Thing.ThingName}. This variable substitutes the connected thing's name in each topic. The iot:Connect statement sets limitations on which devices can establish connections, ensuring that the thing policy variable can only substitute names starting with SiteWiseTutorialDevice.

For more information, see Thing policy variables in the AWS IoT Developer Guide.

Step 2: Create and configure an AWS IoT thing

In this step, register your edge device as an AWS IoT thing and generate your thing’s certificates and keys needed for secure communication with AWS IoT SiteWise Edge. This process establishes the foundation for your device to send third-party data through your MQTT-enabled, V3 gateway.

Console

Ensure completion of the creation and configuration steps for an AWS IoT thing. For detailed instructions, see Step 2 in the Ingest data tutorial.

To verify you have an active AWS IoT thing

1. Navigate to the AWS IoT console.

2. In the left navigation pane, choose All devices, then Things.

3. Choose the thing you created. For example, SiteWiseTutorialDevice1.

4. Under Certificates, confirm that the status is listed as active.

AWS CLI

Ensure completion of the creation and configuration steps for an AWS IoT thing. For detailed instructions, see Step 2 in the Ingest data tutorial.

Use the following AWS CLI command to verify you have an active AWS IoT policy:

 aws iot describe-thing --thing-name "SiteWiseTutorialDevice1"

After completing these steps, you can securely connect your device to AWS IoT SiteWise Edge. You created a local directory to store your certificates and keys you generated for MQTT authentication. Your device is registered as an AWS IoT thing in the AWS IoT console, and your device is prepared to integrate data with SiteWise Edge. You can connect your industrial equipment or other devices to the AWS IoT platform and start ingesting data into SiteWise Edge.

Step 3: Configure your SiteWise Edge MQTT-enabled, V3 gateway

In this step, create your AWS IoT SiteWise Edge MQTT-enabled, V3 gateway and configure it to receive data from the EMQX broker. The gateway acts as a bridge between your devices and AWS IoT. This allows you to process data locally at the edge before sending it to the AWS Cloud. This configuration reduces bandwidth and decreases cloud processing delays.

Console

To create your AWS IoT SiteWise MQTT-enabled, V3 gateway

1. Sign in to the AWS Management Console and open the AWS IoT SiteWise console.

2. In the left navigation pane, choose Edge gateways, then choose Create gateway.

3. Under Deployment target, choose Self-hosted gateway.

4. Under Self-hosted gateway options, choose MQTT-enabled, V3 gateway - recommended.

5. Under Gateway configuration:

   1. In Gateway name, enter a name for your gateway. For example, SiteWise Tutorial Device Gateway.

   2. In Greengrass device OS, select the appropriate option for your device.

6. Under Advanced configuration:

   1. Choose Default setup.

   2. Enter a name for the Greengrass core device or use the name generated by AWS IoT SiteWise.

7. Choose Create gateway.

8. In the confirmation dialog, choose Generate and download to generate an installer for your SiteWise Edge gateway. For more information, see Create a self-hosted SiteWise Edge gateway.

Warning

Store the installer file in a secure location. This file can't be regenerated, and is needed to complete the gateway setup in later steps.

AWS CLI

Use AWS CLI to create a self-hosted gateway. You need to provide a name for the gateway, specify the platform and gateway version. For more information, see CreateGateway in the AWS IoT SiteWise API Reference.

To use this example, replace the user input placeholders with your own information.

aws iotsitewise create-gateway \
    --gateway-name SiteWise Tutorial Device Gateway \
    --gateway-platform greengrassV2={coreDeviceThingName=your-core-device-thing-name, coreDeviceOperatingSystem=LINUX_AMD64} \                        
    --gateway-version 3 \
    [--cli-input-json your-configuration]

• gateway-name – A unique name for the gateway, for example, SiteWise Tutorial Device Gateway.

• gateway-platform – Enter greengrassV2. For more information, see CreateGateway in the AWS IoT SiteWise API Reference.

  • coreDeviceThingName – The name of the AWS IoT thing for your AWS IoT Greengrass V2 core device. For example, SiteWiseTutorialDevice1.

  • coreDeviceOperatingSystem – The operating system of the core device in AWS IoT Greengrass V2. Specifying the operating system is required for gateway-version 3. Options include: LINUX_AARCH64, LINUX_AMD64, and WINDOWS_AMD64.

• gateway-version – The version of the gateway.

  • Use 3 for the gateway version to create an MQTT-enabled, V3 gateway.

• cli-input-json – A JSON file containing request parameters.

Use the following AWS CLI command to verify that your gateway was created successfully:

aws iotsitewise describe-gateway --gateway-id your-gateway-id

Step 4: Install SiteWise Edge gateway software

To install the gateway software, use the installer package that you downloaded in the previous step. The installation process configures the necessary components, starts the Greengrass core service, and registers your device with AWS IoT Greengrass. After installation is complete, verify that your gateway appears in the AWS IoT SiteWise console under Edge gateways and that the Greengrass service is running properly on your device.

For detailed instructions, see Install the AWS IoT SiteWise Edge gateway software on your local device.

Step 5: Configure the EMQX broker to connect to external applications

Configure the EMQX broker to enable secure communication between your IoT devices and external applications. The EMQX broker functions as a central messaging hub that routes data between your IoT devices, gateway, and applications. The EMQX broker ensures reliable message delivery on your gateway and connected applications at the edge. For more information, see Connect external applications to the EMQX broker.

To configure the EMQX broker

EMQX allows you to create authorization rules based on identifiers such as username, IP address, or client ID. This is useful for controlling access to your data. For more information, see Set up authorization rules for AWS IoT SiteWise Edge in EMQX.

After successful deployment, your EMQX broker is configured and ready to securely connect with external applications.

Example: Add CPU, memory, and timestamp JSON payloads

CPU JSON payload

{
  "propertyAlias": "SiteWiseTutorialDevice/cpu",
  "propertyValues": [
    {
      "quality": "GOOD",
      "timestamp": {
        "offsetInNanos": 0,
        "timeInSeconds": 1753206441
      },
      "value": {
        "integerValue": 45.2
      }
    }
  ]
}

Memory JSON payload

{
  "propertyAlias": "SiteWiseTutorialDevice/memory",
  "propertyValues": [
    {
      "quality": "GOOD",
      "timestamp": {
        "offsetInNanos": 0,
        "timeInSeconds": 1753206441
      },
      "value": {
        "integerValue": 67.8
      }
    }
  ]
}

Timestamp JSON payload

{
  "propertyAlias": "SiteWiseTutorialDevice/timestamp",
  "propertyValues": [
    {
      "quality": "GOOD",
      "timestamp": {
        "offsetInNanos": 0,
        "timeInSeconds": 1753206441
      },
      "value": {
        "integerValue": 23.5
      }
    }
  ]
}

The payload defines the required JSON structure that your IoT devices must use to send device data through the EMQX broker to SiteWise Edge. This format ensures that AWS IoT SiteWise can identify your devices and process the sensor readings. After you implement these configurations and payload structures, your wind farm monitoring system is ready to collect and process data.

Step 6: Publish data with Mosquitto

After creating your MQTT-enabled, V3 gateway, configure Eclipse Mosquitto to send test data to SiteWise Edge. Mosquitto is an open-source MQTT message broker that uses the MQTT protocol for lightweight messaging between devices. The Mosquitto client allows you to publish messages to MQTT topics, simulating data from wind farm sensors. Using Mosquitto, simulate device data without requiring any third-party services or additional equipment. For more information, see documentation on the official Eclipse Mosquitto website. In this tutorial, local data from the Ingest data tutorial and fictitious data are being used for demonstration purposes.

Use Mosquitto CLI client to test the SiteWise Edge EMQX broker

Use Mosquitto to publish test data. Open a command line and run the following commands:

Example: CPU property

mosquitto_pub -h localhost -p 1883 -t "SiteWiseTutorialDevice/cpu" -m '{
  "propertyAlias": "SiteWiseTutorialDevice/cpu",
  "propertyValues": [
    {
      "quality": "GOOD",
      "timestamp": {
        "timeInSeconds": 1753206441,
        "offsetInNanos": 0
      },
      "value": {
        "integerValue": 45.2
      }
    }
  ]
}' 

Example: Memory property

mosquitto_pub -h localhost -p 1883 -t "SiteWiseTutorialDevice/memory" -m '{
  "propertyAlias": "SiteWiseTutorialDevice/memory",
  "propertyValues": [
    {
      "quality": "GOOD",
      "timestamp": {
        "timeInSeconds": 1753206441,
        "offsetInNanos": 0
      },
      "value": {
        "integerValue": 72.1
      }
    }
  ]
}'

Example: Timestamp property

mosquitto_pub -h localhost -p 1883 -t "SiteWiseTutorialDevice/timestamp" -m '{
  "propertyAlias": "SiteWiseTutorialDevice/timestamp",
  "propertyValues": [
    {
      "quality": "GOOD",
      "timestamp": {
        "timeInSeconds": 1753206441,
        "offsetInNanos": 0
      },
      "value": {
        "integerValue": 1683000000
      }
    }
  ]
}'

Step 7: Specify destinations

In this step, specify destinations to determine where to direct your source data. Use AWS IoT SiteWise with Amazon S3 buffering as your destination. This option provides a scalable way to store and process your IoT data.

Console

To add destinations

1. Navigate to the AWS IoT SiteWise console and select Edge gateways.

2. Under SiteWise Tutorial Device Gateway, choose Add destinations.

3. Under Destination details, choose AWS IoT SiteWise buffered using Amazon S3. To learn more about destination types, see AWS IoT SiteWise gateway destinations.

4. Under Destination name, enter a name for your destination, for example, SiteWise Tutorial S3 Destination.

5. Under S3 upload settings, enter your S3 bucket location. For example, s3://sitewise-tutorial-mqtt-data-[your-account-id]. To learn more about Amazon S3, see Creating, configuring, and working with Amazon S3 buckets in the Amazon Simple Storage Service User Guide.

6. Under Data upload frequency, enter a value between 1 minute and 30 days. For example, 1 minute.

7. Under Data storage settings:

   1. Deselect Copy data to storage. While this setting is recommended for production environments, you don't need it for this tutorial. When you deselect this option, the Delete data from S3 option is automatically deselected.

8. Choose Add destination.

Note

This tutorial uses a 1-minute interval for testing. After you complete the tutorial, you can adjust this interval to match your production needs or delete it to avoid additional charges.

AWS CLI

Example: Create a new AWS IoT SiteWise destination buffered using Amazon S3

Use the update-gateway-capability-configuration in the AWS CLI Command Reference to configure the publisher. Set the capabilityNamespace parameter to iotsitewise:publisher:3.

{
    "sources": [
      {
        "type": "MQTT"
      }
    ],
    "destinations": [
      {
        "type": "SITEWISE_BUFFERED",
        "name": "your-s3-destination-name",
        "config": {
          "targetBucketArn": "arn:aws:s3:::amzn-s3-demo-bucket/Optional/SomeFolder",
          "publishPolicy": {
            "publishFrequency": "1m",
            "localSizeLimitGB": 10
          },
          "siteWiseImportPolicy": {
            "enableSiteWiseStorageImport": true,
            "enableDeleteAfterImport": true,
            "bulkImportJobRoleArn": "arn:aws:iam::123456789012:role/your-role-name"
          }
        },
        "filters": [
          {
            "type": "PATH",
            "config": {
              "paths": [
                "#"
              ]
            }
          }
        ]
      }
    ]
  }          
      

For more information about destinations, see Add an AWS IoT SiteWise buffered destination using Amazon S3.

Step 8: Specify path filters

In this step, configure path filters to specify which MQTT topics to monitor for your wind farm device data.

Path filters follow the MQTT topic wildcard specification, which supports two special characters:

Console

To configure your path filters

Under Path filters:

1. Navigate to the AWS IoT SiteWise console and select Edge gateways.

2. Under SiteWise Tutorial Device Gateway, choose Add destinations.

3. Choose Add path filters to enter the following path filters manually:

   • SiteWiseTutorialDevice/#

   • windfarm/+/turbine/+/performance/#

   • cpu/+/idle-time

   • cpu/+/interruption-count/+

   • +/memory/consumption

   • timestamp/+/measurement

   • device/+/status/+

   • system/+/performance-log

4. Choose Add destination.

For more information about best practices for path filters, see Best practices for path filters.

AWS CLI

Use the following AWS CLI commands to configure your path filters:

Example 1: Device data using wildcard

{
 "destinations": [
   {
     "name": "All Device Data Destination"
   }
 ],
 "filters": [
   {
     "type": "PATH",
     "config": {
       "paths": [
         "SiteWiseTutorialDevice/#",
         "windfarm/+/turbine/+/performance/#"
       ]
     }
   }
 ]
}

This path filter configuration uses multi-level wildcards (#) to capture all data from the SiteWiseTutorialDevice and all performance data from any turbine in the wind farm.

Example 2: CPU and memory performance

{
 "destinations": [
   {
     "name": "Performance Metrics Destination"
   }
 ],
 "filters": [
   {
     "type": "PATH",
     "config": {
       "paths": [
         "cpu/+/idle-time",
         "+/memory/consumption",
         "cpu/+/interruption-count/+"
       ]
     }
   }
 ]
}

This example captures various CPU metrics (idle time and interruption count) and memory consumption data across devices.

Example 3: Device diagnostics

{
 "destinations": [
   {
     "name": "Device Diagnostics Destination"
   }
 ],
 "filters": [
   {
     "type": "PATH",
     "config": {
       "paths": [
        "device/+/status/+",
        "system/+/performance-log"
       ]
     }
   }
 ]
}  

This configuration uses the + wildcard to capture diagnostic data from multiple devices, specifically system performance logs and device status updates.

These three path filters match the MQTT topics that you use to publish test data with Mosquitto. The filters ensure your SiteWise Edge gateway captures and processes the relevant MQTT messages. For more information on how to add path filters, see Add path filters to AWS IoT SiteWise Edge destinations.

Step 9: Configure your AWS IoT resources

In this step, create the necessary AWS IoT SiteWise asset models and assets to represent your simulated third-party devices and enable data ingestion through your edge gateway.

Before starting this step, you should have completed steps 3 to 8 in the Ingest data tutorial. These steps establish the foundational components to integrate your third-party data through the MQTT-enabled V3 gateway. You also set up rules that define how your sensor data flows into AWS IoT SiteWise, and run a device client script that simulates industrial wind farm data.

To validate your AWS IoT resource configuration

Step 10: Visualize your data

Set up the open-source version of Grafana to visualize your wind farm device data. Grafana is a visualization platform that displays your real-time operational data. These dashboards help you track operational efficiency and identify maintenance needs across your infrastructure. For more information about integration, see Integrate AWS IoT SiteWise with Grafana.

To setup Grafana

After completing these steps, you can build and customize Grafana dashboards to display your wind farm's operational metrics. This enables you to track and analyze your wind farm performance at the edge in real time.

You have completed the tutorial. In this procedure, you configured AWS IoT SiteWise Edge to integrate third-party device data using an MQTT-enabled, V3 gateway. This setup allows you to collect, process, and visualize industrial equipment data at the edge, reducing latency and operational costs. By using the wind farm demo, you collected and processed operational metrics like CPU and memory usage data through your MQTT-enabled, V3 gateway.

To enhance your IoT solution, consider exploring advanced features like anomaly detection by leveraging Detect anomalies with Lookout for Equipment, or integrating with other AWS services like Amazon QuickSight in the Amazon QuickSight User Guide for advanced analytics.

Step 11: Clean up resources after the tutorial

After you complete this tutorial about integrating data into AWS IoT SiteWise Edge, clean up your resources to avoid incurring additional charges.

To uninstall AWS IoT Greengrass Core

Uninstall the AWS IoT Greengrass Core software from your local device. For detailed instructions, see Uninstall the AWS IoT Greengrass Core software in the AWS IoT Greengrass Developer Guide, Version 2.

(Optional) To delete third-party resources

After completing this tutorial, consider shutting down any external resources you created. This helps to prevent incurring charges from third-party providers.

Additional resources

Refer to the following resources for more information: