# Destinations and path filters Destinations in AWS IoT SiteWise Edge provide a flexible and efficient way to manage how your industrial data flows from edge devices to the cloud. This section explains how to configure destinations, use path filters to route specific data streams, and choose the right destination type for your use case. You can use destinations and path filters on self-hosted MQTT-enabled, V3 gateways and gateways used in conjunction with AWS IoT SiteWise Edge application hosted on Siemens Industrial Edge. Destinations and path filters do not work with Classic Streams, V2 gateways. **Topics** + [Understand AWS IoT SiteWise Edge destinations](#source-destination) + [Understand path filters for AWS IoT SiteWise Edge destinations](#destinations-path-filters) + [Add an AWS IoT SiteWise Edge real-time destination](destinations-real-time.md) + [Add an AWS IoT SiteWise buffered destination using Amazon S3](destinations-buffered.md) + [Add path filters to AWS IoT SiteWise Edge destinations](destinations-add-path-filters.md) + [Manage AWS IoT SiteWise Edge destinations](destinations-manage.md) ## Understand AWS IoT SiteWise Edge destinations Use AWS IoT SiteWise Edge destinations to determine where to send your source data. You can choose your data destination based on specific characteristics you need, like cost-effectiveness, low latency, or storage requirements. Integrate device data captured by AWS IoT SiteWise, our partners, or custom applications to publish and subscribe to path filters (topics) at the edge. You can then model, transfer, and store your device data in the cloud. **Note** For full use of all destination features on self-hosted gateways, upgrade to the latest versions of the IoT SiteWise publisher and IoT SiteWise OPC UA collector. Stream support is continued on Classic streams, V2 gateways to maintain compatibility with existing setups. For more information, see [Classic streams, V2 gateways for AWS IoT SiteWise Edge](classic-streams-v2-gateway.md) **Topics** + [How SiteWise Edge destinations enhance data management](#how-destinations-work) + [Destination types](#destination-types) + [Compare destination functionality between gateway versions](#destinations-vs-publisher-component) + [Destination limitations](#destinations-limitiations) + [Use cases for SiteWise Edge destinations](#destinations-use-cases) ### How SiteWise Edge destinations enhance data management Export data from the edge to AWS IoT SiteWise in real time, or in batches using Amazon S3. Destinations enhance flexibility and scalability in your AWS IoT SiteWise environment. Destinations implement a centralized data management model, where sources publish data to a central system. Destinations determine where your data is sent using path filters. Destinations can subscribe to multiple path filters. MQTT-enabled gateways, whether self-hosted or running on Siemens Industrial Edge, use MQTT for local communication and come with a default real-time destination which has filters set to `#`. This means that, by default, all messages on all topics are published to the AWS IoT SiteWise real-time destination. For more information, see [Understand path filters for AWS IoT SiteWise Edge destinationsUnderstand path filters](#destinations-path-filters). You can add one real-time destination in each gateway. ### Destination types When configuring a destination for your gateway, you have two main options: real-time configuration using AWS IoT SiteWise, and a buffered configuration using Amazon S3. Each destination type has its own set of settings and considerations. **AWS IoT SiteWise real-time settings** Choose this to send data directly to AWS IoT SiteWise hot-tier storage to facilitate ingesting and monitoring data in real-time. The real-time settings manage data flow, particularly when a gateway experiences connectivity issues with the cloud. During connection loss, data is temporarily stored locally on the gateway. Once the connection is re-established, the stored data is automatically sent to the cloud. You can adjust various aspects of the data publishing process, such as the maximum amount of data to be stored locally, the rate at which data is sent to the cloud upon reconnection, and when to delete data after the storage reaches its capacity. For more information on AWS IoT SiteWise storage tiers, see, [Manage data storage in AWS IoT SiteWise](manage-data-storage.md). **AWS IoT SiteWise buffered using Amazon S3 settings** This destination type allows you to buffer data locally on the gateway and periodically send it to an Amazon S3 bucket in batches. The data is stored in the efficient Parquet format, which is optimized for analytical workloads. Once the data is in Amazon S3, you can import it into AWS IoT SiteWise for storage, processing, and analysis. Choose this option to ingest data in batches, and store historical data in a cost-effective way. You can configure your preferred Amazon S3 bucket location, and the frequency at which you want data to be uploaded to Amazon S3. You can also choose what to do with the data after ingestion into AWS IoT SiteWise. You can choose to have the data available in both SiteWise and Amazon S3 or you can choose to delete it automatically from Amazon S3. ### Compare destination functionality between gateway versions The destinations feature in MQTT-enabled gateways streamlines data flow management. Destinations simplify data management through centralized configuration of data routing to various endpoints. This approach eliminates the need for complex individual stream setups, making the overall system more flexible and easier to manage. By comparison, the Classic streams, V2 gateway, SiteWise Edge transmits data from data sources to publishers via AWS IoT Greengrass streams, configuring data destinations individually for each data source. With the AWS IoT SiteWise destination feature, the publisher routing configuration is consolidated. Destination configuration allows you to manage destinations and path filters in a centralized manner. You can easily add a destination, manage path filters, delete unnecessary filters or destinations, depending on your needs. Additionally, the destinations feature utilizes MQTT (Message Queuing Telemetry Transport), an industry-standard protocol widely used in industrial IoT applications. This adoption of MQTT helps AWS IoT SiteWise to facilitate easier integration with various devices and systems. ### Destination limitations Current limitations for destinations on SiteWise Edge gateways include: + The data processing pack isn't supported on MQTT-enabled gateways. + Data type support is limited to AWS IoT SiteWise data types. For information on enabling data type conversion, see [Converting unsupported data types](string-conversion.md). ### Use cases for SiteWise Edge destinations SiteWise Edge destinations are utilized in diverse applications. Here are some key examples: **Industrial automation***Real-time monitoring and predictive maintenance* In industrial settings, sensors and devices on the factory floor can publish data to SiteWise Edge. Destinations can be configured to filter and route relevant data, enabling real-time monitoring and analysis of machine performance. You can subscribe to relevant MQTT topics using path filters, process the data, and then publish the processed data. In this way, you can selectively route processed data to AWS cloud analytic services or on-premises systems. Manufacturers can then implement predictive maintenance strategies, optimize production processes, and reduce downtime. **Smart buildings***Energy efficiency and occupancy optimization* Building automation systems generate data streams to monitor and control various aspects of a building, such as HVAC systems, lighting, and access control. With SiteWise Edge, these data streams can be ingested, processed, and routed to different destinations. Facility managers can configure destinations to filter and forward relevant data, enabling advanced capabilities like energy efficiency measures and occupancy optimization while ensuring data privacy and compliance. These use cases demonstrate how the Destinations feature in SiteWise Edge can be leveraged across various industries to ingest, process, and route data efficiently. This enables advanced capabilities such as real-time monitoring, predictive maintenance, energy efficiency, and remote diagnostics while ensuring data privacy and compliance. ## Understand path filters for AWS IoT SiteWise Edge destinations **Topics** + [Path filter requirements](#path-filter-requirements) + [Best practices for path filters](#create-effective-path-filters) + [Path filters for OPC UA servers](#path-filters-opcua) + [Special characters in path filter names](#path-filters-special-characters) Each destination is configured to route data to AWS IoT SiteWise or Amazon S3. Path filters allow you to select specific data to filter when receiving MQTT messages for a destination. Path filters represent the logical names of your data streams, acting as subscriptions to desired MQTT topics. In MQTT, data is organized into topics, which are hierarchical strings separated by forward slashes (`/`). For example, a device might publish temperature data to the topic `home/livingroom/sensor1/temperature`. Here, `home/livingroom/sensor1` represents the path or logical name of the sensor, and `temperature` is the data type being published. You can use path filters to subscribe to specific topics or a range of topics using wildcards (`+` and `#`). The `+` wildcard matches a single level in the topic hierarchy. For example, `home/+/sensor1/temperature` would match `home/livingroom/sensor1/temperature` and `home/bedroom/sensor1/temperature`. The `#` wildcard, when used at the end of a filter, matches multiple levels. You can also use a variety of characters typically not allowed in the MQTT specification within a path filter name. These characters don't function as wildcards when used within a name. AWS IoT SiteWise converts these characters using encoding to ensure MQTT compliance while preserving your original naming structure. This feature is particularly useful for accommodating existing naming conventions from other systems. For more information, see [Special characters in path filter names](#path-filters-special-characters). By carefully selecting the appropriate path filters, you can control which data is sent to a specific destination. Tailor the data flow to your IoT system's requirements using path filters. ### Path filter requirements When entering path filters using the AWS IoT SiteWise console, keep the following in mind: + Path filters are delimited by a new line, with each line representing a separate path filter. + Individual path filters can have between 1 and 65,535 bytes. + A path filter can't be blank. + Null values (U\$10000) are not allowed. + You can enter up to 100 path filters or 65,535 characters at a time, whichever limit is reached first. + The overall limit is 20,000 path filters for all the destinations on a gateway combined. + You can use `%`, `#`, `+`, and `$` characters within path filter names, however AWS IoT SiteWise automatically converts them to URI encoding. ### Best practices for path filters When creating path filters for your AWS IoT SiteWise destinations, consider the following strategies to effectively manage your data. + Structure your filters to mirror your device hierarchy. For example, in a manufacturing setting, `factory/+/machine/#`, captures data from all machines across different production lines. + Use specific levels for device types, locations, or functions. For example, `factory/assembly-line/robot/temperature`. Or, in smart agriculture, `farm/+/crop/+/moisture`, to monitor moisture levels for various crops across different fields. + Leverage wildcards strategically: Use `+` for variations at a single level and `#` to capture all subsequent levels. For example, `building/+/+/energy-consumption`, tracks energy usage across different zones and floors in a building. This assumes the first `+` captures all floors and the second `+` captures all zones. + Balance specificity and flexibility by creating filters that are specific enough to capture relevant data but flexible enough to accommodate future changes. For example, `site/+/equipment-type/+/measurement` allows for addition of new sites or equipment types without changing the filter structure. Test your filters thoroughly to ensure they capture the intended data and align with your IoT system's architecture and goals. ### Path filters for OPC UA servers For OPC UA servers, your path filters must correspond to the OPC UA tag names. The final level of your path filter must match the OPC UA tag name exactly. For example, if your OPC UA tag is `Device1.Temperature`, your path filter might be `factory/line1/Device1.Temperature`. You can use wildcards in the preceding levels, such as `factory/+/Device1.Temperature` to capture the tag across multiple production lines. If you have special characters within your path filter names, see [Special characters in path filter names](#path-filters-special-characters) for more information. ### Special characters in path filter names AWS IoT SiteWise accommodates characters commonly used in industrial protocols like OPC UA, which are typically not allowed in standard MQTT topic names. This feature facilitates smoother integration of industrial systems with MQTT-based architectures. **Note** While our special character handling is helpful for integration and migration, it's recommended to align with standard MQTT naming conventions for new implementations when possible to ensure broader compatibility. When receiving data from industrial sources, AWS IoT SiteWise normalizes topic names using URI encoding for special characters: + `%` becomes `%25` (encoded first as the escape character) + `#` becomes `%23` + `+` becomes `%2B` + `$` becomes `%24` (only when at the start of a topic) This encoding ensures that source data containing these special MQTT characters can be safely used as MQTT topic names while preserving the original industrial naming conventions. **Example : Special characters in path filter names** Here are examples of how industrial topic names might appear in AWS IoT SiteWise path filters: + `Factory1/Line#2/Sensor+3` becomes `Factory1/Line%232/Sensor%2B3` + `Plant%A/Unit$1/Temp` becomes `Plant%25A/Unit%241/Temp` + `Site1/#Section/+Node` becomes `Site1/%23Section/%2BNode` When creating subscriptions or viewing topic names in AWS IoT SiteWise, you'll see the original, unencoded versions. The encoding is handled automatically to ensure MQTT compliance. # Add an AWS IoT SiteWise Edge real-time destination The real-time destination type enables you to stream IoT data directly from your devices and gateways into AWS IoT SiteWise storage in real-time. This option is ideal for use cases that require immediate ingestion and processing of data as it is generated, without the need for batching or buffering. You can only have one real-time destination configured in each gateway, as it streams data continuously to AWS IoT SiteWise. **Note** Duplicate TQVs may result in double charging. **To add a real-time destination** Use the AWS IoT SiteWise console or AWS CLI to add a real-time destination to your SiteWise Edge MQTT-enabled gateway. ------ #### [ Console ] 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the navigation pane, choose **Edge gateways**. 1. Select the gateway to which you want to add a destination. 1. In the **Destinations** section, choose **Add destination**. 1. On the **Add destination** page, enter **Destination details**: 1. A name for your destination in the **Destination name** field. 1. Select the **AWS IoT SiteWise real-time** for the **Destination type**. 1. Configure the gateway publishing order by setting the **Publishing order** to either **Publish older data first** or **Publish newest data first**. By default, the gateway publishes the oldest data first. 1. Use **Maximum batch wait time** to set a maximum time for the publisher to wait before sending a batch of data to AWS IoT SiteWise. This setting applies for each alias. The data is stored locally until either: + The set time has elapsed, or + 10 time-quality-value (TQV) entries are received for the alias Whichever condition is met first triggers the batch to be sent to the cloud. 1. To compress uploaded data, select the **Activate compression when uploading data** check box. Letting the gateway compress your data prior to uploading it to the cloud reduces bandwidth usage. 1. To filter out expired publisher data, select the **Exclude expired data** check box. This selection only sends active and current data to AWS IoT SiteWise. 1. In the **Cutoff period** field, enter the frequency at which data should be considered expired within your dataset. You can determine if the data is counted in terms of minutes or days. The minimum cutoff period is five minutes. The maximum cutoff period is seven days. 1. Optionally configure the **Local storage settings**: 1. Set the **Retention period** frequency – The amount of time the gateway locally stores data that is older than the cutoff period. The minimum retention period is one minute. The maximum retention period is 30 days and greater than or equal to the rotation period. 1. Set the **Rotation period** – The time interval to specify when saving data that is older than the cutoff period for a single file. The gateway transfers one batch of data to the following local directory at the end of each rotation period: `/greengrass/v2/work/aws.iot.SiteWiseEdgePublisher/exports`. The retention must be greater than one minute and equal to the retention period. 1. Provide the **Storage capacity (GB)** value to set the maximum size of data stored locally in GB. If the data exceeds the determined maximum local storage size, the gateway starts deleting the oldest data first. The gateway continues to delete until the size of data stored locally is equal to or less than the quota. The storage capacity must be greater than or equal to one GB. 1. Add path filters to your destination. For more information see, [Add path filters to AWS IoT SiteWise Edge destinations](destinations-add-path-filters.md). For more information, see [Destination types](gw-destinations.md#destination-types). ------ #### [ AWS CLI ] **Example : Create a new AWS IoT SiteWise real-time destination** Use the [UpdateGatewayCapabilityConfiguration](https://docs.aws.amazon.com/iot-sitewise/latest/APIReference/API_UpdateGatewayCapabilityConfiguration.html) API to configure the publisher. Set the `capabilityNamespace` parameter to `iotsitewise:publisher:3`. ``` { "sources": [ { "type": "MQTT" } ], "destinations": [ { "type": "SITEWISE_REALTIME", "name": "your-destination-name", "config": { "publishingOrder": "TIME_ORDER", "enableCompression": true, "maxBatchWaitTime": "10s" }, "filters": [ { "type": "PATH", "config": { "paths": [ "#" ] } } ] } ] } ``` To update an existing AWS IoT SiteWise real-time destination, first use the `DescribeGatewayCapabilityConfiguration` API to find the `destinationId`. **Example : Update an AWS IoT SiteWise real-time destination** Use the [UpdateGatewayCapabilityConfiguration](https://docs.aws.amazon.com/iot-sitewise/latest/APIReference/API_UpdateGatewayCapabilityConfiguration.html) API to configure the publisher. Set the `capabilityNamespace` parameter to `iotsitewise:publisher:3`. ``` { "sources": [ { "type": "MQTT" } ], "destinations": [ { "id": "your-existing-destination-id", "type": "SITEWISE_REALTIME", "name": "your-destination-name", "config": { "publishingOrder": "TIME_ORDER", "enableCompression": true, "dropPolicy": { "cutoffAge": "7d", "exportPolicy": { "retentionPeriod": "7d", "rotationPeriod": "6h", "exportSizeLimitGB": 10 } }, "maxBatchWaitTime": "10s" }, "filters": [ { "type": "PATH", "config": { "paths": [ "#" ] } } ] } ] } ``` The following configuration options are specific to gateways using the `iotsitewise:publisher:3` namespace. `sources` Defines data sources to transfer of data from your industrial equipment to AWS IoT SiteWise. For MQTT-enabled gateways, use `MQTT`. Type: Array of objects Required: Yes `destinations` Defines where to send data. Destinations are either real-time or buffered using Amazon S3. At least one destination object is required, but you can add an empty array. You can have one real-time destination for each gateway. For more information, see [Understand AWS IoT SiteWise Edge destinations](gw-destinations.md#source-destination). Type: Array of objects Required: Yes `id` The unique identifier for the destination. You can either provide an existing destination ID or leave it blank. If you do not specify an ID then a UUID is generated by default. Type: String Required: No `type` Type of destination. Options include: `SITEWISE_REALTIME` and `SITEWISE_BUFFERED`. + `SITEWISE_REALTIME` – Send data directly to AWS IoT SiteWise storage in real-time. + `SITEWISE_BUFFERED` – Send data to Amazon S3 in batches in Parquet format, and then import into AWS IoT SiteWise storage. Type: String Required: Yes `name` A unique name for the destination. Type: String Required: Yes `config` Configuration specific to the destination type in JSON format. The configuration varies between real-time and buffered destinations. Type: Object Required: Yes publishingOrder Determines the order in which data is published. Data publishes based on its timestamp. Options include `TIME_ORDER` and `RECENT_DATA`. + `TIME_ORDER` (default) – Publishes older data first. + `RECENT_DATA` – Publishes newest data first. Type: String Required: No enableCompression When set to `true`, enables data compression before sending to AWS IoT SiteWise. Letting the gateway compress your data prior to uploading it to the cloud reduces bandwidth usage. The default value is `true`. Type: Boolean Required: No dropPolicy Defines how to handle older data. Type: object Required: No + `cutoffAge` The maximum age of data to be published specified in days, hours, and minutes. For example, `7d` or `1d7h16m`. Data older than what you specify is not sent to AWS IoT SiteWise. Data that is earlier than the cutoff period is not published to the cloud. The cutoff age must be between five minutes and seven days. You can use `m`, `h`, and `d` when you specify a cutoff age. Note that `m` represents minutes, `h` represents hours, and `d` represents days. Type: String Required: Yes + `exportPolicy` Defines how to handle data that exceeds the cutoff age. Type: Object Required: No + `retentionPeriod` Your SiteWise Edge gateway deletes any data at the edge that is earlier than the cutoff period from the local storage after it is stored for the specified retention period. The retention period must be between one minute and 30 days, and greater than or equal to the rotation period. You can use `m`, `h`, and `d` when you specify a retention period. Note that `m` represents minutes, `h` represents hours, and `d` represents days. Type: String Required: No + `rotationPeriod` The time interval over which to batch up and save data that is earlier than the cutoff period to a single file. The SiteWise Edge gateway transfers one batch of data to the following local directory at the end of each rotation period: `/greengrass/v2/work/aws.iot.SiteWiseEdgePublisher/exports`. The rotation period must be greater than one minute, and equal to or less than the retention period. You can use `m`, `h`, and `d` when you specify a rotation period. Note that `m` represents minutes, `h` represents hours, and `d` represents days. Type: String Required: No + `exportSizeLimitGB` The maximum allowed size of data stored locally, in GB. If this quota is breached, the SiteWise Edge gateway starts deleting the earliest data until the size of data stored locally is equal to or less than the quota. The value of this parameter must be greater than or equal to 1. Type: Integer Required: No `maxBatchWaitTime` Sets a maximum time for the publisher to wait before sending a batch of data to AWS IoT SiteWise. This setting applies for each alias. The data is stored locally until either: + The set time has elapsed, or + 10 time-quality-value (TQV) entries are received for the alias Use `m`, `h`, and `d` to specify a cutoff time. Note that `m` represents minutes, `h` represents hours, and `d` represents days. Type: String Required: No `filters` Filters to apply to the data. At least one filter is required. Type: String Required: Yes `type` Type of filter. Use `PATH`. Type: String Required: Yes `config` Configuration specific to the filter type in JSON format. At least one object is required, but the array can be empty. Type: Object Required: Yes + `paths` An array of path filters. For more information, see [Understand path filters for AWS IoT SiteWise Edge destinationsUnderstand path filters](gw-destinations.md#destinations-path-filters). The default path is `#`. Type: Array of strings Required: Yes ------ # Add an AWS IoT SiteWise buffered destination using Amazon S3 The buffered destination type allows you to save on ingestion costs into AWS IoT SiteWise if you don't need the data in real-time. It enables you to temporarily store your IoT data in an Amazon S3 bucket before importing it into AWS IoT SiteWise. Or, you can simply upload your data to S3 for storage, regardless of whether you plan to import it to AWS IoT SiteWise. This is useful for batching and buffering data from your devices and gateways before ingesting it into AWS IoT SiteWise. With this option, data is uploaded to the specified S3 bucket in Parquet format at a configured frequency. You can then import this data into AWS IoT SiteWise storage for further analysis and processing. **To add a destination buffered using Amazon S3** Use the AWS IoT SiteWise console or AWS CLI to add a destination that buffers data using Amazon S3 to your SiteWise Edge gateway. ------ #### [ Console ] Use the AWS Management Console to add an AWS IoT SiteWise destination buffered using Amazon S3. 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the navigation pane, choose **Edge gateways**. 1. Select the gateway to which you want to add a destination. 1. In the **Destinations** section, choose **Add destination**. 1. On the **Add destination** page, enter **Destination details**: 1. A name for your destination in the **Destination name** field. 1. Select **AWS IoT SiteWise buffered using Amazon S3** for **Destination type**. AWS IoT SiteWise buffered using Amazon S3 sends data to Amazon Simple Storage Service in batches, in Parquet format, and then imports the data into AWS IoT SiteWise storage. 1. Enter the Amazon S3 URL for the location where you want to store your gateway data. You can browse for the path by choosing **Browse S3**. Once a bucket is added, you can also view the bucket by choosing **View**. 1. Specify how often your gateway should upload data to Amazon S3 by entering a time frame and selecting a time increment for **Data upload frequency**. The frequency value should be greater than 0 and less than or equal to 30 days. 1. In **Data storage settings**, determine what to do with your gateway data after importing it to AWS IoT SiteWise. There are two decisions to make regarding data storage: + If you want to copy imported data into AWS IoT SiteWise storage, select the **Copy data to storage** check box. This option duplicates the imported data from your configured Amazon S3 bucket into AWS IoT SiteWise storage. + If you choose to import your data from your Amazon S3 bucket into AWS IoT SiteWise storage, you can also specify whether the imported data should be deleted after the import is complete. Select the **Delete data from Amazon S3** check box to delete the imported date from the configured Amazon S3 bucket after importing it to AWS IoT SiteWise storage. 1. Add path filters to your destination. For more information see, [Add path filters to AWS IoT SiteWise Edge destinations](destinations-add-path-filters.md). ------ #### [ AWS CLI ] **Example : Create a new AWS IoT SiteWise destination buffered using Amazon S3** Use the [UpdateGatewayCapabilityConfiguration](https://docs.aws.amazon.com/iot-sitewise/latest/APIReference/API_UpdateGatewayCapabilityConfiguration.html) API to configure the publisher. Set the `capabilityNamespace` parameter to `iotsitewise:publisher:3`. ``` { "sources": [ { "type": "MQTT" } ], "destinations": [ { "type": "SITEWISE_BUFFERED", "name": "your-s3-destination-name", "config": { "targetBucketArn": "arn:aws:s3:::amzn-s3-demo-bucket/Optional/SomeFolder", "publishPolicy": { "publishFrequency": "15m", "localSizeLimitGB": 10 }, "siteWiseImportPolicy": { "enableSiteWiseStorageImport": true, "enableDeleteAfterImport": true, "bulkImportJobRoleArn": "arn:aws:iam::123456789012:role/your-role-name" } }, "filters": [ { "type": "PATH", "config": { "paths": [ "#" ] } } ] } ] } ``` **Example : Update an AWS IoT SiteWise destination buffered using Amazon S3** To update an existing AWS IoT SiteWise real-time destination, first use the `DescribeGatewayCapabilityConfiguration` API to find the `destinationId`. The publisher namespace: `iotsitewise:publisher:3` ``` { "sources": [ { "type": "MQTT" } ], "destinations": [ { "id": "your-existing-destination-id", "type": "SITEWISE_BUFFERED", "name": "your-s3-destination-name", "config": { "targetBucketArn": "arn:aws:s3:::amzn-s3-demo-bucket/Optional/SomeFolder", "publishPolicy": { "publishFrequency": "15m", "localSizeLimitGB": 10 }, "siteWiseImportPolicy": { "enableSiteWiseStorageImport": true, "enableDeleteAfterImport": true, "bulkImportJobRoleArn": "arn:aws:iam::123456789012:role/your-role-name" } }, "filters": [ { "type": "PATH", "config": { "paths": [ "#" ] } } ] } ] } ``` The following configuration options are specific to MQTT-enabled gateways using the `iotsitewise:publisher:3` namespace. `sources` Defines data sources to transfer of data from your industrial equipment to AWS IoT SiteWise. For MQTT-enabled gateways, use `MQTT`. Type: Array of objects Required: Yes `destinations` Defines where to send data. Destinations are either real-time or buffered using Amazon S3. At least one destination object is required, but you can add an empty array. You can have one real-time destination for each gateway. For more information, see [Understand AWS IoT SiteWise Edge destinations](gw-destinations.md#source-destination). Type: Array of objects Required: Yes `id` The unique identifier for the destination. You can either provide an existing destination ID or leave it blank to have a new ID automatically generated for the destination. Type: String Required: No `type` Type of destination. Options include: `SITEWISE_REALTIME` and `SITEWISE_BUFFERED`. Choose `SITEWISE_BUFFERED`. + `SITEWISE_REALTIME` (default) – Send data directly to AWS IoT SiteWise storage in real-time. For more information, see [Add an AWS IoT SiteWise Edge real-time destination](destinations-real-time.md). + `SITEWISE_BUFFERED` – Send data to Amazon S3 in batches in Parquet format, and then import into AWS IoT SiteWise storage. Type: String Required: Yes `name` A unique name for the destination. Type: String Required: Yes `config` Configuration specific to the destination type in JSON format. The configuration varies between real-time and buffered destinations. Type: Object Required: Yes `targetBucketArn` The bucket ARN to publish to. Choose the same AWS Region for both AWS IoT SiteWise and Amazon S3. If a prefix is chosen, it must have between 1-255 characters. AWS IoT SiteWise, including the gateway, will have access to the entire specified S3 bucket. We recommend using a dedicated bucket for buffered data ingestion. Type: String Required: Yes `publishPolicy` Details of the publishing policy. Type: Object Required: Yes `publishFrequency` The frequency with which the SiteWise Edge gateway publishes to the Amazon S3 bucket. Data upload frequency to Amazon S3 must be greater than 0 minutes and less than or equal to 30 days. You can use `m`, `h`, and `d` when you specify a publishing frequency age. Note that `m` represents minutes, `h` represents hours, and `d` represents days. The default value is 15 minutes. Type: String Required: Yes `localSizeLimitGB` The maximum size of the files written to local disk in GB. If this threshold is breached, the publisher publishes all buffered data to its destination. Type: Integer Required: Yes `siteWiseImportPolicy` Details of the import policy for importing data to AWS IoT SiteWise. Type: Object Required: Yes `enableSiteWiseStorageImport` Set this to `true` to import data from an Amazon S3 bucket to AWS IoT SiteWise storage. It initially makes a copy of the data in AWS IoT SiteWise. Then, if you set `enableDeleteAfterImport` to true, the data in S3 deletes after copying to AWS IoT SiteWise. Pricing implications apply. The default value is `true`. Type: Boolean Required: Yes `enableDeleteAfterImport` Set this to `true` to delete the file in the Amazon S3 bucket after ingestion into the AWS IoT SiteWise storage. The default value is `true`. Type: Boolean Required: Yes `bulkImportJobRoleArn` The ARN of the IAM role that AWS IoT SiteWise assumes to read buffered data from Amazon S3 during data ingestion. This role is used when an edge device calls on AWS IoT SiteWise APIs to initiate the bulk import process. If `enableSiteWiseStorageImport` is set to `true`, this parameter is required. Type: String Required: No ------ Add path filters for your destination. For more information, see [Add path filters to AWS IoT SiteWise Edge destinations](destinations-add-path-filters.md). # Add path filters to AWS IoT SiteWise Edge destinations Add path filters to a destination. Path filters use MQTT topic syntax, where `#` is a wildcard character that matches any number of levels, and `+` is a wildcard character that matches a single level. You can add multiple destinations to a gateway, each with its own set of path filters subscribed to your equipment telemetry. Siemens Industrial Edge gateways use a prefix for compatibility. For more information, see [Prefixes for path filters](sitewise-edge-on-siemens.md#siemens-path-filters). ------ #### [ Console ] **To add path filters** 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the navigation pane, choose **Edge gateways**. 1. Select the gateway to which you want to add path filters. 1. In the **Path filters** section under **Add destination**, choose **Add path filter**. 1. Enter the path filter that you want this destination to subscribe to. You can use wildcard characters (`#` and `+`) to subscribe to multiple paths. 1. Choose **Add path filter** to add the path filter to the list. 1. Repeat steps to add additional path filters, if needed. 1. Once you have added all of the required path filters, choose **Create**. ------ #### [ AWS CLI for self-hosted gateways ] **Example : Path filter configuration** ``` { "destinations": [ { ... } ], "filters": [ { "type": "PATH", "config": { "paths": [ "home/+/sensor1/temperature", "home/livingroom/sensor1/temperature", "home/livingroom/sensor1/temperature", "building/#" ] } } ] } ``` ------ #### [ AWS CLI for Siemens IEgateways ] **Example : Prefix configuration for path filters** Capture all data by using both the data (`ie/d`) and metadata (`ie/m`) prefixes for each path filter. ``` { "destinations": [ { ... } ], "filters": [ { "type": "PATH", "config": { "paths": [ "ie/d/home/+/sensor12/temperature", "ie/m/home/livingroom/sensor12/temperature", "ie/d/home/livingroom/sensor13/temperature2", "ie/m/home/livingroom/sensor13/temperature2", "ie/d/building/#", "ie/m/building/#" ] } } ] } ``` ------ **Note** Copy path filters between destinations by downloading list of path filters. For more information, see [Download all path filters in a destination (console)](destinations-manage.md#destinations-download-list). ## Upload path filters in bulk To upload path filters in bulk, use a CSV or text file. AWS IoT SiteWise automatically removes exact duplicates when you upload files. For example, `windfarm/site1/` and `windfarm/site1/` are exact duplicates that AWS IoT SiteWise catches because the string is exactly the same. Partial duplicates are not removed and result in additional charges. For example, `windfarm/#` and `windfarm/site1` are overlapping topics because `windfarm/site1` is already encompassed by `windfarm/#`. **Note** Avoid duplicates to prevent additional charges. The uploaded file must be in either .csv or .txt format. It can't contain any headers and should consist of a single column. In the column, list your path filters, with each filter on a separate line. No other information should be included in the file. **File upload requirements** These are additional path filter requirements. + You can upload one .csv or .txt file. Other file formats are not supported. + CSV (.csv) files cannot have headers and should only contain one column. + You can have one path filter on each line. + The uploaded files cannot be empty. + When using `#` as a wildcard, it must be the last character in the topic filter. For example, `topic/#` or as a standalone character at a particular topic level. However, note that `#` can also be used as a regular character within a topic level name, such as `factory/machine#1/topic`. For more information see [Special characters in path filter names](gw-destinations.md#path-filters-special-characters) + You can also use the `+` character. For example, use `factory/+/temp` to get all temperatures for factories instead of `factory/machine2/temp` and `factory/machine3/temp` individually. # Manage AWS IoT SiteWise Edge destinations After adding destinations, you can perform various operations to manage them, such as editing destination configurations, deleting destinations, and managing path filters. ## Edit a destination Select the radio button next to the destination in the table and choose the **Edit** button to edit a destination. ------ #### [ Console ] **To edit a destination using the AWS IoT SiteWise console** 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the left navigation, choose **Edge gateways** in the **Edge** section. 1. Select the appropriate gateway. 1. In the **Destinations** section, choose destination you want to edit and then choose **Edit**. 1. Modify the destination and then choose **Save**. ------ #### [ AWS CLI ] **To edit a destination using AWS CLI** + You can edit a destination by modifying the JSON capability configuration information. ``` aws iotsitewise update-gateway-capability-configuration \ --gateway-id your-gateway-id \ --capability-namespace "iotsitewise:publisher:3" \ --capability-configuration '{ "sources": [ { "type": "MQTT" } ], "destinations": [ { "id": "your-existing-destination-id", "type": "SITEWISE_REALTIME", "name": "your-updated-destination-name", "config": { "publishingOrder": "TIME_ORDER", "enableCompression": true, "dropPolicy": { "cutoffAge": "10d", "exportPolicy": { "retentionPeriod": "10d", "rotationPeriod": "6h", "exportSizeLimitGB": 10 } }, "maxBatchWaitTime": "15s" }, "filters": [ { ... } ] } ] }' ``` **Note** You can't update the destination `type` or `capability-namespace`. For example, you can't switch from a type of `SITEWISE_REALTIME` to `SITEWISE_BUFFERED`. You can have one real-time destination for each MQTT-enabled gateway. ------ ## Delete a destination If you no longer need a destination, you can delete it from your SiteWise Edge gateway. ------ #### [ Console ] **To delete a destination using the AWS IoT SiteWise console** 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the left navigation, choose **Edge gateways** in the **Edge** section. 1. Select the appropriate gateway. 1. In the **Destinations** section, choose destination you want to delete and then choose **Delete**. A confirmation screen appears. 1. To confirm your choice to delete the destination, type "delete" in the confirmation box. ------ #### [ AWS CLI ] **To delete a destination using AWS CLI** + Delete the gateway capability configuration by specifying the gateway ID and modifying the capability configuration to remove the destination you want to delete. ``` aws iotsitewise update-gateway-capability-configuration \ --gateway-id your-gateway-id \ --capability-namespace "iotsitewise:publisher:3" \ --capability-configuration '{ "sources": [ { "type": "MQTT" } ], "destinations": [] }' ``` **Note** The destinations array can be empty (`[]`), but the destinations object itself must be included in the capability configuration. ------ ## Download all path filters in a destination (console) Download a CSV file containing all of your path filters in the AWS IoT SiteWise console. You can use a downloaded list of path filters to easily share path filter lists between gateway destinations. **To download a CSV file of all path filters using the AWS IoT SiteWise console** 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the left navigation, choose **Edge gateways** in the **Edge** section. 1. Select the gateway containing your path filters. 1. Choose either **Add destination** or **Edit destination**. 1. Navigate to the **Path filters** section and choose **Download CSV**. **Note** The CSV file includes all path filters in a particular destination, regardless of which ones you selected from the list of path filters. ## Edit a path filter You can edit individual path filters to refine which data your destination receives. ------ #### [ Console ] Using the AWS IoT SiteWise console, you can edit each individual path filter within each respective text box. **To edit a path filter using the AWS IoT SiteWise console** 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the left navigation, choose **Edge gateways** in the **Edge** section. 1. Select the gateway containing your path filters. 1. Select the appropriate destination. 1. Choose **Edit**. 1. Choose the text box for the row containing the path filter that you want to edit. 1. Update the path filter's text, ensuring that the edited path filter's checkbox is selected. 1. Choose **Save**. ------ #### [ AWS CLI ] To edit path filters for a destination using the AWS CLI, first retrieve the current configuration, modify it, and then update it using the `update-gateway-capability-configuration` command. **To edit a path filter using AWS CLI** 1. Retrieve the current capability configuration: ``` aws iotsitewise describe-gateway-capability-configuration \ --gateway-id your-gateway-id \ --capability-namespace "iotsitewise:publisher:3" \ --query "capabilityConfiguration" ``` 1. Edit the JSON to modify the path filters as needed. 1. Update the capability configuration with the modified path filters: ``` aws iotsitewise update-gateway-capability-configuration \ --gateway-id your-gateway-id \ --capability-namespace "iotsitewise:publisher:3" \ --capability-configuration json-containing-your-updated-path-filters ``` ------ ## Delete a path filter You can delete path filters for a destination to control the data it receives from MQTT sources and data processing pipelines. ------ #### [ Console ] **To delete a path filter using the AWS IoT SiteWise console** 1. Open the [AWS IoT SiteWise console](https://console.aws.amazon.com/iotsitewise/). 1. In the left navigation, choose **Edge gateways** in the **Edge** section. 1. Select the gateway containing your path filters. 1. Select the appropriate destination. 1. Choose **Edit**. 1. On the **Edit destination** screen, in the **Path filters** section, select one or more the path filters to delete. 1. Choose **Delete**. A deletion confirmation message appears. If want to proceed with deleting the path filters, choose **Delete** on the confirmation screen. ------ #### [ AWS CLI ] **To delete a destination using AWS CLI** + Delete a path filter by removing it from the capability configuration. ``` aws iotsitewise update-gateway-capability-configuration \ --gateway-id your-gateway-id \ --capability-namespace "iotsitewise:publisher:3" \ --capability-configuration '{ "sources": [ { "type": "MQTT" } ], "destinations": [ { "id": "your-destination-id", "type": "SITEWISE_REALTIME", "name": "your-destination-name", "config": { ... }, "filters": [ { "type": "PATH", "config": { "paths": [ "/path1", "/path2", "/delete-a-path-to-remove-it" ] } } ] } ] } ``` **Note** The filters array can be empty (`[]`), but the filters object itself must be included in the capability configuration. ------