# AWS IoT Device Tester for FreeRTOS
The IDT for FreeRTOS is a tool to qualify data throughput rate with the FreeRTOS operating system. The device tester (IDT) first opens a USB or UART connection to a device. It then flashes an image of FreeRTOS configured to test the device functionality under various conditions. AWS IoT Device Tester suites are extensible and IDT is used for customer AWS IoT test orchestration.
IDT for FreeRTOS runs on a host computer (Windows, macOS, or Linux) that is connected to the device being tested. IDT configures and orchestrates test cases, and aggregates results. It also provides a command line interface to manage test execution.
## FreeRTOS qualification suite
IDT for FreeRTOS verifies the port of FreeRTOS on your micro-controller, and if it can communicate effectively with AWS IoT in a reliable and secure manner. Specifically, it verifies if the porting layer interfaces for FreeRTOS libraries are implemented correctly. It also performs end-to-end tests with AWS IoT Core. For example, it verifies if your board can send and receive MQTT messages and process them correctly.
FreeRTOS qualification (FRQ) 2.0 suite uses tests cases from FreeRTOS-Libraries-Integration-Tests and Device Advisor defined in the [ FreeRTOS Qualification Guide](https://docs.aws.amazon.com/freertos/latest/qualificationguide/freertos-qualification.html#qualifying-your-device-idt).
IDT for FreeRTOS generates test reports that you can submit to AWS Partner Network (APN) for inclusion of your FreeRTOS devices in the AWS Partner Device Catalog. For more information, see [AWS Device Qualification Program](https://aws.amazon.com/partners/dqp/).
The following diagram shows the test infrastructure setup for FreeRTOS qualification.
![\[Flowchart that shows how AWS IoT Core interacts with your computer and the microcontroller.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/devicetester_afr.png)
IDT for FreeRTOS organizes test resources into test suites and test groups:
+ A test suite is the set of test groups used to verify that a device works with particular versions of FreeRTOS.
+ A test group is the set of individual test cases related to a particular feature, such as BLE and MQTT messaging.
For more information, see [Test suite versions](idt-test-suite-versions.md)
## Understand custom test suites
IDT for FreeRTOS combines a standardized configuration setup and result format with a test suite environment. This environment lets you develop custom test suites for your devices and device software. You can add custom tests for your own internal validation, or provide them to your customers for device verification.
How you configure custom test suites determines the setting configurations that you must provide to your users to run your custom test suites. For more information, see [Develop and run your own IDT test suites](idt-custom-tests.md).
# Supported versions of AWS IoT Device Tester
Supported versions of IDT for FreeRTOS
This topic lists supported versions of AWS IoT Device Tester for FreeRTOS. As a best practice, we recommend that you use the latest version of IDT for FreeRTOS that supports your target version of FreeRTOS. Each version of IDT for FreeRTOS has one or more corresponding versions of FreeRTOS that it supports. We recommend that you download a new version of IDT for FreeRTOS when a new version of FreeRTOS is released.
By downloading the software, you agree to the AWS IoT Device Tester License Agreement contained in the download archive.
**Note**
When you use AWS IoT Device Tester for FreeRTOS, we recommend that you update to the latest patch release of the most recent FreeRTOS-LTS version.
**Important**
As of October 2022, AWS IoT Device Tester for AWS IoT FreeRTOS Qualification (FRQ) 1.0 doesn't generate signed qualification reports. You can't qualify new AWS IoT FreeRTOS devices to list in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices) through the [AWS Device Qualification Program](http://aws.amazon.com/partners/programs/dqp/) using IDT FRQ 1.0 versions. While you can't qualify FreeRTOS devices using IDT FRQ 1.0, you can continue to test your FreeRTOS devices with FRQ 1.0. We recommend that you use [IDT FRQ 2.0](https://docs.aws.amazon.com/freertos/latest/userguide/lts-idt-freertos-qualification.html) to qualify and list FreeRTOS devices in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices).
## Latest version of AWS IoT Device Tester for FreeRTOS
Latest version of IDT for FreeRTOS
Use the following links to download the latest versions of IDT for FreeRTOS.
**Latest version of AWS IoT Device Tester for FreeRTOS**
| **AWS IoT Device Tester version** | **Test suite versions** | **Supported FreeRTOS versions** | **Download links** | **Release date** | **Release notes** |
| --- | --- | --- | --- | --- | --- |
| IDT v4.9.0 | FRQ\$12.5.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | 2023.04.04 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) |
**Note**
We don't recommend that multiple users run IDT from a shared location, such as an NFS directory or a Windows network shared folder. This practice might result in crashes or data corruption. We recommend that you extract the IDT package to a local drive and run the IDT binary on your local workstation.
## Earlier IDT versions for FreeRTOS
Earlier IDT versions
The following earlier versions of IDT for FreeRTOS are also supported.
**Earlier versions of AWS IoT Device Tester for FreeRTOS**
| **AWS IoT Device Tester version** | **Test suite versions** | **Supported FreeRTOS versions** | **Download links** | **Release date** | **Release notes** |
| --- | --- | --- | --- | --- | --- |
| IDT v4.8.1 | FRQ\$12.4.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | 2023.01.23 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) |
| IDT v4.6.0 | FRQ\$12.3.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | 2022.11.16 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) |
| IDT v4.5.11 | FRQ\$12.2.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) | 2022.10.14 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) |
For more information, see [Understand the support policy for AWS IoT Device Tester](idt-support-policy.md).
# Unsupported IDT versions for FreeRTOS
Unsupported IDT versions
This section lists unsupported versions of IDT for FreeRTOS. Unsupported versions do not receive bug fixes or updates. For more information, see [Understand the support policy for AWS IoT Device Tester](idt-support-policy.md).
The following versions of IDT-FreeRTOS are no longer supported.
**Unsupported versions of AWS IoT Device Tester for FreeRTOS**
| **AWS IoT Device Tester version** | **Test suite versions** | **Supported FreeRTOS versions** | **Release date** | **Release notes** |
| --- | --- | --- | --- | --- |
| IDT v4.5.10 | FRQ\$12.1.4 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | 2022.09.02 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.5.9 | FRQ\$12.1.3 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | 2022.08.17 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.5.6 | FRQ\$12.1.2 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | 2022.06.29 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.5.5 | FRQ\$12.1.1 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | 2022.06.06 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.5.5 | FRQ\$12.1.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | 2022.05.31 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.5.4 | FRQ\$12.0.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | 2022.05.09 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.5.2 | FRQ\$11.6.2 | 202107.00 | 2022.01.25 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.0.3 | FRQ\$11.5.1 | 202012.00 | 2021.07.30 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.3.0 | FRQ\$11.6.1 | 202107.00 | 2021.07.26 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.1.0 | FRQ\$11.6.0 | 202107.00 | 2021.07.21 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v4.0.1 | FRQ\$11.4.1 | 202012.00 | 2021.01.19 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v3.4.0 | FRQ\$11.3.0 | 202011.01 | 2020.11.05 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v3.3.0 | FRQ\$11.2.0 | 202007.00 | 2020.09.17 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v3.0.2 | FRQ\$11.0.1 | 202002.00 | | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v1.7.1 | FRQ\$11.0.0 | 202002.00 | | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v1.6.2 | FRQ\$11.0.0 | 202002.00 | | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v1.5.2 | FRQ\$11.0.0 | 201910.00 | | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v1.4.1 | FRQ\$11.0.0 | 201908.00 | | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT v1.3.2 | FRQ\$11.0.0 | 201906.00 | | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) |
| IDT-FreeRTOS v1.2 | FRQ\$11.0.0 | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/idt-unsupported-versions-afr.html) | | Added support for testing FreeRTOS devices with the CMAKE build system. |
| IDT-FreeRTOS v1.1 | FRQ\$11.0.0 | | | |
| IDT-FreeRTOS v1.0 | FRQ\$11.0.0 | | | |
# Download IDT for FreeRTOS
This topic describes the options to download IDT for FreeRTOS. You can either use one of the following software download links or you can follow instructions to programmatically download IDT.
**Important**
As of October 2022, AWS IoT Device Tester for AWS IoT FreeRTOS Qualification (FRQ) 1.0 does not generate signed qualification reports. You cannot qualify new AWS IoT FreeRTOS devices to list in the [AWS Partner Device Catalog ](https://partners.amazonaws.com/qualified-devices)through the [AWS Device Qualification Program](http://aws.amazon.com/partners/programs/dqp/) using IDT FRQ 1.0 versions. While you can't qualify FreeRTOS devices using IDT FRQ 1.0, you can continue to test your FreeRTOS devices with FRQ 1.0. We recommend that you use [IDT FRQ 2.0](https://docs.aws.amazon.com/freertos/latest/userguide/lts-idt-freertos-qualification.html) to qualify and list FreeRTOS devices in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices).
**Topics**
+ [
## Download IDT manually
](#idt-download-options)
+ [
# Download IDT programmatically
](idt-programmatic-download-process.md)
By downloading the software, you agree to the AWS IoT Device Tester License Agreement contained in the download archive.
**Note**
IDT does not support being run by multiple users from a shared location, such as an NFS directory or a Windows network shared folder. We recommend that you extract the IDT package to a local drive and run the IDT binary on your local workstation.
## Download IDT manually
This topic lists supported versions of IDT for FreeRTOS. As a best practice, we recommend that you use the latest version of AWS IoT Device Tester that supports your target version of FreeRTOS. New releases of FreeRTOS might require you to download a new version of AWS IoT Device Tester. You receive a notification when you start a test run if AWS IoT Device Tester is not compatible with the version of FreeRTOS you are using.
See [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md)
# Download IDT programmatically
IDT provides an API operation that you can use to retrieve a URL where you can download IDT programmatically. You can also use this API operation to check if you have the latest version of IDT. This API operation has the following endpoint.
```
https://download.devicetester.iotdevicesecosystem.amazonaws.com/latestidt
```
To call this API operation, you must have permission to perform the **iot-device-tester:LatestIdt** action. Include your AWS signature, with `iot-device-tester` as the Service Name
## API request
HostOs – The operating system of the host machine. Choose from the following options:
+ `mac`
+ `linux`
+ `windows`
TestSuiteType – The type of the test suite. Choose the following option:
`FR` – IDT for FreeRTOS
ProductVersion
(Optional) The version of FreeRTOS. The service returns the latest compatible version of IDT for that version of FreeRTOS. If you don't specify this option, the service returns the latest version of IDT.
## API response
The API response has the following format. The `DownloadURL` includes a zip file.
```
{
"Success": True or False,
"Message": Message,
"LatestBk": {
"Version": The version of the IDT binary,
"TestSuiteVersion": The version of the test suite,
"DownloadURL": The URL to download the IDT Bundle, valid for one hour
}
}
```
## Examples
You can reference the following examples to programmatically download IDT. These examples use credentials that you store in the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. To follow best security practices, don't store your credentials in your code.
**Example**
**Example: Download using cURL version 7.75.0 or later (Mac and Linux)**
If you have cURL version 7.75.0 or later, you can use the `aws-sigv4` flag to sign the API request. This example uses [jq](https://stedolan.github.io/jq/) to parse the download URL from the response.
The `aws-sigv4` flag requires the query parameters of the curl GET request be in the order of **HostOs/ProductVersion/TestSuiteType** or **HostOs/TestSuiteType**. Orders that do not conform, will result in an error of getting mismatched signatures for the Canonical String from the API Gateway.
If the optional parameter **ProductVersion** is included, you must use a supported product version as documented in [Supported versions of AWS IoT Device Tester for FreeRTOS](https://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html).
+ Replace *us-west-2* with your AWS Region. For the list of Region codes, see [Regional endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints).
+ Replace *linux* with your host machine's operating system.
+ Replace *202107.00* with your version of FreeRTOS.
```
url=$(curl --request GET "https://download.devicetester.iotdevicesecosystem.amazonaws.com/latestidt?HostOs=linux&ProductVersion=202107.00&TestSuiteType=FR" \
--user $AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY \
--aws-sigv4 "aws:amz:us-west-2:iot-device-tester" \
| jq -r '.LatestBk["DownloadURL"]')
curl $url --output devicetester.zip
```
**Example**
**Example: Download using an earlier version of cURL (Mac and Linux)**
You can use the following cURL command with an AWS signature that you sign and calculate. For more information about how to sign and calculate an AWS signature, see [Signing AWS API requests](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv.html).
+ Replace *linux* with your host machine's operating system.
+ Replace *Timestamp* with the date and time, such as **20220210T004606Z**.
+ Replace *Date* with the date, such as **20220210**.
+ Replace *AWSRegion* with your AWS Region. For the list of Region codes, see [Regional endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
+ Replace *AWSSignature* with the [AWS signature](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-create-signed-request.html) that you generate.
```
curl --location --request GET 'https://download.devicetester.iotdevicesecosystem.amazonaws.com/latestidt?HostOs=linux&TestSuiteType=FR' \
--header 'X-Amz-Date: Timestamp \
--header 'Authorization: AWS4-HMAC-SHA256 Credential=$AWS_ACCESS_KEY_ID/Date/AWSRegion/iot-device-tester/aws4_request, SignedHeaders=host;x-amz-date, Signature=AWSSignature'
```
**Example**
**Example: Download using a Python script**
This example uses the Python [requests](https://pypi.org/project/requests/) library. This example is adapted from the Python example to [Sign an AWS API request](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv.html) in the *AWS General Reference*.
+ Replace *us-west-2* with your region. For the list of Region codes, see [Regional endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html).
+ Replace *linux* with your host machine's operating system.
```
# Copyright 2010-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# This file is licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License. A copy of the
#License is located at
#
# http://aws.amazon.com/apache2.0/
#
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.
# See: http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
# This version makes a GET request and passes the signature
# in the Authorization header.
import sys, os, base64, datetime, hashlib, hmac
import requests # pip install requests
# ************* REQUEST VALUES *************
method = 'GET'
service = 'iot-device-tester'
host = 'download.devicetester.iotdevicesecosystem.amazonaws.com'
region = 'us-west-2'
endpoint = 'https://download.devicetester.iotdevicesecosystem.amazonaws.com/latestidt'
request_parameters = 'HostOs=linux&TestSuiteType=FR'
# Key derivation functions. See:
# http://docs.aws.amazon.com/general/latest/gr/signature-v4-examples.html#signature-v4-examples-python
def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
def getSignatureKey(key, dateStamp, regionName, serviceName):
kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
kRegion = sign(kDate, regionName)
kService = sign(kRegion, serviceName)
kSigning = sign(kService, 'aws4_request')
return kSigning
# Read AWS access key from env. variables or configuration file. Best practice is NOT
# to embed credentials in code.
access_key = os.environ.get('AWS_ACCESS_KEY_ID')
secret_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
if access_key is None or secret_key is None:
print('No access key is available.')
sys.exit()
# Create a date for headers and the credential string
t = datetime.datetime.utcnow()
amzdate = t.strftime('%Y%m%dT%H%M%SZ')
datestamp = t.strftime('%Y%m%d') # Date w/o time, used in credential scope
# ************* TASK 1: CREATE A CANONICAL REQUEST *************
# http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
# Step 1 is to define the verb (GET, POST, etc.)--already done.
# Step 2: Create canonical URI--the part of the URI from domain to query
# string (use '/' if no path)
canonical_uri = '/latestidt'
# Step 3: Create the canonical query string. In this example (a GET request),
# request parameters are in the query string. Query string values must
# be URL-encoded (space=%20). The parameters must be sorted by name.
# For this example, the query string is pre-formatted in the request_parameters variable.
canonical_querystring = request_parameters
# Step 4: Create the canonical headers and signed headers. Header names
# must be trimmed and lowercase, and sorted in code point order from
# low to high. Note that there is a trailing \n.
canonical_headers = 'host:' + host + '\n' + 'x-amz-date:' + amzdate + '\n'
# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers lists those that you want to be included in the
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = 'host;x-amz-date'
# Step 6: Create payload hash (hash of the request body content). For GET
# requests, the payload is an empty string ("").
payload_hash = hashlib.sha256(('').encode('utf-8')).hexdigest()
# Step 7: Combine elements to create canonical request
canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
# ************* TASK 2: CREATE THE STRING TO SIGN*************
# Match the algorithm to the hashing algorithm you use, either SHA-1 or
# SHA-256 (recommended)
algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request'
string_to_sign = algorithm + '\n' + amzdate + '\n' + credential_scope + '\n' + hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, datestamp, region, service)
# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'), hashlib.sha256).hexdigest()
# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
# The signing information can be either in a query string value or in
# a header named Authorization. This code shows how to use a header.
# Create authorization header and add to request headers
authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' + 'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature
# The request can include any headers, but MUST include "host", "x-amz-date",
# and (for this scenario) "Authorization". "host" and "x-amz-date" must
# be included in the canonical_headers and signed_headers, as noted
# earlier. Order here is not significant.
# Python note: The 'host' header is added automatically by the Python 'requests' library.
headers = {'x-amz-date':amzdate, 'Authorization':authorization_header}
# ************* SEND THE REQUEST *************
request_url = endpoint + '?' + canonical_querystring
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + request_url)
response = requests.get(request_url, headers=headers)
print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % response.status_code)
print(response.text)
download_url = response.json()["LatestBk"]["DownloadURL"]
r = requests.get(download_url)
open('devicetester.zip', 'wb').write(r.content)
```
# IDT with FreeRTOS qualification suite 2.0 (FRQ 2.0)
The FreeRTOS qualification suite 2.0 is an updated version of FreeRTOS qualification suite. We recommend developers to use FRQ 2.0 because it consists of relevant test cases to qualify devices that run FreeRTOS Long Term Support (LTS) libraries.
IDT for FreeRTOS verifies the port of FreeRTOS on your micro-controller, and if it communicates effectively with AWS IoT. Specifically, it verifies the porting layer interfaces with the FreeRTOS libraries, and if FreeRTOS test repositories are implemented correctly. It also performs end-to-end tests with AWS IoT Core. The tests run by IDT for FreeRTOS are defined in the [FreeRTOS GitHub repository](https://github.com/FreeRTOS/FreeRTOS-Libraries-Integration-Tests).
IDT for FreeRTOS run tests as embedded applications that it flashes on the microcontroller device under test. The application binary images include FreeRTOS, the ported FreeRTOS interfaces, and board device drivers. The purpose of the tests is to verify that the ported FreeRTOS interfaces function correctly on top of your device drivers.
IDT for FreeRTOS generates test reports that you can submit to AWS IoT to get your hardware listed on the AWS Partner Device Catalog. For more information, see [AWS Device Qualification Program](https://aws.amazon.com/partners/dqp/).
IDT for FreeRTOS runs on a host computer (Windows, macOS, or Linux) that is connected to the device under testing. IDT configures and orchestrates test cases and aggregates results. It also provides a command line interface to manage running the tests.
In order to test your device, IDT for FreeRTOS creates resources such as AWS IoT things, FreeRTOS groups, Lambda functions. To create these resources, IDT for FreeRTOS uses the AWS credentials configured in the `config.json` to make API calls on your behalf. These resources are provisioned at various times during a test.
When you run IDT for FreeRTOS on your host computer, it performs the following steps:
1. Loads and validates your device and credentials configuration.
1. Performs selected tests with the required local and cloud resources.
1. Cleans up local and cloud resources.
1. Generates tests reports that indicate if your board passed the tests required for qualification.
# Set up the LTS qualification prerequisites
This section describes the prerequisites for testing microcontrollers with AWS IoT Device Tester.
## Prepare for FreeRTOS qualification
**Note**
AWS IoT Device Tester for FreeRTOS strongly recommends using the latest patch release of the most recently FreeRTOS-LTS version.
IDT for FRQ 2.0 is a qualification for FreeRTOS. Before running IDT FRQ 2.0 for qualification, you must complete [Qualifying your board](https://docs.aws.amazon.com/freertos/latest/qualificationguide/freertos-qualification.html) in the *FreeRTOS Qualification Guide*. To port libraries, tests, and setup the `manifest.yml`, see [Porting the FreeRTOS libraries](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-porting.html) in the *FreeRTOS Porting Guide*. FRQ 2.0 contains a different process for qualification. See [Latest changes in qualification](https://docs.aws.amazon.com/freertos/latest/qualificationguide/latest-changes.html) in the *FreeRTOS qualification guide* for details.
The [FreeRTOS-Libraries-Integration-Tests](https://github.com/FreeRTOS/FreeRTOS-Libraries-Integration-Tests) repository must be present for IDT to run. See the [README.md](https://github.com/FreeRTOS/FreeRTOS-Libraries-Integration-Tests/blob/main/README.md) on how to clone and port this repository to your source project. FreeRTOS-Libraries-Integration-Tests must include the `manifest.yml` located in the root of your project, for IDT to run.
**Note**
IDT is dependent on the tests repository's implementation of `UNITY_OUTPUT_CHAR`. The test output logs and the device logs must not interleave with each other. See [ Implementing the library logging macros](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-library-logging-macros.html) section in the *FreeRTOS Porting Guide* for further details.
## Download IDT for FreeRTOS
Every version of FreeRTOS has a corresponding version of IDT for FreeRTOS to perform qualification tests. Download the appropriate version of IDT for FreeRTOS from [ Supported versions of AWS IoT Device Tester for FreeRTOS](https://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html).
Extract IDT for FreeRTOS to a location on the file system where you have read and write permissions. Since Microsoft Windows has a character limit for the path length, extract IDT for FreeRTOS into a root directory such as `C:\` or `D:\`.
**Note**
Multiple users must not run IDT from a shared location, such as an NFS directory or a Windows network shared folder. This will result in crashes or data corruption. We recommend that you extract the IDT package to a local drive.
## Download Git
IDT must have Git installed as a prerequisite to ensure source code integrity.
Follow the instructions in the [GitHub](https://github.com/git-guides/install-git) guide to install Git. To verify the current installed version of Git, enter the command `git --version` at the terminal.
**Warning**
IDT uses Git to align with a directory's status of clean or dirty. If Git is not installed, the `FreeRTOSIntegrity` test groups will either fail, or won't run as expected. If IDT returns an error such as `git executable not found` or `git command not found`, install or re-install Git and try again.
**Topics**
+ [
## Prepare for FreeRTOS qualification
](#idt-preparing-qualification)
+ [
## Download IDT for FreeRTOS
](#idt-download-dev-tester-afr)
+ [
## Download Git
](#idt-download-git)
+ [
## Create an AWS account
](#lts-config-aws-account)
+ [
## AWS IoT Device Tester managed policy
](#managed-policy)
+ [
## (Optional) Install the AWS Command Line Interface
](#install-cli)
## Create an AWS account
**Note**
The full IDT qualification suite is supported only in the following AWS Regions
US East (N. Virginia)
US West (Oregon)
Asia Pacific (Tokyo)
Europe (Ireland)
In order to test your device, IDT for FreeRTOS creates resources like AWS IoT things, FreeRTOS groups and Lambda functions. To create those resources, IDT for FreeRTOS requires you to create and configure an AWS account, and an IAM policy that grants IDT for FreeRTOS permission to access resources on your behalf while running tests.
The following steps are to create and configure your AWS account.
1. If you already have an AWS account, skip to the next step. Else create an [AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).
1. Follow the steps in [ Creating IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html). Do not add permissions or policies at this time.
1. To run OTA qualification tests, go to Step 4. Else go to Step 5.
1. Attach the OTA IAM permissions inline policy to your IAM role.
1.
**Important**
The following policy template grants IDT permission to create roles, create policies, and attach policies to roles. IDT for FreeRTOS uses these permissions for tests that create roles. Although the policy template doesn't provide administrator privileges to the user, the permissions can be used to gain administrator access to your AWS account.
1. Follow the steps below to attach the necessary permissions to your IAM role:
1. On the **Permissions** page, choose **Add permissions**.
1. Choose **Create inline policy**.
1. Choose the **JSON** tab and copy the following permissions in to the **JSON** text box. Use the template under **Most Regions** if you are not in the China region. If you are in the China region, use the template under **Beijing and Ningxia Regions**.
------
#### [ Most Regions ]
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iotdeviceadvisor:*",
"Resource": [
"arn:aws:iotdeviceadvisor:*:*:suiterun/*/*",
"arn:aws:iotdeviceadvisor:*:*:suitedefinition/*"
]
},
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/idt*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "iotdeviceadvisor.amazonaws.com"
}
}
},
{
"Effect": "Allow",
"Action": [
"execute-api:Invoke*",
"iam:ListRoles",
"iot:Connect",
"iot:CreateJob",
"iot:DeleteJob",
"iot:DescribeCertificate",
"iot:DescribeEndpoint",
"iot:DescribeJobExecution",
"iot:DescribeJob",
"iot:DescribeThing",
"iot:GetPolicy",
"iot:ListAttachedPolicies",
"iot:ListCertificates",
"iot:ListPrincipalPolicies",
"iot:ListThingPrincipals",
"iot:ListThings",
"iot:Publish",
"iot:UpdateThingShadow",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:PutRetentionPolicy"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "iotdeviceadvisor:*",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": "logs:DeleteLogGroup",
"Resource": "arn:aws:logs:*:*:log-group:/aws/iot/deviceadvisor/*"
},
{
"Effect": "Allow",
"Action": "logs:GetLogEvents",
"Resource": "arn:aws:logs:*:*:log-group:/aws/iot/deviceadvisor/*:log-stream:*"
},
{
"Effect": "Allow",
"Action": [
"iam:CreatePolicy",
"iam:DetachRolePolicy",
"iam:DeleteRolePolicy",
"iam:DeletePolicy",
"iam:CreateRole",
"iam:DeleteRole",
"iam:AttachRolePolicy"
],
"Resource": [
"arn:aws:iam::*:policy/idt*",
"arn:aws:iam::*:role/idt*"
]
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameters"
],
"Resource": [
"arn:aws:ssm:*::parameter/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:RunInstances",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteTags"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateKeyPair",
"ec2:DeleteKeyPair"
],
"Resource": [
"arn:aws:ec2:*:*:key-pair/idt-ec2-ssh-key-*"
]
},
{
"Effect": "Allow",
"Condition": {
"StringEqualsIgnoreCase": {
"aws:ResourceTag/Owner": "IoTDeviceTester"
}
},
"Action": [
"ec2:TerminateInstances",
"ec2:DeleteSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": [
"*"
]
}
]
}
```
------
------
#### [ Beijing and Ningxia Regions ]
The following policy template can be used in the Beijing and Ningxia Regions.
------
1. When you're finished, choose **Review policy**.
1. Enter **IDTFreeRTOSIAMPermissions** as the policy name.
1. Choose **Create policy**.
1. Attach **AWSIoTDeviceTesterForFreeRTOSFullAccess** to your IAM role.
1. To attach the necessary permissions to your IAM role:
1. On the **Permissions** page, choose **Add permissions**.
1. Choose **Attach policies**.
1. Search for the **AWSIoTDeviceTesterForFreeRTOSFullAccess** policy. Check the box.
1. Choose **Add permissions**.
1. Export credentials for IDT. See [Getting IAM role credentials for CLI access](https://docs.aws.amazon.com/singlesignon/latest/userguide/howtogetcredentials.html) for details.
## AWS IoT Device Tester managed policy
The `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy contains the following AWS IoT Device Tester permissions for version checking, auto update features, and collection of metrics.
+ `iot-device-tester:SupportedVersion`
Grants AWS IoT Device Tester permission to fetch the list of supported products, test suites and IDT versions.
+ `iot-device-tester:LatestIdt`
Grants AWS IoT Device Tester permission to fetch the latest IDT version available for download.
+ `iot-device-tester:CheckVersion`
Grants AWS IoT Device Tester permission to check version compatibility for IDT, test suites and products.
+ `iot-device-tester:DownloadTestSuite`
Grants AWS IoT Device Tester permission to download test suite updates.
+ `iot-device-tester:SendMetrics`
Grants AWS permission to collect metrics about AWS IoT Device Tester internal usage.
## (Optional) Install the AWS Command Line Interface
You might prefer to use the AWS CLI to perform some operations. If you don't have the AWS CLI installed, follow the instructions at [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/installing.html).
Configure the AWS CLI for the AWS Region you want to use by running **aws configure** from a command line. For information about the AWS Regions that support IDT for FreeRTOS, see [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#amazon-freertos-ota-control). For more information about **aws configure** see [ Quick configuration with **aws configure**](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config).
# First test of your microcontroller board
You can use IDT for FreeRTOS to test your implementation of the FreeRTOS libraries. After you have ported the FreeRTOS libraries for your board’s device drivers, use AWS IoT Device Tester to run the qualification tests on your microcontroller board.
## Add library porting layers
To port FreeRTOS for your device, see the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/porting-guide.html). When implementing the FreeRTOS tests repository and porting the FreeRTOS layers, you must provide a `manifest.yml` with paths to each library, including the tests repository. This file will be in the root directory of your source code. See [ manifest file instructions](https://docs.aws.amazon.com/freertos/latest/qualificationguide/afq-checklist-manifest-instr.html) for details.
## Configure your AWS credentials for AWS IoT Device Tester to communicate with the AWS Cloud
You need to configure your AWS credentials for AWS IoT Device Tester to communicate with the AWS Cloud. For more information, see [Set up AWS Credentials and Region for Development](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html). Valid AWS credentials are specified in the `devicetester_extract_location/devicetester_freertos_[win|mac|linux]/configs/config.json` configuration file.
```
"auth": {
"method": "environment"
}
"auth": {
"method": "file",
"credentials": {
"profile": ""
}
}
```
The `auth` attribute of the `config.json` file has a method field that controls AWS authentication, and can be declared as either file or environment. Setting the field to environment pulls your AWS credentials from your host machine’s environment variables. Setting the field to file imports a specified profile from the `.aws/credentials` config file.
**Topics**
+ [
## Add library porting layers
](#lts-add-port-layer)
+ [
## Configure your AWS credentials for AWS IoT Device Tester to communicate with the AWS Cloud
](#lts-cfg-aws-afr)
+ [
# Create a device pool in IDT for FreeRTOS
](lts-cfg-dt-dp.md)
+ [
# Configure build, flash, and test settings
](lts-cfg-dt-ud.md)
# Create a device pool in IDT for FreeRTOS
Devices to be tested are organized in device pools. Each device pool consists of one or more identical devices. You can configure IDT for FreeRTOS to test a single device, or multiple devices in a pool. To accelerate the qualification process, IDT for FreeRTOS can test devices with the same specifications in parallel. It uses a round-robin method to execute a different test group on each device in a device pool.
The `device.json` file has an array in its top level. Each array attribute is a new device pool. Each device pool has a devices array attribute, which has multiple devices declared. In the template, there is a device pool and only one device in that device pool. You can add one or more devices to a device pool by editing the `devices` section of the `device.json` template in the `configs` folder.
**Note**
All devices in the same pool must be of the same technical specification and SKU. To enable parallel builds of the source code for different test groups, IDT for FreeRTOS copies the source code to a results folder inside the IDT for FreeRTOS extracted folder. You must reference the source code path in your build or flash command using the `testdata.sourcePath` variable. IDT for FreeRTOS replaces this variable with a temporary path of the copied source code. For more information, see [IDT for FreeRTOS variables](lts-dt-vars.md).
The following is an example `device.json` file was used to create a device pool with multiple devices.
```
[
{
"id": "pool-id",
"sku": "sku",
"features": [
{
"name": "Wifi",
"value": "Yes | No"
},
{
"name": "Cellular",
"value": "Yes | No"
},
{
"name": "BLE",
"value": "Yes | No"
},
{
"name": "PKCS11",
"value": "RSA | ECC | Both"
},
{
"name": "OTA",
"value": "Yes | No",
"configs": [
{
"name": "OTADataPlaneProtocol",
"value": "MQTT | HTTP | None"
}
]
},
{
"name": "KeyProvisioning",
"value": "Onboard | Import | Both | No"
}
],
"devices": [
{
"id": "device-id",
"connectivity": {
"protocol": "uart",
"serialPort": "/dev/tty*"
},
"secureElementConfig" : {
"publicKeyAsciiHexFilePath": "absolute-path-to/public-key-txt-file: contains-the-hex-bytes-public-key-extracted-from-onboard-private-key",
"publiDeviceCertificateArn": "arn:partition:iot:region:account-id:resourcetype:resource:qualifier",
"secureElementSerialNumber": "secure-element-serialNo-value",
"preProvisioned" : "Yes | No",
"pkcs11JITPCodeVerifyRootCertSupport": "Yes | No"
},
"identifiers": [
{
"name": "serialNo",
"value": "serialNo-value"
}
]
}
]
}
]
```
The following attributes are used in the `device.json` file:
** `id` **
A user-defined alphanumeric ID that uniquely identifies a pool of devices. Devices that belong to a pool must be of the same type. When a suite of tests is running, devices in the pool are used to parallelize the workload.
** `sku` **
An alphanumeric value that uniquely identifies the board you are testing. The SKU is used to track qualified boards.
If you want to list your board in AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process.
** `features` **
An array that contains the device's supported features. AWS IoT Device Tester uses this information to select the qualification tests to run.
Supported values are:
** `Wifi` **
Indicates if your board has Wi-Fi capabilities.
** `Cellular` **
Indicates if your board has cellular capabilities.
** `PKCS11` **
Indicates the public key cryptography algorithm that the board supports. PKCS11 is required for qualification. Supported values are `ECC`, `RSA`, and `Both`. `Both` indicates the board supports both `ECC` and `RSA`.
** `KeyProvisioning` **
Indicates the method of writing a trusted X.509 client certificate onto your board.
Valid values are `Import`, `Onboard`, `Both` and `No`. `Onboard`, `Both`, or `No` key provisioning is required for qualification. `Import` alone is not a valid option for qualification.
+ Use `Import` only if your board allows the import of private keys. Selecting `Import` is not a valid configuration for qualification and should be used only for testing purposes, specifically with PKCS11 test cases. `Onboard`, `Both` or `No` is required for qualification.
+ Use `Onboard` if your board supports on-board private keys (for example, if your device has a secure element, or if you prefer to generate your own device key pair and certificate). Make sure you add a `secureElementConfig` element in each of the device sections and put the absolute path to the public key file in the `publicKeyAsciiHexFilePath` field.
+ Use `Both` if your board supports both importing private keys and on-board key generation for key provisioning.
+ Use `No` if your board doesn't support key provisioning. `No`is only a valid option when your device is also pre-provisioned.
** `OTA` **
Indicates if your board supports over-the-air (OTA) update functionality. The `OtaDataPlaneProtocol` attribute indicates which OTA dataplane protocol the device supports. OTA with either HTTP or MQTT dataplane protocol is required for qualification. To skip running OTA tests while testing, set the OTA feature to `No` and the `OtaDataPlaneProtocol` attribute to `None`. This will not be a qualification run.
** `BLE` **
Indicates if your board supports Bluetooth Low Energy (BLE).
** `devices.id` **
A user-defined unique identifier for the device being tested.
** `devices.connectivity.serialPort` **
The serial port of the host computer used to connect to the devices being tested.
** `devices.secureElementConfig.PublicKeyAsciiHexFilePath` **
Required if your board is NOT `pre-provisioned` or `PublicDeviceCertificateArn` is not provided. Since `Onboard`is a required type of Key Provisioning, this field is currently required for the FullTransportInterfaceTLS test group. If your device is `pre-provisioned`, `PublicKeyAsciiHexFilePath` is optional and need not be included.
The following block is an absolute path to the file that contains the hex bytes public key extracted from `Onboard` private key.
```
3059 3013 0607 2a86 48ce 3d02 0106 082a
8648 ce3d 0301 0703 4200 04cd 6569 ceb8
1bb9 1e72 339f e8cf 60ef 0f9f b473 33ac
6f19 1813 6999 3fa0 c293 5fae 08f1 1ad0
41b7 345c e746 1046 228e 5a5f d787 d571
dcb2 4e8d 75b3 2586 e2cc 0c
```
If your public key is in .der format, you can hex encode the public key directly to generate the hex file.
To generate the hex file from a .der public key, enter the following **xxd** command:
```
xxd -p pubkey.der > outFile
```
If your public key is in .pem format, you can extract the base64 encoded headers and footers and decode it into binary format. Then, you hex encode the binary string to generate the hex file.
To generate a hex file for a .pem public key, do the following:
1. Run the following **base64** command to remove the base64 header and footer from the public key. The decoded key, named `base64key`, is then output to the file `pubkey.der`:
```
base64 —decode base64key > pubkey.der
```
1. Run the following **xxd** command to convert `pubkey.der` to hex format. The resulting key is saved as `outFile`
```
xxd -p pubkey.der > outFile
```
** `devices.secureElementConfig.PublicDeviceCertificateArn` **
The ARN of the certificate from your secure element that is uploaded to AWS IoT Core. For information about uploading your certificate to AWS IoT Core, see [X.509 client certificates](https://docs.aws.amazon.com/iot/latest/developerguide/x509-client-certs.html) in the *AWS IoT Developer Guide*.
** `devices.secureElementConfig.SecureElementSerialNumber` **
(Optional) The serial number of the secure element. The serial number is optionally used to create device certificates for JITR key provisioning.
** `devices.secureElementConfig.preProvisioned` **
(Optional) Set to "Yes" if the device has a pre-provisioned secure element with locked-down credentials, that cannot import, create, or destroy objects. If this attribute is set to **Yes**, you must provide the corresponding pkcs11 labels.
** `devices.secureElementConfig.pkcs11JITPCodeVerifyRootCertSupport` **
(Optional) Set to **Yes** if the device’s corePKCS11 implementation supports storage for JITP. This will enable the JITP `codeverify` test when testing core PKCS 11, and requires code verification key, JITP certificate, and root certificate PKCS 11 labels to be provided.
** `identifiers` **
(Optional) An array of arbitrary name-value pairs. You can use these values in the build and flash commands described in the next section.
# Configure build, flash, and test settings
IDT for FreeRTOS builds and flashes tests on to your board automatically. To enable this, you must configure IDT to run the build and flash commands for your hardware. The build and flash command settings are configured in the `userdata.json` template file located in the `config` folder.
# Configure settings for testing devices
Build, flash, and test settings are made in the `configs/userdata.json` file. The following JSON example shows how you can configure IDT for FreeRTOS to test multiple devices:
```
{
"sourcePath": "",
"retainModifiedSourceDirectories": true | false,
"freeRTOSVersion": "",
"freeRTOSTestParamConfigPath": "{{testData.sourcePath}}/path/from/source/path/to/test_param_config.h",
"freeRTOSTestExecutionConfigPath": "{{testData.sourcePath}}/path/from/source/path/to/test_execution_config.h",
"buildTool": {
"name": "your-build-tool-name",
"version": "your-build-tool-version",
"command": [
" -any-additional-flags {{testData.sourcePath}}"
]
},
"flashTool": {
"name": "your-flash-tool-name",
"version": "your-flash-tool-version",
"command": [
" -any-additional-flags {{testData.sourcePath}} -any-additional-flags"
]
},
"testStartDelayms": 0,
"echoServerConfiguration": {
"keyGenerationMethod": "EC | RSA",
"serverPort": 9000
},
"otaConfiguration": {
"otaE2EFirmwarePath": "{{testData.sourcePath}}/relative-path-to/ota-image-generated-in-build-process",
"otaPALCertificatePath": "/path/to/ota/pal/certificate/on/device",
"deviceFirmwarePath" : "/path/to/firmware/image/name/on/device",
"codeSigningConfiguration": {
"signingMethod": "AWS | Custom",
"signerHashingAlgorithm": "SHA1 | SHA256",
"signerSigningAlgorithm": "RSA | ECDSA",
"signerCertificate": "arn:partition:service:region:account-id:resource:qualifier | /absolute-path-to/signer-certificate-file",
"untrustedSignerCertificate": "arn:partition:service:region:account-id:resourcetype:resource:qualifier",
"signerCertificateFileName": "signerCertificate-file-name",
"compileSignerCertificate": true | false,
// ***********Use signerPlatform if you choose AWS for signingMethod***************
"signerPlatform": "AmazonFreeRTOS-Default | AmazonFreeRTOS-TI-CC3220SF"
]
}
},
**********
This section is used for PKCS #11 labels of private key, public key, device certificate, code verification key, JITP certificate, and root certificate.
When configuring PKCS11, you set up labels and you must provide the labels of the device certificate, public key,
and private key for the key generation type (EC or RSA) it was created with. If your device supports PKCS11 storage of JITP certificate,
code verification key, and root certificate, set 'pkcs11JITPCodeVerifyRootCertSupport' to 'Yes' in device.json and provide the corresponding labels.
**********
"pkcs11LabelConfiguration":{
"pkcs11LabelDevicePrivateKeyForTLS": "",
"pkcs11LabelDevicePublicKeyForTLS": "",
"pkcs11LabelDeviceCertificateForTLS": "",
"pkcs11LabelPreProvisionedECDevicePrivateKeyForTLS": "",
"pkcs11LabelPreProvisionedECDevicePublicKeyForTLS": "",
"pkcs11LabelPreProvisionedECDeviceCertificateForTLS": "",
"pkcs11LabelPreProvisionedRSADevicePrivateKeyForTLS": "",
"pkcs11LabelPreProvisionedRSADevicePublicKeyForTLS": "",
"pkcs11LabelPreProvisionedRSADeviceCertificateForTLS": "",
"pkcs11LabelCodeVerifyKey": "",
"pkcs11LabelJITPCertificate": "",
"pkcs11LabelRootCertificate": ""
}
}
```
The following lists the attributes used in `userdata.json`:
** `sourcePath` **
The path to the root of the ported FreeRTOS source code.
** `retainModifiedSourceDirectories` **
(Optional) Checks if to retain the modified source directories used during building and flashing for debugging purposes. If set to `true`, the modified source directories are named retainedSrc and found within the results log folders in each test group run. If not included, the field defaults to `false`.
** `freeRTOSTestParamConfigPath` **
The path to `test_param_config.h` file for FreeRTOS-Libraries-Integration-Tests integration. This file must use the `{{testData.sourcePath}}` placeholder variable to make it relative to the source code root. AWS IoT Device Tester uses the parameters in this file to configure the tests.
** `freeRTOSTestExecutionConfigPath` **
The path to `test_execution_config.h` file for FreeRTOS-Libraries-Integration-Tests integration. This file must use the `{{testData.sourcePath}}` placeholder variable to make it relative to the repository root. AWS IoT Device Tester uses this file to control which tests must run.
** `freeRTOSVersion` **
The version of FreeRTOS including the patch version used in your implementation. See [Supported versions of AWS IoT Device Tester for FreeRTOS](https://docs.aws.amazon.com/freertos/latest/userguide/dev-test-versions-afr.html) for the FreeRTOS versions compatible with AWS IoT Device Tester for FreeRTOS.
** `buildTool` **
The command to build your source code. All references to the source code path in the build command must be replaced by the AWS IoT Device Tester variable `{{testData.sourcePath}}`. Use the `{{config.idtRootPath}}` placeholder to reference a build script relative to the AWS IoT Device Tester root path.
** `flashTool` **
The command to flash an image to your device. All references to the source code path in the flash command must be replaced by the AWS IoT Device Tester variable `{{testData.sourcePath}}`. Use the `{{config.idtRootPath}}` placeholder to reference a flash script relative to the AWS IoT Device Tester root path.
The new integration tests structure with FRQ 2.0 doesn't require path variables such as `{{enableTests}}` and `{{buildImageName}}`. The OTA End to End tests are run with the config templates provided in the [ FreeRTOS-Libraries-Integration-Tests](https://github.com/FreeRTOS/FreeRTOS-Libraries-Integration-Tests/blob/main/config_template/) GitHub repository. If the files in the GitHub repository are present in your parent source project, the source code isn't changed between tests. If a different build image for OTA End to End is needed, you must build this image in the build script and specify it in the `userdata.json` file specified under `otaConfiguration`.
** `testStartDelayms` **
Specifies how many milliseconds the FreeRTOS test runner will wait before starting to run tests. This can be useful if the device under test begins to output important test information before IDT has a chance to connect and start logging due to network or other latency issues. This value is applicable to FreeRTOS test groups only, and not to other test groups that do not utilize the FreeRTOS test runner, such as the OTA tests. If you receive an error related to **expected 10 but received 5**, this field should be set to 5000.
** `echoServerConfiguration` **
The configuration to setup the echo server for the TLS test. This field is required.
** `keyGenerationMethod` **
The echo server is configured with this option. The options are EC, or RSA.
** `serverPort` **
The port number on which the echo server runs.
** `otaConfiguration` **
The configuration for OTA PAL and OTA E2E tests. This field is required.
**`otaE2EFirmwarePath`**
Path to the OTA bin image that IDT uses for the OTA End to End tests.
** `otaPALCertificatePath` **
The path to the certificate for OTA PAL test on device. This is used to verify the signature. For example, **ecdsa-sha256-signer.crt.pem**.
** `deviceFirmwarePath` **
The path to the hard coded name for the firmware image to boot. If your device does NOT use the file system for firmware boot, specify this field as `'NA'`. If your device uses the file system for firmware boot, specify the path or name to the firmware boot image.
** `codeSigningConfiguration` **
** `signingMethod` **
The code signing method. Possible values are AWS or Custom.
For the Beijing and Ningxia Regions, use Custom. AWS code signing is not supported in that region.
** `signerHashingAlgorithm` **
The hashing algorithm supported on the device. Possible values are `SHA1` or `SHA256`.
** `signerSigningAlgorithm` **
The signing algorithm supported on the device. Possible values are `RSA` or `ECDSA`.
** `signerCertificate` **
The trusted certificate used for OTA. For the AWS code signing method, use the Amazon Resource Name (ARN) for the trusted certificate uploaded to the AWS Certificate Manager. For the Custom code signing method, use the absolute path to the signer certificate file. For information about creating a trusted certificate, see [ Create a code-signing certificate](https://docs.aws.amazon.com/freertos/latest/userguide/ota-code-sign-cert.html).
** `untrustedSignerCertificate` **
The ARN or filepath for a second certificate used in some OTA tests as an untrusted certificate. For information about creating a certificate, see [ Create a code-signing certificate](https://docs.aws.amazon.com//freertos/latest/userguide/ota-code-sign-cert.html).
** `signerCertificateFileName` **
The file name of the code signing certificate on the device. This value must match the file name that you provided when you ran the `aws acm import-certificate` command.
** `compileSignerCertificate` **
Boolean value that determines the status of the signature verification certificate. Valid values are `true` and `false`.
Set this value to **true** if the code signer signature verification certificate is not provisioned or flashed. It must be compiled into the project. AWS IoT Device Tester fetches the trusted certificate and compiles it into `aws_codesigner_certificate.h`.
** `signerPlatform` **
The signing and hashing algorithm that AWS Code Signer uses while creating the OTA update job. Currently, the possible values for this field are `AmazonFreeRTOS-TI-CC3220SF` and `AmazonFreeRTOS-Default`.
+ Choose `AmazonFreeRTOS-TI-CC3220SF` if `SHA1` and `RSA`.
+ Choose `AmazonFreeRTOS-Default` if `SHA256` and `ECDSA`.
+ If you need `SHA256` \$1 `RSA` or `SHA1` \$1 `ECDSA` for your configuration, contact us for further support.
+ Configure `signCommand` if you chose `Custom` for `signingMethod`.
** `signCommand` **
Two placeholders `{{inputImageFilePath}}` and `{{outputSignatureFilePath}}` are required in the command. `{{inputImageFilePath}}` is the file path of the image built by IDT to be signed. `{{outputSignatureFilePath}} `is the file path of the signature which will be generated by the script.
** `pkcs11LabelConfiguration` **
PKCS11 label configuration requires at least one set of labels of device certificate label, public key label, and private key label to run the PKCS11 test groups. The required PKCS11 labels are based on your device configuration in the `device.json` file. If pre-provisioned is set to **Yes** in `device.json`, then the required labels must be one of the below depending on what's chosen for the PKCS11 feature.
+ `PreProvisionedEC`
+ `PreProvisionedRSA`
If pre-provisioned is set to **No** in `device.json`, then the required labels are:
+ `pkcs11LabelDevicePrivateKeyForTLS`
+ `pkcs11LabelDevicePublicKeyForTLS`
+ `pkcs11LabelDeviceCertificateForTLS`
The following three labels are required only if you select **Yes** for `pkcs11JITPCodeVerifyRootCertSupport` in your `device.json` file.
+ `pkcs11LabelCodeVerifyKey`
+ `pkcs11LabelRootCertificate`
+ `pkcs11LabelJITPCertificate`
The values for these fields should match the values defined in the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-porting-pkcs.html).
** `pkcs11LabelDevicePrivateKeyForTLS` **
(Optional) This label is used for the PKCS \$111 label of the private key. For devices with onboard and import support of key provisioning, this label is used for testing. This label may be different than the one defined for the pre-provisioned case. If you have key provisioning set to **No** and pre-provisioned set to **Yes**, in `device.json`, this will be undefined.
** `pkcs11LabelDevicePublicKeyForTLS` **
(Optional) This label is used for the PKCS \$111 label of the public key. For devices with onboard and import support of key provisioning, this label is used for testing. This label may be different than the one defined for pre-provisioned case. If you have key provisioning set to **No** and pre-provisioned set to **Yes**, in `device.json`, this will be undefined.
** `pkcs11LabelDeviceCertificateForTLS` **
(Optional) This label is used for the PKCS \$111 label of the device certificate. For devices with onboard and import support of key provisioning, this label will be used for testing. This label may be different than the one defined for pre-provisioned case. If you have key provisioning set to **No** and pre-provisioned set to **Yes**, in `device.json`, this will be undefined.
** `pkcs11LabelPreProvisionedECDevicePrivateKeyForTLS` **
(Optional) This label is used for the PKCS \$111 label of the private key. For devices with secure elements or hardware limitations, this will have a different label to preserve AWS IoT credentials. If your device supports pre-provisioning with an EC key, provide this label. When preProvisioned is set to **Yes** in `device.json`, this label, `pkcs11LabelPreProvisionedRSADevicePrivateKeyForTLS`, or both must be provided. This label may be different than the one defined for onboard and import cases.
** `pkcs11LabelPreProvisionedECDevicePublicKeyForTLS` **
(Optional) This label is used for the PKCS \$111 label of the public key. For devices with secure elements or hardware limitations, this will have a different label to preserve AWS IoT credentials. If your device supports pre-provisioning with an EC key, provide this label. When preProvisioned is set to **Yes** in `device.json`, this label, `pkcs11LabelPreProvisionedRSADevicePublicKeyForTLS`, or both must be provided. This label may be different than the one defined for onboard and import cases.
** `pkcs11LabelPreProvisionedECDeviceCertificateForTLS` **
(Optional) This label is used for the PKCS \$111 label of the device certificate. For devices with secure elements or hardware limitations, this will have a different label to preserve AWS IoT credentials. If your device supports pre-provisioning with an EC key, provide this label. When preProvisioned is set to **Yes** in `device.json`, this label, `pkcs11LabelPreProvisionedRSADeviceCertificateForTLS`, or both must be provided. This label may be different than the one defined for onboard and import cases.
** `pkcs11LabelPreProvisionedRSADevicePrivateKeyForTLS` **
(Optional) This label is used for the PKCS \$111 label of the private key. For devices with secure elements or hardware limitations, this will have a different label to preserve AWS IoT credentials. If your device supports pre-provisioning with an RSA key, provide this label. When preProvisioned is set to **Yes** in `device.json`, this label, `pkcs11LabelPreProvisionedECDevicePrivateKeyForTLS`, or both must be provided.
** `pkcs11LabelPreProvisionedRSADevicePublicKeyForTLS` **
(Optional) This label is used for the PKCS \$111 label of the public key. For devices with secure elements or hardware limitations, this will have a different label to preserve AWS IoT credentials. If your device supports pre-provisioning with an RSA key, provide this label. When preProvisioned is set to **Yes** in `device.json`, this label, `pkcs11LabelPreProvisionedECDevicePublicKeyForTLS`, or both must be provided.
** `pkcs11LabelPreProvisionedRSADeviceCertificateForTLS` **
(Optional) This label is used for the PKCS \$111 label of the device certificate. For devices with secure elements or hardware limitations, this will have a different label to preserve AWS IoT credentials. If your device supports pre-provisioning with an RSA key, provide this label. When preProvisioned is set to **Yes** in `device.json`, this label, `pkcs11LabelPreProvisionedECDeviceCertificateForTLS`, or both must be provided.
** `pkcs11LabelCodeVerifyKey` **
(Optional) This label is used for the PKCS \$111 label of the code verification key. If your device has PKCS \$111 storage support of the JITP certificate, code verification key, and root certificate, provide this label. When `pkcs11JITPCodeVerifyRootCertSupport` in `device.json` is set to **Yes**, this label must be provided.
** `pkcs11LabelJITPCertificate` **
(Optional) This label is used for the PKCS \$111 label of the JITP certificate. If your device has PKCS \$111 storage support of the JITP certificate, code verification key, and root certificate, provide this label. When `pkcs11JITPCodeVerifyRootCertSupport` in `device.json` is set to **Yes**, this label must be provided.
# IDT for FreeRTOS variables
The commands to build your code and flash the device might require connectivity or other information about your devices to run successfully. AWS IoT Device Tester allows you to reference device information in flash and build commands using [JsonPath](https://goessner.net/articles/JsonPath/). By using simple JsonPath expressions, you can fetch the required information specified in your `device.json` file.
## Path variables
IDT for FreeRTOS defines the following path variables that can be used in command lines and configuration files:
** `{{testData.sourcePath}}` **
Expands to the source code path. If you use this variable, it must be used in both the flash and build commands.
** `{{device.connectivity.serialPort}}` **
Expands to the serial port.
** `{{device.identifiers[?(@.name == 'serialNo')].value[0]}}` **
Expands to the serial number of your device.
** `{{config.idtRootPath}}` **
Expands to the AWS IoT Device Tester root path.
# UI for IDT for FreeRTOS qualification suite 2.0 (FRQ 2.0)
Use the IDT UI to run the FreeRTOS qualification suite
AWS IoT Device Tester for FreeRTOS (IDT for FreeRTOS) includes a web-based user interface (UI) where you can interact with the IDT command line application and related configuration files. You use the IDT for FreeRTOS UI to create a new configuration, or modify an existing one, for your device. You can also use the UI to call the IDT application and run the FreeRTOS tests against your device.
For information about how to use the command line to run qualification tests, see [First test of your microcontroller board](lts-qual-steps.md).
This section describes the prerequisites for the IDT for FreeRTOS UI and how to run qualifications tests from the UI.
**Topics**
+ [
## Set up the IDT prerequisites
](#lts-dev-tester-ui-prereqs)
+ [
# Configure AWS credentials to use the IDT UI
](lts-configure-aws-credentials.md)
+ [
# Open the IDT for FreeRTOS UI
](lts-open-idt-ui.md)
+ [
# Create a new configuration
](lts-create-new-configuration.md)
+ [
# Modify an existing configuration
](lts-modify-existing-configuration.md)
+ [
# Run qualification tests
](lts-run-tests-from-ui.md)
## Set up the IDT prerequisites
To run tests through the AWS IoT Device Tester (IDT) for FreeRTOS UI, you must complete the prerequisites on the [Set up the LTS qualification prerequisites](lts-idt-dev-tester-prereqs.md) page for IDT FreeRTOS Qualification (FRQ) 2.x.
# Configure AWS credentials to use the IDT UI
You must configure your IAM user credentials for the AWS user you created in [Create an AWS account](lts-idt-dev-tester-prereqs.md#lts-config-aws-account). You can specify your credentials in one of two ways:
+ In a credentials file
+ As environment variables
## Configure AWS credentials with a credentials file
IDT uses the same credentials file as the AWS CLI. For more information, see [Configuration and credential files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html).
The location of the credentials file varies based on the operating system you use:
+ **macOS and Linux** – `~/.aws/credentials`
+ **Windows** – `C:\Users\UserName\.aws\credentials`
Add your AWS credentials to the `credentials` file in the following format:
```
[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key
```
**Note**
If you don't use the `default` AWS profile, you must specify the profile name in the IDT for FreeRTOS UI. For more information about profiles, see [ Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
## Configure AWS credentials with environment variables
Environment variables are variables maintained by the operating system and used by system commands. They're not saved if you close the SSH session. The IDT for FreeRTOS UI uses the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to store your AWS credentials.
To set these variables on Linux, macOS, or Unix, use **export**:
```
export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key
```
To set these variables on Windows, use **set**:
```
set AWS_ACCESS_KEY_ID=your_access_key_id
set AWS_SECRET_ACCESS_KEY=your_secret_access_key
```
# Open the IDT for FreeRTOS UI
This topic explains how to open the IDT for FreeRTOS UI to use the FreeRTOS qualification suite.
**To open the IDT for FreeRTOS UI**
1. Download a supported IDT for FreeRTOS version. Then extract the downloaded archive to a directory that you have read and write permissions for.
1. Navigate to the IDT for FreeRTOS installation directory:
```
cd devicetester-extract-location/bin
```
1. Run the following command to open the IDT for FreeRTOS UI:
------
#### [ Linux ]
```
.devicetester_ui_linux_x86-64
```
------
#### [ Windows ]
```
./devicetester_ui_win_x64-64
```
------
#### [ macOS ]
```
./devicetester_ui_mac_x86-64
```
**Note**
In macOS, to allow your system to run the UI, go to **System Preferences -> Security & Privacy**. When you run the tests, you may need to do this three more times. this
------
The IDT for FreeRTOS UI opens in your default browser. The latest three major versions of the following browsers support the UI:
+ Google Chrome
+ Mozilla Firefox
+ Microsoft Edge
+ Apple Safari for macOS
**Note**
For a better experience, we recommend Google Chrome or Mozilla Firefox to access the IDT for FreeRTOS UI. Microsoft Internet Explorer isn't supported by the UI.
**Important**
You must configure your AWS credentials before you open the UI. If you haven't configured your credentials, close the IDT for FreeRTOS UI browser window, follow the steps in [Configure AWS credentials to use the IDT UI](lts-configure-aws-credentials.md), and then reopen the IDT for FreeRTOS UI.
# Create a new configuration
If you're a first-time user, you must create a new configuration to set up the JSON configuration files that IDT for FreeRTOS requires to run tests. You can then run tests or modify the created configuration.
For examples of the `config.json`, `device.json`, and `userdata.json` files, see [First test of your microcontroller board](lts-qual-steps.md).
**To create a new configuration**
1. In the IDT for FreeRTOS UI, open the navigation menu, and choose **Create new configuration**.
![\[Device Tester for FreeRTOS interface with "Create new configuration" button and information about automated self-testing of microcontrollers.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-create-configuration.png)
1. Follow the configuration wizard to enter the IDT configuration settings used to run qualification tests. The wizard configures the following settings in JSON configuration files located in the `devicetester-extract-location/config` directory.
+ **Device settings** – The device pool settings for the devices to be tested. These settings are configured in the `id` and `sku` fields, and the **devices** block for the device pool in the `config.json` file.
![\[Device Tester for FreeRTOS configuration screen with Identifier and SKU fields for configuring a device pool, device settings options like connectivity method, key provisioning, PKCS #11 settings, device details input fields, and controls to add devices or identifiers.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-device-settings.png)
+ **AWS account settings** – The AWS account information that IDT for FreeRTOS uses to create AWS resources during test runs. These settings are configured in the `config.json` file.
![\[AWS account settings page with fields for account region, credentials location as file or environment, and profile name\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-account-settings.png)
+ **FreeRTOS implementation** – The absolute path to the FreeRTOS repository and ported code, and the FreeRTOS version you want to run IDT FRQ on. The paths to the execution and parameter config header files from the `FreeRTOS-Libraries-Integration-Tests` GitHub repository. The build and flash commands for your hardware that allow IDT to build and flash tests onto your board automatically. These settings are configured in the `userdata.json` file.
![\[FreeRTOS implementation configuration section with repository paths, test execution paths, FreeRTOS version, build tool details, and flash tool settings.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-implementation-settings.png)
+ **PKCS \$111 labels and Echo server** – The [PKCS \$111](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-porting-pkcs.html) labels that correspond to the keys provisioned in your hardware based on the key functionality and key provisioning method. The echo server configuration settings for the Transport Interface tests. These settings are configured in the `userdata.json` and `device.json` files.
![\[PKCS #11 labels and Echo server configurations with input fields for key labels, key generation method, and server port number\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-pkcs11-settings.png)
+ **Over-the-air (OTA) updates** – The settings that control OTA functionality tests. These settings are configured in the `features` block of the `device.json` and `userdata.json` files.
![\[OTA update config options: skip tests, data protocols, firmware paths, PAL cert path, code signing, hash/sign algorithms, trusted/untrusted signer certs, signer cert file, compile signer cert, signer platform.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-ota-settings.png)
1. On the **Review** page, verify your configuration information.
![\[Configuration created dialog for Device Tester for FreeRTOS, showing details about creating a new test configuration with options to edit or run tests.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/gsg-configuration-created.png)
After you finish reviewing your configuration, to run your qualification tests, choose **Run tests**.
# Modify an existing configuration
If you have already set up configuration files for IDT for FreeRTOS, you can use the IDT for FreeRTOS UI to modify your existing configuration. The existing configuration files must be located in the `devicetester-extract-location/config` directory.
**To modify a configuration**
1. In the IDT for FreeRTOS UI, open the navigation menu, and choose **Edit existing configuration**.
The configuration dashboard displays information about your existing configuration settings. If a configuration is incorrect or unavailable, the status for that configuration is `Error validating configuration`.
![\[Configuration screen with device, AWS account, FreeRTOS implementation, PKCS labels and echo server, over-the-air updates, and test run settings sections showing valid status.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/modify-existing-configuration.png)
1. To modify an existing configuration setting, complete the following steps:
1. Choose the name of a configuration setting to open its settings page.
1. Modify the settings, and then choose **Save** to regenerate the corresponding configuration file.
1. To modify the IDT for FreeRTOS test run settings, choose **IDT test run settings** in the edit view:
![\[IDT test run settings dialog with options for test selection, skipping test groups, timeout multiplier, and stopping on first failure.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/idt-testrun-settings.png)
After you finish modifying your configuration, verify that all of your configuration settings pass validation. If the status for each configuration setting is `Valid`, you can run your qualification tests with this configuration.
# Run qualification tests
After you create a configuration for the IDT for FreeRTOS UI you can run your qualification tests.
**To run qualification tests**
1. In the navigation menu, choose **Run tests**.
1. Choose **Start tests** to start the test run. By default, all applicable tests are run for your device configuration. IDT for FreeRTOS generates a qualification report when all tests finish.
![\[Device Tester for FreeRTOS interface showing no tests run yet, with options to create new configuration, edit existing configuration, and run tests.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/idt-run-tests.png)
IDT for FreeRTOS runs the qualification tests. It then displays the test run summary and any errors in the **Test runner** console. After the test run is complete, you can view the test results and logs from the following locations:
+ Test results are located in the `devicetester-extract-location/results/execution-id` directory.
+ Test logs are located in the `devicetester-extract-location/results/execution-id/logs` directory.
For more information about test results and logs, see [View the IDT for FreeRTOSresults](view-results-lts.md) and [View the IDT for FreeRTOSlogs](view-logs-lts.md).
![\[Device Tester for FreeRTOS execution log showing tests passed, test groups, and file paths for logs and reports.\]](http://docs.aws.amazon.com/freertos/latest/userguide/images/idt-results.png)
# Run the FreeRTOS qualification 2.0 suite
Use the AWS IoT Device Tester for FreeRTOS executable to interact with IDT for FreeRTOS. The following command line examples show you how to run the qualification tests for a device pool (a set of identical devices).
------
#### [ IDT v4.5.2 and later ]
```
devicetester_[linux | mac | win] run-suite \
--suite-id suite-id \
--group-id group-id \
--pool-id your-device-pool \
--test-id test-id \
--userdata userdata.json
```
Runs a suite of tests on a pool of devices. The `userdata.json` file must be located in the `devicetester_extract_location/devicetester_freertos_[win|mac|linux]/configs/` directory.
**Note**
If you're running IDT for FreeRTOS on Windows, use forward slashes (/) to specify the path to the `userdata.json` file.
Use the following command to run a specific test group:
```
devicetester_[linux | mac | win] run-suite \
--suite-id FRQ_1.99.0 \
--group-id group-id \
--pool-id pool-id \
--userdata userdata.json
```
The `suite-id` and `pool-id` parameters are optional if you're running a single test suite on a single device pool (that is, you have only one device pool defined in your `device.json` file).
Use the following command to run a specific test case in a test group:
```
devicetester_[linux | mac | win_x86-64] run-suite \
--group-id group-id \
--test-id test-id
```
You can use the `list-test-cases` command to list the test cases in a test group.
**IDT for FreeRTOS command line options**
**group-id**
(Optional) The test groups to run, as a comma-separated list. If not specified, IDT runs all test groups in the test suite.
**pool-id**
(Optional) The device pool to test. This is required if you define multiple device pools in `device.json`. If you only have one device pool, you can omit this option.
**suite-id**
(Optional) The test suite version to run. If not specified, IDT uses the latest version in the tests directory on your system.
**test-id**
(Optional) The tests to run, as a comma-separated list. If specified, `group-id` must specify a single group.
**Example**
```
devicetester_[linux | mac | win_x86-64] run-suite --group-id FreeRTOSVersion --test-id FreeRTOSVersion
```
**h**
Use the help option to learn more about `run-suite` options.
**Example**
**Example**
```
devicetester_[linux | mac | win_x86-64] run-suite -h
```
------
## IDT for FreeRTOS commands
The IDT for FreeRTOS command supports the following operations:
------
#### [ IDT v4.5.2 and later ]
** `help` **
Lists information about the specified command.
** `list-groups` **
Lists the groups in a given suite.
** `list-suites` **
Lists the available suites.
** `list-supported-products` **
Lists the supported products and test suite versions.
** `list-supported-versions` **
Lists the FreeRTOS and test suite versions supported by the current IDT version.
** `list-test-cases` **
Lists the test cases in a specified group.
** `run-suite` **
Runs a suite of tests on a pool of devices.
Use the `--suite-id` option to specify a test suite version, or omit it to use the latest version on your system.
Use the `--test-id` to run an individual test case.
**Example**
```
devicetester_[linux | mac | win_x86-64] run-suite --group-id FreeRTOSVersion --test-id FreeRTOSVersion
```
Starting in IDT v3.0.0, IDT checks online for newer test suites. For more information, see [Test suite versions](idt-test-suite-versions.md).
------
# View the IDT for FreeRTOSresults
While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it writes a test run summary to the console and generates two test reports. These reports can be found in `devicetester-extract-location/results/execution-id/`. Both reports capture the results from the qualification test suite execution.
The `awsiotdevicetester_report.xml` is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements:
+ The IDT for FreeRTOS version.
+ The FreeRTOS version that was tested.
+ The features of FreeRTOS that are supported by the device based on the tests passed.
+ The SKU and the device name specified in the `device.json` file.
+ The features of the device specified in the `device.json` file.
+ The aggregate summary of test case results.
+ A breakdown of test case results by libraries that were tested based on the device features.
The `FRQ_Report.xml` is a report in standard [JUnit XML format](https://llg.cubic.org/docs/junit/). You can integrate it into CI/CD platforms like [Jenkins](https://jenkins.io/), [Bamboo](https://www.atlassian.com/software/bamboo), and so on. The report contains the following elements:
+ An aggregate summary of test case results.
+ A breakdown of test case results by libraries that were tested based on the device features.
# Interpret the IDT for FreeRTOS results
The report section in `awsiotdevicetester_report.xml` or `FRQ_Report.xml` lists the results of the tests that are executed.
The first XML tag `` contains the overall summary of the test execution. For example:
``
**Attributes used in the `` tag**
** `name` **
The name of the test suite.
** `time` **
The time, in seconds, it took to run the qualification suite.
** `tests` **
The number of test cases executed.
** `failures` **
The number of test cases that were run, but did not pass.
** `errors` **
The number of test cases that IDT for FreeRTOS couldn't execute.
** `disabled` **
This attribute is not used and can be ignored.
If there are no test case failures or errors, your device meets the technical requirements to run FreeRTOS and can interoperate with AWS IoT services. If you choose to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence.
In the event of test case failures or errors, you can identify the test case that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag shows the test case result summary for a test group.
``
The format is similar to the `` tag, but with an attribute called `skipped` that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each of the test cases that were executed for a test group. For example:
``
**Attributes used in the `` tag**
** `name` **
The name of the product being tested.
** `version` **
The version of the product being tested.
** `features` **
The features validated. Features marked as `required` are required to submit your board for qualification. The following snippet shows how this appears in the `awsiotdevicetester_report.xml` file.
```
```
Features marked as `optional` are not required for qualification. The following snippets show optional features.
```
```
If there are no test failures or errors for the required features, your device meets the technical requirements to run FreeRTOS and can interoperate with AWS IoT services. If you want to list your device in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices), you can use this report as qualification evidence.
In the event of test failures or errors, you can identify the test that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag show the test result summary for a test group. For example:
```
```
The format is similar to the `` tag, but has a `skipped` attribute that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each executed test for a test group. For example:
```
```
**Attributes used in the `` tag**
** `name` **
The name of the test case.
** `attempts` **
The number of times IDT for FreeRTOS executed the test case.
When a test fails or an error occurs, `` or `` tags are added to the `` tag with information for troubleshooting. For example:
```
Reason for the test case failure
Reason for the test case execution error
```
For more information, see [Troubleshoot errors](dt-afr-troubleshooting.md).
# View the IDT for FreeRTOSlogs
You can find logs that IDT for FreeRTOS generates from test execution in `devicetester-extract-location/results/execution-id/logs`. Two sets of logs are generated:
+ `test_manager.log`
Contains logs generated from IDT for FreeRTOS (for example, logs related configuration and report generation).
+ `test_group_id/test_case_id/test_case_id.log`
The log file for a test case, including output from the device under test. The log file is named according to the test group and test case that was run.
# IDT with FreeRTOS qualification suite 1.0 (FRQ 1.0)
**Important**
As of October 2022, AWS IoT Device Tester for AWS IoT FreeRTOS Qualification (FRQ) 1.0 does not generate signed qualification reports. You cannot qualify new AWS IoT FreeRTOS devices to list in the [AWS Partner Device Catalog ](https://partners.amazonaws.com/qualified-devices)through the [AWS Device Qualification Program](http://aws.amazon.com/partners/programs/dqp/) using IDT FRQ 1.0 versions. While you can't qualify FreeRTOS devices using IDT FRQ 1.0, you can continue to test your FreeRTOS devices with FRQ 1.0. We recommend that you use [IDT FRQ 2.0](https://docs.aws.amazon.com/freertos/latest/userguide/lts-idt-freertos-qualification.html) to qualify and list FreeRTOS devices in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices).
You can use IDT for FreeRTOS qualification to verify that the FreeRTOS operating system works locally on your device and can communicate with AWS IoT. Specifically, it verifies that the porting layer interfaces for the FreeRTOS libraries are implemented correctly. It also performs end-to-end tests with AWS IoT Core. For example, it verifies your board can send and receive MQTT messages and process them correctly. The tests run by IDT for FreeRTOS are defined in the [FreeRTOS GitHub repository](https://github.com/aws/amazon-freertos).
The tests run as embedded applications that are flashed onto your board. The application binary images include FreeRTOS, the semiconductor vendor’s ported FreeRTOS interfaces, and board device drivers. The purpose of the tests is to verify the ported FreeRTOS interfaces function correctly on top of the device drivers.
IDT for FreeRTOS generates test reports that you can submit to AWS IoT to add your hardware to the AWS Partner Device Catalog. For more information, see [AWS Device Qualification Program](https://aws.amazon.com/partners/dqp/).
IDT for FreeRTOS runs on a host computer (Windows, macOS, or Linux) that is connected to the board to be tested. IDT executes test cases and aggregates results. It also provides a command line interface to manage test execution.
In addition to testing devices, IDT for FreeRTOS creates resources (for example, AWS IoT things, FreeRTOS groups, Lambda functions, and so on) to facilitate the qualification process. To create these resources, IDT for FreeRTOS uses the AWS credentials configured in the `config.json` to make API calls on your behalf. These resources are provisioned at various times during a test.
When you run IDT for FreeRTOS on your host computer, it performs the following steps:
1. Loads and validates your device and credentials configuration.
1. Performs selected tests with the required local and cloud resources.
1. Cleans up local and cloud resources.
1. Generates tests reports that indicate if your board passed the tests required for qualification.
**Topics**
+ [
# Setup the 1.0 qualification prerequisites
](dev-tester-prereqs.md)
+ [
# First test of your microcontroller board
](qual-steps.md)
+ [
# Use the IDT user interface to run the FreeRTOS qualification suite
](device-tester-ui.md)
+ [
# Run Bluetooth Low Energy tests
](afr-bridgekeeper-dt-bt.md)
+ [
# Run the FreeRTOS qualification suite
](run-tests.md)
+ [
# View the IDT for FreeRTOS results
](view-results-frq.md)
+ [
# Interpret the IDT for FreeRTOS results
](interpreting-results-frq.md)
+ [
# View the IDT for FreeRTOS logs
](view-logs-frq.md)
# Setup the 1.0 qualification prerequisites
This section describes the prerequisites for testing microcontrollers with AWS IoT Device Tester.
## Download FreeRTOS
You can download a release of FreeRTOS from [GitHub](https://github.com/aws/amazon-freertos) with the following command:
```
git clone --branch --recurse-submodules https://github.com/aws/amazon-freertos.git
cd amazon-freertos
git submodule update --checkout --init --recursive
```
where is a version of FreeRTOS (for example, 202007.00) corresponding to an IDT version listed in [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md). This ensures you have the full source code, including submodules, and are using the correct version of IDT for your version of FreeRTOS, and vice versa.
Windows has a path length limitation of 260 characters. The path structure of FreeRTOS is many levels deep, so if you are using Windows, keep your file paths under the 260-character limit. For example, clone FreeRTOS to `C:\FreeRTOS` rather than `C:\Users\username\programs\projects\myproj\FreeRTOS\`.
### FreeRTOS qualification with LTS libraries
+ In order for your microcontroller to be designated as supporting long-term support (LTS) based versions of FreeRTOS in the AWS Partner Device Catalog, you must provide a manifest file. For more information, see the [ FreeRTOS Qualification Checklist](https://docs.aws.amazon.com/freertos/latest/qualificationguide/afq-checklist.html) in the *FreeRTOS Qualification Guide*.
+ In order to validate that your microcontroller supports LTS based versions of FreeRTOS and qualify it for submission to the AWS Partner Device Catalog, you must use AWS IoT Device Tester (IDT) with FreeRTOS Qualification (FRQ) test suite version v1.4.x.
+ Support for LTS based versions of FreeRTOS is limited to the 202012.xx version of FreeRTOS.
## Download IDT for FreeRTOS
Every version of FreeRTOS has a corresponding version of IDT for FreeRTOS to perform qualification tests. Download the appropriate version of IDT for FreeRTOS from [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md).
Extract IDT for FreeRTOS to a location on the file system where you have read and write permissions. Because Microsoft Windows has a character limit for the path length, extract IDT for FreeRTOS into a root directory such as `C:\` or `D:\`.
**Note**
We don't recommend that multiple users run IDT from a shared location, such as an NFS directory or a Windows network shared folder. This may result in crashes or data corruption. We recommend that you extract the IDT package to a local drive.
## Create and configure an AWS account
### Sign up for an AWS account
If you do not have an AWS account, complete the following steps to create one.
**To sign up for an AWS account**
1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup).
1. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad.
When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks).
AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**.
### Create a user with administrative access
After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.
**Secure your AWS account root user**
1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password.
For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*.
1. Turn on multi-factor authentication (MFA) for your root user.
For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*.
**Create a user with administrative access**
1. Enable IAM Identity Center.
For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*.
1. In IAM Identity Center, grant administrative access to a user.
For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*.
**Sign in as the user with administrative access**
+ To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.
For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*.
**Assign access to additional users**
1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.
For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*.
1. Assign users to a group, and then assign single sign-on access to the group.
For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*.
## AWS IoT Device Tester managed policy
The `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy contains the following AWS IoT Device Tester permissions for version checking, auto update features, and collection of metrics.
+ `iot-device-tester:SupportedVersion`
Grants AWS IoT Device Tester permission to fetch the list of supported products, test suites and IDT versions.
+ `iot-device-tester:LatestIdt`
Grants AWS IoT Device Tester permission to fetch the latest IDT version available for download.
+ `iot-device-tester:CheckVersion`
Grants AWS IoT Device Tester permission to check version compatibility for IDT, test suites and products.
+ `iot-device-tester:DownloadTestSuite`
Grants AWS IoT Device Tester permission to download test suite updates.
+ `iot-device-tester:SendMetrics`
Grants AWS permission to collect metrics about AWS IoT Device Tester internal usage.
## (Optional) Install the AWS Command Line Interface
You might prefer to use the AWS CLI to perform some operations. If you don't have the AWS CLI installed, follow the instructions at [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/installing.html).
Configure the AWS CLI for the AWS Region you want to use by running **aws configure** from a command line. For information about the AWS Regions that support IDT for FreeRTOS, see [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#amazon-freertos-ota-control). For more information about **aws configure** see [ Quick configuration with **aws configure**](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config).
# First test of your microcontroller board
You can use IDT for FreeRTOS to test as you port the FreeRTOS interfaces. After you have ported the FreeRTOS interfaces for your board’s device drivers, you use AWS IoT Device Tester to run the qualification tests on your microcontroller board.
## Add library porting layers
To port FreeRTOS for your device, follow the instructions in the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/).
## Configure your AWS credentials
You need to configure your AWS credentials for AWS IoT Device Tester to communicate with the AWS Cloud. For more information, see [Set up AWS Credentials and Region for Development](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html). Valid AWS credentials must be specified in the `devicetester_extract_location/devicetester_afreertos_[win|mac|linux]/configs/config.json` configuration file.
**Topics**
+ [
## Add library porting layers
](#add-port-layer)
+ [
## Configure your AWS credentials
](#cfg-aws-afr)
+ [
# Create a device pool in IDT for FreeRTOS
](cfg-dt-dp.md)
+ [
# Configure build, flash, and test settings
](cfg-dt-ud.md)
# Create a device pool in IDT for FreeRTOS
Devices to be tested are organized in device pools. Each device pool consists of one or more identical devices. You can configure IDT for FreeRTOS to test a single device in a pool or multiple devices in a pool. To accelerate the qualification process, IDT for FreeRTOS can test devices with the same specifications in parallel. It uses a round-robin method to execute a different test group on each device in a device pool.
You can add one or more devices to a device pool by editing the `devices` section of the `device.json` template in the `configs` folder.
**Note**
All devices in the same pool must be of same technical specification and SKU.
To enable parallel builds of the source code for different test groups, IDT for FreeRTOS copies the source code to a results folder inside the IDT for FreeRTOS extracted folder. The source code path in your build or flash command must be referenced using either the `testdata.sourcePath` or `sdkPath` variable. IDT for FreeRTOS replaces this variable with a temporary path of the copied source code. For more information see, [IDT for FreeRTOS variables](dt-vars.md).
The following is an example `device.json` file used to create a device pool with multiple devices.
```
[
{
"id": "pool-id",
"sku": "sku",
"features": [
{
"name": "WIFI",
"value": "Yes | No"
},
{
"name": "Cellular",
"value": "Yes | No"
},
{
"name": "OTA",
"value": "Yes | No",
"configs": [
{
"name": "OTADataPlaneProtocol",
"value": "HTTP | MQTT"
}
]
},
{
"name": "BLE",
"value": "Yes | No"
},
{
"name": "TCP/IP",
"value": "On-chip | Offloaded | No"
},
{
"name": "TLS",
"value": "Yes | No"
},
{
"name": "PKCS11",
"value": "RSA | ECC | Both | No"
},
{
"name": "KeyProvisioning",
"value": "Import | Onboard | No"
}
],
"devices": [
{
"id": "device-id",
"connectivity": {
"protocol": "uart",
"serialPort": "/dev/tty*"
},
***********Remove the section below if the device does not support onboard key generation***************
"secureElementConfig" : {
"publicKeyAsciiHexFilePath": "absolute-path-to/public-key-txt-file: contains-the-hex-bytes-public-key-extracted-from-onboard-private-key",
"secureElementSerialNumber": "secure-element-serialNo-value",
"preProvisioned" : "Yes | No"
},
**********************************************************************************************************
"identifiers": [
{
"name": "serialNo",
"value": "serialNo-value"
}
]
}
]
}
]
```
The following attributes are used in the `device.json` file:
**`id`**
A user-defined alphanumeric ID that uniquely identifies a pool of devices. Devices that belong to a pool must be of the same type. When a suite of tests is running, devices in the pool are used to parallelize the workload.
**`sku`**
An alphanumeric value that uniquely identifies the board you are testing. The SKU is used to track qualified boards.
If you want to list your board in AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process.
**`features`**
An array that contains the device's supported features. AWS IoT Device Tester uses this information to select the qualification tests to run.
Supported values are:
**`TCP/IP`**
Indicates if your board supports a TCP/IP stack and whether it is supported on-chip (MCU) or offloaded to another module. TCP/IP is required for qualification.
**`WIFI`**
Indicates if your board has Wi-Fi capabilities. Must be set to `No` if `Cellular` is set to `Yes`.
**`Cellular`**
Indicates if your board has cellular capabilities. Must be set to `No` if `WIFI` is set to `Yes`. When this feature is set to `Yes`, the FullSecureSockets test will be executed using AWS t2.micro EC2 instances and this may incur additional costs to your account. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/).
**`TLS`**
Indicates if your board supports TLS. TLS is required for qualification.
**`PKCS11`**
Indicates the public key cryptography algorithm that the board supports. PKCS11 is required for qualification. Supported values are `ECC`, `RSA`, `Both` and `No`. `Both` indicates the board supports both the `ECC` and `RSA` algorithms.
**`KeyProvisioning`**
Indicates the method of writing a trusted X.509 client certificate onto your board. Valid values are `Import`, `Onboard` and `No`. Key provisioning is required for qualification.
+ Use `Import` if your board allows the import of private keys. IDT will create a private key and build this to the FreeRTOS source code.
+ Use `Onboard` if your board supports on-board private key generation (for example, if your device has a secure element, or if you prefer to generate your own device key pair and certificate). Make sure you add a `secureElementConfig` element in each of the device sections and put the absolute path to the public key file in the `publicKeyAsciiHexFilePath` field.
+ Use `No` if your board does not support key provisioning.
**`OTA`**
Indicates if your board supports over-the-air (OTA) update functionality. The `OtaDataPlaneProtocol` attribute indicates which OTA dataplane protocol the device supports. The attribute is ignored if the OTA feature is not supported by the device. When `"Both"` is selected, the OTA test execution time is prolonged due to running both MQTT, HTTP, and mixed tests.
Starting with IDT v4.1.0, `OtaDataPlaneProtocol` accepts only `HTTP` and `MQTT` as supported values.
**`BLE`**
Indicates if your board supports Bluetooth Low Energy (BLE).
**`devices.id`**
A user-defined unique identifier for the device being tested.
**`devices.connectivity.protocol`**
The communication protocol used to communicate with this device. Supported value: `uart`.
**`devices.connectivity.serialPort`**
The serial port of the host computer used to connect to the devices being tested.
**`devices.secureElementConfig.PublicKeyAsciiHexFilePath`**
The absolute path to the file that contains the hex bytes public key extracted from onboard private key.
Example format:
```
3059 3013 0607 2a86 48ce 3d02 0106 082a
8648 ce3d 0301 0703 4200 04cd 6569 ceb8
1bb9 1e72 339f e8cf 60ef 0f9f b473 33ac
6f19 1813 6999 3fa0 c293 5fae 08f1 1ad0
41b7 345c e746 1046 228e 5a5f d787 d571
dcb2 4e8d 75b3 2586 e2cc 0c
```
If your public key is in .der format, you can hex encode the public key directly to generate the hex file.
Example command for .der public key to generate hex file:
```
xxd -p pubkey.der > outFile
```
If your public key is in .pem format, you can extract the base64 encoded part, decode it into binary format, and then hex encode it to generate the hex file.
For example, use these commands to generate a hex file for a .pem public key:
1. Take out the base64 encoded part of the key (strip the header and footer ) and store it in a file, for example name it `base64key`, run this command to convert it to .der format:
```
base64 —decode base64key > pubkey.der
```
1. Run the `xxd` command to convert it to hex format.
```
xxd -p pubkey.der > outFile
```
**`devices.secureElementConfig.SecureElementSerialNumber`**
(Optional) The serial number of the secure element. Provide this field when the serial number is printed out along with the device public key when you run the FreeRTOS demo/test project.
**`devices.secureElementConfig.preProvisioned`**
(Optional) Set to "Yes" if the device has a pre-provisioned secure element with locked-down credentials, that cannot import, create, or destroy objects. This configuration takes effect only when `features` has `KeyProvisioning` set to "Onboard", along with `PKCS11` set to "ECC".
**`identifiers`**
(Optional) An array of arbitrary name-value pairs. You can use these values in the build and flash commands described in the next section.
# Configure build, flash, and test settings
For IDT for FreeRTOS to build and flash tests on to your board automatically, you must configure IDT to run the build and flash commands for your hardware. The build and flash command settings are configured in the `userdata.json` template file located in the `config` folder.
# Configure settings for testing devices
Build, flash, and test settings are made in the `configs/userdata.json` file. We support Echo Server configuration by loading both the client and server certificates and keys in the `customPath`. For more information, see [Setting up an echo server](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-echo-server.html) in the *FreeRTOS Porting Guide*. The following JSON example shows how you can configure IDT for FreeRTOS to test multiple devices:
```
{
"sourcePath": "/absolute-path-to/freertos",
"vendorPath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name",
// ***********The sdkConfiguration block below is needed if you are not using the default, unmodified FreeRTOS repo.
// In other words, if you are using the default, unmodified FreeRTOS repo then remove this block***************
"sdkConfiguration": {
"name": "sdk-name",
"version": "sdk-version",
"path": "/absolute-path-to/sdk"
},
"buildTool": {
"name": "your-build-tool-name",
"version": "your-build-tool-version",
"command": [
"{{config.idtRootPath}}/relative-path-to/build-parallel.sh {{testData.sourcePath}} {{enableTests}}"
]
},
"flashTool": {
"name": "your-flash-tool-name",
"version": "your-flash-tool-version",
"command": [
"/{{config.idtRootPath}}/relative-path-to/flash-parallel.sh {{testData.sourcePath}} {{device.connectivity.serialPort}} {{buildImageName}}"
],
"buildImageInfo" : {
"testsImageName": "tests-image-name",
"demosImageName": "demos-image-name"
}
},
"testStartDelayms": 0,
"clientWifiConfig": {
"wifiSSID": "ssid",
"wifiPassword": "password",
"wifiSecurityType": "eWiFiSecurityOpen | eWiFiSecurityWEP | eWiFiSecurityWPA | eWiFiSecurityWPA2 | eWiFiSecurityWPA3"
},
"testWifiConfig": {
"wifiSSID": "ssid",
"wifiPassword": "password",
"wifiSecurityType": "eWiFiSecurityOpen | eWiFiSecurityWEP | eWiFiSecurityWPA | eWiFiSecurityWPA2 | eWiFiSecurityWPA3"
},
//**********
//This section is used to start echo server based on server certificate generation method,
//When certificateGenerationMethod is set as Automatic specify the eccCurveFormat to generate certifcate and key based on curve format,
//When certificateGenerationMethod is set as Custom specify the certificatePath and PrivateKeyPath to be used to start echo server
//**********
"echoServerCertificateConfiguration": {
"certificateGenerationMethod": "Automatic | Custom",
"customPath": {
"clientCertificatePath":"/path/to/clientCertificate",
"clientPrivateKeyPath": "/path/to/clientPrivateKey",
"serverCertificatePath":"/path/to/serverCertificate",
"serverPrivateKeyPath": "/path/to/serverPrivateKey"
},
"eccCurveFormat": "P224 | P256 | P384 | P521"
},
"echoServerConfiguration": {
"securePortForSecureSocket": 33333, // Secure tcp port used by SecureSocket test. Default value is 33333. Ensure that the port configured isn't blocked by the firewall or your corporate network
"insecurePortForSecureSocket": 33334, // Insecure tcp port used by SecureSocket test. Default value is 33334. Ensure that the port configured isn't blocked by the firewall or your corporate network
"insecurePortForWiFi": 33335 // Insecure tcp port used by Wi-Fi test. Default value is 33335. Ensure that the port configured isn't blocked by the firewall or your corporate network
},
"otaConfiguration": {
"otaFirmwareFilePath": "{{testData.sourcePath}}/relative-path-to/ota-image-generated-in-build-process",
"deviceFirmwareFileName": "ota-image-name-on-device",
"otaDemoConfigFilePath": "{{testData.sourcePath}}/relative-path-to/ota-demo-config-header-file",
"codeSigningConfiguration": {
"signingMethod": "AWS | Custom",
"signerHashingAlgorithm": "SHA1 | SHA256",
"signerSigningAlgorithm": "RSA | ECDSA",
"signerCertificate": "arn:partition:service:region:account-id:resource:qualifier | /absolute-path-to/signer-certificate-file",
"signerCertificateFileName": "signerCertificate-file-name",
"compileSignerCertificate": boolean,
// ***********Use signerPlatform if you choose aws for signingMethod***************
"signerPlatform": "AmazonFreeRTOS-Default | AmazonFreeRTOS-TI-CC3220SF",
"untrustedSignerCertificate": "arn:partition:service:region:account-id:resourcetype:resource:qualifier",
// ***********Use signCommand if you choose custom for signingMethod***************
"signCommand": [
"/absolute-path-to/sign.sh {{inputImageFilePath}} {{outputSignatureFilePath}}"
]
}
},
// ***********Remove the section below if you're not configuring CMake***************
"cmakeConfiguration": {
"boardName": "board-name",
"vendorName": "vendor-name",
"compilerName": "compiler-name",
"frToolchainPath": "/path/to/freertos/toolchain",
"cmakeToolchainPath": "/path/to/cmake/toolchain"
},
"freertosFileConfiguration": {
"required": [
{
"configName": "pkcs11Config",
"filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_tests/config_files/core_pkcs11_config.h"
},
{
"configName": "pkcs11TestConfig",
"filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_tests/config_files/iot_test_pkcs11_config.h"
}
],
"optional": [
{
"configName": "otaAgentTestsConfig",
"filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_tests/config_files/ota_config.h"
},
{
"configName": "otaAgentDemosConfig",
"filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_demos/config_files/ota_config.h"
},
{
"configName": "otaDemosConfig",
"filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_demos/config_files/ota_demo_config.h"
}
]
}
}
```
The following lists the attributes used in `userdata.json`:
**`sourcePath`**
The path to the root of the ported FreeRTOS source code. For parallel testing with an SDK, the `sourcePath` can be set using the `{{userData.sdkConfiguration.path}}` place holder. For example:
```
{ "sourcePath":"{{userData.sdkConfiguration.path}}/freertos" }
```
**`vendorPath`**
The path to the vendor specific FreeRTOS code. For serial testing, the `vendorPath` can be set as an absolute path. For example:
```
{ "vendorPath":"C:/path-to-freertos/vendors/espressif/boards/esp32" }
```
For parallel testing, the `vendorPath` can be set using the `{{testData.sourcePath}}` place holder. For example:
```
{ "vendorPath":"{{testData.sourcePath}}/vendors/espressif/boards/esp32" }
```
The `vendorPath` variable is only necessary when running without an SDK, it can be removed otherwise.
When running tests in parallel without an SDK, the `{{testData.sourcePath}}` placeholder must be used in the `vendorPath`, `buildTool`, `flashTool` fields. When running test with a single device, absolute paths must be used in the `vendorPath`, `buildTool`, `flashTool` fields. When running with an SDK, the `{{sdkPath}}` placeholder must be used in the `sourcePath`, `buildTool`, and `flashTool` commands.
**`sdkConfiguration`**
If you are qualifying FreeRTOS with any modifications to files and folder structure beyond what is required for porting, then you will need to configure your SDK information in this block. If you're not qualifying with a ported FreeRTOS inside of an SDK, then you should omit this block entirely.
**`sdkConfiguration.name`**
The name of the SDK you're using with FreeRTOS. If you're not using an SDK, then the entire `sdkConfiguration` block should be omitted.
**`sdkConfiguration.version`**
The version of the SDK you're using with FreeRTOS. If you're not using an SDK, then the entire `sdkConfiguration` block should be omitted.
**`sdkConfiguration.path`**
The absolute path to your SDK directory that contains your FreeRTOS code. If you're not using an SDK, then the entire `sdkConfiguration` block should be omitted.
**`buildTool`**
The full path to your build script (.bat or .sh) that contains the commands to build your source code. All references to the source code path in the build command must be replaced by the AWS IoT Device Tester variable `{{testdata.sourcePath}}` and references to the SDK path should be replaced by `{{sdkPath}}`. Use the `{{config.idtRootPath}}` placeholder to reference the absolute or relative IDT path.
**`testStartDelayms`**
Specifies how many milliseconds the FreeRTOS test runner will wait before starting to run tests. This can be useful if the device under test begins outputting important test information before IDT has a chance to connect and start logging due to network or other latency. The max allowed value is 30000 ms (30 seconds). This value is applicable to FreeRTOS test groups only, and not applicable to other test groups that do not utilize the FreeRTOS test runner, such as the OTA tests.
**`flashTool`**
Full path to your flash script (.sh or .bat) that contains the flash commands for your device. All references to the source code path in the flash command must be replaced by the IDT for FreeRTOS variable `{{testdata.sourcePath}}` and all references to your SDK path must be replaced by the IDT for FreeRTOS variable `{{sdkPath}}`.Use the `{{config.idtRootPath}}` placeholder to reference the absolute or relative IDT path.
**`buildImageInfo`**
**`testsImageName`**
The name of the file produced by the build command when building tests from the `freertos-source/tests` folder.
**`demosImageName`**
The name of the file produced by the build command when building tests from the `freertos-source/demos` folder.
**`clientWifiConfig`**
The client Wi-Fi configuration. The Wi-Fi library tests require an MCU board to connect to two access points. (The two access points can be the same.) This attribute configures the Wi-Fi settings for the first access point. Some of the Wi-Fi test cases expect the access point to have some security and not to be open. Please make sure both access points are on the same subnet as the host computer running IDT.
**`wifi_ssid`**
The Wi-Fi SSID.
**`wifi_password`**
The Wi-Fi password.
**`wifiSecurityType`**
The type of Wi-Fi security used. One of the values:
+ `eWiFiSecurityOpen`
+ `eWiFiSecurityWEP`
+ `eWiFiSecurityWPA`
+ `eWiFiSecurityWPA2`
+ `eWiFiSecurityWPA3`
If your board does not support Wi-Fi, you must still include the `clientWifiConfig` section in your `device.json` file, but you can omit values for these attributes.
**`testWifiConfig`**
The test Wi-Fi configuration. The Wi-Fi library tests require an MCU board to connect to two access points. (The two access points can be the same.) This attribute configures the Wi-Fi setting for the second access point. Some of the Wi-Fi test cases expect the access point to have some security and not to be open. Please make sure both access points are on the same subnet as the host computer running IDT.
**`wifiSSID`**
The Wi-Fi SSID.
**`wifiPassword`**
The Wi-Fi password.
**`wifiSecurityType`**
The type of Wi-Fi security used. One of the values:
+ `eWiFiSecurityOpen`
+ `eWiFiSecurityWEP`
+ `eWiFiSecurityWPA`
+ `eWiFiSecurityWPA2`
+ `eWiFiSecurityWPA3`
If your board does not support Wi-Fi, you must still include the `testWifiConfig` section in your `device.json` file, but you can omit values for these attributes.
**`echoServerCertificateConfiguration`**
The configurable echo server certificate generation placeholder for secure socket tests. This field is required.
**`certificateGenerationMethod`**
Specifies whether the server certificate is generated automatically or provided manually.
**`customPath`**
If `certificateGenerationMethod` is "Custom", `certificatePath` and `privateKeyPath` are required.
**`certificatePath`**
Specifies the filepath for the server certificate.
**`privateKeyPath`**
Specifies the filepath for the private key.
**`eccCurveFormat`**
Specifies the curve format supported by the board. Required when `PKCS11` is set to "ecc" in `device.json`. Valid values are "P224", "P256", "P384", or "P521".
**`echoServerConfiguration`**
The configurable echo server ports for WiFi and secure sockets tests. This field is optional.
**`securePortForSecureSocket`**
The port which is used to setup an echo server with TLS for the secure sockets test. The default value is 33333. Ensure the port configured is not blocked by a firewall or your corporate network.
**`insecurePortForSecureSocket`**
The port which is used to setup echo server without TLS for the secure sockets test. The default value used in the test is 33334. Ensure the port configured is not blocked by a firewall or your corporate network.
**`insecurePortForWiFi`**
The port which is used to setup echo server without TLS for WiFi test. The default value used in the test is 33335. Ensure the port configured is not blocked by a firewall or your corporate network.
**`otaConfiguration`**
The OTA configuration. [Optional]
**`otaFirmwareFilePath`**
The full path to the OTA image created after the build. For example, `{{testData.sourcePath}}/relative-path/to/ota/image/from/source/root`.
**`deviceFirmwareFileName`**
The full file path on the MCU device where the OTA firmware is located. Some devices do not use this field, but you still must provide a value.
**`otaDemoConfigFilePath`**
The full path to `aws_demo_config.h`, found in `afr-source/vendors/vendor/boards/board/aws_demos/config_files/`. These files are included in the porting code template that FreeRTOS provides.
**`codeSigningConfiguration`**
The code signing configuration.
**`signingMethod`**
The code signing method. Possible values are `AWS` or `Custom`.
For the Beijing and Ningxia Regions, use `Custom`. `AWS` code signing isn't supported in these Regions.
**`signerHashingAlgorithm`**
The hashing algorithm supported on the device. Possible values are `SHA1` or `SHA256`.
**`signerSigningAlgorithm`**
The signing algorithm supported on the device. Possible values are `RSA` or `ECDSA`.
**`signerCertificate`**
The trusted certificate used for OTA.
For AWS code signing method, use the Amazon Resource Name (ARN) for the trusted certificate uploaded to the AWS Certificate Manager.
For Custom code signing method, use the absolute path to the signer certificate file.
For more information about creating a trusted certificate, see [Create a code-signing certificate](ota-code-sign-cert.md).
**`signerCertificateFileName`**
The file name of the code signing certificate on the device. This value must match the file name that you provided when you ran the `aws acm import-certificate` command.
For more information, see [Create a code-signing certificate](ota-code-sign-cert.md).
**`compileSignerCertificate`**
Set to `true` if the code signer signature verification certificate isn't provisioned or flashed, so it must be compiled into the project. AWS IoT Device Tester fetches the trusted certificate and compiles it into `aws_codesigner_certifiate.h`.
**`untrustedSignerCertificate`**
The ARN or filepath for a second certificate used in some OTA tests as an untrusted certificate. For more information about creating a certificate, see [ Create a code-signing certificate](https://docs.aws.amazon.com/freertos/latest/userguide/ota-code-sign-cert.html).
**`signerPlatform`**
The signing and hashing algorithm that AWS Code Signer uses while creating the OTA update job. Currently, the possible values for this field are `AmazonFreeRTOS-TI-CC3220SF` and `AmazonFreeRTOS-Default`.
+ Choose `AmazonFreeRTOS-TI-CC3220SF` if `SHA1` and `RSA`.
+ Choose `AmazonFreeRTOS-Default` if `SHA256` and `ECDSA`.
If you need `SHA256` \$1 `RSA` or `SHA1` \$1 `ECDSA` for your configuration, contact us for further support.
Configure `signCommand` if you chose `Custom` for `signingMethod`.
**`signCommand`**
The command used to perform custom code signing. You can find the template in the `/configs/script_templates` directory.
Two placeholders `{{inputImageFilePath}}` and `{{outputSignatureFilePath}}` are required in the command. `{{inputImageFilePath}}` is the file path of the image built by IDT to be signed. `{{outputSignatureFilePath}}` is the file path of the signature which will be generated by the script.
**`cmakeConfiguration`**
CMake configuration [Optional]
To execute CMake test cases, you must provide the board name, vendor name, and either the `frToolchainPath` or `compilerName`. You may also provide the `cmakeToolchainPath` if you have a custom path to the CMake toolchain.
**`boardName`**
The name of the board under test. The board name should be the same as the folder name under `path/to/afr/source/code/vendors/vendor/boards/board`.
**`vendorName`**
The vendor name for the board under test. The vendor should be the same as the folder name under `path/to/afr/source/code/vendors/vendor`.
**`compilerName`**
The compiler name.
**`frToolchainPath`**
The fully-qualified path to the compiler toolchain
**`cmakeToolchainPath` **
The fully-qualified path to the CMake toolchain. This field is optional
**`freertosFileConfiguration`**
The configuration of the FreeRTOS files that IDT modifies before running tests.
**`required`**
This section specifies required tests whose config files you have moved, for example, PKCS11, TLS, and so on.
**`configName`**
The name of the test that is being configured.
**`filePath`**
The absolute path to the configuration files within the `freertos` repo. Use the `{{testData.sourcePath}}` variable to define the path.
**`optional`**
This section specifies optional tests whose config files you have moved, for example OTA, WiFi, and so on.
**`configName`**
The name of the test that is being configured.
**`filePath`**
The absolute path to the configuration files within the `freertos` repo. Use the `{{testData.sourcePath}}` variable to define the path.
**Note**
To execute CMake test cases, you must provide the board name, vendor name, and either the `afrToolchainPath` or `compilerName`. You may also provide `cmakeToolchainPath` if you have a custom path to the CMake toolchain.
# IDT for FreeRTOS variables
The commands to build your code and flash the device might require connectivity or other information about your devices to run successfully. AWS IoT Device Tester allows you to reference device information in flash and build commands using [JsonPath](https://goessner.net/articles/JsonPath/). By using simple JsonPath expressions, you can fetch the required information specified in your `device.json` file.
## Path variables
IDT for FreeRTOS defines the following path variables that can be used in command lines and configuration files:
**`{{testData.sourcePath}}`**
Expands to the source code path. If you use this variable, it must be used in both the flash and build commands.
**`{{sdkPath}}`**
Expands to the value in your `userData.sdkConfiguration.path` when used in the build and flash commands.
**`{{device.connectivity.serialPort}}`**
Expands to the serial port.
**`{{device.identifiers[?(@.name == 'serialNo')].value[0]}}`**
Expands to the serial number of your device.
**`{{enableTests}}`**
Integer value indicating whether the build is for tests (value 1) or demos (value 0).
**`{{buildImageName}}`**
The file name of the image built by the build command.
**`{{otaCodeSignerPemFile}}`**
PEM file for the OTA code signer.
**`{{config.idtRootPath}}`**
Expands to the AWS IoT Device Tester root path. This variable replaces the absolute path for IDT when used by the build and flash commands.
# Use the IDT user interface to run the FreeRTOS qualification suite
Use the IDT UI to run the FreeRTOS qualification suite
Starting with IDT v4.3.0, AWS IoT Device Tester for FreeRTOS (IDT-FreeRTOS) includes a web-based user interface that enables you to interact with the IDT command line executable and related configuration files. You can use the IDT-FreeRTOS UI to create a new configuration to run IDT tests, or to modify an existing configuration. You can also use the UI to invoke the IDT executable and run tests.
The IDT-FreeRTOS UI provides the following functions:
+ Simplify setting up configuration files for IDT-FreeRTOS tests.
+ Simplify using IDT-FreeRTOS to run qualification tests.
For information about the using the command line to run qualification tests, see [First test of your microcontroller board](qual-steps.md).
This section describes the prerequisites for using the IDT-FreeRTOS UI, and shows you how to get started running qualification tests in the UI.
**Topics**
+ [
# Set up the prerequisites to run the FreeRTOS qualification suite
](dev-tester-ui-prereqs.md)
+ [
# Get started with the IDT-FreeRTOS UI
](dev-tester-ui-getting-started.md)
# Set up the prerequisites to run the FreeRTOS qualification suite
Prerequisites
This section describes the prerequisites for testing microcontrollers with AWS IoT Device Tester.
**Topics**
+ [
## Use a supported web browser
](#idt-ui-supported-web-browser)
+ [
## Download FreeRTOS
](#ui-download-afr)
+ [
## Download IDT for FreeRTOS
](#ui-download-dev-tester-afr)
+ [
## Create and configure an AWS account
](#ui-config-aws-account)
+ [
## AWS IoT Device Tester managed policy
](#ui-managed-policy)
## Use a supported web browser
The IDT-FreeRTOS UI supports the following web browsers.
| Browser | Version |
| --- | --- |
| Google Chrome | Latest three major versions |
| Mozilla Firefox | Latest three major versions |
| Microsoft Edge | Latest three major versions |
| Apple Safari for macOS | Latest three major versions |
We recommend that you use Google Chrome or Mozilla Firefox for a better experience.
**Note**
The IDT-FreeRTOS UI doesn't support Microsoft Internet Explorer.
## Download FreeRTOS
You can download a release of FreeRTOS from [GitHub](https://github.com/aws/amazon-freertos) with the following command:
```
git clone --branch --recurse-submodules https://github.com/aws/amazon-freertos.git
cd amazon-freertos
git submodule update --checkout --init --recursive
```
where is a version of FreeRTOS (for example, 202007.00) corresponding to an IDT version listed in [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md). This ensures you have the full source code, including submodules, and are using the correct version of IDT for your version of FreeRTOS, and vice versa.
Windows has a path length limitation of 260 characters. The path structure of FreeRTOS is many levels deep, so if you're using Windows, keep your file paths under the 260-character limit. For example, clone FreeRTOS to `C:\FreeRTOS` rather than `C:\Users\username\programs\projects\myproj\FreeRTOS\`.
### Considerations for LTS qualification (qualification for FreeRTOS that uses LTS libraries)
+ In order for your microcontroller to be designated as supporting long-term support (LTS) based versions of FreeRTOS in the AWS Partner Device Catalog, you must provide a manifest file. For more information, see the [FreeRTOS Qualification Checklist](https://docs.aws.amazon.com/freertos/latest/qualificationguide/afq-checklist.html) in the *FreeRTOS Qualification Guide*.
+ In order to validate that your microcontroller supports LTS based versions of FreeRTOS and qualify it for submission to the AWS Partner Device Catalog, you must use AWS IoT Device Tester (IDT) with FreeRTOS Qualification (FRQ) test suite version v1.4.x.
+ Support for LTS based versions of FreeRTOS is limited to the 202012.xx version of FreeRTOS.
## Download IDT for FreeRTOS
Every version of FreeRTOS has a corresponding version of IDT for FreeRTOS for performing qualification tests. Download the appropriate version of IDT for FreeRTOS from [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md).
Extract IDT for FreeRTOS to a location on the file system where you have read and write permissions. Because Microsoft Windows has a character limit for the path length, extract IDT for FreeRTOS into a root directory such as `C:\` or `D:\`.
**Note**
We recommend that you extract the IDT package to a local drive.Allowing multiple users to run IDT from a shared location, such as an NFS directory or a Windows network shared folder, might result in the system not responding or data corruption.
## Create and configure an AWS account
### Sign up for an AWS account
If you do not have an AWS account, complete the following steps to create one.
**To sign up for an AWS account**
1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup).
1. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad.
When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks).
AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**.
### Create a user with administrative access
After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.
**Secure your AWS account root user**
1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password.
For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*.
1. Turn on multi-factor authentication (MFA) for your root user.
For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*.
**Create a user with administrative access**
1. Enable IAM Identity Center.
For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*.
1. In IAM Identity Center, grant administrative access to a user.
For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*.
**Sign in as the user with administrative access**
+ To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.
For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*.
**Assign access to additional users**
1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.
For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*.
1. Assign users to a group, and then assign single sign-on access to the group.
For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*.
## AWS IoT Device Tester managed policy
To enable device tester to run and to collect metrics, the `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy contains the following permissions:
+ `iot-device-tester:SupportedVersion`
Grants permission to get the list of FreeRTOS versions and test suite versions supported by IDT, so that they're available from the AWS CLI.
+ `iot-device-tester:LatestIdt`
Grants permission to get the latest AWS IoT Device Tester version that is available for download.
+ `iot-device-tester:CheckVersion`
Grants permission to check that a combination of product, test suite, and AWS IoT Device Tester versions are compatible.
+ `iot-device-tester:DownloadTestSuite`
Grants permission to AWS IoT Device Tester to download test suites.
+ `iot-device-tester:SendMetrics`
Grants permission to publish AWS IoT Device Tester usage metrics data.
# Get started with the IDT-FreeRTOS UI
This section shows you how to use the IDT-FreeRTOS UI to create or modify your configuration, and then shows you how to run tests.
**Topics**
+ [
## Configure AWS credentials
](#configure-aws-credentials)
+ [
## Open the IDT-FreeRTOS UI
](#open-idt-ui)
+ [
## Create a new configuration
](#create-new-configuration)
+ [
## Modify an existing configuration
](#modify-existing-configuration)
+ [
## Run qualification tests
](#run-tests-from-ui)
## Configure AWS credentials
You must configure credentials for the AWS user that you created in [Create and configure an AWS account](dev-tester-ui-prereqs.md#ui-config-aws-account). You can specify your credentials in one of two ways:
+ In a credentials file
+ As environment variables
### Configure AWS credentials with a credentials file
IDT uses the same credentials file as the AWS CLI. For more information, see [Configuration and credential files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html).
The location of the credentials file varies, depending on the operating system you're using:
+ macOS, Linux: `~/.aws/credentials`
+ Windows: `C:\Users\UserName\.aws\credentials`
Add your AWS credentials to the `credentials` file in the following format:
```
[default]
aws_access_key_id =
aws_secret_access_key =
```
**Note**
If you don't use the `default` AWS profile, be sure to specify the profile name in the IDT-FreeRTOS UI. For more information about profiles, see [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
### Configure AWS credentials with environment variables
Environment variables are variables maintained by the operating system and used by system commands. They're not saved if you close the SSH session. The IDT-FreeRTOS UI uses the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to store your AWS credentials.
To set these variables on Linux, macOS, or Unix, use **export**:
```
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
```
To set these variables on Windows, use **set**:
```
set AWS_ACCESS_KEY_ID=
set AWS_SECRET_ACCESS_KEY=
```
## Open the IDT-FreeRTOS UI
**To open the IDT-FreeRTOS UI**
1. Download a supported IDT-FreeRTOS version and extract the downloaded archive into a location on your file system where you have read and write permissions.
1. Run the following command to navigate to the IDT-FreeRTOS installation directory:
```
cd devicetester-extract-location/bin
```
1. Run the following command to open the IDT-FreeRTOS UI:
------
#### [ Linux ]
```
.devicetestergui_linux_x86-64.exe
```
------
#### [ Windows ]
```
./devicetestergui_win_x64-64
```
------
#### [ macOS ]
```
./devicetestergui_mac_x86-64
```
**Note**
On Mac, to allow your system to run the UI, go to **System Preferences -> Security & Privacy**. When you run the tests, you may need to do this three more times.
------
The IDT-FreeRTOS UI opens in your default browser. For information about supported browsers, see [Use a supported web browser](dev-tester-ui-prereqs.md#idt-ui-supported-web-browser).
## Create a new configuration
If you're a first-time user, then you must create a new configuration to set up the JSON configuration files that IDT-FreeRTOS requires to run tests. You can then run tests or modify the configuration that was created.
For examples of the `config.json`, `device.json`, and `userdata.json` files, see [First test of your microcontroller board](qual-steps.md). For an example of the `resource.json` file that is used only for running Bluetooth Low Energy (BLE) tests, see [Run Bluetooth Low Energy tests](afr-bridgekeeper-dt-bt.md).
**To create a new configuration**
1. In the IDT-FreeRTOS UI, open the navigation menu, and then choose **Create new configuration**.
**Important**
You must configure your AWS credentials before you open the UI. If you haven't configured your credentials, close the IDT-FreeRTOS UI browser window, follow the steps in [Configure AWS credentials](#configure-aws-credentials), and then reopen the IDT-FreeRTOS UI.
1. Follow the configuration wizard to enter the IDT configuration settings that are used to run qualification tests. The wizard configures the following settings in JSON configuration files that are located in the `devicetester-extract-location/config` directory.
+ **AWS settings**—The AWS account information that IDT-FreeRTOS uses to create AWS resources during test runs. These settings are configured in the `config.json` file.
+ **FreeRTOS repository**—The absolute path to the FreeRTOS repository and ported code, and the type of qualification you want to perform. These settings are configured in the `userdata.json` file.
You must port FreeRTOS for your device before you can run qualification tests. For more information, see the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/)
+ **Build and flash**—The build and flash commands for your hardware that allow IDT to build and flash tests on to your board automatically. These settings are configured in the `userdata.json` file.
+ **Devices**—The device pool settings for the devices to be tested. These settings are configured in `id` and `sku` fields, and the `devices` block for the device pool in the `device.json` file.
+ **Networking**—The settings to test network communication support for your devices. These settings are configured in the `features` block of the `device.json` file, and in the `clientWifiConfig` and `testWifiConfig` blocks in the `userdata.json` file.
+ **Echo server**—The echo server configuration settings for secure socket tests. These settings are configured in the `userdata.json` file.
For more information about echo server configuration, see [https://docs.aws.amazon.com/freertos/latest/portingguide/afr-echo-server.html](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-echo-server.html).
+ **CMake**—(Optional) The settings to run CMake build functionality tests. This configuration is required only if you're using CMake as your build system. These settings are configured in the `userdata.json` file.
+ **BLE**—The settings to run Bluetooth Low Energy functionality tests. These settings are configured in the `features` block of the `device.json` file and in the `resource.json` file.
+ **OTA**—The settings to run OTA functionality tests. These settings are configured in the `features` block of the `device.json` file and in the `userdata.json` file.
1. On the **Review** page, verify your configuration information.
After you finish reviewing your configuration, to run your qualification tests, choose **Run tests**.
## Modify an existing configuration
If you have already set up configuration files for IDT, then you can use the IDT-FreeRTOS UI to modify your existing configuration. Make sure that your existing configuration files are available in the `devicetester-extract-location/config` directory.
**To modify a new configuration**
1. In the IDT-FreeRTOS UI, open the navigation menu, and then choose **Edit existing configuration**.
The configuration dashboard displays information about your existing configuration settings. If a configuration is incorrect or unavailable, the status for that configuration is `Error validating configuration`.
1. To modify an existing configuration setting, complete the following steps:
1. Choose the name of a configuration setting to open its settings page.
1. Modify the settings, and then choose **Save** to regenerate the corresponding configuration file.
After you finish modifying your configuration, verify that all of your configuration settings pass validation. If the status for each configuration setting is `Valid`, you can run your qualification tests using this configuration.
## Run qualification tests
After you have created a configuration for IDT-FreeRTOS, you can run your qualification tests.
**To run qualification tests**
1. Validate your configuration.
1. In the navigation menu, choose **Run tests**.
1. To start the test run, choose **Start tests**.
IDT-FreeRTOS runs the qualification tests, and displays the test run summary and any errors in the **Test runner** console. After the test run is complete, you can view the test results and logs from the following locations:
+ Test results are located in the `devicetester-extract-location/results/execution-id` directory.
+ Test logs are located in the `devicetester-extract-location/results/execution-id/logs` directory.
For more information about test results and logs, see [View the IDT for FreeRTOS results](view-results-frq.md) and [View the IDT for FreeRTOS logs](view-logs-frq.md).
# Run Bluetooth Low Energy tests
This section describes how to set up and run Bluetooth Low Energy tests using AWS IoT Device Tester for FreeRTOS.
Bluetooth tests are not required for core qualification. If you do not want to test your device with FreeRTOS Bluetooth support you may skip this setup, be sure to leave the BLE feature in device.json set to `No`.
## Prerequisites
+ Follow the instructions in [First test of your microcontroller board](qual-steps.md).
+ A Raspberry Pi 4B or 3B\$1. (Required to run the Raspberry Pi BLE companion application)
+ A micro SD card and SD card adapter for the Raspberry Pi software.
## Raspberry Pi setup
To test the BLE capabilities of the device under test (DUT), you must have a Raspberry Pi Model 4B or 3B\$1.
**To set up your Raspberry Pi to run BLE tests**
1. Download one of the custom Yocto images that contains the software required to perform the tests.
+ [ Image for Raspberry Pi 4B](https://docs.aws.amazon.com/freertos/latest/userguide/freertos/IDTFR_BLE_RaspberryPi4B_1.0.0_2021-04-13.rpi-sd.img)
+ [ Image for Raspberry Pi 3B\$1](https://docs.aws.amazon.com/freertos/latest/userguide/freertos/IDTFR_BLE_RaspberryPi3Bplus_1.0.0_2021-04-13.rpi-sd.img)
**Note**
The Yocto image should only be used for testing with AWS IoT Device Tester for FreeRTOS and not for any other purpose.
1. Flash the yocto image onto the SD card for Raspberry Pi.
1. Using an SD card-writing tool such as [Etcher](https://www.balena.io/etcher), flash the downloaded `image-name.rpi-sd.img` file onto the SD card. Because the operating system image is large, this step might take some time. Then eject your SD card from your computer and insert the microSD card into your Raspberry Pi.
1. Configure your Raspberry Pi.
1. For the first boot, we recommend that you connect the Raspberry Pi to a monitor, keyboard, and mouse.
1. Connect your Raspberry Pi to a micro USB power source.
1. Sign in using the default credentials. For user ID, enter **root**. For password, enter **idtafr**.
1. Using an Ethernet or Wi-Fi connection, connect the Raspberry Pi to your network.
1. To connect your Raspberry Pi over Wi-Fi, open `/etc/wpa_supplicant.conf` on the Raspberry Pi and add your Wi-Fi credentials to the `Network` configuration.
```
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=0
update_config=1
network={
scan_ssid=1
ssid="your-wifi-ssid"
psk="your-wifi-password"
}
```
1. Run `ifup wlan0` to start the Wi-Fi connection. It might take a minute to connect to your Wi-Fi network.
1. For an Ethernet connection, run `ifconfig eth0`. For a Wi-Fi connection, run `ifconfig wlan0`. Make a note of the IP address, which appears as `inet addr` in the command output. You need the IP address later in this procedure.
1. (Optional) The tests execute commands on the Raspberry Pi over SSH using the default credentials for the yocto image. For additional security, we recommend that you set up public key authentication for SSH and disable password-based SSH.
1. Create an SSH key using the OpenSSL `ssh-keygen` command. If you already have an SSK key pair on your host computer, it is a best practice to create a new one to allow AWS IoT Device Tester for FreeRTOS to sign in to your Raspberry Pi.
**Note**
Windows does not come with an installed SSH client. For information about how to install an SSH client on Windows, see [Download SSH Software](https://www.ssh.com/ssh/#sec-Download-client-software).
1. The `ssh-keygen` command prompts you for a name and path to store the key pair. By default, the key pair files are named `id_rsa` (private key) and `id_rsa.pub` (public key). On macOS and Linux, the default location of these files is `~/.ssh/`. On Windows, the default location is `C:\Users\user-name`.
1. When you are prompted for a key phrase, just press ENTER to continue.
1. To add your SSH key onto your Raspberry Pi so AWS IoT Device Tester for FreeRTOS can sign into the device, use the `ssh-copy-id` command from your host computer. This command adds your public key into the `~/.ssh/authorized_keys` file on your Raspberry Pi.
`ssh-copy-id root@raspberry-pi-ip-address`
1. When prompted for a password, enter **idtafr**. This is the default password for the yocto image.
**Note**
The `ssh-copy-id` command assumes the public key is named `id_rsa.pub`. On macOS and Linux, the default location is ` ~/.ssh/`. On Windows, the default location is `C:\Users\user-name\.ssh`. If you gave the public key a different name or stored it in a different location, you must specify the fully qualified path to your SSH public key using the `-i` option to `ssh-copy-id` (for example, `ssh-copy-id -i ~/my/path/myKey.pub`). For more information about creating SSH keys and copying public keys, see [SSH-COPY-ID](https://www.ssh.com/ssh/copy-id).
1. To test that the public key authentication is working, run `ssh -i /my/path/myKey root@raspberry-pi-device-ip`.
If you are not prompted for a password, your public key authentication is working.
1. Verify that you can sign in to your Raspberry Pi using a public key, and then disable password-based SSH.
1. On the Raspberry Pi, edit the `/etc/ssh/sshd_config` file.
1. Set the `PasswordAuthentication` attribute to `no`.
1. Save and close the `sshd_config` file.
1. Reload the SSH server by running `/etc/init.d/sshd reload`.
1. Create a `resource.json` file.
1. In the directory in which you extracted AWS IoT Device Tester, create a file named `resource.json`.
1. Add the following information about your Raspberry Pi to the file, replacing *rasp-pi-ip-address* with the IP address of your Raspberry Pi.
```
[
{
"id": "ble-test-raspberry-pi",
"features": [
{"name":"ble", "version":"4.2"}
],
"devices": [
{
"id": "ble-test-raspberry-pi-1",
"connectivity": {
"protocol": "ssh",
"ip": "rasp-pi-ip-address"
}
}
]
}
]
```
1. If you didn't choose to use public key authentication for SSH, add the following to the `connectivity` section of the `resource.json` file.
```
"connectivity": {
"protocol": "ssh",
"ip": "rasp-pi-ip-address",
"auth": {
"method": "password",
"credentials": {
"user": "root",
"password": "idtafr"
}
}
}
```
1. (Optional) If you chose to use public key authentication for SSH, add the following to the `connectivity` section of the `resource.json` file.
```
"connectivity": {
"protocol": "ssh",
"ip": "rasp-pi-ip-address",
"auth": {
"method": "pki",
"credentials": {
"user": "root",
"privKeyPath": "location-of-private-key"
}
}
}
```
## FreeRTOS device setup
In your `device.json` file, set the `BLE` feature to `Yes`. If you are starting with a `device.json` file from before Bluetooth tests were available, you need to add the feature for BLE to the `features` array:
```
{
...
"features": [
{
"name": "BLE",
"value": "Yes"
},
...
}
```
## Run the BLE tests
After you have enabled the BLE feature in `device.json`, the BLE tests run when you run `devicetester_[linux | mac | win_x86-64] run-suite` *without specifying a group-id*.
If you want to run the BLE tests separately, you can specify the group ID for BLE: `devicetester_[linux | mac | win_x86-64] run-suite --userdata path-to-userdata/userdata.json --group-id FullBLE`.
For the most reliable performance, place your Raspberry Pi close to the device under test (DUT).
## Troubleshoot BLE tests
Make sure you have followed the steps in [First test of your microcontroller board](qual-steps.md). If tests other than BLE are failing, then the problem is most likely not due to the Bluetooth configuration.
# Run the FreeRTOS qualification suite
You use the AWS IoT Device Tester for FreeRTOS executable to interact with IDT for FreeRTOS. The following command line examples show you how to run the qualification tests for a device pool (a set of identical devices).
------
#### [ IDT v3.0.0 and later ]
```
devicetester_[linux | mac | win] run-suite \
--suite-id suite-id \
--group-id group-id \
--pool-id your-device-pool \
--test-id test-id \
--upgrade-test-suite y|n \
--update-idt y|n \
--update-managed-policy y|n \
--userdata userdata.json
```
Runs a suite of tests on a pool of devices. The `userdata.json` file must be located in the `devicetester_extract_location/devicetester_afreertos_[win|mac|linux]/configs/` directory.
**Note**
If you're running IDT for FreeRTOS on Windows, use forward slashes (/) to specify the path to the `userdata.json` file.
Use the following command to run a specific test group:
```
devicetester_[linux | mac | win] run-suite \
--suite-id FRQ_1.0.1 \
--group-id group-id \
--pool-id pool-id \
--userdata userdata.json
```
The `suite-id` and `pool-id` parameters are optional if you're running a single test suite on a single device pool (that is, you have only one device pool defined in your `device.json` file).
Use the following command to run a specific test case in a test group:
```
devicetester_[linux | mac | win_x86-64] run-suite \
--group-id group-id \
--test-id test-id
```
You can use the `list-test-cases` command to list the test cases in a test group. IDT for FreeRTOS command line options
**group-id**
(Optional) The test groups to run, as a comma-separated list. If not specified, IDT runs all test groups in the test suite.
**pool-id**
(Optional) The device pool to test. This is required if you define multiple device pools in `device.json`. If you only have one device pool, you can omit this option.
**suite-id**
(Optional) The test suite version to run. If not specified, IDT uses the latest version in the tests directory on your system.
Starting in IDT v3.0.0, IDT checks online for newer test suites. For more information, see [Test suite versions](idt-test-suite-versions.md).
**test-id**
(Optional) The tests to run, as a comma-separated list. If specified, `group-id` must specify a single group.
**Example**
```
devicetester_[linux | mac | win_x86-64] run-suite --group-id mqtt --test-id mqtt_test
```
**update-idt**
(Optional) If this parameter is not set and a newer IDT version is available, you will be prompted to update IDT. If this parameter is set to `Y`, IDT will stop test execution if it detects that a newer version is available. If this parameter is set to `N`, IDT will continue the test execution.
**update-managed-policy**
(Optional) If this parameter is not used and IDT detects that your managed policy isn't up-to-date, you will be prompted to update your managed policy. If this parameter is set to `Y`, IDT will stop test execution if it detects that your managed policy isn't up-to-date. If this parameter is set to `N`, IDT will continue the test execution.
**upgrade-test-suite**
(Optional) If not used, and a newer test suite version is available, you're prompted to download it. To hide the prompt, specify `y` to always download the latest test suite, or `n` to use the test suite specified or the latest version on your system.
**Example**
**Example**
To always download and use the latest test suite, use the following command.
```
devicetester_[linux | mac | win_x86-64] run-suite --userdata userdata file --group-id group ID --upgrade-test-suite y
```
To use the latest test suite on your system, use the following command.
```
devicetester_[linux | mac | win_x86-64] run-suite --userdata userdata file --group-id group ID --upgrade-test-suite n
```
**h**
Use the help option to learn more about `run-suite` options.
**Example**
**Example**
```
devicetester_[linux | mac | win_x86-64] run-suite -h
```
------
#### [ IDT v1.7.0 and earlier ]
```
devicetester_[linux | mac | win] run-suite \
--suite-id suite-id \
--pool-id your-device-pool \
--userdata userdata.json
```
The `userdata.json` file should be located in the `devicetester_extract_location/devicetester_afreertos_[win|mac|linux]/configs/` directory.
**Note**
If you are running IDT for FreeRTOS on Windows, use forward slashes (/) to specify the path to the `userdata.json` file.
Use the following command to run a specific test group.
```
devicetester_[linux | mac | win] run-suite \
--suite-id FRQ_1 --group-id group-id \
--pool-id pool-id \
--userdata userdata.json
```
`suite-id` and `pool-id` are optional if you are running a single test suite on a single device pool (that is, you have only one device pool defined in your `device.json` file).IDT for FreeRTOS command line options
**group-id**
(Optional) Specifies the test group.
**pool-id**
Specifies the device pool to test. If you only have one device pool, you can omit this option.
**suite-id**
(Optional) Specifies the test suite to run.
------
## IDT for FreeRTOS commands
The IDT for FreeRTOS command supports the following operations:
------
#### [ IDT v3.0.0 and later ]
**`help`**
Lists information about the specified command.
**`list-groups`**
Lists the groups in a given suite.
**`list-suites`**
Lists the available suites.
**`list-supported-products`**
Lists the supported products and test suite versions.
**`list-supported-versions`**
Lists the FreeRTOS and test suite versions supported by the current IDT version.
**`list-test-cases`**
Lists the test cases in a specified group.
**`run-suite`**
Runs a suite of tests on a pool of devices.
Use the `--suite-id` option to specify a test suite version, or omit it to use the latest version on your system.
Use the `--test-id` to run an individual test case.
**Example**
```
devicetester_[linux | mac | win_x86-64] run-suite --group-id mqtt --test-id mqtt_test
```
For a complete list of options see [Run the FreeRTOS qualification suite](#run-tests).
Starting in IDT v3.0.0, IDT checks online for newer test suites. For more information, see [Test suite versions](idt-test-suite-versions.md).
------
#### [ IDT v1.7.0 and earlier ]
**`help`**
Lists information about the specified command.
**`list-groups`**
Lists the groups in a given suite.
**`list-suites`**
Lists the available suites.
**`run-suite`**
Runs a suite of tests on a pool of devices.
------
## Test for re-qualification
As new versions of IDT for FreeRTOS qualification tests are released, or as you update your board-specific packages or device drivers, you can use IDT for FreeRTOS to test your microcontroller boards. For subsequent qualifications, make sure that you have the latest versions of FreeRTOS and IDT for FreeRTOS and run the qualification tests again.
# View the IDT for FreeRTOS results
While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it writes a test run summary to the console and generates two test reports. These reports can be found in `devicetester-extract-location/results/execution-id/`. Both reports capture the results from the qualification test suite execution.
The `awsiotdevicetester_report.xml` is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements:
+ The IDT for FreeRTOS version.
+ The FreeRTOS version that was tested.
+ The features of FreeRTOS that are supported by the device based on the tests passed.
+ The SKU and the device name specified in the `device.json` file.
+ The features of the device specified in the `device.json` file.
+ The aggregate summary of test case results.
+ A breakdown of test case results by libraries that were tested based on the device features (for example, FullWiFi, FullMQTT, and so on).
+ Whether this qualification of FreeRTOS is for version 202012.00 that uses LTS libraries.
The `FRQ_Report.xml` is a report in standard [JUnit XML format](https://llg.cubic.org/docs/junit/). You can integrate it into CI/CD platforms like [Jenkins](https://jenkins.io/), [Bamboo](https://www.atlassian.com/software/bamboo), and so on. The report contains the following elements:
+ An aggregate summary of test case results.
+ A breakdown of test case results by libraries that were tested based on the device features.
# Interpret the IDT for FreeRTOS results
The report section in `awsiotdevicetester_report.xml` or `FRQ_Report.xml` lists the results of the tests that are executed.
The first XML tag `` contains the overall summary of the test execution. For example:
``
**Attributes used in the `` tag**
**`name`**
The name of the test suite.
**`time`**
The time, in seconds, it took to run the qualification suite.
**`tests`**
The number of test cases executed.
**`failures`**
The number of test cases that were run, but did not pass.
**`errors`**
The number of test cases that IDT for FreeRTOS couldn't execute.
**`disabled`**
This attribute is not used and can be ignored.
If there are no test case failures or errors, your device meets the technical requirements to run FreeRTOS and can interoperate with AWS IoT services. If you choose to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence.
In the event of test case failures or errors, you can identify the test case that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag shows the test case result summary for a test group.
``
The format is similar to the `` tag, but with an attribute called `skipped` that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each of the test cases that were executed for a test group. For example:
``
**Attributes used in the `` tag**
**`name`**
The name of the product being tested.
**`version`**
The version of the product being tested.
**`sdk`**
If you ran IDT with an SDK, this block contains the name and version of your SDK. If you didn't run IDT with an SDK, then this block contains:
```
N/A
N/A
```
**`features`**
The features validated. Features marked as `required` are required to submit your board for qualification. The following snippet shows how this appears in the `awsiotdevicetester_report.xml` file.
```
```
Features marked as `optional` are not required for qualification. The following snippets show optional features.
```
```
If there are no test failures or errors for the required features, your device meets the technical requirements to run FreeRTOS and can interoperate with AWS IoT services. If you want to list your device in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices), you can use this report as qualification evidence.
In the event of test failures or errors, you can identify the test that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag show the test result summary for a test group. For example:
```
```
The format is similar to the `` tag, but has a `skipped` attribute that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each executed test for a test group. For example:
```
```
**`lts`**
True if you are qualifying for a version of FreeRTOS that uses LTS libraries, false otherwise.
**Attributes used in the `` tag**
**`name`**
The name of the test case.
**`attempts`**
The number of times IDT for FreeRTOS executed the test case.
When a test fails or an error occurs, `` or `` tags are added to the `` tag with information for troubleshooting. For example:
```
Reason for the test case failure
Reason for the test case execution error
```
For more information, see [Troubleshoot errors](dt-afr-troubleshooting.md).
# View the IDT for FreeRTOS logs
You can find logs that IDT for FreeRTOS generates from test execution in `devicetester-extract-location/results/execution-id/logs`. Two sets of logs are generated:
**`test_manager.log`**
Contains logs generated from IDT for FreeRTOS (for example, logs related configuration and report generation).
**`test_group_id__test_case_id.log` (for example, `FullMQTT__Full_MQTT.log`)**
The log file for a test case, including output from the device under test. The log file is named according to the test group and test case that was run.
# Develop and run your own IDT test suites
Starting in IDT v4.0.0, IDT for FreeRTOS combines a standardized configuration setup and result format with a test suite environment that enables you to develop custom test suites for your devices and device software. You can add custom tests for your own internal validation or provide them to your customers for device verification.
Use IDT to develop and run custom test suites, as follows:
****To develop custom test suites****
+ Create test suites with custom test logic for the device that you want to test.
+ Provide IDT with your custom test suites to test runners. Include information about specific settings configurations for your test suites.
****To run custom test suites****
+ Set up the device that you want to test.
+ Implement the settings configurations as required by the test suites that you want to use.
+ Use IDT to run your custom test suites.
+ View the test results and execution logs for the tests run by IDT.
## Download latest version of AWS IoT Device Tester for FreeRTOS
Download the latest version of IDT for FreeRTOS
Download the [latest version](dev-test-versions-afr.md#idt-latest-version-afr) of IDT and extract the software into a location on your file system where you have read and write permissions.
**Note**
IDT does not support being run by multiple users from a shared location, such as an NFS directory or a Windows network shared folder. We recommend that you extract the IDT package to a local drive and run the IDT binary on your local workstation.
Windows has a path length limitation of 260 characters. If you are using Windows, extract IDT to a root directory like `C:\ ` or `D:\` to keep your paths under the 260 character limit.
## Test suite workflow
Test suites are composed of three types of files:
+ Configuration files that provide IDT with information on how to execute the test suite.
+ Test executable files that IDT uses to run test cases.
+ Additional files required to run tests.
Complete the following basic steps to create custom IDT tests:
1. [Create configuration files](idt-json-config.md) for your test suite.
1. [Create test case executables](test-executables.md) that contain the test logic for your test suite.
1. Verify and document the [configuration information required for test runners](set-config-custom.md) to run the test suite.
1. Verify that IDT can run your test suite and produce [test results](run-tests-custom.md) as expected.
To quickly build a sample custom suite and run it, follow the instructions in [Tutorial: Build and run the sample IDT test suite](build-sample-suite.md).
To get started creating a custom test suite in Python, see [Tutorial: Develop a simple IDT test suite](create-custom-tests.md).
# Tutorial: Build and run the sample IDT test suite
The AWS IoT Device Tester download includes the source code for a sample test suite. You can complete this tutorial to build and run the sample test suite to understand how you can use AWS IoT Device Tester for FreeRTOS to run custom test suites. Although this tutorial uses SSH, it is useful to learn how to use AWS IoT Device Tester with FreeRTOS devices.
In this tutorial, you will complete the following steps:
1. [Build the sample test suite](build-sample.md)
1. [Use IDT to run the sample test suite](run-sample.md)
**Topics**
+ [
# Set up the prerequisites for the sample test suite
](prereqs-tutorial-sample.md)
+ [
# Configure device information for IDT
](configure-idt-sample.md)
+ [
# Build the sample test suite
](build-sample.md)
+ [
# Use IDT to run the sample test suite
](run-sample.md)
+ [
# Troubleshoot errors
](tutorial-troubleshooting-custom.md)
# Set up the prerequisites for the sample test suite
To complete this tutorial, you need the following:
+
**Host computer requirements**
+ Latest version of AWS IoT Device Tester
+ [Python](https://docs.python.org/3/) 3.7 or later
To check the version of Python installed on your computer, run the following command:
```
python3 --version
```
On Windows, if using this command returns an error, then use `python --version` instead. If the returned version number is 3.7 or greater, then run the following command in a Powershell terminal to set `python3` as an alias for your `python` command.
```
Set-Alias -Name "python3" -Value "python"
```
If no version information is returned or if the version number is less than 3.7, follow the instructions in [Downloading Python](https://wiki.python.org/moin/BeginnersGuide/Download) to install Python 3.7\$1. For more information, see the [Python documentation](https://docs.python.org/3/).
+ [urllib3](https://urllib3.readthedocs.io/en/latest/)
To verify that `urllib3` is installed correctly, run the following command:
```
python3 -c 'import urllib3'
```
If `urllib3` is not installed, run the following command to install it:
```
python3 -m pip install urllib3
```
+
**Device requirements**
+ A device with a Linux operating system and a network connection to the same network as your host computer.
We recommend that you use a [Raspberry Pi](https://www.raspberrypi.org/) with Raspberry Pi OS. Make sure you set up [SSH](https://www.raspberrypi.com/documentation/computers/remote-access.html) on your Raspberry Pi to remotely connect to it.
# Configure device information for IDT
Configure your device information for IDT to run the test. You must update the `device.json` template located in the `/configs` folder with the following information.
```
[
{
"id": "pool",
"sku": "N/A",
"devices": [
{
"id": "",
"connectivity": {
"protocol": "ssh",
"ip": "",
"port": "",
"auth": {
"method": "pki | password",
"credentials": {
"user": "",
"privKeyPath": "/path/to/private/key",
"password": ""
}
}
}
}
]
}
]
```
In the `devices` object, provide the following information:
**`id`**
A user-defined unique identifier for your device.
**`connectivity.ip`**
The IP address of your device.
**`connectivity.port`**
Optional. The port number to use for SSH connections to your device.
**`connectivity.auth`**
Authentication information for the connection.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.auth.method`**
The authentication method used to access a device over the given connectivity protocol.
Supported values are:
+ `pki`
+ `password`
**`connectivity.auth.credentials`**
The credentials used for authentication.
**`connectivity.auth.credentials.user`**
The user name used to sign in to your device.
**`connectivity.auth.credentials.privKeyPath`**
The full path to the private key used to sign in to your device.
This value applies only if `connectivity.auth.method` is set to `pki`.
**`devices.connectivity.auth.credentials.password`**
The password used for signing in to your device.
This value applies only if `connectivity.auth.method` is set to `password`.
**Note**
Specify `privKeyPath` only if `method` is set to `pki`.
Specify `password` only if `method` is set to `password`.
# Build the sample test suite
The `/samples/python` folder contains sample configuration files, source code, and the IDT Client SDK that you can combine into a test suite using the provided build scripts. The following directory tree shows the location of these sample files:
```
├── ...
├── tests
├── samples
│ ├── ...
│ └── python
│ ├── configuration
│ ├── src
│ └── build-scripts
│ ├── build.sh
│ └── build.ps1
└── sdks
├── ...
└── python
└── idt_client
```
To build the test suite, run the following commands on your host computer:
------
#### [ Windows ]
```
cd /samples/python/build-scripts
./build.ps1
```
------
#### [ Linux, macOS, or UNIX ]
```
cd /samples/python/build-scripts
./build.sh
```
------
This creates the sample test suite in the `IDTSampleSuitePython_1.0.0` folder within the `/tests` folder. Review the files in the `IDTSampleSuitePython_1.0.0` folder to understand how the sample test suite is structured and to see various examples of test case executables and test configuration files.
**Note**
The sample test suite includes python source code. Do not include sensitive information in your test suite code.
Next step: Use IDT to [run the sample test suite](run-sample.md) that you created.
# Use IDT to run the sample test suite
To run the sample test suite, run the following commands on your host computer:
```
cd /bin
./devicetester_[linux | mac | win_x86-64] run-suite --suite-id IDTSampleSuitePython
```
IDT runs the sample test suite and streams the results to the console. When the test has finished running, you see the following information:
```
========== Test Summary ==========
Execution Time: 5s
Tests Completed: 4
Tests Passed: 4
Tests Failed: 0
Tests Skipped: 0
----------------------------------
Test Groups:
sample_group: PASSED
----------------------------------
Path to AWS IoT Device Tester Report: /path/to/devicetester/results/87e673c6-1226-11eb-9269-8c8590419f30/awsiotdevicetester_report.xml
Path to Test Execution Logs: /path/to/devicetester/results/87e673c6-1226-11eb-9269-8c8590419f30/logs
Path to Aggregated JUnit Report: /path/to/devicetester/results/87e673c6-1226-11eb-9269-8c8590419f30/IDTSampleSuitePython_Report.xml
```
# Troubleshoot errors
Use the following information to help resolve any issues with completing the tutorial.
**Test case does not run successfully**
+ If the test does not run successfully, IDT streams the error logs to the console that can help you troubleshoot the test run. Make sure that you meet all the [prerequisites](prereqs-tutorial-sample.md) for this tutorial.
**Cannot connect to the device under test**
Verify the following:
+ Your `device.json` file contains the correct IP address, port, and authentication information.
+ You can connect to your device over SSH from your host computer.
# Tutorial: Develop a simple IDT test suite
A test suite combines the following:
+ Test executable that contain the test logic
+ Configuration files that describe the test suite
This tutorial shows you how to use IDT for FreeRTOS to develop a Python test suite that contains a single test case. Although this tutorial uses SSH, it is useful to learn how to use AWS IoT Device Tester with FreeRTOS devices.
In this tutorial, you will complete the following steps:
1. [Create a test suite directory](test-suite-dir.md)
1. [Create configuration files](test-suite-json.md)
1. [Create the test case executable](test-suite-exe.md)
1. [Run the test suite](run-test-suite.md)
Follow the steps below to complete a tutorial for developing a simple IDT test suite.
**Topics**
+ [
# Set up the prerequisites for a simple IDT test suite
](prereqs-tutorial-custom.md)
+ [
# Create a test suite directory
](test-suite-dir.md)
+ [
# Create configuration files
](test-suite-json.md)
+ [
# Get the IDT client SDK
](add-idt-sdk.md)
+ [
# Create the test case executable
](test-suite-exe.md)
+ [
# Configure device information for IDT
](configure-idt-sample2.md)
+ [
# Run the test suite
](run-test-suite.md)
+ [
# Troubleshoot errors
](tutorial-troubleshooting.md)
+ [
# Create IDT test suite configuration files
](idt-json-config.md)
+ [
# Configure the IDT test orchestrator
](idt-test-orchestrator.md)
+ [
# Configure the IDT state machine
](idt-state-machine.md)
+ [
# Create IDT test case executable
](test-executables.md)
+ [
# Use the IDT context
](idt-context.md)
+ [
# Configure settings for test runners
](set-config-custom.md)
+ [
# Debug and run custom test suites
](run-tests-custom.md)
+ [
# Review IDT test results and logs
](idt-review-results-logs.md)
+ [
# Submit IDT usage metrics
](idt-usage-metrics.md)
# Set up the prerequisites for a simple IDT test suite
To complete this tutorial, you need the following:
+
**Host computer requirements**
+ Latest version of AWS IoT Device Tester
+ [Python](https://www.python.org/downloads/) 3.7 or later
To check the version of Python installed on your computer, run the following command:
```
python3 --version
```
On Windows, if using this command returns an error, then use `python --version` instead. If the returned version number is 3.7 or greater, then run the following command in a Powershell terminal to set `python3` as an alias for your `python` command.
```
Set-Alias -Name "python3" -Value "python"
```
If no version information is returned or if the version number is less than 3.7, follow the instructions in [Downloading Python](https://wiki.python.org/moin/BeginnersGuide/Download) to install Python 3.7\$1. For more information, see the [Python documentation](https://docs.python.org/3/).
+ [urllib3](https://urllib3.readthedocs.io/en/latest/)
To verify that `urllib3` is installed correctly, run the following command:
```
python3 -c 'import urllib3'
```
If `urllib3` is not installed, run the following command to install it:
```
python3 -m pip install urllib3
```
+
**Device requirements**
+ A device with a Linux operating system and a network connection to the same network as your host computer.
We recommend that you use a [Raspberry Pi](https://www.raspberrypi.org/) with Raspberry Pi OS. Make sure you set up [SSH](https://www.raspberrypi.com/documentation/computers/remote-access.html) on your Raspberry Pi to remotely connect to it.
# Create a test suite directory
IDT logically separates test cases into test groups within each test suite. Each test case must be inside a test group. For this tutorial, create a folder called `MyTestSuite_1.0.0` and create the following directory tree within this folder:
```
MyTestSuite_1.0.0
└── suite
└── myTestGroup
└── myTestCase
```
# Create configuration files
Your test suite must contain the following required [configuration files](idt-json-config.md):
**Required files**
**`suite.json`**
Contains information about the test suite. See [Configure suite.json](idt-json-config.md#suite-json).
**`group.json`**
Contains information about a test group. You must create a `group.json` file for each test group in your test suite. See [Configure group.json](idt-json-config.md#group-json).
**`test.json`**
Contains information about a test case. You must create a `test.json` file for each test case in your test suite. See [Configure test.json](idt-json-config.md#test-json).
1. In the `MyTestSuite_1.0.0/suite` folder, create a `suite.json` file with the following structure:
```
{
"id": "MyTestSuite_1.0.0",
"title": "My Test Suite",
"details": "This is my test suite.",
"userDataRequired": false
}
```
1. In the `MyTestSuite_1.0.0/myTestGroup` folder, create a `group.json` file with the following structure:
```
{
"id": "MyTestGroup",
"title": "My Test Group",
"details": "This is my test group.",
"optional": false
}
```
1. In the `MyTestSuite_1.0.0/myTestGroup/myTestCase` folder, create a `test.json` file with the following structure:
```
{
"id": "MyTestCase",
"title": "My Test Case",
"details": "This is my test case.",
"execution": {
"timeout": 300000,
"linux": {
"cmd": "python3",
"args": [
"myTestCase.py"
]
},
"mac": {
"cmd": "python3",
"args": [
"myTestCase.py"
]
},
"win": {
"cmd": "python3",
"args": [
"myTestCase.py"
]
}
}
}
```
The directory tree for your `MyTestSuite_1.0.0` folder should now look like the following:
```
MyTestSuite_1.0.0
└── suite
├── suite.json
└── myTestGroup
├── group.json
└── myTestCase
└── test.json
```
# Get the IDT client SDK
You use the [IDT client SDK](test-executables.md#idt-client-sdk) to enable IDT to interact with the device under test and to report test results. For this tutorial, you will use the Python version of the SDK.
From the `/sdks/python/` folder, copy the `idt_client` folder to your `MyTestSuite_1.0.0/suite/myTestGroup/myTestCase` folder.
To verify that the SDK was successfully copied, run the following command.
```
cd MyTestSuite_1.0.0/suite/myTestGroup/myTestCase
python3 -c 'import idt_client'
```
# Create the test case executable
Test case executables contain the test logic that you want to run. A test suite can contain multiple test case executables. For this tutorial, you will create only one test case executable.
1. Create the test suite file.
In the `MyTestSuite_1.0.0/suite/myTestGroup/myTestCase` folder, create a `myTestCase.py` file with the following content:
```
from idt_client import *
def main():
# Use the client SDK to communicate with IDT
client = Client()
if __name__ == "__main__":
main()
```
1. Use client SDK functions to add the following test logic to your `myTestCase.py` file:
1. Run an SSH command on the device under test.
```
from idt_client import *
def main():
# Use the client SDK to communicate with IDT
client = Client()
# Create an execute on device request
exec_req = ExecuteOnDeviceRequest(ExecuteOnDeviceCommand("echo 'hello world'"))
# Run the command
exec_resp = client.execute_on_device(exec_req)
# Print the standard output
print(exec_resp.stdout)
if __name__ == "__main__":
main()
```
1. Send the test result to IDT.
```
from idt_client import *
def main():
# Use the client SDK to communicate with IDT
client = Client()
# Create an execute on device request
exec_req = ExecuteOnDeviceRequest(ExecuteOnDeviceCommand("echo 'hello world'"))
# Run the command
exec_resp = client.execute_on_device(exec_req)
# Print the standard output
print(exec_resp.stdout)
# Create a send result request
sr_req = SendResultRequest(TestResult(passed=True))
# Send the result
client.send_result(sr_req)
if __name__ == "__main__":
main()
```
# Configure device information for IDT
Configure your device information for IDT to run the test. You must update the `device.json` template located in the `/configs` folder with the following information.
```
[
{
"id": "pool",
"sku": "N/A",
"devices": [
{
"id": "",
"connectivity": {
"protocol": "ssh",
"ip": "",
"port": "",
"auth": {
"method": "pki | password",
"credentials": {
"user": "",
"privKeyPath": "/path/to/private/key",
"password": ""
}
}
}
}
]
}
]
```
In the `devices` object, provide the following information:
**`id`**
A user-defined unique identifier for your device.
**`connectivity.ip`**
The IP address of your device.
**`connectivity.port`**
Optional. The port number to use for SSH connections to your device.
**`connectivity.auth`**
Authentication information for the connection.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.auth.method`**
The authentication method used to access a device over the given connectivity protocol.
Supported values are:
+ `pki`
+ `password`
**`connectivity.auth.credentials`**
The credentials used for authentication.
**`connectivity.auth.credentials.user`**
The user name used to sign in to your device.
**`connectivity.auth.credentials.privKeyPath`**
The full path to the private key used to sign in to your device.
This value applies only if `connectivity.auth.method` is set to `pki`.
**`devices.connectivity.auth.credentials.password`**
The password used for signing in to your device.
This value applies only if `connectivity.auth.method` is set to `password`.
**Note**
Specify `privKeyPath` only if `method` is set to `pki`.
Specify `password` only if `method` is set to `password`.
# Run the test suite
After you create your test suite, you want to make sure that it functions as expected. Complete the following steps to run the test suite with your existing device pool to do so.
1. Copy your `MyTestSuite_1.0.0` folder into `/tests`.
1. Run the following commands:
```
cd /bin
./devicetester_[linux | mac | win_x86-64] run-suite --suite-id MyTestSuite
```
IDT runs your test suite and streams the results to the console. When the test has finished running, you see the following information:
```
time="2020-10-19T09:24:47-07:00" level=info msg=Using pool: pool
time="2020-10-19T09:24:47-07:00" level=info msg=Using test suite "MyTestSuite_1.0.0" for execution
time="2020-10-19T09:24:47-07:00" level=info msg=b'hello world\n' suiteId=MyTestSuite groupId=myTestGroup testCaseId=myTestCase deviceId=my-device executionId=9a52f362-1227-11eb-86c9-8c8590419f30
time="2020-10-19T09:24:47-07:00" level=info msg=All tests finished. executionId=9a52f362-1227-11eb-86c9-8c8590419f30
time="2020-10-19T09:24:48-07:00" level=info msg=
========== Test Summary ==========
Execution Time: 1s
Tests Completed: 1
Tests Passed: 1
Tests Failed: 0
Tests Skipped: 0
----------------------------------
Test Groups:
myTestGroup: PASSED
----------------------------------
Path to AWS IoT Device Tester Report: /path/to/devicetester/results/9a52f362-1227-11eb-86c9-8c8590419f30/awsiotdevicetester_report.xml
Path to Test Execution Logs: /path/to/devicetester/results/9a52f362-1227-11eb-86c9-8c8590419f30/logs
Path to Aggregated JUnit Report: /path/to/devicetester/results/9a52f362-1227-11eb-86c9-8c8590419f30/MyTestSuite_Report.xml
```
# Troubleshoot errors
Use the following information to help resolve any issues with completing the tutorial.
**Test case does not run successfully**
If the test does not run successfully, IDT streams the error logs to the console that can help you troubleshoot the test run. Before you check the error logs, verify the following:
+ The IDT client SDK is in the correct folder as described in [Get the IDT client SDK](add-idt-sdk.md).
+ You meet all the prerequisites for this tutorial. For more information, see [Set up the prerequisites for a simple IDT test suite](prereqs-tutorial-custom.md).
**Cannot connect to the device under test**
Verify the following:
+ Your `device.json` file contains the correct IP address, port, and authentication information.
+ You can connect to your device over SSH from your host computer.
# Create IDT test suite configuration files
This section describes the formats in which you create configuration files that you include when you write a custom test suite.
**Required configuration files**
**`suite.json`**
Contains information about the test suite. See [Configure suite.json](#suite-json).
**`group.json`**
Contains information about a test group. You must create a `group.json` file for each test group in your test suite. See [Configure group.json](#group-json).
**`test.json`**
Contains information about a test case. You must create a `test.json` file for each test case in your test suite. See [Configure test.json](#test-json).
**Optional configuration files**
**`test_orchestrator.yaml` or `state_machine.json`**
Defines how tests are run when IDT runs the test suite. SSe [Configure test\$1orchestrator.yaml](#test-orchestrator-config).
Starting in IDT v4.5.2, you use the `test_orchestrator.yaml` file to define the test workflow. In previous versions of IDT, you use the `state_machine.json` file. For information about the state machine, see [Configure the IDT state machine](idt-state-machine.md).
**`userdata_schema.json`**
Defines the schema for the [`userdata.json` file](set-config-custom.md#userdata-config-custom) that test runners can include in their setting configuration. The `userdata.json` file is used for any additional configuration information that is required to run the test but is not present in the `device.json` file. See [Configure userdata\$1schema.json](#userdata-schema-json).
Configuration files are placed in your `` as shown here.
```
└── suite
├── suite.json
├── test_orchestrator.yaml
├── userdata_schema.json
├──
├── group.json
├──
└── test.json
```
## Configure suite.json
The `suite.json` file sets environment variables and determines whether user data is required to run the test suite. Use the following template to configure your `/suite/suite.json` file:
```
{
"id": "_",
"title": "",
"details": "",
"userDataRequired": true | false,
"environmentVariables": [
{
"key": "",
"value": "",
},
...
{
"key": "",
"value": "",
}
]
}
```
All fields that contain values are required as described here:
**`id`**
A unique user-defined ID for the test suite. The value of `id` must match the name of the test suite folder in which the `suite.json` file is located. The suite name and suite version must also meet the following requirements:
+ `` cannot contain underscores.
+ `` is denoted as `x.x.x`, where `x` is a number.
The ID is shown in IDT-generated test reports.
**`title`**
A user-defined name for the product or feature being tested by this test suite. The name is displayed in the IDT CLI for test runners.
**`details`**
A short description of the purpose of the test suite.
**`userDataRequired`**
Defines whether test runners need to include custom information in a `userdata.json` file. If you set this value to `true`, you must also include the [`userdata_schema.json` file](#userdata-schema-json) in your test suite folder.
**`environmentVariables`**
Optional. An array of environment variables to set for this test suite.
**`environmentVariables.key`**
The name of the environment variable.
**`environmentVariables.value`**
The value of the environment variable.
## Configure group.json
The `group.json` file defines whether a test group is required or optional. Use the following template to configure your `/suite//group.json` file:
```
{
"id": "",
"title": "",
"details": "",
"optional": true | false,
}
```
All fields that contain values are required as described here:
**`id`**
A unique user-defined ID for the test group. The value of `id` must match the name of the test group folder in which the `group.json` file is located and should not have underscores (`_`). The ID is used in IDT-generated test reports.
**`title`**
A descriptive name for the test group. The name is displayed in the IDT CLI for test runners.
**`details`**
A short description of the purpose of the test group.
**`optional`**
Optional. Set to `true` to display this test group as an optional group after IDT finishes running required tests. Default value is `false`.
## Configure test.json
The `test.json` file determines the test case executables and the environment variables that are used by a test case. For more information about creating test case executables, see [Create IDT test case executable](test-executables.md).
Use the following template to configure your `/suite///test.json` file:
```
{
"id": "",
"title": "",
"details": "",
"requireDUT": true | false,
"requiredResources": [
{
"name": "",
"features": [
{
"name": "",
"version": "",
"jobSlots":
}
]
}
],
"execution": {
"timeout": ,
"mac": {
"cmd": "/path/to/executable",
"args": [
""
],
},
"linux": {
"cmd": "/path/to/executable",
"args": [
""
],
},
"win": {
"cmd": "/path/to/executable",
"args": [
""
]
}
},
"environmentVariables": [
{
"key": "",
"value": "",
}
]
}
```
All fields that contain values are required as described here:
**`id`**
A unique user-defined ID for the test case. The value of `id` must match the name of the test case folder in which the `test.json` file is located and should not have underscores (`_`). The ID is used in IDT-generated test reports.
**`title`**
A descriptive name for the test case. The name is displayed in the IDT CLI for test runners.
**`details`**
A short description of the purpose of the test case.
**`requireDUT`**
Optional. Set to `true` if a device is required to run this test, otherwise set to `false`. Default value is `true`. Test runners will configure the devices they will use to run the test in their `device.json` file.
**`requiredResources`**
Optional. An array that provides information about resource devices needed to run this test.
**`requiredResources.name`**
The unique name to give the resource device when this test is running.
**`requiredResources.features`**
An array of user-defined resource device features.
**`requiredResources.features.name`**
The name of the feature. The device feature for which you want to use this device. This name is matched against the feature name provided by the test runner in the `resource.json` file.
**`requiredResources.features.version`**
Optional. The version of the feature. This value is matched against the feature version provided by the test runner in the `resource.json` file. If a version is not provided, then the feature is not checked. If a version number is not required for the feature, leave this field blank.
**`requiredResources.features.jobSlots`**
Optional. The number of simultaneous tests that this feature can support. The default value is `1`. If you want IDT to use distinct devices for individual features, then we recommend that you set this value to `1`.
**`execution.timeout`**
The amount of time (in milliseconds) that IDT waits for the test to finish running. For more information about setting this value, see [Create IDT test case executable](test-executables.md).
**`execution.os`**
The test case executables to run based on the operating system of the host computer that runs IDT. Supported values are `linux`, `mac`, and `win`.
**`execution.os.cmd`**
The path to the test case executable that you want to run for the specified operating system. This location must be in the system path.
**`execution.os.args`**
Optional. The arguments to provide to run the test case executable.
**`environmentVariables`**
Optional. An array of environment variables set for this test case.
**`environmentVariables.key`**
The name of the environment variable.
**`environmentVariables.value`**
The value of the environment variable.
If you specify the same environment variable in the `test.json` file and in the `suite.json` file, the value in the `test.json` file takes precedence.
## Configure test\$1orchestrator.yaml
A test orchestrator is a construct that controls the test suite execution flow. It determines the starting state of a test suite, manages state transitions based on user-defined rules, and continues to transition through those states until it reaches the end state.
If your test suite doesn't include a user-defined test orchestrator, IDT will generate a test orchestrator for you.
The default test orchestrator performs the following functions:
+ Provides test runners with the ability to select and run specific test groups, instead of the entire test suite.
+ If specific test groups are not selected, runs every test group in the test suite in a random order.
+ Generates reports and prints a console summary that shows the test results for each test group and test case.
For more information about how the IDT test orchestrator functions, see [Configure the IDT test orchestrator](idt-test-orchestrator.md).
## Configure userdata\$1schema.json
The `userdata_schema.json` file determines the schema in which test runners provide user data. User data is required if your test suite requires information that is not present in the `device.json` file. For example, your tests might need Wi-Fi network credentials, specific open ports, or certificates that a user must provide. This information can be provided to IDT as an input parameter called `userdata`, the value for which is a `userdata.json` file, that users create in their `/config` folder. The format of the `userdata.json` file is based on the `userdata_schema.json` file that you include in the test suite.
To indicate that test runners must provide a `userdata.json` file:
1. In the `suite.json` file, set `userDataRequired` to `true`.
1. In your ``, create a `userdata_schema.json` file.
1. Edit the `userdata_schema.json` file to create a valid [IETF Draft v4 JSON Schema](https://json-schema.org/specification-links#draft-4).
When IDT runs your test suite, it automatically reads the schema and uses it to validate the `userdata.json` file provided by the test runner. If valid, the contents of the `userdata.json` file are available in both the [IDT context](idt-context.md) and in the [test orchestrator context](idt-test-orchestrator.md#idt-test-orchestrator-context).
# Configure the IDT test orchestrator
Starting in IDT v4.5.2, IDT includes a new *test orchestrator* component. The test orchestrator is an IDT component that controls the test suite execution flow, and generates the test report after IDT finishes running all tests. The test orchestrator determines test selection and the order in which tests are run based on user-defined rules.
If your test suite doesn't include a user-defined test orchestrator, IDT will generate a test orchestrator for you.
The default test orchestrator performs the following functions:
+ Provides test runners with the ability to select and run specific test groups, instead of the entire test suite.
+ If specific test groups are not selected, runs every test group in the test suite in a random order.
+ Generates reports and prints a console summary that shows the test results for each test group and test case.
The test orchestrator replaces the IDT state machine. We strongly recommend that you use the test orchestrator to develop your test suites instead of the IDT state machine. The test orchestrator provides the following improved features:
+ Uses a declarative format compared to the imperative format that the IDT state machine uses. This allows you to specify which tests you want to run and when you want to run them.
+ Manages specific group handling, report generation, error handling, and result tracking so that you aren't required to manually manage these actions.
+ Uses the YAML format, which supports comments by default.
+ Requires 80 percent less disk space than the test orchestrator to define the same workflow.
+ Adds pre-test validation to verify that your workflow definition doesn't contain incorrect test IDs or circular dependencies.
## Test orchestrator format
You can use the following template to configure your own `custom-test-suite-folder/suite/test_orchestrator.yaml` file:
```
Aliases:
string: context-expression
ConditionalTests:
- Condition: context-expression
Tests:
- test-descriptor
Order:
- - group-descriptor
- group-descriptor
Features:
- Name: feature-name
Value: support-description
Condition: context-expression
Tests:
- test-descriptor
OneOfTests:
- test-descriptor
IsRequired: boolean
```
All fields that contain values are required as described here:
`Aliases`
Optional. User-defined strings that map to context expressions. Aliases allow you to generate friendly names to identify context expressions in your test orchestrator configuration. This is especially useful if you're creating complex context expressions or expressions that you use in multiple places.
You can use context expressions to store context queries that allow you to access data from other IDT configurations. For more information, see [Access data in the context](idt-context.md#accessing-context-data).
**Example**
**Example**
```
Aliases:
FizzChosen: "'{{$pool.features[?(@.name == 'Fizz')].value[0]}}' == 'yes'"
BuzzChosen: "'{{$pool.features[?(@.name == 'Buzz')].value[0]}}' == 'yes'"
FizzBuzzChosen: "'{{$aliases.FizzChosen}}' && '{{$aliases.BuzzChosen}}'"
```
`ConditionalTests`
Optional. A list of conditions, and the corresponding test cases that are run when each condition is satisfied. Each condition can have multiple test cases; however, you can assign a given test case to only one condition.
By default, IDT runs any test case that isn't assigned to a condition in this list. If you don't specify this section, IDT runs all test groups in the test suite.
Each item in the `ConditionalTests` list includes the following parameters:
`Condition`
A context expression that evaluates to a Boolean value. If the evaluated value is true, IDT runs the test cases that are specified in the `Tests` parameter.
`Tests`
The list of test descriptors.
Each test descriptor uses the test group ID and one or more test case IDs to identify the individual tests to run from a specific test group. The test descriptor uses the following format:
```
GroupId: group-id
CaseIds: [test-id, test-id] # optional
```
**Example**
**Example**
The following example uses generic context expressions that you can define as `Aliases`.
```
ConditionalTests:
- Condition: "{{$aliases.Condition1}}"
Tests:
- GroupId: A
- GroupId: B
- Condition: "{{$aliases.Condition2}}"
Tests:
- GroupId: D
- Condition: "{{$aliases.Condition1}} || {{$aliases.Condition2}}"
Tests:
- GroupId: C
```
Based on the defined conditions, IDT selects test groups as follows:
+ If `Condition1` is true, IDT runs the tests in test groups A, B, and C.
+ If `Condition2` is true, IDT runs the tests in test groups C and D.
`Order`
Optional. The order in which to run tests. You specify the test order at the test group level. If you don't specify this section, IDT runs all applicable test groups in a random order. The value of `Order` is a list of group descriptor lists. Any test group that you don't list in `Order`, can be run in parallel with any other listed test group.
Each group descriptor list contains one of more group descriptors, and identifies the order in which to run the groups that are specified in each descriptor. You can use the following formats to define individual group descriptors:
+ `group-id`—The group ID of an existing test group.
+ `[group-id, group-id]`—List of test groups that can be run in any order relative to each other.
+ `"*"`—Wildcard. This is equivalent to the list of all test groups that are not already specified in the current group descriptor list.
The value for `Order` must also meet the following requirements:
+ Test group IDs that you specify in a group descriptor must exist in your test suite.
+ Each group descriptor list must include at least one test group.
+ Each group descriptor list must contain unique group IDs. You cannot repeat a test group ID within individual group descriptors.
+ A group descriptor list can have at most one wildcard group descriptor. The wildcard group descriptor must be the first or the last item in the list.
**Example**
**Example**
For a test suite that contains test groups A, B, C, D, and E, the following list of examples shows different ways to specify that IDT should first run test group A, then run test group B, and then run test groups C, D, and E in any order.
+
```
Order:
- - A
- B
- [C, D, E]
```
+
```
Order:
- - A
- B
- "*"
```
+
```
Order:
- - A
- B
- - B
- C
- - B
- D
- - B
- E
```
`Features`
Optional. The list of product features that you want IDT to add to the `awsiotdevicetester_report.xml` file. If you don't specify this section, IDT won't add any product features to the report.
A product feature is user-defined information about specific criteria that a device might meet. For example, the MQTT product feature can designate that the device publishes MQTT messages properly. In `awsiotdevicetester_report.xml`, product features are set as `supported`, `not-supported`, or a custom user-defined value, based on whether specified tests passed.
Each item in the `Features` list consists of the following parameters:
`Name`
The name of the feature.
`Value`
Optional. The custom value that you want to use in the report instead of `supported`. If this value is not specified, then based IDT sets the feature value to `supported` or `not-supported` based on test results. If you test the same feature with different conditions, you can use a custom value for each instance of that feature in the `Features` list, and IDT concatenates the feature values for supported conditions. For more information, see
`Condition`
A context expression that evaluates to a Boolean value. If the evaluated value is true, IDT adds the feature to the test report after it finishes running the test suite. If the evaluated value is false, the test is not included in the report.
`Tests`
Optional. The list of test descriptors. All of the tests that are specified in this list must pass for the feature to be supported.
Each test descriptor in this list uses the test group ID and one or more test case IDs to identify the individual tests to run from a specific test group. The test descriptor uses the following format:
```
GroupId: group-id
CaseIds: [test-id, test-id] # optional
```
You must specify either `Tests` or `OneOfTests` for each feature in the `Features` list.
`OneOfTests`
Optional. The list of test descriptors. At least one of the tests that are specified in this list must pass for the feature to be supported.
Each test descriptor in this list uses the test group ID and one or more test case IDs to identify the individual tests to run from a specific test group. The test descriptor uses the following format:
```
GroupId: group-id
CaseIds: [test-id, test-id] # optional
```
You must specify either `Tests` or `OneOfTests` for each feature in the `Features` list.
`IsRequired`
The boolean value that defines whether the feature is required in the test report. The default value is `false`.
## Test orchestrator context
The test orchestrator context is a read-only JSON document that contains data that is available to the test orchestrator during execution. The test orchestrator context is accessible only from the test orchestrator, and contains information that determines the test flow. For example, you can use information configured by test runners in the `userdata.json` file to determine whether a specific test is required to run.
The test orchestrator context uses the following format:
```
{
"pool": {
},
"userData": {
},
"config": {
}
}
```
`pool`
Information about the device pool selected for the test run. For a selected device pool, this information is retrieved from the corresponding top-level device pool array element defined in the `device.json` file.
`userData`
Information in the `userdata.json` file.
`config`
Information in the `config.json` file.
You can query the context using JSONPath notation. The syntax for JSONPath queries in state definitions is `{{query}}`. When you access data from the test orchestrator context, make sure that each value evaluates to a string, a number, or a Boolean.
For more information about using JSONPath notation to access data from the context, see [Use the IDT context](idt-context.md).
# Configure the IDT state machine
**Important**
Starting in IDT v4.5.2, this state machine is deprecated. We strongly recommend that you use the new test orchestrator. For more information, see [Configure the IDT test orchestrator](idt-test-orchestrator.md).
A state machine is a construct that controls the test suite execution flow. It determines the starting state of a test suite, manages state transitions based on user-defined rules, and continues to transition through those states until it reaches the end state.
If your test suite doesn't include a user-defined state machine, IDT will generate a state machine for you. The default state machine performs the following functions:
+ Provides test runners with the ability to select and run specific test groups, instead of the entire test suite.
+ If specific test groups are not selected, runs every test group in the test suite in a random order.
+ Generates reports and prints a console summary that shows the test results for each test group and test case.
The state machine for an IDT test suite must meet the following criteria:
+ Each state corresponds to an action for IDT to take, such as to run a test group or product a report file.
+ Transitioning to a state executes the action associated with the state.
+ Each state defines the transition rule for the next state.
+ The end state must be either `Succeed` or `Fail`.
## State machine format
You can use the following template to configure your own `/suite/state_machine.json` file:
```
{
"Comment": "",
"StartAt": "",
"States": {
"": {
"Type": "",
// Additional state configuration
}
// Required states
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
}
```
All fields that contain values are required as described here:
**`Comment`**
A description of the state machine.
**`StartAt`**
The name of the state at which IDT starts running the test suite. The value of `StartAt` must be set to one of the states listed in the `States` object.
**`States`**
An object that maps user-defined state names to valid IDT states. Each States.*state-name* object contains the definition of a valid state mapped to the *state-name*.
The `States` object must include the `Succeed` and `Fail` states. For information about valid states, see [Valid states and state definitions](#valid-states).
## Valid states and state definitions
This section describes the state definitions of all of the valid states that can be used in the IDT state machine. Some of the following states support configurations at the test case level. However, we recommend that you configure state transition rules at the test group level instead of the test case level unless absolutely necessary.
**Topics**
+ [
### RunTask
](#state-runtask)
+ [
### Choice
](#state-choice)
+ [
### Parallel
](#state-parallel)
+ [
### AddProductFeatures
](#state-addproductfeatures)
+ [
### Report
](#state-report)
+ [
### LogMessage
](#state-logmessage)
+ [
### SelectGroup
](#state-selectgroup)
+ [
### Fail
](#state-fail)
+ [
### Succeed
](#state-succeed)
### RunTask
The `RunTask` state runs test cases from a test group defined in the test suite.
```
{
"Type": "RunTask",
"Next": "",
"TestGroup": "",
"TestCases": [
""
],
"ResultVar": ""
}
```
All fields that contain values are required as described here:
**`Next`**
The name of the state to transition to after executing the actions in the current state.
**`TestGroup`**
Optional. The ID of the test group to run. If this value is not specified, then IDT runs the test group that the test runner selects.
**`TestCases`**
Optional. An array of test case IDs from the group specified in `TestGroup`. Based on the values of `TestGroup` and `TestCases`, IDT determines the test execution behavior as follows:
+ When both `TestGroup` and `TestCases` are specified, IDT runs the specified test cases from the test group.
+ When `TestCases` are specified but `TestGroup` is not specified, IDT runs the specified test cases.
+ When `TestGroup` is specified, but `TestCases` is not specified, IDT runs all of the test cases within the specified test group.
+ When neither `TestGroup` or `TestCases` is specified, IDT runs all test cases from the test group that the test runner selects from the IDT CLI. To enable group selection for test runners, you must include both `RunTask` and `Choice` states in your `statemachine.json` file. For an example of how this works, see [Example state machine: Run user-selected test groups](#allow-specific-groups).
For more information about enabling IDT CLI commands for test runners, see [Enable IDT CLI commands](test-executables.md#idt-cli-coop).
**`ResultVar`**
The name of the context variable to set with the results of the test run. Do not specify this value if you did not specify a value for `TestGroup`. IDT sets the value of the variable that you define in `ResultVar` to `true` or `false` based on the following:
+ If the variable name is of the form `text_text_passed`, then the value is set to whether all tests in the first test group passed or were skipped.
+ In all other cases, the value is set to whether all tests in all test groups passed or were skipped.
Typically, you will use `RunTask` state to specify a test group ID without specifying individual test case IDs, so that IDT will run all of the test cases in the specified test group. All test cases that are run by this state run in parallel, in a random order. However, if all of the test cases require a device to run, and only a single device is available, then the test cases will run sequentially instead.
**Error handling**
If any of the specified test groups or test case IDs are not valid, then this state issues the `RunTaskError` execution error. If the state encounters an execution error, then it also sets the `hasExecutionError` variable in the state machine context to `true`.
### Choice
The `Choice` state lets you dynamically set the next state to transition to based on user-defined conditions.
```
{
"Type": "Choice",
"Default": "",
"FallthroughOnError": true | false,
"Choices": [
{
"Expression": "",
"Next": ""
}
]
}
```
All fields that contain values are required as described here:
**`Default`**
The default state to transition to if none of the expressions defined in `Choices` can be evaluated to `true`.
**`FallthroughOnError`**
Optional. Specifies the behavior when the state encounters an error in evaluating expressions. Set to `true` if you want to skip an expression if the evaluation results in an error. If no expressions match, then the state machine transitions to the `Default` state. If the `FallthroughOnError` value is not specified, it defaults to `false`.
**`Choices`**
An array of expressions and states to determine which state to transition to after executing the actions in the current state.
**`Choices.Expression`**
An expression string that evaluates to a boolean value. If the expression evaluates to `true`, then the state machine transitions to the state defined in `Choices.Next`. Expression strings retrieve values from the state machine context and then perform operations on them to arrive at a boolean value. For information about accessing the state machine context, see [State machine context](#state-machine-context).
**`Choices.Next`**
The name of the state to transition to if the expression defined in `Choices.Expression` evaluates to `true`.
**Error handling**
The `Choice` state can require error handling in the following cases:
+ Some variables in the choice expressions don’t exist in the state machine context.
+ The result of an expression is not a boolean value.
+ The result of a JSON lookup is not a string, number, or boolean.
You cannot use a `Catch` block to handle errors in this state. If you want to stop executing the state machine when it encounters an error, you must set `FallthroughOnError` to `false`. However, we recommend that you set `FallthroughOnError` to `true`, and depending on your use case, do one of the following:
+ If a variable you are accessing is expected to not exist in some cases, then use the value of `Default` and additional `Choices` blocks to specify the next state.
+ If a variable that you are accessing should always exist, then set the `Default` state to `Fail`.
### Parallel
The `Parallel` state lets you define and run new state machines in parallel with each other.
```
{
"Type": "Parallel",
"Next": "",
"Branches": [
]
}
```
All fields that contain values are required as described here:
**`Next`**
The name of the state to transition to after executing the actions in the current state.
**`Branches`**
An array of state machine definitions to run. Each state machine definition must contain its own `StartAt`, `Succeed`, and `Fail` states. The state machine definitions in this array cannot reference states outside of their own definition.
Because each branch state machine shares the same state machine context, setting variables in one branch and then reading those variables from another branch might result in unexpected behavior.
The `Parallel` state moves to the next state only after it runs all of the branch state machines. Each state that requires a device will wait to run until the device is available. If multiple devices are available, this state runs test cases from multiple groups in parallel. If enough devices are not available, then test cases will run sequentially. Because test cases are run in a random order when they run in parallel, different devices might be used to run tests from the same test group.
**Error handling**
Make sure that both the branch state machine and the parent state machine transition to the `Fail` state to handle execution errors.
Because branch state machines do not transmit execution errors to the parent state machine, you cannot use a `Catch` block to handle execution errors in branch state machines. Instead, use the `hasExecutionErrors` value in the shared state machine context. For an example of how this works, see [Example state machine: Run two test groups in parallel](#run-in-parallel).
### AddProductFeatures
The `AddProductFeatures` state lets you add product features to the `awsiotdevicetester_report.xml` file generated by IDT.
A product feature is user-defined information about specific criteria that a device might meet. For example, the `MQTT` product feature can designate that the device publishes MQTT messages properly. In the report, product features are set as `supported`, `not-supported`, or a custom value, based on whether specified tests passed.
**Note**
The `AddProductFeatures` state does not generate reports by itself. This state must transition to the [`Report` state](#state-report) to generate reports.
```
{
"Type": "Parallel",
"Next": "",
"Features": [
{
"Feature": "",
"Groups": [
""
],
"OneOfGroups": [
""
],
"TestCases": [
""
],
"IsRequired": true | false,
"ExecutionMethods": [
""
]
}
]
}
```
All fields that contain values are required as described here:
**`Next`**
The name of the state to transition to after executing the actions in the current state.
**`Features`**
An array of product features to show in the `awsiotdevicetester_report.xml` file.
**`Feature`**
The name of the feature
**`FeatureValue`**
Optional. The custom value to use in the report instead of `supported`. If this value is not specified, then based on test results, the feature value is set to `supported` or `not-supported`.
If you use a custom value for `FeatureValue`, you can test the same feature with different conditions, and IDT concatenates the feature values for the supported conditions. For example, the following excerpt shows the `MyFeature` feature with two separate feature values:
```
...
{
"Feature": "MyFeature",
"FeatureValue": "first-feature-supported",
"Groups": ["first-feature-group"]
},
{
"Feature": "MyFeature",
"FeatureValue": "second-feature-supported",
"Groups": ["second-feature-group"]
},
...
```
If both test groups pass, then the feature value is set to `first-feature-supported, second-feature-supported`.
**`Groups`**
Optional. An array of test group IDs. All tests within each specified test group must pass for the feature to be supported.
**`OneOfGroups`**
Optional. An array of test group IDs. All tests within at least one of the specified test groups must pass for the feature to be supported.
**`TestCases`**
Optional. An array of test case IDs. If you specify this value, then the following apply:
+ All of the specified test cases must pass for the feature to be supported.
+ `Groups` must contain only one test group ID.
+ `OneOfGroups` must not be specified.
**`IsRequired`**
Optional. Set to `false` to mark this feature as an optional feature in the report. The default value is `true`.
**`ExecutionMethods`**
Optional. An array of execution methods that match the `protocol` value specified in the `device.json` file. If this value is specified, then test runners must specify a `protocol` value that matches one of the values in this array to include the feature in the report. If this value is not specified, then the feature will always be included in the report.
To use the `AddProductFeatures` state, you must set the value of `ResultVar` in the `RunTask` state to one of the following values:
+ If you specified individual test case IDs, then set `ResultVar` to `group-id_test-id_passed`.
+ If you did not specify individual test case IDs, then set `ResultVar` to `group-id_passed`.
The `AddProductFeatures` state checks for test results in the following manner:
+ If you did not specify any test case IDs, then the result for each test group is determined from the value of the `group-id_passed` variable in the state machine context.
+ If you did specify test case IDs, then the result for each of the tests is determined from the value of the `group-id_test-id_passed` variable in the state machine context.
**Error handling**
If a group ID provided in this state is not a valid group ID, then this state results in the `AddProductFeaturesError` execution error. If the state encounters an execution error, then it also sets the `hasExecutionErrors` variable in the state machine context to `true`.
### Report
The `Report` state generates the `suite-name_Report.xml` and `awsiotdevicetester_report.xml` files. This state also streams the report to the console.
```
{
"Type": "Report",
"Next": ""
}
```
All fields that contain values are required as described here:
**`Next`**
The name of the state to transition to after executing the actions in the current state.
You should always transition to the `Report` state towards the end of the test execution flow so that test runners can view test results. Typically, the next state after this state is `Succeed`.
**Error handling**
If this state encounters issues with generating the reports, then it issues the `ReportError` execution error.
### LogMessage
The `LogMessage` state generates the `test_manager.log` file and streams the log message to the console.
```
{
"Type": "LogMessage",
"Next": ""
"Level": "info | warn | error"
"Message": ""
}
```
All fields that contain values are required as described here:
**`Next`**
The name of the state to transition to after executing the actions in the current state.
**`Level`**
The error level at which to create the log message. If you specify a level that is not valid, this state generates an error message and discards it.
**`Message`**
The message to log.
### SelectGroup
The `SelectGroup` state updates the state machine context to indicate which groups are selected. The values set by this state are used by any subsequent `Choice` states.
```
{
"Type": "SelectGroup",
"Next": ""
"TestGroups": [
"
]
}
```
All fields that contain values are required as described here:
**`Next`**
The name of the state to transition to after executing the actions in the current state.
**`TestGroups`**
An array of test groups that will be marked as selected. For each test group ID in this array, the `group-id_selected` variable is set to `true` in the context. Make sure that you provide valid test group IDs because IDT does not validate whether the specified groups exist.
### Fail
The `Fail` state indicates that the state machine did not execute correctly. This is an end state for the state machine, and each state machine definition must include this state.
```
{
"Type": "Fail"
}
```
### Succeed
The `Succeed` state indicates that the state machine executed correctly. This is an end state for the state machine, and each state machine definition must include this state.
```
{
"Type": "Succeed"
}
```
## State machine context
The state machine context is a read-only JSON document that contains data that is available to the state machine during execution. The state machine context is accessible only from the state machine, and contains information that determines the test flow. For example, you can use information configured by test runners in the `userdata.json` file to determine whether a specific test is required to run.
The state machine context uses the following format:
```
{
"pool": {
},
"userData": {
},
"config": {
},
"suiteFailed": true | false,
"specificTestGroups": [
""
],
"specificTestCases": [
""
],
"hasExecutionErrors": true
}
```
**`pool`**
Information about the device pool selected for the test run. For a selected device pool, this information is retrieved from the corresponding top-level device pool array element defined in the `device.json` file.
**`userData`**
Information in the `userdata.json` file.
**`config`**
Information pin the `config.json` file.
**`suiteFailed`**
The value is set to `false` when the state machine starts. If a test group fails in a `RunTask` state, then this value is set to `true` for the remaining duration of the state machine execution.
**`specificTestGroups`**
If the test runner selects specific test groups to run instead of the entire test suite, this key is created and contains the list of specific test group IDs.
**`specificTestCases`**
If the test runner selects specific test cases to run instead of the entire test suite, this key is created and contains the list of specific test case IDs.
**`hasExecutionErrors`**
Does not exit when the state machine starts. If any state encounters an execution errors, this variable is created and set to `true` for the remaining duration of the state machine execution.
You can query the context using JSONPath notation. The syntax for JSONPath queries in state definitions is `{{$.query}}`. You can use JSONPath queries as placeholder strings within some states. IDT replaces the placeholder strings with the value of the evaluated JSONPath query from the context. You can use placeholders for the following values:
+ The `TestCases` value in `RunTask` states.
+ The `Expression` value `Choice` state.
When you access data from the state machine context, make sure the following conditions are met:
+ Your JSON paths must begin with `$.`
+ Each value must evaluate to a string, a number, or a boolean.
For more information about using JSONPath notation to access data from the context, see [Use the IDT context](idt-context.md).
## Execution errors
Execution errors are errors in the state machine definition that the state machine encounters when executing a state. IDT logs information about each error in the `test_manager.log` file and streams the log message to the console.
You can use the following methods to handle execution errors:
+ Add a [`Catch` block](#catch) in the state definition.
+ Check the value of the [`hasExecutionErrors` value](#context) in the state machine context.
### Catch
To use `Catch`, add the following to your state definition:
```
"Catch": [
{
"ErrorEquals": [
""
]
"Next": ""
}
]
```
All fields that contain values are required as described here:
**`Catch.ErrorEquals`**
An array of the error types to catch. If an execution error matches one of the specified values, then the state machine transitions to the state specified in `Catch.Next`. See each state definition for information about the type of error it produces.
**`Catch.Next`**
The next state to transition to if the current state encounters an execution error that matches one of the values specified in `Catch.ErrorEquals` .
Catch blocks are handled sequentially until one matches. If the no errors match the ones listed in the Catch blocks, then the state machines continues to execute. Because execution errors are a result of incorrect state definitions, we recommend that you transition to the Fail state when a state encounters an execution error.
### hasExecutionError
When some states encounter execution errors, in addition to issuing the error, they also set the `hasExecutionError` value to `true` in the state machine context. You can use this value to detect when an error occurs, and then use a `Choice` state to transition the state machine to the `Fail` state.
This method has the following characteristics.
+ The state machine does not start with any value assigned to `hasExecutionError`, and this value is not available until a particular state sets it. This means that you must explicitly set the `FallthroughOnError` to `false` for the `Choice` states that access this value to prevent the state machine from stopping if no execution errors occur.
+ Once it is set to `true`, `hasExecutionError` is never set to false or removed from the context. This means that this value is useful only the first time that it is set to `true`, and for all subsequent states, it does not provide a meaningful value.
+ The `hasExecutionError` value is shared with all branch state machines in the `Parallel` state, which can result in unexpected results depending on the order in which it is accessed.
Because of these characteristics, we do not recommend that you use this method if you can use a Catch block instead.
## Example state machines
This section provides some example state machine configurations.
**Topics**
+ [
### Example state machine: Run a single test group
](#single-test-group)
+ [
### Example state machine: Run user-selected test groups
](#allow-specific-groups)
+ [
### Example state machine: Run a single test group with product features
](#run-with-product-features)
+ [
### Example state machine: Run two test groups in parallel
](#run-in-parallel)
### Example state machine: Run a single test group
This state machine:
+ Runs the test group with id `GroupA`, which must be present in the suite in a `group.json` file.
+ Checks for execution errors and transitions to `Fail` if any are found.
+ Generates a report and transitions to `Succeed` if there are no errors, and `Fail` otherwise.
```
{
"Comment": "Runs a single group and then generates a report.",
"StartAt": "RunGroupA",
"States": {
"RunGroupA": {
"Type": "RunTask",
"Next": "Report",
"TestGroup": "GroupA",
"Catch": [
{
"ErrorEquals": [
"RunTaskError"
],
"Next": "Fail"
}
]
},
"Report": {
"Type": "Report",
"Next": "Succeed",
"Catch": [
{
"ErrorEquals": [
"ReportError"
],
"Next": "Fail"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
}
```
### Example state machine: Run user-selected test groups
This state machine:
+ Checks if the test runner selected specific test groups. The state machine does not check for specific test cases because test runners cannot select test cases without also selecting a test group.
+ If test groups are selected:
+ Runs the test cases within the selected test groups. To do so, the state machine does not explicitly specify any test groups or test cases in the `RunTask` state.
+ Generates a report after running all tests and exits.
+ If test groups are not selected:
+ Runs tests in test group `GroupA`.
+ Generates reports and exits.
```
{
"Comment": "Runs specific groups if the test runner chose to do that, otherwise runs GroupA.",
"StartAt": "SpecificGroupsCheck",
"States": {
"SpecificGroupsCheck": {
"Type": "Choice",
"Default": "RunGroupA",
"FallthroughOnError": true,
"Choices": [
{
"Expression": "{{$.specificTestGroups[0]}} != ''",
"Next": "RunSpecificGroups"
}
]
},
"RunSpecificGroups": {
"Type": "RunTask",
"Next": "Report",
"Catch": [
{
"ErrorEquals": [
"RunTaskError"
],
"Next": "Fail"
}
]
},
"RunGroupA": {
"Type": "RunTask",
"Next": "Report",
"TestGroup": "GroupA",
"Catch": [
{
"ErrorEquals": [
"RunTaskError"
],
"Next": "Fail"
}
]
},
"Report": {
"Type": "Report",
"Next": "Succeed",
"Catch": [
{
"ErrorEquals": [
"ReportError"
],
"Next": "Fail"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
}
```
### Example state machine: Run a single test group with product features
This state machine:
+ Runs the test group `GroupA`.
+ Checks for execution errors and transitions to `Fail` if any are found.
+ Adds the `FeatureThatDependsOnGroupA` feature to the `awsiotdevicetester_report.xml` file:
+ If `GroupA` passes, the feature is set to `supported`.
+ The feature is not marked optional in the report.
+ Generates a report and transitions to `Succeed` if there are no errors, and `Fail` otherwise
```
{
"Comment": "Runs GroupA and adds product features based on GroupA",
"StartAt": "RunGroupA",
"States": {
"RunGroupA": {
"Type": "RunTask",
"Next": "AddProductFeatures",
"TestGroup": "GroupA",
"ResultVar": "GroupA_passed",
"Catch": [
{
"ErrorEquals": [
"RunTaskError"
],
"Next": "Fail"
}
]
},
"AddProductFeatures": {
"Type": "AddProductFeatures",
"Next": "Report",
"Features": [
{
"Feature": "FeatureThatDependsOnGroupA",
"Groups": [
"GroupA"
],
"IsRequired": true
}
]
},
"Report": {
"Type": "Report",
"Next": "Succeed",
"Catch": [
{
"ErrorEquals": [
"ReportError"
],
"Next": "Fail"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
}
```
### Example state machine: Run two test groups in parallel
This state machine:
+ Runs the `GroupA` and `GroupB` test groups in parallel. The `ResultVar` variables stored in the context by the `RunTask` states in the branch state machines by are available to the `AddProductFeatures` state.
+ Checks for execution errors and transitions to `Fail` if any are found. This state machine does not use a `Catch` block because that method does not detect execution errors in branch state machines.
+ Adds features to the `awsiotdevicetester_report.xml` file based on the groups that pass
+ If `GroupA` passes, the feature is set to `supported`.
+ The feature is not marked optional in the report.
+ Generates a report and transitions to `Succeed` if there are no errors, and `Fail` otherwise
If two devices are configured in the device pool, both `GroupA` and `GroupB` can run at the same time. However, if either `GroupA` or `GroupB` has multiple tests in it, then both devices may be allocated to those tests. If only one device is configured, the test groups will run sequentially.
```
{
"Comment": "Runs GroupA and GroupB in parallel",
"StartAt": "RunGroupAAndB",
"States": {
"RunGroupAAndB": {
"Type": "Parallel",
"Next": "CheckForErrors",
"Branches": [
{
"Comment": "Run GroupA state machine",
"StartAt": "RunGroupA",
"States": {
"RunGroupA": {
"Type": "RunTask",
"Next": "Succeed",
"TestGroup": "GroupA",
"ResultVar": "GroupA_passed",
"Catch": [
{
"ErrorEquals": [
"RunTaskError"
],
"Next": "Fail"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
},
{
"Comment": "Run GroupB state machine",
"StartAt": "RunGroupB",
"States": {
"RunGroupA": {
"Type": "RunTask",
"Next": "Succeed",
"TestGroup": "GroupB",
"ResultVar": "GroupB_passed",
"Catch": [
{
"ErrorEquals": [
"RunTaskError"
],
"Next": "Fail"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
}
]
},
"CheckForErrors": {
"Type": "Choice",
"Default": "AddProductFeatures",
"FallthroughOnError": true,
"Choices": [
{
"Expression": "{{$.hasExecutionErrors}} == true",
"Next": "Fail"
}
]
},
"AddProductFeatures": {
"Type": "AddProductFeatures",
"Next": "Report",
"Features": [
{
"Feature": "FeatureThatDependsOnGroupA",
"Groups": [
"GroupA"
],
"IsRequired": true
},
{
"Feature": "FeatureThatDependsOnGroupB",
"Groups": [
"GroupB"
],
"IsRequired": true
}
]
},
"Report": {
"Type": "Report",
"Next": "Succeed",
"Catch": [
{
"ErrorEquals": [
"ReportError"
],
"Next": "Fail"
}
]
},
"Succeed": {
"Type": "Succeed"
},
"Fail": {
"Type": "Fail"
}
}
}
```
# Create IDT test case executable
You can create and place test case executable in a test suite folder in the following ways:
+ For test suites that use arguments or environment variables from the `test.json` files to determine which tests to run, you can create a single test case executable for the entire test suite, or a test executable for each test group in the test suite.
+ For a test suite where you want to run specific tests based on specified commands, you create one test case executable for each test case in the test suite.
As a test writer, you can determine which approach is appropriate for your use case and structure your test case executable accordingly. Make sure that your provide the correct test case executable path in each `test.json` file, and that the specified executable runs correctly.
When all devices are ready for a test case to run, IDT reads the following files:
+ The `test.json` for the selected test case determines the processes to start and the environment variables to set.
+ The `suite.json` for the test suite determines the environment variables to set.
IDT starts the required test executable process based on the commands and arguments specified in the `test.json` file, and passes the required environment variables to the process.
## Use the IDT Client SDK
The IDT Client SDKs let you simplify how you write test logic in your test executable with API commands that you can use interact with IDT and your devices under test. IDT currently provides the following SDKs:
+ IDT Client SDK for Python
+ IDT Client SDK for Go
+ IDT Client SDK for Java
These SDKs are located in the `/sdks` folder. When you create a new test case executable, you must copy the SDK that you want to use to the folder that contains your test case executable and reference the SDK in your code. This section provides a brief description of the available API commands that you can use in your test case executables.
**Topics**
+ [
### Device interaction
](#api-device-interaction)
+ [
### IDT interaction
](#api-idt-interaction)
+ [
### Host interaction
](#api-host-interaction)
### Device interaction
The following commands enable you to communicate with the device under test without having to implement any additional device interaction and connectivity management functions.
**`ExecuteOnDevice`**
Allows test suites to run shell commands on a device that support SSH or Docker shell connections.
**`CopyToDevice`**
Allows test suites to copy a local file from the host machine that runs IDT to a specified location on a device that supports SSH or Docker shell connections.
**`ReadFromDevice`**
Allows test suites to read from the serial port of devices that support UART connections.
**Note**
Because IDT does not manage direct connections to devices that are made using device access information from the context, we recommend using these device interaction API commands in your test case executables. However, if these commands do not meet your test case requirements, then you can retrieve device access information from the IDT context and use it to make a direct connection to the device from the test suite.
To make a direct connection, retrieve the information in the `device.connectivity` and the `resource.devices.connectivity` fields for your device under test and for resource devices, respectively. For more information about using the IDT context, see [Use the IDT context](idt-context.md).
### IDT interaction
The following commands enable your test suites to communicate with IDT.
**`PollForNotifications`**
Allows test suites to check for notifications from IDT.
**`GetContextValue ` and `GetContextString`**
Allows test suites to retrieve values from the IDT context. For more information, see [Use the IDT context](idt-context.md).
**`SendResult`**
Allows test suites to report test case results to IDT. This command must be called at the end of each test case in a test suite.
### Host interaction
The following command enable your test suites to communicate with the host machine.
**`PollForNotifications`**
Allows test suites to check for notifications from IDT.
**`GetContextValue` and `GetContextString`**
Allows test suites to retrieve values from the IDT context. For more information, see [Use the IDT context](idt-context.md).
**`ExecuteOnHost`**
Allows test suites to run commands on the local machine and lets IDT manage the test case executable lifecycle.
## Enable IDT CLI commands
The `run-suite` command IDT CLI provides several options that let test runner customize test execution. To allow test runners to use these options to run your custom test suite, you implement support for the IDT CLI. If you do not implement support, test runners will still be able to run tests, but some CLI options will not function correctly. To provide an ideal customer experience, we recommend that you implement support for the following arguments for the `run-suite` command in the IDT CLI:
**`timeout-multiplier`**
Specifies a value greater than 1.0 that will be applied to all timeouts while running tests.
Test runners can use this argument to increase the timeout for the test cases that they want to run. When a test runner specifies this argument in their `run-suite` command, IDT uses it to calculate the value of the IDT\$1TEST\$1TIMEOUT environment variable and sets the `config.timeoutMultiplier` field in the IDT context. To support this argument, you must do the following:
+ Instead of directly using the timeout value from the `test.json` file, read the IDT\$1TEST\$1TIMEOUT environment variable to obtain the correctly calculated timeout value.
+ Retrieve the `config.timeoutMultiplier` value from the IDT context and apply it to long running timeouts.
For more information about exiting early because of timeout events, see [Specify exit behavior](#test-exec-exiting).
**`stop-on-first-failure`**
Specifies that IDT should stop running all tests if it encounters a failure.
When a test runner specifies this argument in their `run-suite` command, IDT will stop running tests as soon as it encounters a failure. However, if test cases are running in parallel, then this can lead to unexpected results. To implement support, make sure that if IDT encounters this event, your test logic instructs all running test cases to stop, clean up temporary resources, and report a test result to IDT. For more information about exiting early on failures, see [Specify exit behavior](#test-exec-exiting).
**`group-id` and `test-id`**
Specifies that IDT should run only the selected test groups or test cases.
Test runners can use these arguments with their `run-suite` command to specify the following test execution behavior:
+ Run all tests inside the specified test groups.
+ Run a selection of tests from within a specified test group.
To support these arguments, the state machine for your test suite must include a specific set of `RunTask` and `Choice` states in your state machine. If you are not using a custom state machine, then the default IDT state machine includes the required states for you and you do not need to take additional action. However, if you are using a custom state machine, then use [Example state machine: Run user-selected test groups](idt-state-machine.md#allow-specific-groups) as a sample to add the required states in your state machine.
For more information about IDT CLI commands, see [Debug and run custom test suites](run-tests-custom.md).
## Write event logs
While the test is running, you send data to `stdout` and `stderr` to write event logs and error messages to the console. For information about the format of console messages, see [Console message format](idt-review-results-logs.md#idt-console-format).
When the IDT finishes running the test suite, this information is also available in the `test_manager.log` file located in the `/results//logs` folder.
You can configure each test case to write the logs from its test run, including logs from the device under test, to the `_` file located in the `/results/execution-id/logs` folder. To do this, retrieve the path to the log file from the IDT context with the `testData.logFilePath` query, create a file at that path, and write the content that you want to it. IDT automatically updates the path based on the test case that is running. If you choose not to create the log file for a test case, then no file is generated for that test case.
You can also set up your text executable to create additional log files as needed in the `/logs` folder. We recommend that you specify unique prefixes for log file names so your files don't get overwritten.
## Report results to IDT
IDT writes test results to the `awsiotdevicetester_report.xml` and the `suite-name_report.xml` files. These report files are located in `/results//`. Both reports capture the results from the test suite execution. For more information about the schemas that IDT uses for these reports, see [Review IDT test results and logs](idt-review-results-logs.md)
To populate the contents of the `suite-name_report.xml` file, you must use the `SendResult` command to report test results to IDT before the test execution finishes. If IDT cannot locate the results of a test, it issues an error for the test case. The following Python excerpt shows the commands to send a test result to IDT:
```
request-variable = SendResultRequest(TestResult(result))
client.send_result(request-variable)
```
If you do not report results through the API, IDT looks for test results in the test artifacts folder. The path to this folder is stored in the `testData.testArtifactsPath` filed in the IDT context. In this folder, IDT uses the first alphabetically sorted XML file it locates as the test result.
If your test logic produces JUnit XML results, you can write the test results to an XML file in the artifacts folder to directly provide the results to IDT instead of parsing the results and then using the API to submit them to IDT.
If you use this method, make sure that your test logic accurately summarizes the test results and format your result file in the same format as the `suite-name_report.xml` file. IDT does not perform any validation of the data that you provide, with the following exceptions:
+ IDT ignores all properties of the `testsuites` tag. Instead, it calculates the tag properties from other reported test group results.
+ At least one `testsuite` tag must exist within `testsuites`.
Because IDT uses the same artifacts folder for all test cases and does not delete result files between test runs, this method might also lead to erroneous reporting if IDT reads the incorrect file. We recommend that you use the same name for the generated XML results file across all test cases to overwrite the results for each test case and make sure that the correct results are available for IDT to use. Although you can use a mixed approach to reporting in your test suite, that is, use an XML result file for some test cases and submit results through the API for others, we do not recommend this approach.
## Specify exit behavior
Configure your text executable to always exit with an exit code of 0, even if a test case reports a failure or an error result. Use non-zero exit codes only to indicate that a test case did not run or if the test case executable could not communicate any results to IDT. When IDT receives a non-zero exit code, it marks the test case has having encountered an error that prevented it from running.
IDT might request or expect a test case to stop running before it has finished in the following events. Use this information to configure your test case executable to detect each of these events from the test case:
****Timeout****
Occurs when a test case runs for longer than the timeout value specified in the `test.json` file. If the test runner used the `timeout-multiplier` argument to specify a timeout multiplier, then IDT calculates the timeout value with the multiplier.
To detect this event, use the IDT\$1TEST\$1TIMEOUT environment variable. When a test runner launches a test, IDT sets the value of the IDT\$1TEST\$1TIMEOUT environment variable to the calculated timeout value (in seconds) and passes the variable to the test case executable. You can read the variable value to set an appropriate timer.
****Interrupt****
Occurs when the test runner interrupts IDT. For example, by pressing Ctrl\$1C.
Because terminals propagate signals to all child processes, you can simply configure a signal handler in your test cases to detect interrupt signals.
Alternatively, you can periodically poll the API to check the value of the `CancellationRequested` boolean in the `PollForNotifications` API response. When IDT receives an interrupt signal, it sets the value of the `CancellationRequested` boolean to `true`.
****Stop on first failure****
Occurs when a test case that is running in parallel with the current test case fails and the test runner used the `stop-on-first-failure` argument to specify that IDT should stop when it encounters any failure.
To detect this event, you can periodically poll the API to check the value of the `CancellationRequested` boolean in the `PollForNotifications` API response. When IDT encounters a failure and is configured to stop on first failure, it sets the value of the `CancellationRequested` boolean to `true`.
When any of these events occur, IDT waits for 5 minutes for any currently running test cases to finish running. If all running test cases do not exit within 5 minutes, IDT forces each of their processes to stop. If IDT has not received test results before the processes end, it will mark the test cases as having timed out. As a best practice, you should ensure that your test cases perform the following actions when they encounter one of the events:
1. Stop running normal test logic.
1. Clean up any temporary resources, such as test artifacts on the device under test.
1. Report a test result to IDT, such as a test failure or an error.
1. Exit.
# Use the IDT context
When IDT runs a test suite, the test suite can access a set of data that can be used to determine how each test runs. This data is called the IDT context. For example, user data configuration provided by test runners in a `userdata.json` file is made available to test suites in the IDT context.
The IDT context can be considered a read-only JSON document. Test suites can retrieve data from and write data to the context using standard JSON data types like objects, arrays, numbers and so on.
## Context schema
The IDT context uses the following format:
```
{
"config": {
"timeoutMultiplier": timeout-multiplier,
"idtRootPath":
},
"device": {
},
"devicePool": {
},
"resource": {
"devices": [
{
"name": ""
}
]
},
"testData": {
"awsCredentials": {
"awsAccessKeyId": "",
"awsSecretAccessKey": "",
"awsSessionToken": ""
},
"logFilePath": "/path/to/log/file"
},
"userData": {
}
}
```
**`config`**
Information from the [`config.json` file](set-config-custom.md#config-json-custom). The `config` field also contains the following additional fields:
**`config.timeoutMultiplier`**
The multiplier for the any timeout value used by the test suite. This value is specified by the test runner from the IDT CLI. The default value is `1`.
**`config.idRootPath`**
This value is a placeholder for the absolute path value of IDT while configuring the `userdata.json` file. This is used by the build and flash commands.
**`device`**
Information about the device selected for the test run. This information is equivalent to the `devices` array element in the [`device.json` file](set-config-custom.md#device-config-custom) for the selected device.
**`devicePool`**
Information about the device pool selected for the test run. This information is equivalent to the top-level device pool array element defined in the `device.json` file for the selected device pool.
**`resource`**
Information about resource devices from the `resource.json` file.
**`resource.devices`**
This information is equivalent to the `devices` array defined in the `resource.json` file. Each `devices` element includes the following additional field:
**`resource.device.name`**
The name of the resource device. This value is set to the `requiredResource.name` value in the `test.json` file.
**`testData.awsCredentials`**
The AWS credentials used by the test to connect to the AWS cloud. This information is obtained from the `config.json` file.
**`testData.logFilePath`**
The path to the log file to which the test case writes log messages. The test suite creates this file if it doesn't exist.
**`userData`**
Information provided by the test runner in the [`userdata.json` file](set-config-custom.md#userdata-config-custom).
## Access data in the context
You can query the context using JSONPath notation from your configuration files and from your text executable with the `GetContextValue` and `GetContextString` APIs. The syntax for JSONPath strings to access the IDT context varies as follows:
+ In `suite.json` and `test.json`, you use `{{query}}`. That is, do not use the root element `$.` to start your expression.
+ In `statemachine.json`, you use `{{$.query}}`.
+ In API commands, you use `query` or `{{$.query}}`, depending on the command. For more information, see the inline documentation in the SDKs.
The following table describes the operators in a typical foobar JSONPath expression:
| Operator | Description |
| --- | --- |
| \$1 | The root element. Because the top-level context value for IDT is an object, you will typically use \$1. to start your queries. |
| .childName | Accesses the child element with name childName from an object. If applied to an array, yields a new array with this operator applied to each element. The element name is case sensitive. For example, the query to access the awsRegion value in the config object is \$1.config.awsRegion. |
| [start:end] | Filters elements from an array, retrieving items beginning from the start index and going up to the end index, both inclusive. |
| [index1, index2, ... , indexN] | Filters elements from an array, retrieving items from only the specified indices. |
| [?(expr)] | Filters elements from an array using the expr expression. This expression must evaluate to a boolean value. |
To create filter expressions, use the following syntax:
```
| operator |
```
In this syntax:
+ `jsonpath` is a JSONPath that uses standard JSON syntax.
+ `value` is any custom value that uses standard JSON syntax.
+ `operator` is one of the following operators:
+ `<` (Less than)
+ `<=` (Less than or equal to)
+ `==` (Equal to)
If the JSONPath or value in your expression is an array, boolean, or object value, then this is the only supported binary operator that you can use.
+ `>=` (Greater than or equal to)
+ `>` (Greater than)
+ `=~` (Regular expression match). To use this operator in a filter expression, the JSONPath or value on the left side of your expression must evaluate to a string and the right side must be a pattern value that follows the [RE2 syntax](https://github.com/google/re2/wiki/Syntax).
You can use JSONPath queries in the form \$1\$1*query*\$1\$1 as placeholder strings within the `args` and `environmentVariables` fields in `test.json` files and within the `environmentVariables` fields in `suite.json` files. IDT performs a context lookup and populates the fields with the evaluated value of the query. For example, in the `suite.json` file, you can use placeholder strings to specify environment variable values that change with each test case and IDT will populate the environment variables with the correct value for each test case. However, when you use placeholder strings in `test.json` and `suite.json` files, the following considerations apply for your queries:
+ You must each occurrence of the `devicePool` key in your query in all lower case. That is, use `devicepool` instead.
+ For arrays, you can use only arrays of strings. In addition, arrays use a non-standard `item1, item2,...,itemN` format. If the array contains only one element, then it is serialized as `item`, making it indistinguishable from a string field.
+ You cannot use placeholders to retrieve objects from the context.
Because of these considerations, we recommend that whenever possible, you use the API to access the context in your test logic instead of placeholder strings in `test.json` and `suite.json` files. However, in some cases it might be more convenient to use JSONPath placeholders to retrieve single strings to set as environment variables.
# Configure settings for test runners
To run custom test suites, test runners must configure their settings based on the test suite that they want to run. Settings are specified based on configuration file templates located in the `/configs/` folder. If required, test runners must also set up AWS credentials that IDT will use to connect to the AWS cloud.
As a test writer, you will need to configure these files to [debug your test suite](run-tests-custom.md). You must provide instructions to test runners so that they can configure the following settings as needed to run your test suites.
## Configure device.json
The `device.json` file contains information about the devices that tests are run on (for example, IP address, login information, operating system, and CPU architecture).
Test runners can provide this information using the following template `device.json` file located in the `/configs/` folder.
```
[
{
"id": "",
"sku": "",
"features": [
{
"name": "",
"value": "",
"configs": [
{
"name": "",
"value": ""
}
],
}
],
"devices": [
{
"id": "",
"pairedResource": "", //used for no-op protocol
"connectivity": {
"protocol": "ssh | uart | docker | no-op",
// ssh
"ip": "",
"port": ,
"publicKeyPath": "",
"auth": {
"method": "pki | password",
"credentials": {
"user": "",
// pki
"privKeyPath": "/path/to/private/key",
// password
"password": "",
}
},
// uart
"serialPort": "",
// docker
"containerId": "",
"containerUser": "",
}
}
]
}
]
```
All fields that contain values are required as described here:
**`id`**
A user-defined alphanumeric ID that uniquely identifies a collection of devices called a *device pool*. Devices that belong to a pool must have identical hardware. When you run a suite of tests, devices in the pool are used to parallelize the workload. Multiple devices are used to run different tests.
**`sku`**
An alphanumeric value that uniquely identifies the device under test. The SKU is used to track qualified devices.
If you want to list your board in the AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process.
**`features`**
Optional. An array that contains the device's supported features. Device features are user-defined values that you configure in your test suite. You must provide your test runners with information about the feature names and values to include in the `device.json` file. For example, if you want to test a device that functions as an MQTT server for other devices, then you can configure your test logic to validate specific supported levels for a feature named `MQTT_QoS`. Test runners provide this feature name and set the feature value to the QoS levels supported by their device. You can retrieve the provided information from the [IDT context](idt-context.md) with the `devicePool.features` query, or from the [state machine context](idt-state-machine.md#state-machine-context) with the `pool.features` query.
**`features.name`**
The name of the feature.
**`features.value`**
The supported feature values.
**`features.configs`**
Configuration settings, if needed, for the feature.
**`features.config.name`**
The name of the configuration setting.
**`features.config.value`**
The supported setting values.
**`devices`**
An array of devices in the pool to be tested. At least one device is required.
**`devices.id`**
A user-defined unique identifier for the device being tested.
**`devices.pairedResource`**
A user-defined unique identifier for a resource device. This value is required when you test devices using the `no-op` connectivity protocol.
**`connectivity.protocol`**
The communication protocol used to communicate with this device. Each device in a pool must use the same protocol.
Currently, the only supported values are `ssh` and `uart` for physical devices, `docker` for Docker containers, and `no-op` for devices who don't have a direct connection with the IDT host machine but require a resource device as physical middleware to communicate with the host machine.
For no-op devices, you configure the resource device ID in `devices.pairedResource`. You must also specify this ID in the `resource.json` file. The paired device must be a device that is physically paired with the device under test. After IDT identifies and connects to the paired resource device, IDT will not connect to other resource devices according to the features described in the `test.json` file.
**`connectivity.ip`**
The IP address of the device being tested.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.port`**
Optional. The port number to use for SSH connections.
The default value is 22.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.publicKeyPath`**
Optional. The full path to the public key used to authenticate connections to the device under test. When you specify the `publicKeyPath`, IDT validates the device’s public key when it establishes an SSH connection to the device under test. If this value is not specified, IDT creates an SSH connection, but doesn’t validate the device’s public key.
We strongly recommend that you specify the path to the public key, and that you use a secure method to fetch this public key. For standard command line-based SSH clients, the public key is provided in the `known_hosts` file. If you specify a separate public key file, this file must use the same format as the `known_hosts` file, that is, `ip-address key-type public-key`.
**`connectivity.auth`**
Authentication information for the connection.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.auth.method`**
The authentication method used to access a device over the given connectivity protocol.
Supported values are:
+ `pki`
+ `password`
**`connectivity.auth.credentials`**
The credentials used for authentication.
**`connectivity.auth.credentials.password`**
The password used for signing in to the device being tested.
This value applies only if `connectivity.auth.method` is set to `password`.
**`connectivity.auth.credentials.privKeyPath`**
The full path to the private key used to sign in to the device under test.
This value applies only if `connectivity.auth.method` is set to `pki`.
**`connectivity.auth.credentials.user`**
The user name for signing in to the device being tested.
**`connectivity.serialPort`**
Optional. The serial port to which the device is connected.
This property applies only if `connectivity.protocol` is set to `uart`.
**`connectivity.containerId`**
The container ID or name of the Docker container being tested.
This property applies only if `connectivity.protocol` is set to `docker`.
**`connectivity.containerUser`**
Optional. The name of the user to user inside the container. The default value is the user provided in the Dockerfile.
The default value is 22.
This property applies only if `connectivity.protocol` is set to `docker`.
To check if test runners configure the incorrect device connection for a test, you can retrieve `pool.Devices[0].Connectivity.Protocol` from the state machine context and compare it to the expected value in a `Choice` state. If an incorrect protocol is used, then print a message using the `LogMessage` state and transition to the `Fail` state.
Alternatively, you can use error handling code to report a test failure for incorrect device types.
## (Optional) Configure userdata.json
The `userdata.json` file contains any additional information that is required by a test suite but is not specified in the `device.json` file. The format of this file depends on the [`userdata_scheme.json` file](idt-json-config.md#userdata-schema-json) that is defined in the test suite. If you are a test writer, make sure you provide this information to users who will run the test suites that you write.
## (Optional) Configure resource.json
The `resource.json` file contains information about any devices that will be used as resource devices. Resource devices are devices that are required to test certain capabilities of a device under test. For example, to test a device's Bluetooth capability, you might use a resource device to test that your device can connect to it successfully. Resource devices are optional, and you can require as many resources devices as you need. As a test writer, you use the [test.json file](idt-json-config.md#test-json) to define the resource device features that are required for a test. Test runners then use the `resource.json` file to provide a pool of resource devices that have the required features. Make sure you provide this information to users who will run the test suites that you write.
Test runners can provide this information using the following template `resource.json` file located in the `/configs/` folder.
```
[
{
"id": "",
"features": [
{
"name": "",
"version": "",
"jobSlots":
}
],
"devices": [
{
"id": "",
"connectivity": {
"protocol": "ssh | uart | docker",
// ssh
"ip": "",
"port": ,
"publicKeyPath": "",
"auth": {
"method": "pki | password",
"credentials": {
"user": "",
// pki
"privKeyPath": "/path/to/private/key",
// password
"password": "",
}
},
// uart
"serialPort": "",
// docker
"containerId": "",
"containerUser": "",
}
}
]
}
]
```
All fields that contain values are required as described here:
**`id`**
A user-defined alphanumeric ID that uniquely identifies a collection of devices called a *device pool*. Devices that belong to a pool must have identical hardware. When you run a suite of tests, devices in the pool are used to parallelize the workload. Multiple devices are used to run different tests.
**`features`**
Optional. An array that contains the device's supported features. The information required in this field is defined in the [test.json files](idt-json-config.md#test-json) in the test suite and determines which tests to run and how to run those tests. If the test suite does not require any features, then this field is not required.
**`features.name`**
The name of the feature.
**`features.version`**
The feature version.
**`features.jobSlots`**
Setting to indicate how many tests can concurrently use the device. The default value is `1`.
**`devices`**
An array of devices in the pool to be tested. At least one device is required.
**`devices.id`**
A user-defined unique identifier for the device being tested.
**`connectivity.protocol`**
The communication protocol used to communicate with this device. Each device in a pool must use the same protocol.
Currently, the only supported values are `ssh` and `uart` for physical devices, and `docker` for Docker containers.
**`connectivity.ip`**
The IP address of the device being tested.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.port`**
Optional. The port number to use for SSH connections.
The default value is 22.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.publicKeyPath`**
Optional. The full path to the public key used to authenticate connections to the device under test. When you specify the `publicKeyPath`, IDT validates the device’s public key when it establishes an SSH connection to the device under test. If this value is not specified, IDT creates an SSH connection, but doesn’t validate the device’s public key.
We strongly recommend that you specify the path to the public key, and that you use a secure method to fetch this public key. For standard command line-based SSH clients, the public key is provided in the `known_hosts` file. If you specify a separate public key file, this file must use the same format as the `known_hosts` file, that is, `ip-address key-type public-key`.
**`connectivity.auth`**
Authentication information for the connection.
This property applies only if `connectivity.protocol` is set to `ssh`.
**`connectivity.auth.method`**
The authentication method used to access a device over the given connectivity protocol.
Supported values are:
+ `pki`
+ `password`
**`connectivity.auth.credentials`**
The credentials used for authentication.
**`connectivity.auth.credentials.password`**
The password used for signing in to the device being tested.
This value applies only if `connectivity.auth.method` is set to `password`.
**`connectivity.auth.credentials.privKeyPath`**
The full path to the private key used to sign in to the device under test.
This value applies only if `connectivity.auth.method` is set to `pki`.
**`connectivity.auth.credentials.user`**
The user name for signing in to the device being tested.
**`connectivity.serialPort`**
Optional. The serial port to which the device is connected.
This property applies only if `connectivity.protocol` is set to `uart`.
**`connectivity.containerId`**
The container ID or name of the Docker container being tested.
This property applies only if `connectivity.protocol` is set to `docker`.
**`connectivity.containerUser`**
Optional. The name of the user to user inside the container. The default value is the user provided in the Dockerfile.
The default value is 22.
This property applies only if `connectivity.protocol` is set to `docker`.
## (Optional) Configure config.json
The `config.json` file contains configuration information for IDT. Typically, test runners will not need to modify this file except to provide their AWS user credentials for IDT, and optionally, an AWS region. If AWS credentials with required permissions are provided AWS IoT Device Tester collects and submits usage metrics to AWS. This is an opt-in feature and is used to improve IDT functionality. For more information, see [Submit IDT usage metrics](idt-usage-metrics.md).
Test runners can configure their AWS credentials in one of the following ways:
+ **Credentials file**
IDT uses the same credentials file as the AWS CLI. For more information, see [Configuration and credential files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html).
The location of the credentials file varies, depending on the operating system you are using:
+ macOS, Linux: `~/.aws/credentials`
+ Windows: `C:\Users\UserName\.aws\credentials`
+ **Environment variables**
Environment variables are variables maintained by the operating system and used by system commands. Variables defined during an SSH session are not available after that session is closed. IDT can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to store AWS credentials
To set these variables on Linux, macOS, or Unix, use **export**:
```
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
```
To set these variables on Windows, use **set**:
```
set AWS_ACCESS_KEY_ID=
set AWS_SECRET_ACCESS_KEY=
```
To configure AWS credentials for IDT, test runners edit the `auth` section in the `config.json` file located in the `/configs/` folder.
```
{
"log": {
"location": "logs"
},
"configFiles": {
"root": "configs",
"device": "configs/device.json"
},
"testPath": "tests",
"reportPath": "results",
"awsRegion": "",
"auth": {
"method": "file | environment",
"credentials": {
"profile": ""
}
}
}
]
```
All fields that contain values are required as described here:
**Note**
All paths in this file are defined relative to the **.
**`log.location`**
The path to the logs folder in the **.
**`configFiles.root`**
The path to the folder that contains the configuration files.
**`configFiles.device`**
The path to the `device.json` file.
**`testPath`**
The path to the folder that contains test suites.
**`reportPath`**
The path to the folder that will contain test results after IDT runs a test suite.
**`awsRegion`**
Optional. The AWS region that test suites will use. If not set, then test suites will use the default region specified in each test suite.
**`auth.method`**
The method IDT uses to retrieve AWS credentials. Supported values are `file` to retrieve credentials from a credentials file, and `environment` to retrieve credentials using environment variables.
**`auth.credentials.profile`**
The credentials profile to use from the credentials file. This property applies only if `auth.method` is set to `file`.
# Debug and run custom test suites
After the [required configuration](set-config-custom.md) is set, IDT can run your test suite. The runtime of the full test suite depends on the hardware and the composition of the test suite. For reference, it takes approximately 30 minutes to complete the full FreeRTOS qualification test suite on a Raspberry Pi 3B.
As you write your test suite, you can use IDT to run the test suite in debug mode to check your code before you run it or provide it to test runners.
## Run IDT in debug mode
Because test suites depend on IDT to interact with devices, provide the context, and receive results, you cannot simply debug your test suites in an IDE without any IDT interaction. To do so, the IDT CLI provides the `debug-test-suite` command that lets you run IDT in debug mode. Run the following command to view the available options for `debug-test-suite`:
```
devicetester_[linux | mac | win_x86-64] debug-test-suite -h
```
When you run IDT in debug mode, IDT does not actually launch the test suite or run the test orchestrator; instead, it interacts with your IDE to responds to requests made from the test suite running in the IDE and prints the logs to the console. IDT does not time out and waits to exit until manually interrupted. In debug mode, IDT also does not run the test orchestrator and will not generate any report files. To debug your test suite, you must use your IDE to provide some information that IDT usually obtains from the configuration files. Make sure you provide the following information:
+ Environment variables and arguments for each test. IDT will not read this information from `test.json` or `suite.json`.
+ Arguments to select resource devices. IDT will not read this information from `test.json`.
To debug your test suites, complete the following steps:
1. Create the setting configuration files that are required to run the test suite. For example, if your test suite requires the `device.json`, `resource.json`, and `user data.json`, make sure you configure all of them as needed.
1. Run the following command to place IDT in debug mode and select any devices that are required to run the test.
```
devicetester_[linux | mac | win_x86-64] debug-test-suite [options]
```
After you run this command, IDT waits for requests from the test suite and then responds to them. IDT also generates the environment variables that are required for the case process for the IDT Client SDK.
1. In your IDE, use the `run` or `debug` configuration to do the following:
1. Set the values of the IDT-generated environment variables.
1. Set the value of any environment variables or arguments that you specified in your `test.json` and `suite.json` file.
1. Set breakpoints as needed.
1. Run the test suite in your IDE.
You can debug and re-run the test suite as many times as needed. IDT does not time out in debug mode.
1. After you complete debugging, interrupt IDT to exit debug mode.
## IDT CLI commands to run tests
The following section describes the IDT CLI commands:
------
#### [ IDT v4.0.0 ]
**`help`**
Lists information about the specified command.
**`list-groups`**
Lists the groups in a given test suite.
**`list-suites`**
Lists the available test suites.
**`list-supported-products`**
Lists the supported products for your version of IDT, in this case FreeRTOS versions, and FreeRTOS qualification test suite versions available for the current IDT version.
**`list-test-cases`**
Lists the test cases in a given test group. The following option is supported:
+ `group-id`. The test group to search for. This option is required and must specify a single group.
**`run-suite`**
Runs a suite of tests on a pool of devices. The following are some commonly used options:
+ `suite-id`. The test suite version to run. If not specified, IDT uses the latest version in the `tests` folder.
+ `group-id`. The test groups to run, as a comma-separated list. If not specified, IDT runs all test groups in the test suite.
+ `test-id`. The test cases to run, as a comma-separated list. When specified, `group-id` must specify a single group.
+ `pool-id`. The device pool to test. Test runners must specify a pool if they have multiple device pools defined in your `device.json` file.
+ `timeout-multiplier`. Configures IDT to modify the test execution timeout specified in the `test.json` file for a test with a user-defined multiplier.
+ `stop-on-first-failure`. Configures IDT to stop execution on the first failure. This option should be used with `group-id` to debug the specified test groups.
+ `userdata`. Sets the file that contains user data information required to run the test suite. This is required only if `userdataRequired` is set to true in the `suite.json` file for the test suite.
For more information about `run-suite` options, use the `help` option:
```
devicetester_[linux | mac | win_x86-64] run-suite -h
```
**`debug-test-suite`**
Run the test suite in debug mode. For more information, see [Run IDT in debug mode](#idt-debug-mode).
------
# Review IDT test results and logs
This section describes the format in which IDT generates console logs and test reports.
## Console message format
AWS IoT Device Tester uses a standard format for printing messages to the console when it starts a test suite. The following excerpt shows an example of a console message generated by IDT.
```
[INFO] [2000-01-02 03:04:05]: Using suite: MyTestSuite_1.0.0 executionId=9a52f362-1227-11eb-86c9-8c8590419f30
```
Most console messages consist of the following fields:
**`time`**
A full ISO 8601 timestamp for the logged event.
**`level`**
The message level for the logged event. Typically, the logged message level is one of `info`, `warn`, or `error`. IDT issues a `fatal` or `panic` message if it encounters an expected event that causes it to exit early.
**`msg`**
The logged message.
**`executionId`**
A unique ID string for the current IDT process. This ID is used to differentiate between individual IDT runs.
Console messages generated from a test suite provide additional information about the device under test and the test suite, test group, and test cases that IDT runs. The following excerpt shows an example of a console message generated from a test suite.
```
[INFO] [2000-01-02 03:04:05]: Hello world! suiteId=MyTestSuitegroupId=myTestGroup testCaseId=myTestCase deviceId=my-deviceexecutionId=9a52f362-1227-11eb-86c9-8c8590419f30
```
The test-suite specific part of the console message contains the following fields:
**`suiteId`**
The name of the test suite currently running.
**`groupId`**
The ID of the test group currently running.
**`testCaseId`**
The ID of the test case current running.
**`deviceId`**
A ID of the device under test that the current test case is using.
The test summary contains information about the test suite, the test results for each group that was run, and the locations of the generated logs and report files. The following example shows a test summary message.
```
========== Test Summary ==========
Execution Time: 5m00s
Tests Completed: 4
Tests Passed: 3
Tests Failed: 1
Tests Skipped: 0
----------------------------------
Test Groups:
GroupA: PASSED
GroupB: FAILED
----------------------------------
Failed Tests:
Group Name: GroupB
Test Name: TestB1
Reason: Something bad happened
----------------------------------
Path to AWS IoT Device Tester Report: /path/to/awsiotdevicetester_report.xml
Path to Test Execution Logs: /path/to/logs
Path to Aggregated JUnit Report: /path/to/MyTestSuite_Report.xml
```
## AWS IoT Device Tester report schema
`awsiotdevicetester_report.xml` is a signed report that contains the following information:
+ The IDT version.
+ The test suite version.
+ The report signature and key used to sign the report.
+ The device SKU and the device pool name specified in the `device.json` file.
+ The product version and the device features that were tested.
+ The aggregate summary of test results. This information is the same as that contained in the `suite-name_report.xml` file.
```
idt-version
test-suite-version
signature
keyname
execution-id
start-time
end-time
product-name
product-version
device-sku
device-name
ssh | uart | docker
```
The `awsiotdevicetester_report.xml` file contains an `` tag that contains information about the product being tested and the product features that were validated after running a suite of tests.
**Attributes used in the `` tag**
**`name`**
The name of the product being tested.
**`version`**
The version of the product being tested.
**`features`**
The features validated. Features marked as `required` are required for the test suite to validate the device. The following snippet shows how this information appears in the `awsiotdevicetester_report.xml` file.
```
```
Features marked as `optional` are not required for validation. The following snippets show optional features.
```
```
## Test suite report schema
The `suite-name_Result.xml` report is in [JUnit XML format](https://llg.cubic.org/docs/junit/). You can integrate it into continuous integration and deployment platforms like [Jenkins](https://jenkins.io/), [Bamboo](https://www.atlassian.com/software/bamboo), and so on. The report contains an aggregate summary of test results.
```
reason
reason
reason
```
The report section in both the `awsiotdevicetester_report.xml` or `suite-name_report.xml` lists the tests that were run and the results.
The first XML tag `` contains the summary of the test execution. For example:
```
```
**Attributes used in the `` tag**
**`name`**
The name of the test suite.
**`time`**
The time, in seconds, it took to run the test suite.
**`tests`**
The number of tests executed.
**`failures`**
The number of tests that were run, but did not pass.
**`errors`**
The number of tests that IDT couldn't execute.
**`disabled`**
This attribute is not used and can be ignored.
In the event of test failures or errors, you can identify the test that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag show the test result summary for a test group. For example:
```
```
The format is similar to the `` tag, but with a `skipped` attribute that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each executed test for a test group. For example:
```
```
**Attributes used in the `` tag**
**`name`**
The name of the test.
**`attempts`**
The number of times IDT executed the test case.
When a test fails or an error occurs, `` or `` tags are added to the `` tag with information for troubleshooting. For example:
```
Reason for the test failure
Reason for the test execution error
```
# Submit IDT usage metrics
If you provide AWS credentials with required permissions, AWS IoT Device Tester collects and submits usage metrics to AWS. This is an opt-in feature and is used to improve IDT functionality. IDT collects information such as the following:
+ The AWS account ID used to run IDT
+ The IDT CLI commands used to run tests
+ The test suite that are run
+ The test suites in the ** folder
+ The number of devices configured in the device pool
+ Test case names and run times
+ Test result information, such as whether tests passed, failed, encountered errors, or were skipped
+ Product features tested
+ IDT exit behavior, such as unexpected or early exits
All of the information that IDT sends is also logged to a `metrics.log` file in the `/results//` folder. You can view the log file to see the information that was collected during a test run. This file is generated only if you choose to collect usage metrics.
To disable metrics collection, you do not need to take additional action. Simply do not store your AWS credentials, and if you do have stored AWS credentials, do not configure the `config.json` file to access them.
## Sign up for an AWS account
If you do not have an AWS account, complete the following steps to create one.
**To sign up for an AWS account**
1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup).
1. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad.
When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks).
AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**.
## Create a user with administrative access
After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks.
**Secure your AWS account root user**
1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password.
For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*.
1. Turn on multi-factor authentication (MFA) for your root user.
For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*.
**Create a user with administrative access**
1. Enable IAM Identity Center.
For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*.
1. In IAM Identity Center, grant administrative access to a user.
For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*.
**Sign in as the user with administrative access**
+ To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user.
For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*.
**Assign access to additional users**
1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions.
For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*.
1. Assign users to a group, and then assign single sign-on access to the group.
For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*.
To provide access, add permissions to your users, groups, or roles:
+ Users and groups in AWS IAM Identity Center:
Create a permission set. Follow the instructions in [Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/howtocreatepermissionset.html) in the *AWS IAM Identity Center User Guide*.
+ Users managed in IAM through an identity provider:
Create a role for identity federation. Follow the instructions in [Create a role for a third-party identity provider (federation)](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_roles_create_for-idp.html) in the *IAM User Guide*.
+ IAM users:
+ Create a role that your user can assume. Follow the instructions in [Create a role for an IAM user](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_roles_create_for-user.html) in the *IAM User Guide*.
+ (Not recommended) Attach a policy directly to a user or add a user to a user group. Follow the instructions in [Adding permissions to a user (console)](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) in the *IAM User Guide*.
## Provide AWS credentials to IDT
To allow IDT to access your AWS credentials and submit metrics to AWS, do the following:
1. Store the AWS credentials for your IAM user as environment variables or in a credentials file:
1. To use environment variables, run the following command:
```
AWS_ACCESS_KEY_ID=access-key
AWS_SECRET_ACCESS_KEY=secret-access-key
```
1. To use the credentials file, add the following information to the `.aws/credentials file:`
```
[profile-name]
aws_access_key_id=access-key
aws_secret_access_key=secret-access-key
```
1. Configure the `auth` section of the `config.json` file. For more information, see [(Optional) Configure config.json](set-config-custom.md#config-json-custom).
# Maintain test suite versions
Test suite versions
IDT for FreeRTOS organizes test resources into test suites and test groups:
+ A test suite is the set of test groups used to verify that a device works with particular versions of FreeRTOS.
+ A test group is the set of individual tests related to a particular feature, such as BLE and MQTT messaging.
Starting in IDT v3.0.0, test suites are versioned using a `major`.`minor`.`patch` format starting from 1.0.0. When you download IDT, the package includes the latest test suite version.
When you start IDT in the command line interface, IDT checks whether a newer test suite version is available. If so, it prompts you to update to the new version. You can choose to update or continue with your current tests.
**Note**
IDT supports the three latest test suite versions for qualification. For more information, see [Understand the support policy for AWS IoT Device Tester](idt-support-policy.md).
You can download test suites by using the `upgrade-test-suite` command. Or, you can use the optional parameter `-upgrade-test-suite flag` when you start IDT where *flag* can be '`y`' to always download the latest version, or '`n`' to use the existing version.
You can also run the `list-supported-versions` command to list the FreeRTOS and test suite versions that are supported by the current version of IDT.
New tests might introduce new IDT configuration settings. If the settings are optional, IDT notifies you and continues running the tests. If the settings are required, IDT notifies you and stops running. After you configure the settings, you can continue to run the tests.
# Troubleshoot errors
Each test suite execution has a unique execution ID that is used to create a folder named `results/execution-id` in the `results` directory. Individual test group logs are under the `results/execution-id/logs` directory. Use the IDT for FreeRTOS console output to find the execution id, test case id, and test group id of the test case that failed and then open the log file for that test case named `results/execution-id/logs/test_group_id__test_case_id.log`. The information in this file includes:
+ Full build and flash command output.
+ Test execution output.
+ More verbose IDT for FreeRTOS console output.
We recommend the following workflow for troubleshooting:
1. If you see the error "*user/role* is not authorized to access this resource", make sure that you configure permissions as specified in [Create and configure an AWS account](dev-tester-prereqs.md#config-aws-account).
1. Read the console output to find information, such as execution UUID and currently executing tasks.
1. Look in the `FRQ_Report.xml` file for error statements from each test. This directory contains execution logs of each test group.
1. Look in the log files under `/results/execution-id/logs`.
1. Investigate one of the following problem areas:
+ Device configuration, such as JSON configuration files in the `/configs/` folder.
+ Device interface. Check the logs to determine which interface is failing.
+ Device tooling. Make sure that the toolchains for building and flashing the device are installed and configured correctly.
+ For FRQ 1.x.x make sure that a clean, cloned version of the FreeRTOS source code is available. FreeRTOS releases are tagged according to the FreeRTOS version. To clone a specific version of the code, use the following commands:
```
git clone --branch version-number https://github.com/aws/amazon-freertos.git
cd amazon-freertos
git submodule update --checkout --init --recursive
```
## Troubleshoot device configuration
When you use IDT for FreeRTOS, you must get the correct configuration files in place before you execute the binary. If you're getting parsing and configuration errors, your first step should be to locate and use a configuration template appropriate for your environment. These templates are located in the `IDT_ROOT/configs` directory.
If you are still having issues, see the following debugging process.
### Where do I look?
Start by reading the console output to find information, such as the execution UUID, which is referenced as `execution-id` in this documentation.
Next, look in the `FRQ_Report.xml` file in the `/results/execution-id` directory. This file contains all of the test cases that were run and error snippets for each failure. To get all of the execution logs, look for the file `/results/execution-id/logs/test_group_id__test_case_id.log` for each test case.
### IDT error codes
The following table explains the error codes generated by IDT for FreeRTOS:
| Error Code | Error Code Name | Possible Root Cause | Troubleshooting |
| --- | --- | --- | --- |
| 201 | InvalidInputError | Fields in `device.json`, `config.json`, or `userdata.json` are either missing or in an incorrect format. | Make sure required fields are not missing and are in required format in listed files. For more information, see [First test of your microcontroller board](qual-steps.md). |
| 202 | ValidationError | Fields in `device.json`, `config.json`, or `userdata.json` contain invalid values. | Check the error message on the right hand side of the error code in the report: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dt-afr-troubleshooting.html) |
| 203 | CopySourceCodeError | Unable to copy FreeRTOS source code to specified directory. | Verify the following items: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dt-afr-troubleshooting.html) |
| 204 | BuildSourceError | Unable to compile the FreeRTOS source code. | Verify the following items: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/dt-afr-troubleshooting.html) |
| 205 | FlashOrRunTestError | IDT FreeRTOS is unable to flash or run FreeRTOS on your DUT. | Verify the information under `flashTool` in your `userdata.json` file is correct. For more information, see [Configure build, flash, and test settings](cfg-dt-ud.md). |
| 206 | StartEchoServerError | IDT FreeRTOS is unable to start echo server for the WiFi or secure sockets tests. | Verify the ports configured under `echoServerConfiguration` in your `userdata.json` file are not in use or blocked by firewall or network settings. |
### Debug config file parsing errors
Occasionally, a typo in a JSON configuration can lead to parsing errors. Most of the time, the issue is a result of omitting a bracket, comma, or quote from your JSON file. IDT for FreeRTOS performs JSON validation and prints debugging information. It prints the line where the error occurred, the line number, and the column number of the syntax error. This information should be enough to help you fix the error, but if you are still having issues locating the error, you can perform validation manually in your IDE, a text editor such as Atom or Sublime, or through an online tool like JSONLint.
### Debug test results parse errors
While running a test group from [ FreeRTOS-Libraries-Integration-Tests](https://github.com/FreeRTOS/FreeRTOS-Libraries-Integration-Tests), such as **FullTransportInterfaceTLS, FullPKCS11\$1Core, FullPKCS11\$1Onboard\$1ECC, FullPKCS11\$1Onboard\$1RSA, FullPKCS11\$1PreProvisioned\$1ECC, FullPKCS11\$1PreProvisioned\$1RSA**, or **OTACore**, IDT for FreeRTOS parses the test results from the test device with the serial connection. Sometimes, extra serial outputs on the device may interfere with the parsing of the test results.
In the above mentioned case, strange test case failure reasons like strings originating from unrelated device outputs are output. The IDT for FreeRTOS test case log file (which includes all the serial output IDT for FreeRTOS has received during the test) may show the following:
```
TEST(Full_PKCS11_Capabilities, PKCS11_Capabilities)
PASS
```
In the above example, the unrelated device output prevents IDT for FreeRTOS from detecting the test result which is **PASS**.
Check the following to ensure optimal testing.
+ Make sure the logging macros used on the device are thread safe. See [Implementing the library logging macros](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-library-logging-macros.html) for more information.
+ Make sure there are minimal outputs to the serial connection during the tests. Other device outputs can be a problem even if your logging macros are properly thread safe, because the test results will output in separate calls during testing.
An IDT for FreeRTOS test case log would ideally show an uninterrupted test results output like below:
```
---------STARTING TESTS---------
TEST(Full_OTA_PAL, otaPal_CloseFile_ValidSignature) PASS
TEST(Full_OTA_PAL, otaPal_CloseFile_InvalidSignatureBlockWritten) PASS
-----------------------
2 Tests 0 Failures 0 Ignored
```
### Debug integrity check failures
If using FRQ 1.x.x version of FreeRTOS the following integrity checks apply.
When you run the FreeRTOSIntegrity test group and you encounter failures, first make sure that you haven't modified any of the `freertos` directory files. If you haven’t, and are still seeing issues, make sure you are using the correct branch. If you run IDT's `list-supported-products` command, you can find which tagged branch of the `freertos` repo you should be using.
If you cloned the correct tagged branch of the `freertos` repo and still have issues, make sure you have also run the `submodule update` command. The clone workflow for the `freertos` repo is as follows.
```
git clone --branch version-number https://github.com/aws/amazon-freertos.git
cd amazon-freertos
git submodule update --checkout —init —recursive
```
The list of files the integrity checker looks for are in the `checksums.json` file in your `freertos` directory. To qualify a FreeRTOS port without any modifications to files and the folder structure, make sure that none of the files listed in the '`exhaustive`' and '`minimal`' sections of the `checksums.json` file have been modified. To run with an SDK configured, verify that none of the files under the '`minimal`' section have been modified.
If you run IDT with an SDK and have modified some files in your `freertos` directory, then make sure you correctly configure your SDK in your `userdata` file. Otherwise, the Integrity checker will verify all files in the `freertos` directory.
### Debug FullWiFi test group failures
If you are using FRQ 1.x.x and encounter failures in the FullWiFi test group, and the "`AFQP_WiFiConnectMultipleAP`" test fails, this could be because both access points aren't in the same subnet as the host computer running IDT. Make sure that both access points are in the same subnet as the host computer running IDT.
### Debug "required parameter missing" errors
Because new features are being added to IDT for FreeRTOS, changes to the configuration files might be introduced. Using an old configuration file might break your configuration. If this happens, the `test_group_id__test_case_id.log` file under the `results/execution-id/logs` directory explicitly lists all missing parameters. IDT for FreeRTOS validates your JSON configuration file schemas to ensure that the latest supported version has been used.
### Debug "test could not start" errors
You might see errors that point to failures during test start. Because there are several possible causes, check the following areas for correctness:
+ Make sure that the pool name you've included in your execution command actually exists. This is referenced directly from your `device.json` file.
+ Make sure that the device or devices in your pool have correct configuration parameters.
### Debug "unable to find start of test results" errors
You might see errors when IDT attempts to parse the results output by the device under test. There are several possible causes, so check the following areas for correctness:
+ Make sure that the device under test has a stable connection to your host machine. You can check the log file for a test that shows these errors to see what IDT is receiving.
+ If using FRQ 1.x.x, and the device under test is connected via a slow network or other interface, or you do not see the "---------STARTING TESTS---------" flag in a FreeRTOS test group log along with other FreeRTOS test group outputs, you can try increasing the value of `testStartDelayms` in your userdata configuration. For more information, see [Configure build, flash, and test settings](cfg-dt-ud.md).
### Debug a "Test failure:expected \$1\$1 results but saw \$1\$1\$1" errors
You might see errors that point to a test failure during testing. The test expects a certain number of results, and does not see it during testing. Some FreeRTOS tests run before IDT sees the output from the device. If you see this error, you can try increasing the value of `testStartDelayms` in your *userdata* configuration. For more information, see [Configure build, flash, and test settings](lts-cfg-dt-ud.md).
### Debug a "\$1\$1\$1\$1\$1\$1\$1\$1 was unselected due to ConditionalTests constraints" errors
This means that you are running a test on a device pool that is incompatible with the test. This may happen with the OTA E2E tests. For example, while running the `OTADataplaneMQTT` test group and in your `device.json` config file, you have chosen OTA as **No** or `OTADataPlaneProtocol` as **HTTP**. The test group chosen to run must match your `device.json` capability selections.
### Debug an IDT timeout during device output monitoring
IDT can timeout due to a number of reasons. If a timeout happens during the device output monitoring phase of a test, and you can see the results inside of the IDT test case log, it means that the results were incorrectly parsed by IDT. One reason could be the interleaved log messages in the middle of the test results. If this is the case, please refer to the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-porting-ota.html) for further details on how the UNITY logs should be setup.
Another reason for a timeout during device output monitoring could be a device rebooting after a single TLS test case failure. The device then runs the flashed image and causes an infinite loop which is seen in the logs. If this happens, make sure your device does not reboot after a test failure.
### Debug a "not authorized to access resource" error
You might see the error "*user/role* is not authorized to access this resource" in the terminal output or in the `test_manager.log` file under `/results/execution-id/logs`. To resolve this issue, attach the `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy to your test user. For more information, see [Create and configure an AWS account](dev-tester-prereqs.md#config-aws-account).
### Debug network test errors
For network-based tests, IDT starts an echo server that binds to a non-reserved port on the host machine. If you are running into errors due to timeouts or unavailable connections in the WiFi or secure sockets tests, make sure that your network is configured to allow traffic to configured ports in the 1024 - 49151 range.
The secure sockets test uses ports 33333 and 33334 by default. The WiFi tests uses port 33335 by default. If these three ports are in use or blocked by firewall or network, you can choose to use different ports in userdata.json for testing. For more information, see [Configure build, flash, and test settings](cfg-dt-ud.md). You can use the following commands to check whether a specific port is in use:
+ Windows: `netsh advfirewall firewall show rule name=all | grep port`
+ Linux: `sudo netstat -pan | grep port`
+ macOS: `netstat -nat | grep port`
### OTA update failures due to same version payload
If OTA test cases are failing due to the same version being on the device after an OTA was performed, it may be due to your build system (e.g. cmake) not noticing IDT's changes to the FreeRTOS source code and not building an updated binary. This causes OTA to be performed with the same binary that is currently on the device, and the test to fail. To troubleshoot OTA update failures, start by making sure that you are using the latest supported version of your build system.
### OTA test failure on `PresignedUrlExpired` test case
One prerequisite of this test is that the OTA update time should be more than 60 seconds, otherwise the test would fail. If this occurs, the following error message is found in the log: "Test takes less than 60 seconds (url expired time) to finish. Please reach out to us."
### Debug device interface and port errors
This section contains information about the device interfaces IDT uses to connect to your devices.
#### Supported platforms
IDT supports Linux, macOS, and Windows. All three platforms have different naming schemes for serial devices that are attached to them:
+ Linux: `/dev/tty*`
+ macOS: `/dev/tty.*` or `/dev/cu.*`
+ Windows: COM\$1
To check your device port:
+ For Linux/macOS, open a terminal and run `ls /dev/tty*`.
+ For macOS, open a terminal and run `ls /dev/tty.*` or `ls /dev/cu.*`.
+ For Windows, open Device Manager and expand the serial devices group.
To verify which device is connected to a port:
+ For Linux, make sure that the `udev` package is installed, and then run `udevadm info –name=PORT`. This utility prints the device driver information that helps you verify you are using the correct port.
+ For macOS, open Launchpad and search for **System Information**.
+ For Windows, open Device Manager and expand the serial devices group.
#### Device interfaces
Each embedded device is different, which means that they can have one or more serial ports. It is common for devices to have two ports when connected to a machine:
+ A data port for flashing the device.
+ A read port to read output.
You must set the correct read port in your `device.json` file. Otherwise, reading output from the device might fail.
In the case of multiple ports, make sure to use the read port of the device in your `device.json` file. For example, if you plug in an Espressif WRover device and the two ports assigned to it are `/dev/ttyUSB0` and `/dev/ttyUSB1`, use `/dev/ttyUSB1` in your `device.json` file.
For Windows, follow the same logic.
#### Reading device data
IDT for FreeRTOS uses individual device build and flash tooling to specify port configuration. If you are testing your device and don't get output, try the following default settings:
+ Baud rate: 115200
+ Data bits: 8
+ Parity: None
+ Stop bits: 1
+ Flow control: None
These settings are handled by IDT for FreeRTOS. You do not have to set them. However, you can use the same method to manually read device output. On Linux or macOS, you can do this with the `screen` command. On Windows, you can use a program such as TeraTerm.
`Screen: screen /dev/cu.usbserial 115200`
`TeraTerm: Use the above-provided settings to set the fields explicitly in the GUI.`
### Development toolchain problems
This section discusses problems that can occur with your toolchain.
#### Code Composer Studio on Ubuntu
Newer versions of Ubuntu (17.10 and 18.04) have a version of the `glibc` package that is not compatible with Code Composer Studio 7.*x* versions. We recommended that you install Code Composer Studio version 8.2 or later.
Symptoms of incompatibility might include:
+ FreeRTOS failing to build or flash to your device.
+ The Code Composer Studio installer might freeze.
+ No log output is displayed in the console during the build or flash process.
+ Build command attempts to launch in GUI mode even when invoked as headless.
### Logging
IDT for FreeRTOS logs are placed in a single location. From the root IDT directory, these files are available under `results/execution-id/`:
+ `FRQ_Report.xml`
+ `awsiotdevicetester_report.xml`
+ `logs/test_group_id__test_case_id.log`
`FRQ_Report.xml` and `logs/test_group_id__test_case_id.log` are the most important logs to examine. `FRQ_Report.xml` contains information about which test cases failed with a specific error message. You can then use `logs/test_group_id__test_case_id.log` to dig further into the problem to get better context.
#### Console errors
When AWS IoT Device Tester is run, failures are reported to the console with brief messages. Look in `results/execution-id/logs/test_group_id__test_case_id.log` to learn more about the error.
#### Log errors
Each test suite execution has a unique execution ID that is used to create a folder named `results/execution-id`. Individual test case logs are under the `results/execution-id/logs` directory. Use the output of the IDT for FreeRTOS console to find the execution id, test case id, and test group id of the test case that failed. Then use this information to find and open the log file for that test case named `results/execution-id/logs/test_group_id__test_case_id.log` The information in this file includes the full build and flash command output, test execution output, and more verbose AWS IoT Device Tester console output.
#### S3 bucket issues
If you press CTRL\$1C while running IDT, IDT will start a clean up process. Part of that clean up is to remove Amazon S3 resources that have been created as a part of the IDT tests. If the clean up can't finish, you might run into an issue where you have too many Amazon S3 buckets that have been created. This means the next time that you run IDT the tests will start to fail.
If you press CTRL\$1C to stop IDT, you must let it finish the clean up process to avoid this issue. You can also delete the Amazon S3 buckets from your account that were created manually.
## Troubleshoot timeout errors
If you see timeout errors while running a test suite, increase the timeout by specifying a timeout multiplier factor. This factor is applied to the default timeout value. Any value configured for this flag must be greater than or equal to 1.0. To use the timeout multiplier, use the flag `--timeout-multiplier` when running the test suite.
**Example**
```
./devicetester_linux run-suite --suite-id FRQ_1.0.1 --pool-id DevicePool1 --timeout-multiplier 2.5
```
```
./devicetester_linux run-suite --suite-id FRQ_1 --pool-id DevicePool1 --timeout-multiplier 2.5
```
## Cellular feature and AWS charges
When the `Cellular` feature is set to `Yes` in your `device.JSON` file, FullSecureSockets will use t.micro EC2 instances for running tests and this may incur additional costs to your AWS account. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/).
## Qualification report generation policy
Qualification reports are only generated by AWS IoT Device Tester (IDT) versions that support FreeRTOS versions released within the last two years. If you have questions about the support policy, please contact [AWS Support](https://aws.amazon.com/contact-us/).
# Understand the AWS Managed policy for AWS IoT Device Tester
AWS Managed policy for AWS IoT Device TesterAWS managed policies
Added information about AWS managed policies for AWS IoT Device Tester.
An AWS managed policy is a standalone policy that is created and administered by AWS. AWS managed policies are designed to provide permissions for many common use cases so that you can start assigning permissions to users, groups, and roles.
Keep in mind that AWS managed policies might not grant least-privilege permissions for your specific use cases because they're available for all AWS customers to use. We recommend that you reduce permissions further by defining [ customer managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#customer-managed-policies) that are specific to your use cases.
You cannot change the permissions defined in AWS managed policies. If AWS updates the permissions defined in an AWS managed policy, the update affects all principal identities (users, groups, and roles) that the policy is attached to. AWS is most likely to update an AWS managed policy when a new AWS service is launched or new API operations become available for existing services.
For more information, see [AWS managed policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_managed-vs-inline.html#aws-managed-policies) in the *IAM User Guide*.
**Topics**
+ [
## AWS managed policy: AWSIoTDeviceTesterForFreeRTOSFullAccess
](#aws-managed-policies-AWSIoTDT)
+ [
## Updates to AWS managed policies
](#aws-managed-policy-updates)
## AWS managed policy: AWSIoTDeviceTesterForFreeRTOSFullAccess
Managed policy
The `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy contains the following AWS IoT Device Tester permissions for version checking, auto update features, and collection of metrics.
**Permission details**
This policy includes the following permissions:
+ `iot-device-tester:SupportedVersion`
Grants AWS IoT Device Tester permission to fetch the list of supported products, test suites and IDT versions.
+ `iot-device-tester:LatestIdt`
Grants AWS IoT Device Tester permission to fetch the latest IDT version available for download.
+ `iot-device-tester:CheckVersion`
Grants AWS IoT Device Tester permission to check version compatibility for IDT, test suites and products.
+ `iot-device-tester:DownloadTestSuite`
Grants AWS IoT Device Tester permission to download test suite updates.
+ `iot-device-tester:SendMetrics`
Grants AWS permission to collect metrics about AWS IoT Device Tester internal usage.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::*:role/idt-*",
"Condition": {
"StringEquals": {
"iam:PassedToService": "iot.amazonaws.com"
}
}
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"iot:DeleteThing",
"iot:AttachThingPrincipal",
"iot:DeleteCertificate",
"iot:GetRegistrationCode",
"iot:CreatePolicy",
"iot:UpdateCACertificate",
"s3:ListBucket",
"iot:DescribeEndpoint",
"iot:CreateOTAUpdate",
"iot:CreateStream",
"signer:ListSigningJobs",
"acm:ListCertificates",
"iot:CreateKeysAndCertificate",
"iot:UpdateCertificate",
"iot:CreateCertificateFromCsr",
"iot:DetachThingPrincipal",
"iot:RegisterCACertificate",
"iot:CreateThing",
"iam:ListRoles",
"iot:RegisterCertificate",
"iot:DeleteCACertificate",
"signer:PutSigningProfile",
"s3:ListAllMyBuckets",
"signer:ListSigningPlatforms",
"iot-device-tester:SendMetrics",
"iot-device-tester:SupportedVersion",
"iot-device-tester:LatestIdt",
"iot-device-tester:CheckVersion",
"iot-device-tester:DownloadTestSuite"
],
"Resource": "*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": [
"iam:GetRole",
"signer:StartSigningJob",
"acm:GetCertificate",
"signer:DescribeSigningJob",
"s3:CreateBucket",
"execute-api:Invoke",
"s3:DeleteBucket",
"s3:PutBucketVersioning",
"signer:CancelSigningProfile"
],
"Resource": [
"arn:aws:execute-api:us-east-1:098862408343:9xpmnvs5h4/prod/POST/metrics",
"arn:aws:signer:*:*:/signing-profiles/*",
"arn:aws:signer:*:*:/signing-jobs/*",
"arn:aws:iam::*:role/idt-*",
"arn:aws:acm:*:*:certificate/*",
"arn:aws:s3:::idt-*",
"arn:aws:s3:::afr-ota*"
]
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"iot:DeleteStream",
"iot:DeleteCertificate",
"iot:AttachPolicy",
"iot:DetachPolicy",
"iot:DeletePolicy",
"s3:ListBucketVersions",
"iot:UpdateCertificate",
"iot:GetOTAUpdate",
"iot:DeleteOTAUpdate",
"iot:DescribeJobExecution"
],
"Resource": [
"arn:aws:s3:::afr-ota*",
"arn:aws:iot:*:*:thinggroup/idt*",
"arn:aws:iam::*:role/idt-*"
]
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": [
"iot:DeleteCertificate",
"iot:AttachPolicy",
"iot:DetachPolicy",
"s3:DeleteObjectVersion",
"iot:DeleteOTAUpdate",
"s3:PutObject",
"s3:GetObject",
"iot:DeleteStream",
"iot:DeletePolicy",
"s3:DeleteObject",
"iot:UpdateCertificate",
"iot:GetOTAUpdate",
"s3:GetObjectVersion",
"iot:DescribeJobExecution"
],
"Resource": [
"arn:aws:s3:::afr-ota*/*",
"arn:aws:s3:::idt-*/*",
"arn:aws:iot:*:*:policy/idt*",
"arn:aws:iam::*:role/idt-*",
"arn:aws:iot:*:*:otaupdate/idt*",
"arn:aws:iot:*:*:thing/idt*",
"arn:aws:iot:*:*:cert/*",
"arn:aws:iot:*:*:job/*",
"arn:aws:iot:*:*:stream/*"
]
},
{
"Sid": "VisualEditor5",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::afr-ota*/*",
"arn:aws:s3:::idt-*/*"
]
},
{
"Sid": "VisualEditor6",
"Effect": "Allow",
"Action": [
"iot:CancelJobExecution"
],
"Resource": [
"arn:aws:iot:*:*:job/*",
"arn:aws:iot:*:*:thing/idt*"
]
},
{
"Sid": "VisualEditor7",
"Effect": "Allow",
"Action": [
"ec2:TerminateInstances"
],
"Resource": [
"arn:aws:ec2:*:*:instance/*"
],
"Condition": {
"StringEquals": {
"ec2:ResourceTag/Owner": "IoTDeviceTester"
}
}
},
{
"Sid": "VisualEditor8",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:DeleteSecurityGroup"
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*"
],
"Condition": {
"StringEquals": {
"ec2:ResourceTag/Owner": "IoTDeviceTester"
}
}
},
{
"Sid": "VisualEditor9",
"Effect": "Allow",
"Action": [
"ec2:RunInstances"
],
"Resource": [
"arn:aws:ec2:*:*:instance/*"
],
"Condition": {
"StringEquals": {
"aws:RequestTag/Owner": "IoTDeviceTester"
}
}
},
{
"Sid": "VisualEditor10",
"Effect": "Allow",
"Action": [
"ec2:RunInstances"
],
"Resource": [
"arn:aws:ec2:*:*:image/*",
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:volume/*",
"arn:aws:ec2:*:*:key-pair/*",
"arn:aws:ec2:*:*:placement-group/*",
"arn:aws:ec2:*:*:snapshot/*",
"arn:aws:ec2:*:*:network-interface/*",
"arn:aws:ec2:*:*:subnet/*"
]
},
{
"Sid": "VisualEditor11",
"Effect": "Allow",
"Action": [
"ec2:CreateSecurityGroup"
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*"
],
"Condition": {
"StringEquals": {
"aws:RequestTag/Owner": "IoTDeviceTester"
}
}
},
{
"Sid": "VisualEditor12",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeSecurityGroups",
"ssm:DescribeParameters",
"ssm:GetParameters"
],
"Resource": "*"
},
{
"Sid": "VisualEditor13",
"Effect": "Allow",
"Action": [
"ec2:CreateTags"
],
"Resource": [
"arn:aws:ec2:*:*:security-group/*",
"arn:aws:ec2:*:*:instance/*"
],
"Condition": {
"ForAnyValue:StringEquals": {
"aws:TagKeys": [
"Owner"
]
},
"StringEquals": {
"ec2:CreateAction": [
"RunInstances",
"CreateSecurityGroup"
]
}
}
}
]
}
```
------
## Updates to AWS managed policies
Policy updates
You can view details about updates to AWS managed policies for AWS IoT Device Tester from the time this service began tracking these changes.
| Version | Change | Description | Date |
| --- | --- | --- | --- |
| 7 (Latest) | Restructured the `ec2:CreateTags` conditions. | Removing usage of `ForAnyValues`. | 6/14/2023 |
| 6 | Removed `freertos:ListHardwarePlatforms` from the policy. | Removing permissions as this action is deprecated as of March 1st, 2023. | 6/2/2023 |
| 5 | Added permissions to run echo server tests using EC2. | This is for starting and stopping an EC2 instance in customers' AWS accounts. | 12/15/2020 |
| 4 | Added `iot:CancelJobExecution`. | This permission cancels OTA jobs. | 7/17/2020 |
| 3 | Added the following permissions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/security-iam-aws-managed-policies.html) | [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/freertos/latest/userguide/security-iam-aws-managed-policies.html) | 3/23/2020 |
| 2 | Added `iot-device-tester:SendMetrics` permissions. | Grants AWS permission to collect metrics about AWS IoT Device Tester internal usage. | 2/18/2020 |
| 1 | Initial version. | | 2/12/2020 |
# Understand the support policy for AWS IoT Device Tester
Support policy
**Important**
As of October 2022, AWS IoT Device Tester for AWS IoT FreeRTOS Qualification (FRQ) 1.0 does not generate signed qualification reports. You cannot qualify new AWS IoT FreeRTOS devices to list in the [AWS Partner Device Catalog ](https://partners.amazonaws.com/qualified-devices)through the [AWS Device Qualification Program](http://aws.amazon.com/partners/programs/dqp/) using IDT FRQ 1.0 versions. While you can't qualify FreeRTOS devices using IDT FRQ 1.0, you can continue to test your FreeRTOS devices with FRQ 1.0. We recommend that you use [IDT FRQ 2.0](https://docs.aws.amazon.com/freertos/latest/userguide/lts-idt-freertos-qualification.html) to qualify and list FreeRTOS devices in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices).
AWS IoT Device Tester for FreeRTOS is a test automation tool to validate FreeRTOS port to devices. Additionally you can [qualify](https://aws.amazon.com/partners/programs/dqp/) your FreeRTOS devices and list them on the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices) . The AWS IoT Device Tester for FreeRTOS supports validation and qualification of FreeRTOS Long Term Supported (LTS) libraries available on GitHub at [FreeRTOS/FreeRTOS-LTS ](https://github.com/FreeRTOS/FreeRTOS-LTS), and FreeRTOS mainline available at [FreeRTOS/FreeRTOS](https://github.com/FreeRTOS/FreeRTOS). We recommend that you use the most recent versions of both the FreeRTOS and AWS IoT Device Tester for FreeRTOS to validate and qualify your devices.
For FreeRTOS-LTS, IDT supports the validation and qualification of FreeRTOS 202210 LTS version. See here for more information on [FreeRTOS LTS releases](https://www.freertos.org/lts-libraries.html) and their maintenance time line. Once the support period of these LTS releases ends, you can still continue validation, but IDT will not generate a report, that will allow you to submit your device for qualification.
For the mainline FreeRTOS available at [FreeRTOS/FreeRTOS ](https://github.com/FreeRTOS/FreeRTOS), we support the validation and qualification of all the versions released in the past six months, or the previous two versions of FreeRTOS if released more than six months apart. See here for [currently supported versions]( https://docs.aws.amazon.com//freertos/latest/userguide/dev-test-versions-afr.html). For unsupported versions of FreeRTOS, you can still continue validation, but IDT will not generate a report, that will allow you to submit your device for qualification.
See [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md) for the latest supported IDT and FreeRTOS versions. You can use any of the supported versions of AWS IoT Device Tester with the corresponding version of FreeRTOS to test or qualify your device. If you continue to use the [Unsupported IDT versions for FreeRTOS](idt-unsupported-versions-afr.md), you will not receive the latest bug fixes or updates.
For questions about the support policy, contact [AWS Customer Support](https://aws.amazon.com/contact-us/).