# IDT with FreeRTOS qualification suite 1.0 (FRQ 1.0) **Important** As of October 2022, AWS IoT Device Tester for AWS IoT FreeRTOS Qualification (FRQ) 1.0 does not generate signed qualification reports. You cannot qualify new AWS IoT FreeRTOS devices to list in the [AWS Partner Device Catalog ](https://partners.amazonaws.com/qualified-devices)through the [AWS Device Qualification Program](http://aws.amazon.com/partners/programs/dqp/) using IDT FRQ 1.0 versions. While you can't qualify FreeRTOS devices using IDT FRQ 1.0, you can continue to test your FreeRTOS devices with FRQ 1.0. We recommend that you use [IDT FRQ 2.0](https://docs.aws.amazon.com/freertos/latest/userguide/lts-idt-freertos-qualification.html) to qualify and list FreeRTOS devices in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices). You can use IDT for FreeRTOS qualification to verify that the FreeRTOS operating system works locally on your device and can communicate with AWS IoT. Specifically, it verifies that the porting layer interfaces for the FreeRTOS libraries are implemented correctly. It also performs end-to-end tests with AWS IoT Core. For example, it verifies your board can send and receive MQTT messages and process them correctly. The tests run by IDT for FreeRTOS are defined in the [FreeRTOS GitHub repository](https://github.com/aws/amazon-freertos). The tests run as embedded applications that are flashed onto your board. The application binary images include FreeRTOS, the semiconductor vendor’s ported FreeRTOS interfaces, and board device drivers. The purpose of the tests is to verify the ported FreeRTOS interfaces function correctly on top of the device drivers. IDT for FreeRTOS generates test reports that you can submit to AWS IoT to add your hardware to the AWS Partner Device Catalog. For more information, see [AWS Device Qualification Program](https://aws.amazon.com/partners/dqp/). IDT for FreeRTOS runs on a host computer (Windows, macOS, or Linux) that is connected to the board to be tested. IDT executes test cases and aggregates results. It also provides a command line interface to manage test execution. In addition to testing devices, IDT for FreeRTOS creates resources (for example, AWS IoT things, FreeRTOS groups, Lambda functions, and so on) to facilitate the qualification process. To create these resources, IDT for FreeRTOS uses the AWS credentials configured in the `config.json` to make API calls on your behalf. These resources are provisioned at various times during a test. When you run IDT for FreeRTOS on your host computer, it performs the following steps: 1. Loads and validates your device and credentials configuration. 1. Performs selected tests with the required local and cloud resources. 1. Cleans up local and cloud resources. 1. Generates tests reports that indicate if your board passed the tests required for qualification. **Topics** + [Setup the 1.0 qualification prerequisites](dev-tester-prereqs.md) + [First test of your microcontroller board](qual-steps.md) + [Use the IDT user interface to run the FreeRTOS qualification suite](device-tester-ui.md) + [Run Bluetooth Low Energy tests](afr-bridgekeeper-dt-bt.md) + [Run the FreeRTOS qualification suite](run-tests.md) + [View the IDT for FreeRTOS results](view-results-frq.md) + [Interpret the IDT for FreeRTOS results](interpreting-results-frq.md) + [View the IDT for FreeRTOS logs](view-logs-frq.md) # Setup the 1.0 qualification prerequisites This section describes the prerequisites for testing microcontrollers with AWS IoT Device Tester. ## Download FreeRTOS You can download a release of FreeRTOS from [GitHub](https://github.com/aws/amazon-freertos) with the following command: ``` git clone --branch --recurse-submodules https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout --init --recursive ``` where is a version of FreeRTOS (for example, 202007.00) corresponding to an IDT version listed in [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md). This ensures you have the full source code, including submodules, and are using the correct version of IDT for your version of FreeRTOS, and vice versa. Windows has a path length limitation of 260 characters. The path structure of FreeRTOS is many levels deep, so if you are using Windows, keep your file paths under the 260-character limit. For example, clone FreeRTOS to `C:\FreeRTOS` rather than `C:\Users\username\programs\projects\myproj\FreeRTOS\`. ### FreeRTOS qualification with LTS libraries + In order for your microcontroller to be designated as supporting long-term support (LTS) based versions of FreeRTOS in the AWS Partner Device Catalog, you must provide a manifest file. For more information, see the [ FreeRTOS Qualification Checklist](https://docs.aws.amazon.com/freertos/latest/qualificationguide/afq-checklist.html) in the *FreeRTOS Qualification Guide*. + In order to validate that your microcontroller supports LTS based versions of FreeRTOS and qualify it for submission to the AWS Partner Device Catalog, you must use AWS IoT Device Tester (IDT) with FreeRTOS Qualification (FRQ) test suite version v1.4.x. + Support for LTS based versions of FreeRTOS is limited to the 202012.xx version of FreeRTOS. ## Download IDT for FreeRTOS Every version of FreeRTOS has a corresponding version of IDT for FreeRTOS to perform qualification tests. Download the appropriate version of IDT for FreeRTOS from [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md). Extract IDT for FreeRTOS to a location on the file system where you have read and write permissions. Because Microsoft Windows has a character limit for the path length, extract IDT for FreeRTOS into a root directory such as `C:\` or `D:\`. **Note** We don't recommend that multiple users run IDT from a shared location, such as an NFS directory or a Windows network shared folder. This may result in crashes or data corruption. We recommend that you extract the IDT package to a local drive. ## Create and configure an AWS account ### Sign up for an AWS account If you do not have an AWS account, complete the following steps to create one. **To sign up for an AWS account** 1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup). 1. Follow the online instructions. Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad. When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks). AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**. ### Create a user with administrative access After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks. **Secure your AWS account root user** 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password. For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*. 1. Turn on multi-factor authentication (MFA) for your root user. For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*. **Create a user with administrative access** 1. Enable IAM Identity Center. For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*. 1. In IAM Identity Center, grant administrative access to a user. For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*. **Sign in as the user with administrative access** + To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*. **Assign access to additional users** 1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions. For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*. 1. Assign users to a group, and then assign single sign-on access to the group. For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*. ## AWS IoT Device Tester managed policy The `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy contains the following AWS IoT Device Tester permissions for version checking, auto update features, and collection of metrics. + `iot-device-tester:SupportedVersion` Grants AWS IoT Device Tester permission to fetch the list of supported products, test suites and IDT versions. + `iot-device-tester:LatestIdt` Grants AWS IoT Device Tester permission to fetch the latest IDT version available for download. + `iot-device-tester:CheckVersion` Grants AWS IoT Device Tester permission to check version compatibility for IDT, test suites and products. + `iot-device-tester:DownloadTestSuite` Grants AWS IoT Device Tester permission to download test suite updates. + `iot-device-tester:SendMetrics` Grants AWS permission to collect metrics about AWS IoT Device Tester internal usage. ## (Optional) Install the AWS Command Line Interface You might prefer to use the AWS CLI to perform some operations. If you don't have the AWS CLI installed, follow the instructions at [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/installing.html). Configure the AWS CLI for the AWS Region you want to use by running **aws configure** from a command line. For information about the AWS Regions that support IDT for FreeRTOS, see [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#amazon-freertos-ota-control). For more information about **aws configure** see [ Quick configuration with **aws configure**](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config). # First test of your microcontroller board You can use IDT for FreeRTOS to test as you port the FreeRTOS interfaces. After you have ported the FreeRTOS interfaces for your board’s device drivers, you use AWS IoT Device Tester to run the qualification tests on your microcontroller board. ## Add library porting layers To port FreeRTOS for your device, follow the instructions in the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/). ## Configure your AWS credentials You need to configure your AWS credentials for AWS IoT Device Tester to communicate with the AWS Cloud. For more information, see [Set up AWS Credentials and Region for Development](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/setup-credentials.html). Valid AWS credentials must be specified in the `devicetester_extract_location/devicetester_afreertos_[win|mac|linux]/configs/config.json` configuration file. **Topics** + [Add library porting layers](#add-port-layer) + [Configure your AWS credentials](#cfg-aws-afr) + [Create a device pool in IDT for FreeRTOS](cfg-dt-dp.md) + [Configure build, flash, and test settings](cfg-dt-ud.md) # Create a device pool in IDT for FreeRTOS Devices to be tested are organized in device pools. Each device pool consists of one or more identical devices. You can configure IDT for FreeRTOS to test a single device in a pool or multiple devices in a pool. To accelerate the qualification process, IDT for FreeRTOS can test devices with the same specifications in parallel. It uses a round-robin method to execute a different test group on each device in a device pool. You can add one or more devices to a device pool by editing the `devices` section of the `device.json` template in the `configs` folder. **Note** All devices in the same pool must be of same technical specification and SKU. To enable parallel builds of the source code for different test groups, IDT for FreeRTOS copies the source code to a results folder inside the IDT for FreeRTOS extracted folder. The source code path in your build or flash command must be referenced using either the `testdata.sourcePath` or `sdkPath` variable. IDT for FreeRTOS replaces this variable with a temporary path of the copied source code. For more information see, [IDT for FreeRTOS variables](dt-vars.md). The following is an example `device.json` file used to create a device pool with multiple devices. ``` [ { "id": "pool-id", "sku": "sku", "features": [ { "name": "WIFI", "value": "Yes | No" }, { "name": "Cellular", "value": "Yes | No" }, { "name": "OTA", "value": "Yes | No", "configs": [ { "name": "OTADataPlaneProtocol", "value": "HTTP | MQTT" } ] }, { "name": "BLE", "value": "Yes | No" }, { "name": "TCP/IP", "value": "On-chip | Offloaded | No" }, { "name": "TLS", "value": "Yes | No" }, { "name": "PKCS11", "value": "RSA | ECC | Both | No" }, { "name": "KeyProvisioning", "value": "Import | Onboard | No" } ], "devices": [ { "id": "device-id", "connectivity": { "protocol": "uart", "serialPort": "/dev/tty*" }, ***********Remove the section below if the device does not support onboard key generation*************** "secureElementConfig" : { "publicKeyAsciiHexFilePath": "absolute-path-to/public-key-txt-file: contains-the-hex-bytes-public-key-extracted-from-onboard-private-key", "secureElementSerialNumber": "secure-element-serialNo-value", "preProvisioned" : "Yes | No" }, ********************************************************************************************************** "identifiers": [ { "name": "serialNo", "value": "serialNo-value" } ] } ] } ] ``` The following attributes are used in the `device.json` file: **`id`** A user-defined alphanumeric ID that uniquely identifies a pool of devices. Devices that belong to a pool must be of the same type. When a suite of tests is running, devices in the pool are used to parallelize the workload. **`sku`** An alphanumeric value that uniquely identifies the board you are testing. The SKU is used to track qualified boards. If you want to list your board in AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process. **`features`** An array that contains the device's supported features. AWS IoT Device Tester uses this information to select the qualification tests to run. Supported values are: **`TCP/IP`** Indicates if your board supports a TCP/IP stack and whether it is supported on-chip (MCU) or offloaded to another module. TCP/IP is required for qualification. **`WIFI`** Indicates if your board has Wi-Fi capabilities. Must be set to `No` if `Cellular` is set to `Yes`. **`Cellular`** Indicates if your board has cellular capabilities. Must be set to `No` if `WIFI` is set to `Yes`. When this feature is set to `Yes`, the FullSecureSockets test will be executed using AWS t2.micro EC2 instances and this may incur additional costs to your account. For more information, see [Amazon EC2 pricing](https://aws.amazon.com/ec2/pricing/). **`TLS`** Indicates if your board supports TLS. TLS is required for qualification. **`PKCS11`** Indicates the public key cryptography algorithm that the board supports. PKCS11 is required for qualification. Supported values are `ECC`, `RSA`, `Both` and `No`. `Both` indicates the board supports both the `ECC` and `RSA` algorithms. **`KeyProvisioning`** Indicates the method of writing a trusted X.509 client certificate onto your board. Valid values are `Import`, `Onboard` and `No`. Key provisioning is required for qualification. + Use `Import` if your board allows the import of private keys. IDT will create a private key and build this to the FreeRTOS source code. + Use `Onboard` if your board supports on-board private key generation (for example, if your device has a secure element, or if you prefer to generate your own device key pair and certificate). Make sure you add a `secureElementConfig` element in each of the device sections and put the absolute path to the public key file in the `publicKeyAsciiHexFilePath` field. + Use `No` if your board does not support key provisioning. **`OTA`** Indicates if your board supports over-the-air (OTA) update functionality. The `OtaDataPlaneProtocol` attribute indicates which OTA dataplane protocol the device supports. The attribute is ignored if the OTA feature is not supported by the device. When `"Both"` is selected, the OTA test execution time is prolonged due to running both MQTT, HTTP, and mixed tests. Starting with IDT v4.1.0, `OtaDataPlaneProtocol` accepts only `HTTP` and `MQTT` as supported values. **`BLE`** Indicates if your board supports Bluetooth Low Energy (BLE). **`devices.id`** A user-defined unique identifier for the device being tested. **`devices.connectivity.protocol`** The communication protocol used to communicate with this device. Supported value: `uart`. **`devices.connectivity.serialPort`** The serial port of the host computer used to connect to the devices being tested. **`devices.secureElementConfig.PublicKeyAsciiHexFilePath`** The absolute path to the file that contains the hex bytes public key extracted from onboard private key. Example format: ``` 3059 3013 0607 2a86 48ce 3d02 0106 082a 8648 ce3d 0301 0703 4200 04cd 6569 ceb8 1bb9 1e72 339f e8cf 60ef 0f9f b473 33ac 6f19 1813 6999 3fa0 c293 5fae 08f1 1ad0 41b7 345c e746 1046 228e 5a5f d787 d571 dcb2 4e8d 75b3 2586 e2cc 0c ``` If your public key is in .der format, you can hex encode the public key directly to generate the hex file. Example command for .der public key to generate hex file: ``` xxd -p pubkey.der > outFile ``` If your public key is in .pem format, you can extract the base64 encoded part, decode it into binary format, and then hex encode it to generate the hex file. For example, use these commands to generate a hex file for a .pem public key: 1. Take out the base64 encoded part of the key (strip the header and footer ) and store it in a file, for example name it `base64key`, run this command to convert it to .der format: ``` base64 —decode base64key > pubkey.der ``` 1. Run the `xxd` command to convert it to hex format. ``` xxd -p pubkey.der > outFile ``` **`devices.secureElementConfig.SecureElementSerialNumber`** (Optional) The serial number of the secure element. Provide this field when the serial number is printed out along with the device public key when you run the FreeRTOS demo/test project. **`devices.secureElementConfig.preProvisioned`** (Optional) Set to "Yes" if the device has a pre-provisioned secure element with locked-down credentials, that cannot import, create, or destroy objects. This configuration takes effect only when `features` has `KeyProvisioning` set to "Onboard", along with `PKCS11` set to "ECC". **`identifiers`** (Optional) An array of arbitrary name-value pairs. You can use these values in the build and flash commands described in the next section. # Configure build, flash, and test settings For IDT for FreeRTOS to build and flash tests on to your board automatically, you must configure IDT to run the build and flash commands for your hardware. The build and flash command settings are configured in the `userdata.json` template file located in the `config` folder. # Configure settings for testing devices Build, flash, and test settings are made in the `configs/userdata.json` file. We support Echo Server configuration by loading both the client and server certificates and keys in the `customPath`. For more information, see [Setting up an echo server](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-echo-server.html) in the *FreeRTOS Porting Guide*. The following JSON example shows how you can configure IDT for FreeRTOS to test multiple devices: ``` { "sourcePath": "/absolute-path-to/freertos", "vendorPath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name", // ***********The sdkConfiguration block below is needed if you are not using the default, unmodified FreeRTOS repo. // In other words, if you are using the default, unmodified FreeRTOS repo then remove this block*************** "sdkConfiguration": { "name": "sdk-name", "version": "sdk-version", "path": "/absolute-path-to/sdk" }, "buildTool": { "name": "your-build-tool-name", "version": "your-build-tool-version", "command": [ "{{config.idtRootPath}}/relative-path-to/build-parallel.sh {{testData.sourcePath}} {{enableTests}}" ] }, "flashTool": { "name": "your-flash-tool-name", "version": "your-flash-tool-version", "command": [ "/{{config.idtRootPath}}/relative-path-to/flash-parallel.sh {{testData.sourcePath}} {{device.connectivity.serialPort}} {{buildImageName}}" ], "buildImageInfo" : { "testsImageName": "tests-image-name", "demosImageName": "demos-image-name" } }, "testStartDelayms": 0, "clientWifiConfig": { "wifiSSID": "ssid", "wifiPassword": "password", "wifiSecurityType": "eWiFiSecurityOpen | eWiFiSecurityWEP | eWiFiSecurityWPA | eWiFiSecurityWPA2 | eWiFiSecurityWPA3" }, "testWifiConfig": { "wifiSSID": "ssid", "wifiPassword": "password", "wifiSecurityType": "eWiFiSecurityOpen | eWiFiSecurityWEP | eWiFiSecurityWPA | eWiFiSecurityWPA2 | eWiFiSecurityWPA3" }, //********** //This section is used to start echo server based on server certificate generation method, //When certificateGenerationMethod is set as Automatic specify the eccCurveFormat to generate certifcate and key based on curve format, //When certificateGenerationMethod is set as Custom specify the certificatePath and PrivateKeyPath to be used to start echo server //********** "echoServerCertificateConfiguration": { "certificateGenerationMethod": "Automatic | Custom", "customPath": { "clientCertificatePath":"/path/to/clientCertificate", "clientPrivateKeyPath": "/path/to/clientPrivateKey", "serverCertificatePath":"/path/to/serverCertificate", "serverPrivateKeyPath": "/path/to/serverPrivateKey" }, "eccCurveFormat": "P224 | P256 | P384 | P521" }, "echoServerConfiguration": { "securePortForSecureSocket": 33333, // Secure tcp port used by SecureSocket test. Default value is 33333. Ensure that the port configured isn't blocked by the firewall or your corporate network "insecurePortForSecureSocket": 33334, // Insecure tcp port used by SecureSocket test. Default value is 33334. Ensure that the port configured isn't blocked by the firewall or your corporate network "insecurePortForWiFi": 33335 // Insecure tcp port used by Wi-Fi test. Default value is 33335. Ensure that the port configured isn't blocked by the firewall or your corporate network }, "otaConfiguration": { "otaFirmwareFilePath": "{{testData.sourcePath}}/relative-path-to/ota-image-generated-in-build-process", "deviceFirmwareFileName": "ota-image-name-on-device", "otaDemoConfigFilePath": "{{testData.sourcePath}}/relative-path-to/ota-demo-config-header-file", "codeSigningConfiguration": { "signingMethod": "AWS | Custom", "signerHashingAlgorithm": "SHA1 | SHA256", "signerSigningAlgorithm": "RSA | ECDSA", "signerCertificate": "arn:partition:service:region:account-id:resource:qualifier | /absolute-path-to/signer-certificate-file", "signerCertificateFileName": "signerCertificate-file-name", "compileSignerCertificate": boolean, // ***********Use signerPlatform if you choose aws for signingMethod*************** "signerPlatform": "AmazonFreeRTOS-Default | AmazonFreeRTOS-TI-CC3220SF", "untrustedSignerCertificate": "arn:partition:service:region:account-id:resourcetype:resource:qualifier", // ***********Use signCommand if you choose custom for signingMethod*************** "signCommand": [ "/absolute-path-to/sign.sh {{inputImageFilePath}} {{outputSignatureFilePath}}" ] } }, // ***********Remove the section below if you're not configuring CMake*************** "cmakeConfiguration": { "boardName": "board-name", "vendorName": "vendor-name", "compilerName": "compiler-name", "frToolchainPath": "/path/to/freertos/toolchain", "cmakeToolchainPath": "/path/to/cmake/toolchain" }, "freertosFileConfiguration": { "required": [ { "configName": "pkcs11Config", "filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_tests/config_files/core_pkcs11_config.h" }, { "configName": "pkcs11TestConfig", "filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_tests/config_files/iot_test_pkcs11_config.h" } ], "optional": [ { "configName": "otaAgentTestsConfig", "filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_tests/config_files/ota_config.h" }, { "configName": "otaAgentDemosConfig", "filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_demos/config_files/ota_config.h" }, { "configName": "otaDemosConfig", "filePath": "{{testData.sourcePath}}/vendors/vendor-name/boards/board-name/aws_demos/config_files/ota_demo_config.h" } ] } } ``` The following lists the attributes used in `userdata.json`: **`sourcePath`** The path to the root of the ported FreeRTOS source code. For parallel testing with an SDK, the `sourcePath` can be set using the `{{userData.sdkConfiguration.path}}` place holder. For example: ``` { "sourcePath":"{{userData.sdkConfiguration.path}}/freertos" } ``` **`vendorPath`** The path to the vendor specific FreeRTOS code. For serial testing, the `vendorPath` can be set as an absolute path. For example: ``` { "vendorPath":"C:/path-to-freertos/vendors/espressif/boards/esp32" } ``` For parallel testing, the `vendorPath` can be set using the `{{testData.sourcePath}}` place holder. For example: ``` { "vendorPath":"{{testData.sourcePath}}/vendors/espressif/boards/esp32" } ``` The `vendorPath` variable is only necessary when running without an SDK, it can be removed otherwise. When running tests in parallel without an SDK, the `{{testData.sourcePath}}` placeholder must be used in the `vendorPath`, `buildTool`, `flashTool` fields. When running test with a single device, absolute paths must be used in the `vendorPath`, `buildTool`, `flashTool` fields. When running with an SDK, the `{{sdkPath}}` placeholder must be used in the `sourcePath`, `buildTool`, and `flashTool` commands. **`sdkConfiguration`** If you are qualifying FreeRTOS with any modifications to files and folder structure beyond what is required for porting, then you will need to configure your SDK information in this block. If you're not qualifying with a ported FreeRTOS inside of an SDK, then you should omit this block entirely. **`sdkConfiguration.name`** The name of the SDK you're using with FreeRTOS. If you're not using an SDK, then the entire `sdkConfiguration` block should be omitted. **`sdkConfiguration.version`** The version of the SDK you're using with FreeRTOS. If you're not using an SDK, then the entire `sdkConfiguration` block should be omitted. **`sdkConfiguration.path`** The absolute path to your SDK directory that contains your FreeRTOS code. If you're not using an SDK, then the entire `sdkConfiguration` block should be omitted. **`buildTool`** The full path to your build script (.bat or .sh) that contains the commands to build your source code. All references to the source code path in the build command must be replaced by the AWS IoT Device Tester variable `{{testdata.sourcePath}}` and references to the SDK path should be replaced by `{{sdkPath}}`. Use the `{{config.idtRootPath}}` placeholder to reference the absolute or relative IDT path. **`testStartDelayms`** Specifies how many milliseconds the FreeRTOS test runner will wait before starting to run tests. This can be useful if the device under test begins outputting important test information before IDT has a chance to connect and start logging due to network or other latency. The max allowed value is 30000 ms (30 seconds). This value is applicable to FreeRTOS test groups only, and not applicable to other test groups that do not utilize the FreeRTOS test runner, such as the OTA tests. **`flashTool`** Full path to your flash script (.sh or .bat) that contains the flash commands for your device. All references to the source code path in the flash command must be replaced by the IDT for FreeRTOS variable `{{testdata.sourcePath}}` and all references to your SDK path must be replaced by the IDT for FreeRTOS variable `{{sdkPath}}`.Use the `{{config.idtRootPath}}` placeholder to reference the absolute or relative IDT path. **`buildImageInfo`** **`testsImageName`** The name of the file produced by the build command when building tests from the `freertos-source/tests` folder. **`demosImageName`** The name of the file produced by the build command when building tests from the `freertos-source/demos` folder. **`clientWifiConfig`** The client Wi-Fi configuration. The Wi-Fi library tests require an MCU board to connect to two access points. (The two access points can be the same.) This attribute configures the Wi-Fi settings for the first access point. Some of the Wi-Fi test cases expect the access point to have some security and not to be open. Please make sure both access points are on the same subnet as the host computer running IDT. **`wifi_ssid`** The Wi-Fi SSID. **`wifi_password`** The Wi-Fi password. **`wifiSecurityType`** The type of Wi-Fi security used. One of the values: + `eWiFiSecurityOpen` + `eWiFiSecurityWEP` + `eWiFiSecurityWPA` + `eWiFiSecurityWPA2` + `eWiFiSecurityWPA3` If your board does not support Wi-Fi, you must still include the `clientWifiConfig` section in your `device.json` file, but you can omit values for these attributes. **`testWifiConfig`** The test Wi-Fi configuration. The Wi-Fi library tests require an MCU board to connect to two access points. (The two access points can be the same.) This attribute configures the Wi-Fi setting for the second access point. Some of the Wi-Fi test cases expect the access point to have some security and not to be open. Please make sure both access points are on the same subnet as the host computer running IDT. **`wifiSSID`** The Wi-Fi SSID. **`wifiPassword`** The Wi-Fi password. **`wifiSecurityType`** The type of Wi-Fi security used. One of the values: + `eWiFiSecurityOpen` + `eWiFiSecurityWEP` + `eWiFiSecurityWPA` + `eWiFiSecurityWPA2` + `eWiFiSecurityWPA3` If your board does not support Wi-Fi, you must still include the `testWifiConfig` section in your `device.json` file, but you can omit values for these attributes. **`echoServerCertificateConfiguration`** The configurable echo server certificate generation placeholder for secure socket tests. This field is required. **`certificateGenerationMethod`** Specifies whether the server certificate is generated automatically or provided manually. **`customPath`** If `certificateGenerationMethod` is "Custom", `certificatePath` and `privateKeyPath` are required. **`certificatePath`** Specifies the filepath for the server certificate. **`privateKeyPath`** Specifies the filepath for the private key. **`eccCurveFormat`** Specifies the curve format supported by the board. Required when `PKCS11` is set to "ecc" in `device.json`. Valid values are "P224", "P256", "P384", or "P521". **`echoServerConfiguration`** The configurable echo server ports for WiFi and secure sockets tests. This field is optional. **`securePortForSecureSocket`** The port which is used to setup an echo server with TLS for the secure sockets test. The default value is 33333. Ensure the port configured is not blocked by a firewall or your corporate network. **`insecurePortForSecureSocket`** The port which is used to setup echo server without TLS for the secure sockets test. The default value used in the test is 33334. Ensure the port configured is not blocked by a firewall or your corporate network. **`insecurePortForWiFi`** The port which is used to setup echo server without TLS for WiFi test. The default value used in the test is 33335. Ensure the port configured is not blocked by a firewall or your corporate network. **`otaConfiguration`** The OTA configuration. [Optional] **`otaFirmwareFilePath`** The full path to the OTA image created after the build. For example, `{{testData.sourcePath}}/relative-path/to/ota/image/from/source/root`. **`deviceFirmwareFileName`** The full file path on the MCU device where the OTA firmware is located. Some devices do not use this field, but you still must provide a value. **`otaDemoConfigFilePath`** The full path to `aws_demo_config.h`, found in `afr-source/vendors/vendor/boards/board/aws_demos/config_files/`. These files are included in the porting code template that FreeRTOS provides. **`codeSigningConfiguration`** The code signing configuration. **`signingMethod`** The code signing method. Possible values are `AWS` or `Custom`. For the Beijing and Ningxia Regions, use `Custom`. `AWS` code signing isn't supported in these Regions. **`signerHashingAlgorithm`** The hashing algorithm supported on the device. Possible values are `SHA1` or `SHA256`. **`signerSigningAlgorithm`** The signing algorithm supported on the device. Possible values are `RSA` or `ECDSA`. **`signerCertificate`** The trusted certificate used for OTA. For AWS code signing method, use the Amazon Resource Name (ARN) for the trusted certificate uploaded to the AWS Certificate Manager. For Custom code signing method, use the absolute path to the signer certificate file. For more information about creating a trusted certificate, see [Create a code-signing certificate](ota-code-sign-cert.md). **`signerCertificateFileName`** The file name of the code signing certificate on the device. This value must match the file name that you provided when you ran the `aws acm import-certificate` command. For more information, see [Create a code-signing certificate](ota-code-sign-cert.md). **`compileSignerCertificate`** Set to `true` if the code signer signature verification certificate isn't provisioned or flashed, so it must be compiled into the project. AWS IoT Device Tester fetches the trusted certificate and compiles it into `aws_codesigner_certifiate.h`. **`untrustedSignerCertificate`** The ARN or filepath for a second certificate used in some OTA tests as an untrusted certificate. For more information about creating a certificate, see [ Create a code-signing certificate](https://docs.aws.amazon.com/freertos/latest/userguide/ota-code-sign-cert.html). **`signerPlatform`** The signing and hashing algorithm that AWS Code Signer uses while creating the OTA update job. Currently, the possible values for this field are `AmazonFreeRTOS-TI-CC3220SF` and `AmazonFreeRTOS-Default`. + Choose `AmazonFreeRTOS-TI-CC3220SF` if `SHA1` and `RSA`. + Choose `AmazonFreeRTOS-Default` if `SHA256` and `ECDSA`. If you need `SHA256` \$1 `RSA` or `SHA1` \$1 `ECDSA` for your configuration, contact us for further support. Configure `signCommand` if you chose `Custom` for `signingMethod`. **`signCommand`** The command used to perform custom code signing. You can find the template in the `/configs/script_templates` directory. Two placeholders `{{inputImageFilePath}}` and `{{outputSignatureFilePath}}` are required in the command. `{{inputImageFilePath}}` is the file path of the image built by IDT to be signed. `{{outputSignatureFilePath}}` is the file path of the signature which will be generated by the script. **`cmakeConfiguration`** CMake configuration [Optional] To execute CMake test cases, you must provide the board name, vendor name, and either the `frToolchainPath` or `compilerName`. You may also provide the `cmakeToolchainPath` if you have a custom path to the CMake toolchain. **`boardName`** The name of the board under test. The board name should be the same as the folder name under `path/to/afr/source/code/vendors/vendor/boards/board`. **`vendorName`** The vendor name for the board under test. The vendor should be the same as the folder name under `path/to/afr/source/code/vendors/vendor`. **`compilerName`** The compiler name. **`frToolchainPath`** The fully-qualified path to the compiler toolchain **`cmakeToolchainPath` ** The fully-qualified path to the CMake toolchain. This field is optional **`freertosFileConfiguration`** The configuration of the FreeRTOS files that IDT modifies before running tests. **`required`** This section specifies required tests whose config files you have moved, for example, PKCS11, TLS, and so on. **`configName`** The name of the test that is being configured. **`filePath`** The absolute path to the configuration files within the `freertos` repo. Use the `{{testData.sourcePath}}` variable to define the path. **`optional`** This section specifies optional tests whose config files you have moved, for example OTA, WiFi, and so on. **`configName`** The name of the test that is being configured. **`filePath`** The absolute path to the configuration files within the `freertos` repo. Use the `{{testData.sourcePath}}` variable to define the path. **Note** To execute CMake test cases, you must provide the board name, vendor name, and either the `afrToolchainPath` or `compilerName`. You may also provide `cmakeToolchainPath` if you have a custom path to the CMake toolchain. # IDT for FreeRTOS variables The commands to build your code and flash the device might require connectivity or other information about your devices to run successfully. AWS IoT Device Tester allows you to reference device information in flash and build commands using [JsonPath](https://goessner.net/articles/JsonPath/). By using simple JsonPath expressions, you can fetch the required information specified in your `device.json` file. ## Path variables IDT for FreeRTOS defines the following path variables that can be used in command lines and configuration files: **`{{testData.sourcePath}}`** Expands to the source code path. If you use this variable, it must be used in both the flash and build commands. **`{{sdkPath}}`** Expands to the value in your `userData.sdkConfiguration.path` when used in the build and flash commands. **`{{device.connectivity.serialPort}}`** Expands to the serial port. **`{{device.identifiers[?(@.name == 'serialNo')].value[0]}}`** Expands to the serial number of your device. **`{{enableTests}}`** Integer value indicating whether the build is for tests (value 1) or demos (value 0). **`{{buildImageName}}`** The file name of the image built by the build command. **`{{otaCodeSignerPemFile}}`** PEM file for the OTA code signer. **`{{config.idtRootPath}}`** Expands to the AWS IoT Device Tester root path. This variable replaces the absolute path for IDT when used by the build and flash commands. # Use the IDT user interface to run the FreeRTOS qualification suite Starting with IDT v4.3.0, AWS IoT Device Tester for FreeRTOS (IDT-FreeRTOS) includes a web-based user interface that enables you to interact with the IDT command line executable and related configuration files. You can use the IDT-FreeRTOS UI to create a new configuration to run IDT tests, or to modify an existing configuration. You can also use the UI to invoke the IDT executable and run tests. The IDT-FreeRTOS UI provides the following functions: + Simplify setting up configuration files for IDT-FreeRTOS tests. + Simplify using IDT-FreeRTOS to run qualification tests. For information about the using the command line to run qualification tests, see [First test of your microcontroller board](qual-steps.md). This section describes the prerequisites for using the IDT-FreeRTOS UI, and shows you how to get started running qualification tests in the UI. **Topics** + [Set up the prerequisites to run the FreeRTOS qualification suite](dev-tester-ui-prereqs.md) + [Get started with the IDT-FreeRTOS UI](dev-tester-ui-getting-started.md) # Set up the prerequisites to run the FreeRTOS qualification suite This section describes the prerequisites for testing microcontrollers with AWS IoT Device Tester. **Topics** + [Use a supported web browser](#idt-ui-supported-web-browser) + [Download FreeRTOS](#ui-download-afr) + [Download IDT for FreeRTOS](#ui-download-dev-tester-afr) + [Create and configure an AWS account](#ui-config-aws-account) + [AWS IoT Device Tester managed policy](#ui-managed-policy) ## Use a supported web browser The IDT-FreeRTOS UI supports the following web browsers. | Browser | Version | | --- | --- | | Google Chrome | Latest three major versions | | Mozilla Firefox | Latest three major versions | | Microsoft Edge | Latest three major versions | | Apple Safari for macOS | Latest three major versions | We recommend that you use Google Chrome or Mozilla Firefox for a better experience. **Note** The IDT-FreeRTOS UI doesn't support Microsoft Internet Explorer. ## Download FreeRTOS You can download a release of FreeRTOS from [GitHub](https://github.com/aws/amazon-freertos) with the following command: ``` git clone --branch --recurse-submodules https://github.com/aws/amazon-freertos.git cd amazon-freertos git submodule update --checkout --init --recursive ``` where is a version of FreeRTOS (for example, 202007.00) corresponding to an IDT version listed in [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md). This ensures you have the full source code, including submodules, and are using the correct version of IDT for your version of FreeRTOS, and vice versa. Windows has a path length limitation of 260 characters. The path structure of FreeRTOS is many levels deep, so if you're using Windows, keep your file paths under the 260-character limit. For example, clone FreeRTOS to `C:\FreeRTOS` rather than `C:\Users\username\programs\projects\myproj\FreeRTOS\`. ### Considerations for LTS qualification (qualification for FreeRTOS that uses LTS libraries) + In order for your microcontroller to be designated as supporting long-term support (LTS) based versions of FreeRTOS in the AWS Partner Device Catalog, you must provide a manifest file. For more information, see the [FreeRTOS Qualification Checklist](https://docs.aws.amazon.com/freertos/latest/qualificationguide/afq-checklist.html) in the *FreeRTOS Qualification Guide*. + In order to validate that your microcontroller supports LTS based versions of FreeRTOS and qualify it for submission to the AWS Partner Device Catalog, you must use AWS IoT Device Tester (IDT) with FreeRTOS Qualification (FRQ) test suite version v1.4.x. + Support for LTS based versions of FreeRTOS is limited to the 202012.xx version of FreeRTOS. ## Download IDT for FreeRTOS Every version of FreeRTOS has a corresponding version of IDT for FreeRTOS for performing qualification tests. Download the appropriate version of IDT for FreeRTOS from [Supported versions of AWS IoT Device Tester](dev-test-versions-afr.md). Extract IDT for FreeRTOS to a location on the file system where you have read and write permissions. Because Microsoft Windows has a character limit for the path length, extract IDT for FreeRTOS into a root directory such as `C:\` or `D:\`. **Note** We recommend that you extract the IDT package to a local drive.Allowing multiple users to run IDT from a shared location, such as an NFS directory or a Windows network shared folder, might result in the system not responding or data corruption. ## Create and configure an AWS account ### Sign up for an AWS account If you do not have an AWS account, complete the following steps to create one. **To sign up for an AWS account** 1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup). 1. Follow the online instructions. Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad. When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks). AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**. ### Create a user with administrative access After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks. **Secure your AWS account root user** 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password. For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*. 1. Turn on multi-factor authentication (MFA) for your root user. For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*. **Create a user with administrative access** 1. Enable IAM Identity Center. For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*. 1. In IAM Identity Center, grant administrative access to a user. For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*. **Sign in as the user with administrative access** + To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*. **Assign access to additional users** 1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions. For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*. 1. Assign users to a group, and then assign single sign-on access to the group. For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*. ## AWS IoT Device Tester managed policy To enable device tester to run and to collect metrics, the `AWSIoTDeviceTesterForFreeRTOSFullAccess` managed policy contains the following permissions: + `iot-device-tester:SupportedVersion` Grants permission to get the list of FreeRTOS versions and test suite versions supported by IDT, so that they're available from the AWS CLI. + `iot-device-tester:LatestIdt` Grants permission to get the latest AWS IoT Device Tester version that is available for download. + `iot-device-tester:CheckVersion` Grants permission to check that a combination of product, test suite, and AWS IoT Device Tester versions are compatible. + `iot-device-tester:DownloadTestSuite` Grants permission to AWS IoT Device Tester to download test suites. + `iot-device-tester:SendMetrics` Grants permission to publish AWS IoT Device Tester usage metrics data. # Get started with the IDT-FreeRTOS UI This section shows you how to use the IDT-FreeRTOS UI to create or modify your configuration, and then shows you how to run tests. **Topics** + [Configure AWS credentials](#configure-aws-credentials) + [Open the IDT-FreeRTOS UI](#open-idt-ui) + [Create a new configuration](#create-new-configuration) + [Modify an existing configuration](#modify-existing-configuration) + [Run qualification tests](#run-tests-from-ui) ## Configure AWS credentials You must configure credentials for the AWS user that you created in [Create and configure an AWS account](dev-tester-ui-prereqs.md#ui-config-aws-account). You can specify your credentials in one of two ways: + In a credentials file + As environment variables ### Configure AWS credentials with a credentials file IDT uses the same credentials file as the AWS CLI. For more information, see [Configuration and credential files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html). The location of the credentials file varies, depending on the operating system you're using: + macOS, Linux: `~/.aws/credentials` + Windows: `C:\Users\UserName\.aws\credentials` Add your AWS credentials to the `credentials` file in the following format: ``` [default] aws_access_key_id = aws_secret_access_key = ``` **Note** If you don't use the `default` AWS profile, be sure to specify the profile name in the IDT-FreeRTOS UI. For more information about profiles, see [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). ### Configure AWS credentials with environment variables Environment variables are variables maintained by the operating system and used by system commands. They're not saved if you close the SSH session. The IDT-FreeRTOS UI uses the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to store your AWS credentials. To set these variables on Linux, macOS, or Unix, use **export**: ``` export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` To set these variables on Windows, use **set**: ``` set AWS_ACCESS_KEY_ID= set AWS_SECRET_ACCESS_KEY= ``` ## Open the IDT-FreeRTOS UI **To open the IDT-FreeRTOS UI** 1. Download a supported IDT-FreeRTOS version and extract the downloaded archive into a location on your file system where you have read and write permissions. 1. Run the following command to navigate to the IDT-FreeRTOS installation directory: ``` cd devicetester-extract-location/bin ``` 1. Run the following command to open the IDT-FreeRTOS UI: ------ #### [ Linux ] ``` .devicetestergui_linux_x86-64.exe ``` ------ #### [ Windows ] ``` ./devicetestergui_win_x64-64 ``` ------ #### [ macOS ] ``` ./devicetestergui_mac_x86-64 ``` **Note** On Mac, to allow your system to run the UI, go to **System Preferences -> Security & Privacy**. When you run the tests, you may need to do this three more times. ------ The IDT-FreeRTOS UI opens in your default browser. For information about supported browsers, see [Use a supported web browser](dev-tester-ui-prereqs.md#idt-ui-supported-web-browser). ## Create a new configuration If you're a first-time user, then you must create a new configuration to set up the JSON configuration files that IDT-FreeRTOS requires to run tests. You can then run tests or modify the configuration that was created. For examples of the `config.json`, `device.json`, and `userdata.json` files, see [First test of your microcontroller board](qual-steps.md). For an example of the `resource.json` file that is used only for running Bluetooth Low Energy (BLE) tests, see [Run Bluetooth Low Energy tests](afr-bridgekeeper-dt-bt.md). **To create a new configuration** 1. In the IDT-FreeRTOS UI, open the navigation menu, and then choose **Create new configuration**. **Important** You must configure your AWS credentials before you open the UI. If you haven't configured your credentials, close the IDT-FreeRTOS UI browser window, follow the steps in [Configure AWS credentials](#configure-aws-credentials), and then reopen the IDT-FreeRTOS UI. 1. Follow the configuration wizard to enter the IDT configuration settings that are used to run qualification tests. The wizard configures the following settings in JSON configuration files that are located in the `devicetester-extract-location/config` directory. + **AWS settings**—The AWS account information that IDT-FreeRTOS uses to create AWS resources during test runs. These settings are configured in the `config.json` file. + **FreeRTOS repository**—The absolute path to the FreeRTOS repository and ported code, and the type of qualification you want to perform. These settings are configured in the `userdata.json` file. You must port FreeRTOS for your device before you can run qualification tests. For more information, see the [FreeRTOS Porting Guide](https://docs.aws.amazon.com/freertos/latest/portingguide/) + **Build and flash**—The build and flash commands for your hardware that allow IDT to build and flash tests on to your board automatically. These settings are configured in the `userdata.json` file. + **Devices**—The device pool settings for the devices to be tested. These settings are configured in `id` and `sku` fields, and the `devices` block for the device pool in the `device.json` file. + **Networking**—The settings to test network communication support for your devices. These settings are configured in the `features` block of the `device.json` file, and in the `clientWifiConfig` and `testWifiConfig` blocks in the `userdata.json` file. + **Echo server**—The echo server configuration settings for secure socket tests. These settings are configured in the `userdata.json` file. For more information about echo server configuration, see [https://docs.aws.amazon.com/freertos/latest/portingguide/afr-echo-server.html](https://docs.aws.amazon.com/freertos/latest/portingguide/afr-echo-server.html). + **CMake**—(Optional) The settings to run CMake build functionality tests. This configuration is required only if you're using CMake as your build system. These settings are configured in the `userdata.json` file. + **BLE**—The settings to run Bluetooth Low Energy functionality tests. These settings are configured in the `features` block of the `device.json` file and in the `resource.json` file. + **OTA**—The settings to run OTA functionality tests. These settings are configured in the `features` block of the `device.json` file and in the `userdata.json` file. 1. On the **Review** page, verify your configuration information. After you finish reviewing your configuration, to run your qualification tests, choose **Run tests**. ## Modify an existing configuration If you have already set up configuration files for IDT, then you can use the IDT-FreeRTOS UI to modify your existing configuration. Make sure that your existing configuration files are available in the `devicetester-extract-location/config` directory. **To modify a new configuration** 1. In the IDT-FreeRTOS UI, open the navigation menu, and then choose **Edit existing configuration**. The configuration dashboard displays information about your existing configuration settings. If a configuration is incorrect or unavailable, the status for that configuration is `Error validating configuration`. 1. To modify an existing configuration setting, complete the following steps: 1. Choose the name of a configuration setting to open its settings page. 1. Modify the settings, and then choose **Save** to regenerate the corresponding configuration file. After you finish modifying your configuration, verify that all of your configuration settings pass validation. If the status for each configuration setting is `Valid`, you can run your qualification tests using this configuration. ## Run qualification tests After you have created a configuration for IDT-FreeRTOS, you can run your qualification tests. **To run qualification tests** 1. Validate your configuration. 1. In the navigation menu, choose **Run tests**. 1. To start the test run, choose **Start tests**. IDT-FreeRTOS runs the qualification tests, and displays the test run summary and any errors in the **Test runner** console. After the test run is complete, you can view the test results and logs from the following locations: + Test results are located in the `devicetester-extract-location/results/execution-id` directory. + Test logs are located in the `devicetester-extract-location/results/execution-id/logs` directory. For more information about test results and logs, see [View the IDT for FreeRTOS results](view-results-frq.md) and [View the IDT for FreeRTOS logs](view-logs-frq.md). # Run Bluetooth Low Energy tests This section describes how to set up and run Bluetooth Low Energy tests using AWS IoT Device Tester for FreeRTOS. Bluetooth tests are not required for core qualification. If you do not want to test your device with FreeRTOS Bluetooth support you may skip this setup, be sure to leave the BLE feature in device.json set to `No`. ## Prerequisites + Follow the instructions in [First test of your microcontroller board](qual-steps.md). + A Raspberry Pi 4B or 3B\$1. (Required to run the Raspberry Pi BLE companion application) + A micro SD card and SD card adapter for the Raspberry Pi software.   ## Raspberry Pi setup To test the BLE capabilities of the device under test (DUT), you must have a Raspberry Pi Model 4B or 3B\$1. **To set up your Raspberry Pi to run BLE tests** 1. Download one of the custom Yocto images that contains the software required to perform the tests. + [ Image for Raspberry Pi 4B](https://docs.aws.amazon.com/freertos/latest/userguide/freertos/IDTFR_BLE_RaspberryPi4B_1.0.0_2021-04-13.rpi-sd.img) + [ Image for Raspberry Pi 3B\$1](https://docs.aws.amazon.com/freertos/latest/userguide/freertos/IDTFR_BLE_RaspberryPi3Bplus_1.0.0_2021-04-13.rpi-sd.img) **Note** The Yocto image should only be used for testing with AWS IoT Device Tester for FreeRTOS and not for any other purpose. 1. Flash the yocto image onto the SD card for Raspberry Pi. 1. Using an SD card-writing tool such as [Etcher](https://www.balena.io/etcher), flash the downloaded `image-name.rpi-sd.img` file onto the SD card. Because the operating system image is large, this step might take some time. Then eject your SD card from your computer and insert the microSD card into your Raspberry Pi. 1. Configure your Raspberry Pi. 1. For the first boot, we recommend that you connect the Raspberry Pi to a monitor, keyboard, and mouse. 1. Connect your Raspberry Pi to a micro USB power source. 1. Sign in using the default credentials. For user ID, enter **root**. For password, enter **idtafr**. 1. Using an Ethernet or Wi-Fi connection, connect the Raspberry Pi to your network. 1. To connect your Raspberry Pi over Wi-Fi, open `/etc/wpa_supplicant.conf` on the Raspberry Pi and add your Wi-Fi credentials to the `Network` configuration. ``` ctrl_interface=/var/run/wpa_supplicant ctrl_interface_group=0 update_config=1 network={ scan_ssid=1 ssid="your-wifi-ssid" psk="your-wifi-password" } ``` 1. Run `ifup wlan0` to start the Wi-Fi connection. It might take a minute to connect to your Wi-Fi network. 1. For an Ethernet connection, run `ifconfig eth0`. For a Wi-Fi connection, run `ifconfig wlan0`. Make a note of the IP address, which appears as `inet addr` in the command output. You need the IP address later in this procedure. 1. (Optional) The tests execute commands on the Raspberry Pi over SSH using the default credentials for the yocto image. For additional security, we recommend that you set up public key authentication for SSH and disable password-based SSH. 1. Create an SSH key using the OpenSSL `ssh-keygen` command. If you already have an SSK key pair on your host computer, it is a best practice to create a new one to allow AWS IoT Device Tester for FreeRTOS to sign in to your Raspberry Pi. **Note** Windows does not come with an installed SSH client. For information about how to install an SSH client on Windows, see [Download SSH Software](https://www.ssh.com/ssh/#sec-Download-client-software). 1. The `ssh-keygen` command prompts you for a name and path to store the key pair. By default, the key pair files are named `id_rsa` (private key) and `id_rsa.pub` (public key). On macOS and Linux, the default location of these files is `~/.ssh/`. On Windows, the default location is `C:\Users\user-name`. 1. When you are prompted for a key phrase, just press ENTER to continue. 1. To add your SSH key onto your Raspberry Pi so AWS IoT Device Tester for FreeRTOS can sign into the device, use the `ssh-copy-id` command from your host computer. This command adds your public key into the `~/.ssh/authorized_keys` file on your Raspberry Pi. `ssh-copy-id root@raspberry-pi-ip-address` 1. When prompted for a password, enter **idtafr**. This is the default password for the yocto image. **Note** The `ssh-copy-id` command assumes the public key is named `id_rsa.pub`. On macOS and Linux, the default location is ` ~/.ssh/`. On Windows, the default location is `C:\Users\user-name\.ssh`. If you gave the public key a different name or stored it in a different location, you must specify the fully qualified path to your SSH public key using the `-i` option to `ssh-copy-id` (for example, `ssh-copy-id -i ~/my/path/myKey.pub`). For more information about creating SSH keys and copying public keys, see [SSH-COPY-ID](https://www.ssh.com/ssh/copy-id). 1. To test that the public key authentication is working, run `ssh -i /my/path/myKey root@raspberry-pi-device-ip`. If you are not prompted for a password, your public key authentication is working. 1. Verify that you can sign in to your Raspberry Pi using a public key, and then disable password-based SSH. 1. On the Raspberry Pi, edit the `/etc/ssh/sshd_config` file. 1. Set the `PasswordAuthentication` attribute to `no`. 1. Save and close the `sshd_config` file. 1. Reload the SSH server by running `/etc/init.d/sshd reload`. 1. Create a `resource.json` file. 1. In the directory in which you extracted AWS IoT Device Tester, create a file named `resource.json`. 1. Add the following information about your Raspberry Pi to the file, replacing *rasp-pi-ip-address* with the IP address of your Raspberry Pi. ``` [ { "id": "ble-test-raspberry-pi", "features": [ {"name":"ble", "version":"4.2"} ], "devices": [ { "id": "ble-test-raspberry-pi-1", "connectivity": { "protocol": "ssh", "ip": "rasp-pi-ip-address" } } ] } ] ``` 1. If you didn't choose to use public key authentication for SSH, add the following to the `connectivity` section of the `resource.json` file. ``` "connectivity": { "protocol": "ssh", "ip": "rasp-pi-ip-address", "auth": { "method": "password", "credentials": { "user": "root", "password": "idtafr" } } } ``` 1. (Optional) If you chose to use public key authentication for SSH, add the following to the `connectivity` section of the `resource.json` file. ``` "connectivity": { "protocol": "ssh", "ip": "rasp-pi-ip-address", "auth": { "method": "pki", "credentials": { "user": "root", "privKeyPath": "location-of-private-key" } } } ``` ## FreeRTOS device setup In your `device.json` file, set the `BLE` feature to `Yes`. If you are starting with a `device.json` file from before Bluetooth tests were available, you need to add the feature for BLE to the `features` array: ``` { ... "features": [ { "name": "BLE", "value": "Yes" }, ... } ``` ## Run the BLE tests After you have enabled the BLE feature in `device.json`, the BLE tests run when you run `devicetester_[linux | mac | win_x86-64] run-suite` *without specifying a group-id*. If you want to run the BLE tests separately, you can specify the group ID for BLE: `devicetester_[linux | mac | win_x86-64] run-suite --userdata path-to-userdata/userdata.json --group-id FullBLE`. For the most reliable performance, place your Raspberry Pi close to the device under test (DUT). ## Troubleshoot BLE tests Make sure you have followed the steps in [First test of your microcontroller board](qual-steps.md). If tests other than BLE are failing, then the problem is most likely not due to the Bluetooth configuration. # Run the FreeRTOS qualification suite You use the AWS IoT Device Tester for FreeRTOS executable to interact with IDT for FreeRTOS. The following command line examples show you how to run the qualification tests for a device pool (a set of identical devices). ------ #### [ IDT v3.0.0 and later ] ``` devicetester_[linux | mac | win] run-suite \ --suite-id suite-id \ --group-id group-id \ --pool-id your-device-pool \ --test-id test-id \ --upgrade-test-suite y|n \ --update-idt y|n \ --update-managed-policy y|n \ --userdata userdata.json ``` Runs a suite of tests on a pool of devices. The `userdata.json` file must be located in the `devicetester_extract_location/devicetester_afreertos_[win|mac|linux]/configs/` directory. **Note** If you're running IDT for FreeRTOS on Windows, use forward slashes (/) to specify the path to the `userdata.json` file. Use the following command to run a specific test group: ``` devicetester_[linux | mac | win] run-suite \ --suite-id FRQ_1.0.1 \ --group-id group-id \ --pool-id pool-id \ --userdata userdata.json ``` The `suite-id` and `pool-id` parameters are optional if you're running a single test suite on a single device pool (that is, you have only one device pool defined in your `device.json` file). Use the following command to run a specific test case in a test group: ``` devicetester_[linux | mac | win_x86-64] run-suite \ --group-id group-id \ --test-id test-id ``` You can use the `list-test-cases` command to list the test cases in a test group. IDT for FreeRTOS command line options **group-id** (Optional) The test groups to run, as a comma-separated list. If not specified, IDT runs all test groups in the test suite. **pool-id** (Optional) The device pool to test. This is required if you define multiple device pools in `device.json`. If you only have one device pool, you can omit this option. **suite-id** (Optional) The test suite version to run. If not specified, IDT uses the latest version in the tests directory on your system. Starting in IDT v3.0.0, IDT checks online for newer test suites. For more information, see [Test suite versions](idt-test-suite-versions.md). **test-id** (Optional) The tests to run, as a comma-separated list. If specified, `group-id` must specify a single group. **Example** ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id mqtt --test-id mqtt_test ``` **update-idt** (Optional) If this parameter is not set and a newer IDT version is available, you will be prompted to update IDT. If this parameter is set to `Y`, IDT will stop test execution if it detects that a newer version is available. If this parameter is set to `N`, IDT will continue the test execution. **update-managed-policy** (Optional) If this parameter is not used and IDT detects that your managed policy isn't up-to-date, you will be prompted to update your managed policy. If this parameter is set to `Y`, IDT will stop test execution if it detects that your managed policy isn't up-to-date. If this parameter is set to `N`, IDT will continue the test execution. **upgrade-test-suite** (Optional) If not used, and a newer test suite version is available, you're prompted to download it. To hide the prompt, specify `y` to always download the latest test suite, or `n` to use the test suite specified or the latest version on your system. **Example** **Example** To always download and use the latest test suite, use the following command. ``` devicetester_[linux | mac | win_x86-64] run-suite --userdata userdata file --group-id group ID --upgrade-test-suite y ``` To use the latest test suite on your system, use the following command. ``` devicetester_[linux | mac | win_x86-64] run-suite --userdata userdata file --group-id group ID --upgrade-test-suite n ``` **h** Use the help option to learn more about `run-suite` options. **Example** **Example** ``` devicetester_[linux | mac | win_x86-64] run-suite -h ``` ------ #### [ IDT v1.7.0 and earlier ] ``` devicetester_[linux | mac | win] run-suite \ --suite-id suite-id \ --pool-id your-device-pool \ --userdata userdata.json ``` The `userdata.json` file should be located in the `devicetester_extract_location/devicetester_afreertos_[win|mac|linux]/configs/` directory. **Note** If you are running IDT for FreeRTOS on Windows, use forward slashes (/) to specify the path to the `userdata.json` file. Use the following command to run a specific test group. ``` devicetester_[linux | mac | win] run-suite \ --suite-id FRQ_1 --group-id group-id \ --pool-id pool-id \ --userdata userdata.json ``` `suite-id` and `pool-id` are optional if you are running a single test suite on a single device pool (that is, you have only one device pool defined in your `device.json` file).IDT for FreeRTOS command line options **group-id** (Optional) Specifies the test group. **pool-id** Specifies the device pool to test. If you only have one device pool, you can omit this option. **suite-id** (Optional) Specifies the test suite to run. ------ ## IDT for FreeRTOS commands The IDT for FreeRTOS command supports the following operations: ------ #### [ IDT v3.0.0 and later ] **`help`** Lists information about the specified command. **`list-groups`** Lists the groups in a given suite. **`list-suites`** Lists the available suites. **`list-supported-products`** Lists the supported products and test suite versions. **`list-supported-versions`** Lists the FreeRTOS and test suite versions supported by the current IDT version. **`list-test-cases`** Lists the test cases in a specified group. **`run-suite`** Runs a suite of tests on a pool of devices. Use the `--suite-id` option to specify a test suite version, or omit it to use the latest version on your system. Use the `--test-id` to run an individual test case. **Example** ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id mqtt --test-id mqtt_test ``` For a complete list of options see [Run the FreeRTOS qualification suite](#run-tests). Starting in IDT v3.0.0, IDT checks online for newer test suites. For more information, see [Test suite versions](idt-test-suite-versions.md). ------ #### [ IDT v1.7.0 and earlier ] **`help`** Lists information about the specified command. **`list-groups`** Lists the groups in a given suite. **`list-suites`** Lists the available suites. **`run-suite`** Runs a suite of tests on a pool of devices. ------ ## Test for re-qualification As new versions of IDT for FreeRTOS qualification tests are released, or as you update your board-specific packages or device drivers, you can use IDT for FreeRTOS to test your microcontroller boards. For subsequent qualifications, make sure that you have the latest versions of FreeRTOS and IDT for FreeRTOS and run the qualification tests again. # View the IDT for FreeRTOS results While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it writes a test run summary to the console and generates two test reports. These reports can be found in `devicetester-extract-location/results/execution-id/`. Both reports capture the results from the qualification test suite execution. The `awsiotdevicetester_report.xml` is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements: + The IDT for FreeRTOS version. + The FreeRTOS version that was tested. + The features of FreeRTOS that are supported by the device based on the tests passed. + The SKU and the device name specified in the `device.json` file. + The features of the device specified in the `device.json` file. + The aggregate summary of test case results. + A breakdown of test case results by libraries that were tested based on the device features (for example, FullWiFi, FullMQTT, and so on). + Whether this qualification of FreeRTOS is for version 202012.00 that uses LTS libraries. The `FRQ_Report.xml` is a report in standard [JUnit XML format](https://llg.cubic.org/docs/junit/). You can integrate it into CI/CD platforms like [Jenkins](https://jenkins.io/), [Bamboo](https://www.atlassian.com/software/bamboo), and so on. The report contains the following elements: + An aggregate summary of test case results. + A breakdown of test case results by libraries that were tested based on the device features. # Interpret the IDT for FreeRTOS results The report section in `awsiotdevicetester_report.xml` or `FRQ_Report.xml` lists the results of the tests that are executed. The first XML tag `` contains the overall summary of the test execution. For example: `` **Attributes used in the `` tag** **`name`** The name of the test suite. **`time`** The time, in seconds, it took to run the qualification suite. **`tests`** The number of test cases executed. **`failures`** The number of test cases that were run, but did not pass. **`errors`** The number of test cases that IDT for FreeRTOS couldn't execute. **`disabled`** This attribute is not used and can be ignored. If there are no test case failures or errors, your device meets the technical requirements to run FreeRTOS and can interoperate with AWS IoT services. If you choose to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence. In the event of test case failures or errors, you can identify the test case that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag shows the test case result summary for a test group. `` The format is similar to the `` tag, but with an attribute called `skipped` that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each of the test cases that were executed for a test group. For example: `` **Attributes used in the `` tag** **`name`** The name of the product being tested. **`version`** The version of the product being tested. **`sdk`** If you ran IDT with an SDK, this block contains the name and version of your SDK. If you didn't run IDT with an SDK, then this block contains: ``` N/A N/A ``` **`features`** The features validated. Features marked as `required` are required to submit your board for qualification. The following snippet shows how this appears in the `awsiotdevicetester_report.xml` file. ``` ``` Features marked as `optional` are not required for qualification. The following snippets show optional features. ``` ``` If there are no test failures or errors for the required features, your device meets the technical requirements to run FreeRTOS and can interoperate with AWS IoT services. If you want to list your device in the [AWS Partner Device Catalog](https://partners.amazonaws.com/qualified-devices), you can use this report as qualification evidence. In the event of test failures or errors, you can identify the test that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag show the test result summary for a test group. For example: ``` ``` The format is similar to the `` tag, but has a `skipped` attribute that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each executed test for a test group. For example: ``` ``` **`lts`** True if you are qualifying for a version of FreeRTOS that uses LTS libraries, false otherwise. **Attributes used in the `` tag** **`name`** The name of the test case. **`attempts`** The number of times IDT for FreeRTOS executed the test case. When a test fails or an error occurs, `` or `` tags are added to the `` tag with information for troubleshooting. For example: ``` Reason for the test case failure Reason for the test case execution error ``` For more information, see [Troubleshoot errors](dt-afr-troubleshooting.md). # View the IDT for FreeRTOS logs You can find logs that IDT for FreeRTOS generates from test execution in `devicetester-extract-location/results/execution-id/logs`. Two sets of logs are generated: **`test_manager.log`** Contains logs generated from IDT for FreeRTOS (for example, logs related configuration and report generation). **`test_group_id__test_case_id.log` (for example, `FullMQTT__Full_MQTT.log`)** The log file for a test case, including output from the device under test. The log file is named according to the test group and test case that was run.