End of support notice: On October 7th, 2026, AWS will discontinue support for AWS IoT Greengrass Version 1. After October 7th, 2026, you will no longer be able to access the AWS IoT Greengrass V1 resources. For more information, please visit [Migrate from AWS IoT Greengrass Version 1](https://docs.aws.amazon.com/greengrass/v2/developerguide/migrate-from-v1.html). # Use IDT to run the AWS IoT Greengrass qualification suite You can use AWS IoT Device Tester (IDT) for AWS IoT Greengrass to verify that the AWS IoT Greengrass Core software runs on your hardware and can communicate with the AWS Cloud. It also performs end-to-end tests with AWS IoT Core. For example, it verifies that your device can send and receive MQTT messages and process them correctly. Because AWS IoT Greengrass Version 1 has been moved into [maintenance mode](https://docs.aws.amazon.com/greengrass/v1/developerguide/maintenance-policy.html), IDT for AWS IoT Greengrass V1 no longer generates signed qualification reports. If you want to add your hardware to the AWS Partner Device Catalog, run the AWS IoT Greengrass V2 qualification suite to generate test reports that you can submit to AWS IoT. For more information, see [AWS Device Qualification Program](https://aws.amazon.com/partners/dqp/) and [Supported versions of IDT for AWS IoT Greengrass V2](https://docs.aws.amazon.com/greengrass/v2/developerguide/dev-test-versions.html). In addition to testing devices, IDT for AWS IoT Greengrass creates resources (for example, AWS IoT things, AWS IoT Greengrass groups, Lambda functions, and so on) in your AWS account to facilitate the qualification process. To create these resources, IDT for AWS IoT Greengrass uses the AWS credentials configured in the `config.json` file to make API calls on your behalf. These resources are provisioned at various times during a test. When you use IDT for AWS IoT Greengrass to run the AWS IoT Greengrass qualification suite, IDT performs the following steps: 1. Loads and validates your device and credential configurations. 1. Performs selected tests with the required local and cloud resources. 1. Cleans up local and cloud resources. 1. Generates tests reports that indicate if your device passed the tests required for qualification. ## Test suite versions IDT for AWS IoT Greengrass organizes tests into test suites and test groups. + A test suite is the set of test groups used to verify that a device works with particular versions of AWS IoT Greengrass. + A test group is the set of individual tests related to a particular feature, such as Greengrass group deployments and MQTT messaging. Starting in IDT v3.0.0, test suites are versioned using a `major.minor.patch` format, for example `GGQ_1.0.0`. When you download IDT, the package includes the latest test suite version. **Important** IDT supports the three latest test suite versions for device qualification. For more information, see [Support policy for AWS IoT Device Tester for AWS IoT Greengrass V1](idt-support-policy.md). You can run `list-supported-products` to list the versions of AWS IoT Greengrass and test suites that are supported by your current version of IDT. Tests from unsupported test suite versions are not valid for device qualification. IDT doesn't print qualification reports for unsupported versions. ### Updates to IDT configuration settings New tests might introduce new IDT configuration settings. + If the settings are optional, IDT continues running the tests. + If the settings are required, IDT notifies you and stops running. After you configure the settings, restart the test run. Configuration settings are located in the `/configs` folder. For more information, see [Configure IDT settings to run the AWS IoT Greengrass qualification suite](set-config.md). If an updated test suite version adds configuration settings, IDT creates a copy of the original configuration file in `/configs`. ## Test group descriptions ------ #### [ IDT v2.0.0 and later ] **Required Test Groups for Core Qualification** These test groups are required to qualify your AWS IoT Greengrass device for the AWS Partner Device Catalog. AWS IoT Greengrass Core Dependencies Validates that your device meets all software and hardware requirements for the AWS IoT Greengrass Core software. The `Software Packages Dependencies` test case in this test group is not applicable when testing in a [Docker container](docker-config-setup.md). Deployment Validates that Lambda functions can be deployed on your device. MQTT Verifies the AWS IoT Greengrass message router functionality by checking local communication between the Greengrass core and client devices, which are local IoT devices. Over-the-Air (OTA) Validates that your device can successfully perform an OTA update of the AWS IoT Greengrass Core software. This test group is not applicable when testing in a [Docker container](docker-config-setup.md). Version Checks that the version of AWS IoT Greengrass provided is compatible with the AWS IoT Device Tester version you are using. **Optional Test Groups** These test groups are optional. If you choose to qualify for optional tests, your device is listed with additional capabilities in the AWS Partner Device Catalog. Container Dependencies Validates that the device meets all of the software and hardware requirements to run Lambda functions in container mode on a Greengrass core. This test group is not applicable when testing in a [Docker container](docker-config-setup.md). Deployment Container Validates that Lambda functions can be deployed on the device and run in container mode on a Greengrass core. This test group is not applicable when testing in a [Docker container](docker-config-setup.md). Docker Dependencies (Supported for IDT v2.2.0 and later) Validates that the device meets all the required technical dependencies to use the Greengrass Docker application deployment connector to run containers This test group is not applicable when testing in a [Docker container](docker-config-setup.md). Hardware Security Integration (HSI) Verifies that the provided HSI shared library can interface with the hardware security module (HSM) and implements the required PKCS\$111 APIs correctly. The HSM and shared library must be able to sign a CSR, perform TLS operations, and provide the correct key lengths and public key algorithm. Stream Manager Dependencies (Supported for IDT v2.2.0 and later) Validates that the device meets all of the required technical dependencies to run AWS IoT Greengrass stream manager. Machine Learning Dependencies (Supported for IDT v3.1.0 and later) Validates that the device meets all of the required technical dependencies to perform ML inference locally. Machine Learning Inference Tests (Supported for IDT v3.1.0 and later) Validates that ML inference can be performed on the given device under test. For more information, see [Optional: Configuring your device for ML qualification](idt-ml-qualification.md). Machine Learning Inference Container Tests (Supported for IDT v3.1.0 and later) Validates that ML inference can be performed on the given device under test and run in container mode on a Greengrass core. For more information, see [Optional: Configuring your device for ML qualification](idt-ml-qualification.md). ------ #### [ IDT v1.3.3 and earlier ] **Required Test Groups for Core Qualification** These tests are required to qualify your AWS IoT Greengrass device for the AWS Partner Device Catalog. AWS IoT Greengrass Core Dependencies Validates that your device meets all software and hardware requirements for the AWS IoT Greengrass Core software. Combination (Device Security Interaction) Verifies the functionality of the device certificate manager and IP detection on the Greengrass core device by changing connectivity information on the Greengrass group in the cloud. The test group rotates the AWS IoT Greengrass server certificate and verifies that AWS IoT Greengrass allows connections. Deployment (Required for IDT v1.2 and earlier) Validates that Lambda functions can be deployed on your device. Device Certificate Manager (DCM) Verifies that the AWS IoT Greengrass device certificate manager can generate a server certificate on startup and rotate certificates if they are close to expiration. IP Detection (IPD) Verifies that core connectivity information is updated when there are IP address changes in a Greengrass core device. For more information, see [Activate automatic IP detection](gg-core.md#ip-auto-detect). Logging Verifies that the AWS IoT Greengrass logging service can write to a log file using a user Lambda function written in Python. MQTT Verifies the AWS IoT Greengrass message router functionality by sending messages on a topic that is routed to two Lambda functions. Native Verifies that AWS IoT Greengrass can run native (compiled) Lambda functions. Over-the-Air (OTA) Validates that your device can successfully perform a OTA update of the AWS IoT Greengrass Core software. Penetration Validates that the AWS IoT Greengrass Core software fails to start if hard link/soft link protection and [seccomp](https://www.kernel.org/doc/Documentation/prctl/seccomp_filter.txt) are not enabled. It is also used to verify other security-related features. Shadow Verifies local shadow and shadow cloud-syncing functionality. Spooler Validates that the MQTT messages are queued with the default spooler configuration. Token Exchange Service (TES) Verifies that AWS IoT Greengrass can exchange its core certificate for valid AWS credentials. Version Checks that the version of AWS IoT Greengrass provided is compatible with the AWS IoT Device Tester version you are using. **Optional Test Groups** These tests are optional. If you choose to qualify for optional tests, your device is listed with additional capabilities in the AWS Partner Device Catalog. Container Dependencies Checks that the device meets all of the required dependencies to run Lambda functions in container mode. Hardware Security Integration (HSI) Verifies that the provided HSI shared library can interface with the hardware security module (HSM) and implements the required PKCS\$111 APIs correctly. The HSM and shared library must be able to sign a CSR, perform TLS operations, and provide the correct key lengths and public key algorithm. Local Resource Access Verifies the local resource access (LRA) feature of AWS IoT Greengrass by providing access to local files and directories owned by various Linux users and groups to containerized Lambda functions through AWS IoT Greengrass LRA APIs. Lambda functions should be allowed or denied access to local resources based on local resource access configuration. Network Verifies that socket connections can be established from a Lambda function. These socket connections should be allowed or denied based on the Greengrass core configuration. ------ # Prerequisites for running the AWS IoT Greengrass qualification suite This section describes the prerequisites for using AWS IoT Device Tester (IDT) for AWS IoT Greengrass to run the AWS IoT Greengrass qualification suite. ## Download the latest version of AWS IoT Device Tester for AWS IoT Greengrass Download the [latest version](dev-test-versions.md) of IDT and extract the software into a location on your file system where you have read and write permissions. **Note** IDT does not support being run by multiple users from a shared location, such as an NFS directory or a Windows network shared folder. We recommend that you extract the IDT package to a local drive and run the IDT binary on your local workstation. Windows has a path length limitation of 260 characters. If you are using Windows, extract IDT to a root directory like `C:\ ` or `D:\` to keep your paths under the 260 character limit. ## Create and configure an AWS account Before you can use IDT for AWS IoT Greengrass, you must perform the following steps: 1. [Create an AWS account.]() If you already have an AWS account, skip to step 2. 1. [Configure permissions for IDT.]() These account permissions allow IDT to access AWS services and create AWS resources, such as AWS IoT things, Greengrass groups, and Lambda functions, on your behalf. To create these resources, IDT for AWS IoT Greengrass uses the AWS credentials configured in the `config.json` file to make API calls on your behalf. These resources are provisioned at various times during a test. **Note** Although most tests qualify for [Amazon Web Services Free Tier](https://aws.amazon.com/free), you must provide a credit card when you sign up for an AWS account. For more information, see [ Why do I need a payment method if my account is covered by the Free Tier?](https://aws.amazon.com/premiumsupport/knowledge-center/free-tier-payment-method/). ### Step 1: Create an AWS account In this step, create and configure an AWS account. If you already have an AWS account, skip to [Step 2: Configure permissions for IDT](#configure-idt-permissions). #### Sign up for an AWS account If you do not have an AWS account, complete the following steps to create one. **To sign up for an AWS account** 1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup). 1. Follow the online instructions. Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad. When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks). AWS sends you a confirmation email after the sign-up process is complete. At any time, you can view your current account activity and manage your account by going to [https://aws.amazon.com/](https://aws.amazon.com/) and choosing **My Account**. #### Create a user with administrative access After you sign up for an AWS account, secure your AWS account root user, enable AWS IAM Identity Center, and create an administrative user so that you don't use the root user for everyday tasks. **Secure your AWS account root user** 1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) as the account owner by choosing **Root user** and entering your AWS account email address. On the next page, enter your password. For help signing in by using root user, see [Signing in as the root user](https://docs.aws.amazon.com/signin/latest/userguide/console-sign-in-tutorials.html#introduction-to-root-user-sign-in-tutorial) in the *AWS Sign-In User Guide*. 1. Turn on multi-factor authentication (MFA) for your root user. For instructions, see [Enable a virtual MFA device for your AWS account root user (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/enable-virt-mfa-for-root.html) in the *IAM User Guide*. **Create a user with administrative access** 1. Enable IAM Identity Center. For instructions, see [Enabling AWS IAM Identity Center](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-set-up-for-idc.html) in the *AWS IAM Identity Center User Guide*. 1. In IAM Identity Center, grant administrative access to a user. For a tutorial about using the IAM Identity Center directory as your identity source, see [ Configure user access with the default IAM Identity Center directory](https://docs.aws.amazon.com//singlesignon/latest/userguide/quick-start-default-idc.html) in the *AWS IAM Identity Center User Guide*. **Sign in as the user with administrative access** + To sign in with your IAM Identity Center user, use the sign-in URL that was sent to your email address when you created the IAM Identity Center user. For help signing in using an IAM Identity Center user, see [Signing in to the AWS access portal](https://docs.aws.amazon.com/signin/latest/userguide/iam-id-center-sign-in-tutorial.html) in the *AWS Sign-In User Guide*. **Assign access to additional users** 1. In IAM Identity Center, create a permission set that follows the best practice of applying least-privilege permissions. For instructions, see [ Create a permission set](https://docs.aws.amazon.com//singlesignon/latest/userguide/get-started-create-a-permission-set.html) in the *AWS IAM Identity Center User Guide*. 1. Assign users to a group, and then assign single sign-on access to the group. For instructions, see [ Add groups](https://docs.aws.amazon.com//singlesignon/latest/userguide/addgroups.html) in the *AWS IAM Identity Center User Guide*. ### Step 2: Configure permissions for IDT In this step, configure the permissions that IDT for AWS IoT Greengrass uses to run tests and collect IDT usage data. You can use the AWS Management Console or AWS Command Line Interface (AWS CLI) to create an IAM policy and a test user for IDT, and then attach policies to the user. If you already created a test user for IDT, skip to [Configure your device to run IDT tests](device-config-setup.md) or [Optional: Configuring your Docker container for IDT for AWS IoT Greengrass](docker-config-setup.md). + [To Configure Permissions for IDT (Console)](#configure-idt-permissions-console) + [To Configure Permissions for IDT (AWS CLI)](#configure-idt-permissions-cli) **To configure permissions for IDT (console)** Follow these steps to use the console to configure permissions for IDT for AWS IoT Greengrass. 1. Sign in to the [IAM console](https://console.aws.amazon.com/iam). 1. Create a customer managed policy that grants permissions to create roles with specific permissions. 1. In the navigation pane, choose **Policies**, and then choose **Create policy**. 1. On the **JSON** tab, replace the placeholder content with the following policy. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "ManageRolePoliciesForIDTGreengrass", "Effect": "Allow", "Action": [ "iam:DetachRolePolicy", "iam:AttachRolePolicy" ], "Resource": [ "arn:aws:iam::*:role/idt-*", "arn:aws:iam::*:role/GreengrassServiceRole" ], "Condition": { "ArnEquals": { "iam:PolicyARN": [ "arn:aws:iam::aws:policy/service-role/AWSGreengrassResourceAccessRolePolicy", "arn:aws:iam::aws:policy/service-role/GreengrassOTAUpdateArtifactAccess", "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" ] } } }, { "Sid": "ManageRolesForIDTGreengrass", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:DeleteRole", "iam:PassRole", "iam:GetRole" ], "Resource": [ "arn:aws:iam::123456789012:role/idt-*", "arn:aws:iam::123456789012:role/GreengrassServiceRole" ] } ] } ``` ------ **Important** The following policy grants permission to create and manage roles required by IDT for AWS IoT Greengrass. This includes permissions to attach the following AWS managed policies: [AWSGreengrassResourceAccessRolePolicy](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/service-role/AWSGreengrassResourceAccessRolePolicy) [GreengrassOTAUpdateArtifactAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/service-role/GreengrassOTAUpdateArtifactAccess) [AWSLambdaBasicExecutionRole](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole) 1. Choose **Next: Tags**. 1. Choose **Next: Review**. 1. For **Name**, enter **IDTGreengrassIAMPermissions**. Under **Summary**, review the permissions granted by your policy. 1. Choose **Create policy**. 1. Create an IAM user and attach the permissions required by IDT for AWS IoT Greengrass. 1. Create an IAM user. Follow steps 1 through 5 in [Creating IAM users (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) in the *IAM User Guide*. 1. Attach the permissions to your IAM user: 1. On the **Set permissions ** page, choose **Attach existing policies directly**. 1. Search for the **IDTGreengrassIAMPermissions** policy that you created in the previous step. Select the check box. 1. Search for the **AWSIoTDeviceTesterForGreengrassFullAccess** policy. Select the check box. **Note** The [AWSIoTDeviceTesterForGreengrassFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSIoTDeviceTesterForGreengrassFullAccess) is an AWS managed policy that defines the permissions IDT requires to create and access AWS resources used for testing. For more information, see [AWS managed policy for AWS IoT Device Tester](#idt-managed-policy). 1. Choose **Next: Tags**. 1. Choose **Next: Review** to view a summary of your choices. 1. Choose **Create user**. 1. To view the user's access keys (access key IDs and secret access keys), choose **Show** next to the password and access key. To save the access keys, choose **Download.csv** and save the file to a secure location. You use this information later to configure your AWS credentials file. 1. Next step: Configure your [physical device](device-config-setup.md).   **To configure permissions for IDT (AWS CLI)** Follow these steps to use the AWS CLI to configure permissions for IDT for AWS IoT Greengrass. If you already configured permissions in the console, skip to [Configure your device to run IDT tests](device-config-setup.md) or [Optional: Configuring your Docker container for IDT for AWS IoT Greengrass](docker-config-setup.md). 1. On your computer, install and configure the AWS CLI if it's not already installed. Follow the steps in [ Installing the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) in the *AWS Command Line Interface User Guide*. **Note** The AWS CLI is an open source tool that you can use to interact with AWS services from your command-line shell. 1. Create a customer managed policy that grants permissions to manage IDT and AWS IoT Greengrass roles. ------ #### [ Linux, macOS, or Unix ] ``` aws iam create-policy --policy-name IDTGreengrassIAMPermissions --policy-document '{ "Version":"2012-10-17", "Statement": [ { "Sid": "ManageRolePoliciesForIDTGreengrass", "Effect": "Allow", "Action": [ "iam:DetachRolePolicy", "iam:AttachRolePolicy" ], "Resource": [ "arn:aws:iam::*:role/idt-*", "arn:aws:iam::*:role/GreengrassServiceRole" ], "Condition": { "ArnEquals": { "iam:PolicyARN": [ "arn:aws:iam::aws:policy/service-role/AWSGreengrassResourceAccessRolePolicy", "arn:aws:iam::aws:policy/service-role/GreengrassOTAUpdateArtifactAccess", "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole" ] } } }, { "Sid": "ManageRolesForIDTGreengrass", "Effect": "Allow", "Action": [ "iam:CreateRole", "iam:DeleteRole", "iam:PassRole", "iam:GetRole" ], "Resource": [ "arn:aws:iam::123456789012:role/idt-*", "arn:aws:iam::123456789012:role/GreengrassServiceRole" ] } ] }' ``` ------ #### [ Windows command prompt ] ``` aws iam create-policy --policy-name IDTGreengrassIAMPermissions --policy-document '{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"ManageRolePoliciesForIDTGreengrass\",\"Effect\": \"Allow\",\"Action\": [\"iam:DetachRolePolicy\", \"iam:AttachRolePolicy\"], \"Resource\": [\"arn:aws:iam::*:role/idt-*\",\"arn:aws:iam::*:role/GreengrassServiceRole\"],\"Condition\": {\"ArnEquals\": {\"iam:PolicyARN\": [\"arn:aws:iam::aws:policy/service-role/AWSGreengrassResourceAccessRolePolicy\",\"arn:aws:iam::aws:policy/service-role/GreengrassOTAUpdateArtifactAccess\",\"arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole\"]}}},{\"Sid\": \"ManageRolesForIDTGreengrass\",\"Effect\": \"Allow\",\"Action\": [\"iam:CreateRole\",\"iam:DeleteRole\", \"iam:PassRole\", \"iam:GetRole\"],\"Resource\": [\"arn:aws:iam::*:role/idt-*\",\"arn:aws:iam::*:role/GreengrassServiceRole\"]}]}' ``` **Note** This step includes a Windows command prompt example because it uses a different JSON syntax than Linux, macOS, or Unix terminal commands. ------ 1. Create an IAM user and attach the permissions required by IDT for AWS IoT Greengrass. 1. Create an IAM user. In this example setup, the user is named `IDTGreengrassUser`. ``` aws iam create-user --user-name IDTGreengrassUser ``` 1. Attach the `IDTGreengrassIAMPermissions` policy you created in step 2 to your IAM user. Replace ** in the command with the ID of your AWS account. ``` aws iam attach-user-policy --user-name IDTGreengrassUser --policy-arn arn:aws:iam:::policy/IDTGreengrassIAMPermissions ``` 1. Attach the `AWSIoTDeviceTesterForGreengrassFullAccess` policy to your IAM user. ``` aws iam attach-user-policy --user-name IDTGreengrassUser --policy-arn arn:aws:iam::aws:policy/AWSIoTDeviceTesterForGreengrassFullAccess ``` **Note** The [AWSIoTDeviceTesterForGreengrassFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSIoTDeviceTesterForGreengrassFullAccess) is an AWS managed policy that defines the permissions IDT requires to create and access AWS resources used for testing. For more information, see [AWS managed policy for AWS IoT Device Tester](#idt-managed-policy). 1. Create a secret access key for the user. ``` aws iam create-access-key --user-name IDTGreengrassUser ``` Store the output in a secure location. You use this information later to configure your AWS credentials file. 1. Next step: Configure your [physical device](device-config-setup.md). ## AWS managed policy for AWS IoT Device Tester The [AWSIoTDeviceTesterForGreengrassFullAccess](https://console.aws.amazon.com/iam/home#/policies/arn:aws:iam::aws:policy/AWSIoTDeviceTesterForGreengrassFullAccess) managed policy allows IDT to run operations and collect usage metrics. This policy grants the following IDT permissions: + `iot-device-tester:CheckVersion`. Check whether a set of AWS IoT Greengrass, test suite, and IDT versions are compatible. + `iot-device-tester:DownloadTestSuite`. Download test suites. + `iot-device-tester:LatestIdt`. Get information about the latest IDT version that is available for download. + `iot-device-tester:SendMetrics`. Publish usage data that IDT collects about your tests. + `iot-device-tester:SupportedVersion`. Get the list of AWS IoT Greengrass and test suite versions that are supported by IDT. This information is displayed in the command-line window. # Configure your device to run IDT tests To configure your device you must install AWS IoT Greengrass dependencies, configure the AWS IoT Greengrass Core software, configure your host computer to access your device, and configure user permissions on your device. ## Verify AWS IoT Greengrass dependencies on the device under test Before IDT for AWS IoT Greengrass can test your devices, make sure that you have set up your device as described in [Getting started with AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-gs.html). For information about supported platforms, see [Supported platforms](https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html#gg-platforms). ## Configure the AWS IoT Greengrass software IDT for AWS IoT Greengrass tests your device for compatibility with a specific version of AWS IoT Greengrass. IDT provides two options for testing AWS IoT Greengrass on your devices: + Download and use a version of the [AWS IoT Greengrass Core software](what-is-gg.md#gg-core-download-tab). IDT installs the software for you. + Use a version of the AWS IoT Greengrass Core software already installed on your device. **Note** Each version of AWS IoT Greengrass has a corresponding IDT version. You must download the version of IDT that corresponds to the version of AWS IoT Greengrass you are using. The following sections describe these options. You only need to do one. ### Option 1: Download the AWS IoT Greengrass Core software and configure AWS IoT Device Tester to use it You can download the AWS IoT Greengrass Core software from the [AWS IoT Greengrass Core Software](what-is-gg.md#gg-core-download-tab) downloads page. 1. Find the correct architecture and Linux distribution, and then choose **Download**. 1. Copy the tar.gz file to the `/products/greengrass/ggc`. **Note** Do not change the name of the AWS IoT Greengrass tar.gz file. Do not place multiple files in this directory for the same operating system and architecture. For example having both `greengrass-linux-armv7l-1.7.1.tar.gz` and `greengrass-linux-armv7l-1.8.1.tar.gz` files in that directory will cause the tests to fail. ### Option 2: Use an existing installation of AWS IoT Greengrass with AWS IoT Device Tester Configure IDT to test the AWS IoT Greengrass Core software installed on your device by adding the `greengrassLocation` attribute to the `device.json` file in the `/configs` folder. For example: ``` "greengrassLocation" : "" ``` For more information about the `device.json` file, see [Configure device.json](set-config.md#device-config). On Linux devices, the default location of the AWS IoT Greengrass Core software is `/greengrass`. **Note** Your device should have an installation of the AWS IoT Greengrass Core software that has not been started. Make sure you have added the `ggc_user` user and `ggc_group` on your device. For more information, see [Environment setup for AWS IoT Greengrass](https://docs.aws.amazon.com/greengrass/latest/developerguide/module1.html). ## Configure your host computer to access your device under test IDT runs on your host computer and must be able to use SSH to connect to your device. There are two options to allow IDT to gain SSH access to your devices under test: 1. Follow the instructions here to create an SSH key pair and authorize your key to sign in to your device under test without specifying a password. 1. Provide a user name and password for each device in the `device.json` file. For more information, see [Configure device.json](set-config.md#device-config). You can use any SSL implementation to create an SSH key. The following instructions show you how to use [SSH-KEYGEN](https://www.ssh.com/ssh/keygen/) or [ PuTTYgen](https://www.ssh.com/ssh/putty/windows/puttygen) (for Windows). If you are using another SSL implementation, refer to the documentation for that implementation. IDT uses SSH keys to authenticate with your device under test. **To create an SSH key with SSH-KEYGEN** 1. Create an SSH key. You can use the Open SSH **ssh-keygen** command to create an SSH key pair. If you already have an SSH key pair on your host computer, it is a best practice to create a SSH key pair specifically for IDT. This way, after you have completed testing, your host computer can no longer connect to your device without entering a password. It also allows you to restrict access to the remote device to only those who need it. **Note** Windows does not have an installed SSH client. For information about installing an SSH client on Windows, see [Download SSH Client Software](https://www.ssh.com/ssh/#sec-Download-client-software). The **ssh-keygen** command prompts you for a name and path to store the key pair. By default, the key pair files are named `id_rsa` (private key) and `id_rsa.pub` (public key). On macOS and Linux, the default location of these files is `~/.ssh/`. On Windows, the default location is `C:\Users\\.ssh`. When prompted, enter a key phrase to protect your SSH key. For more information, see [Generate a New SSH key](https://www.ssh.com/ssh/keygen/). 1. Add authorized SSH keys to your device under test. IDT must use your SSH private key to sign in to your device under test. To authorize your SSH private key to sign in to your device under test, use the **ssh-copy-id** command from your host computer. This command adds your public key into the `~/.ssh/authorized_keys` file on your device under test. For example: **\$1 ssh-copy-id **@**** Where *remote-ssh-user* is the user name used to sign in to your device under test and *remote-device-ip* is the IP address of the device under test to run tests against. For example: **ssh-copy-id pi@192.168.1.5** When prompted, enter the password for the user name you specified in the **ssh-copy-id** command. **ssh-copy-id** assumes the public key is named `id_rsa.pub` and is stored the default location (on macOS and Linux, `~/.ssh/` and on Windows, `C:\Users\\.ssh`). If you gave the public key a different name or stored it in a different location, you must specify the fully qualified path to your SSH public key using the **-i** option to **ssh-copy-id** (for example, **ssh-copy-id -i \$1/my/path/myKey.pub**). For more information about creating SSH keys and copying public keys, see [SSH-COPY-ID](https://www.ssh.com/ssh/copy-id). **To create an SSH key using PuTTYgen (Windows only)** 1. Make sure you have the OpenSSH server and client installed on your device under test. For more information, see [OpenSSH](https://www.openssh.com/). 1. Install [PuTTYgen](https://www.puttygen.com/) on your device under test. 1. Open PuTTYgen. 1. Choose **Generate** and move your mouse cursor inside the box to generate a private key. 1. From the **Conversions** menu, choose **Export OpenSSH key**, and save the private key with a `.pem` file extension. 1. Add the public key to the `/home//.ssh/authorized_keys` file on device under test. 1. Copy the public key text from the PuTTYgen window. 1. Use PuTTY to create a session on your device under test. 1. From a command prompt or Windows Powershell window, run the following command: **C:/**/putty.exe -ssh **@**** 1. When prompted, enter your device's password. 1. Use vi or another text editor to append the public key to the `/home//.ssh/authorized_keys` file on your device under test. 1. Update your `device.json` file with your user name, the IP address, and path to the private key file that you just saved on your host computer for each device under test. For more information, see [Configure device.json](set-config.md#device-config). Make sure you provide the full path and file name to the private key and use forward slashes ('/'). For example, for the Windows path `C:\DT\privatekey.pem`, use `C:/DT/privatekey.pem` in the `device.json` file. ## Configure user permissions on your device IDT performs operations on various directories and files in a device under test. Some of these operations require elevated permissions (using **sudo**). To automate these operations, IDT for AWS IoT Greengrass must be able to run commands with sudo without being prompted for a password. Follow these steps on the device under test to allow sudo access without being prompted for a password. **Note** `username` refers to the SSH user used by IDT to access the device under test. **To add the user to the sudo group** 1. On the device under test, run `sudo usermod -aG sudo `. 1. Sign out and then sign back in for changes to take effect. 1. To verify your user name was added successfully, run **sudo echo test**. If you are not prompted for a password, your user is configured correctly. 1. Open the `/etc/sudoers` file and add the following line to the end of the file: ` ALL=(ALL) NOPASSWD: ALL` ## Configure your device to test optional features The following topics describe how to configure your devices to run IDT tests for optional features. Follow these configuration steps only if you want to test these features. Otherwise, continue to [Configure IDT settings to run the AWS IoT Greengrass qualification suite](set-config.md). **Topics** + [Verify AWS IoT Greengrass dependencies on the device under test](#install-gg-dependencies) + [Configure the AWS IoT Greengrass software](#config-gg) + [Configure your host computer to access your device under test](#configure-host) + [Configure user permissions on your device](#root-access) + [Configure your device to test optional features](#optional-feature-config) + [Optional: Configuring your Docker container for IDT for AWS IoT Greengrass](docker-config-setup.md) + [Optional: Configuring your device for ML qualification](idt-ml-qualification.md) # Optional: Configuring your Docker container for IDT for AWS IoT Greengrass AWS IoT Greengrass provides a Docker image and Dockerfile that make it easier to run the AWS IoT Greengrass Core software in a Docker container. After you set up the AWS IoT Greengrass container, you can run IDT tests. Currently, only x86\$164 Docker architectures are supported to run IDT for AWS IoT Greengrass. This feature requires IDT v2.3.0 or later. The process of setting up the Docker container to run IDT tests depends on whether you use the Docker image or Dockerfile provided by AWS IoT Greengrass. + [Use the Docker image](#docker-config-setup-docker-image). The Docker image has the AWS IoT Greengrass Core software and dependencies installed. + [Use the Dockerfile](#docker-config-setup-dockerfile). The Dockerfile contains source code you can use to build custom AWS IoT Greengrass container images. The image can be modified to run on different platform architectures or to reduce the image size. **Note** AWS IoT Greengrass doesn't provide Dockerfiles or Docker images for AWS IoT Greengrass core software version 1.11.1. To run IDT tests on your own custom container images, your image must include the dependencies defined in the Dockerfile provided by AWS IoT Greengrass. The following features aren't available when you run AWS IoT Greengrass in a Docker container: + [Connectors](connectors.md) that run in **Greengrass container** mode. To run a connector in a Docker container, the connector must run in **No container** mode. To find connectors that support **No container** mode, see [AWS-provided Greengrass connectors](connectors-list.md). Some of these connectors have an isolation mode parameter that you must set to **No container**. + [Local device and volume resources](access-local-resources.md). Your user-defined Lambda functions that run in the Docker container must access devices and volumes on the core directly. ## Configure the Docker image provided by AWS IoT Greengrass Follow these steps to configure the AWS IoT Greengrass Docker image to run IDT tests. **Prerequisities** Before you start this tutorial, you must do the following. + You must install the following software and versions on your host computer based on the AWS Command Line Interface (AWS CLI) version that you choose. ------ #### [ AWS CLI version 2 ] + [Docker](https://docs.docker.com/install/) version 18.09 or later. Earlier versions might also work, but we recommend 18.09 or later. + AWS CLI version 2.0.0 or later. + To install the AWS CLI version 2, see [Installing the AWS CLI version 2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html). + To configure the AWS CLI, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). **Note** To upgrade to a later AWS CLI version 2 on a Windows computer, you must repeat the [MSI installation](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html) process. ------ #### [ AWS CLI version 1 ] + [Docker](https://docs.docker.com/install/) version 18.09 or later. Earlier versions might also work, but we recommend 18.09 or later. + [Python](https://www.python.org/downloads/) version 3.6 or later. + [pip](https://pip.pypa.io/en/stable/installing) version 18.1 or later. + AWS CLI version 1.17.10 or later + To install the AWS CLI version 1, see [Installing the AWS CLI version 1](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv1.html). + To configure the AWS CLI, see [Configuring the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). + To upgrade to the latest version of the AWS CLI version 1, run the following command. ``` pip install awscli --upgrade --user ``` **Note** If you use the [MSI installation](https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html#msi-on-windows) of the AWS CLI version 1 on Windows, be aware of the following: If the AWS CLI version 1 installation fails to install botocore, try using the [Python and pip installation](https://docs.aws.amazon.com/cli/latest/userguide/awscli-install-windows.html#awscli-install-windows-pip). To upgrade to a later AWS CLI version 1, you must repeat the MSI installation process. ------ + To access Amazon Elastic Container Registry (Amazon ECR) resources, you must grant the following permission. + Amazon ECR requires users to grant the `ecr:GetAuthorizationToken` permission through an AWS Identity and Access Management (IAM) policy before they can authenticate to a registry and push or pull images from an Amazon ECR repository. For more information, see [Amazon ECR Repository Policy Examples](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policy-examples.html) and [Accessing One Amazon ECR Repository](https://docs.aws.amazon.com/AmazonECR/latest/userguide/security_iam_id-based-policy-examples.html#security_iam_id-based-policy-examples-access-one-bucket) in the *Amazon Elastic Container Registry User Guide*.   1. Download the Docker image and configure the container. You can download the prebuilt image from [Docker Hub](https://hub.docker.com/r/amazon/aws-iot-greengrass) or [Amazon Elastic Container Registry](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html) (Amazon ECR) and run it on Windows, macOS, and Linux (x86\$164) platforms. To download the Docker image from Amazon ECR, complete all of the steps in [Step 1: Get the AWS IoT Greengrass container image from Amazon ECR](run-gg-in-docker-container.md#docker-pull-image). Then, return to this topic to continue the configuration. 1. Linux users only: Make sure the user that runs IDT has permission to run Docker commands. For more information, see [Manage Docker as a non-root user](https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user) in the Docker documentation. 1. To run the AWS IoT Greengrass container, use the command for your operating system: ------ #### [ Linux ] ``` docker run --rm --init -it -d --name aws-iot-greengrass \ -p 8883:8883 \ -v : \ : ``` + Replace ** with the path to the kernel configuration file on the host and ** with the path where the volume is mounted in the container. The kernel config file on the host is usually located in `/proc/config.gz` or `/boot/config-`. You can run `uname -r` to find the ** value. **Example:** To mount the config file from `/boot/config-` ``` -v /boot/config-4.15.0-74-generic:/boot/config-4.15.0-74-generic \ ``` **Example:** To mount the config file from `proc/config.gz` ``` -v /proc/config.gz:/proc/config.gz \ ``` + Replace **:** in the command with the name of the repository and tag of the target image. **Example:** To point to the latest version of the AWS IoT Greengrass Core software ``` 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest ``` To get the list of AWS IoT Greengrass Docker images, run the following command. ``` aws ecr list-images --region us-west-2 --registry-id 216483018798 --repository-name aws-iot-greengrass ``` ------ #### [ macOS ] ``` docker run --rm --init -it -d --name aws-iot-greengrass \ -p 8883:8883 \ : ``` + Replace **:** in the command with the name of the repository and tag of the target image. **Example:** To point to the latest version of the AWS IoT Greengrass Core software ``` 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest ``` To get the list of AWS IoT Greengrass Docker images, run the following command: ``` aws ecr list-images --region us-west-2 --registry-id 216483018798 --repository-name aws-iot-greengrass ``` ------ #### [ Windows ] ``` docker run --rm --init -it -d --name aws-iot-greengrass \ -p 8883:8883 \ : ``` + Replace **:** in the command with the name of the repository and tag of the target image. **Example:** To point to the latest version of the AWS IoT Greengrass Core software ``` 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest ``` To get the list of AWS IoT Greengrass Docker images, run the following command: ``` aws ecr list-images --region us-west-2 --registry-id 216483018798 --repository-name aws-iot-greengrass ``` ------ **Important** When testing with IDT, do not include the `--entrypoint /greengrass-entrypoint.sh \` argument that's used to run the image for general AWS IoT Greengrass use. 1. Next step: [Configure your AWS credentials and `device.json` file](set-config.md). ## Configure the dockerfile provided by AWS IoT Greengrass Follow these steps to configure the Docker image built from the AWS IoT Greengrass Dockerfile to run IDT tests. 1. From [AWS IoT Greengrass Docker software](what-is-gg.md#gg-docker-download), download the Dockerfile package to your host computer and extract it. 1. Open `README.md`. The next three steps refer to sections in this file. 1. Make sure that you meet the requirements in the **Prerequisites** section. 1. Linux users only: Complete the **Enable Symlink and Hardlink Protection** and **Enable IPv4 Network Forwarding** steps. 1. To build the Docker image, complete all of the steps in **Step 1. Build the AWS IoT Greengrass Docker Image**. Then, return to this topic to continue the configuration. 1. To run the AWS IoT Greengrass container, use the command for your operating system: ------ #### [ Linux ] ``` docker run --rm --init -it -d --name aws-iot-greengrass \ -p 8883:8883 \ -v : \ : ``` + Replace ** with the path to the kernel configuration file on the host and ** with the path where the volume is mounted in the container. The kernel config file on the host is usually located in `/proc/config.gz` or `/boot/config-`. You can run `uname -r` to find the ** value. **Example:** To mount the config file from `/boot/config-` ``` -v /boot/config-4.15.0-74-generic:/boot/config-4.15.0-74-generic \ ``` **Example:** To mount the config file from `proc/config.gz` ``` -v /proc/config.gz:/proc/config.gz \ ``` + Replace **:** in the command with the name of the repository and tag of the target image. **Example:** To point to the latest version of the AWS IoT Greengrass Core software ``` 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest ``` To get the list of AWS IoT Greengrass Docker images, run the following command. ``` aws ecr list-images --region us-west-2 --registry-id 216483018798 --repository-name aws-iot-greengrass ``` ------ #### [ macOS ] ``` docker run --rm --init -it -d --name aws-iot-greengrass \ -p 8883:8883 \ : ``` + Replace **:** in the command with the name of the repository and tag of the target image. **Example:** To point to the latest version of the AWS IoT Greengrass Core software ``` 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest ``` To get the list of AWS IoT Greengrass Docker images, run the following command: ``` aws ecr list-images --region us-west-2 --registry-id 216483018798 --repository-name aws-iot-greengrass ``` ------ #### [ Windows ] ``` docker run --rm --init -it -d --name aws-iot-greengrass \ -p 8883:8883 \ : ``` + Replace **:** in the command with the name of the repository and tag of the target image. **Example:** To point to the latest version of the AWS IoT Greengrass Core software ``` 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest ``` To get the list of AWS IoT Greengrass Docker images, run the following command: ``` aws ecr list-images --region us-west-2 --registry-id 216483018798 --repository-name aws-iot-greengrass ``` ------ **Important** When testing with IDT, do not include the `--entrypoint /greengrass-entrypoint.sh \` argument that's used to run the image for general AWS IoT Greengrass use. 1. Next step: [Configure your AWS credentials and `device.json` file](set-config.md). ## Troubleshooting your Docker container setup for IDT for AWS IoT Greengrass Use the following information to help troubleshoot issues with running a Docker container for IDT for AWS IoT Greengrass testing. ### WARNING: Error loading config file:/home/user/.docker/config.json - stat /home//.docker/config.json: permission denied If you get this error when running `docker` commands on Linux, run the following command. Replace ** in the following command with the user that runs IDT. ``` sudo chown : /home//.docker -R sudo chmod g+rwx /home//.docker -R ``` # Optional: Configuring your device for ML qualification IDT for AWS IoT Greengrass provides machine learning (ML) qualification tests to validate that your devices can perform ML inference locally using cloud-trained models. To run ML qualification tests, you must first configure your devices as described in [Configure your device to run IDT tests](device-config-setup.md). Then, follow the steps in this topic to install dependencies for the ML frameworks that you want to run. IDT v3.1.0 or later is required to run tests for ML qualification. ## Installing ML framework dependencies All ML framework dependencies must be installed under the `/usr/local/lib/python3.x/site-packages` directory. To make sure they are installed under the correct directory, we recommend that you use `sudo` root permissions when installing the dependencies. Virtual environments are not supported for qualification tests. **Note** If you're testing Lambda functions that run with [containerization](lambda-group-config.md#lambda-containerization-considerations) (in **Greengrass container** mode), creating symlinks for Python libraries under `/usr/local/lib/python3.x` isn't supported. To avoid errors, you must install the dependencies under the correct directory. Follow the steps to install the dependencies for your target framework: + [Install MXNet dependencies](#ml-qualification-mxnet-dependencies) + [Install TensorFlow dependencies](#ml-qualification-tensorflow-dependencies) + [Install DLR dependencies](#ml-qualification-dlr-dependencies)   ## Install Apache MXNet dependencies IDT qualification tests for this framework have the following dependencies: + Python 3.6 or Python 3.7. **Note** If you're using Python 3.6, you must create a symbolic link from Python 3.7 to Python 3.6 binaries. This configures your device to meet the Python requirement for AWS IoT Greengrass. For example: ``` sudo ln -s path-to-python-3.6/python3.6 path-to-python-3.7/python3.7 ``` + Apache MXNet v1.2.1 or later. + NumPy. The version must be compatible with your MXNet version. ### Installing MXNet Follow the instructions in the MXNet documentation to [install MXNet](https://mxnet.apache.org/get_started/?platform=linux&language=python&processor=cpu&environ=pip&). **Note** If Python 2.x and Python 3.x are both installed on your device, use Python 3.x in the commands that you run to install the dependencies. ### Validating the MXNet installation Choose one of the following options to validate the MXNet installation. #### Option 1: SSH into your device and run scripts 1. SSH into your device. 1. Run the following scripts to verify that the dependencies are correctly installed. ``` sudo python3.7 -c "import mxnet; print(mxnet.__version__)" ``` ``` sudo python3.7 -c "import numpy; print(numpy.__version__)" ``` The output prints the version number and the script should exit without error. #### Option 2: Run the IDT dependency test 1. Make sure that `device.json` is configured for ML qualification. For more information, see [Configure device.json for ML qualification](set-config.md#device-json-ml-qualification). 1. Run the dependencies test for the framework. ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id mldependencies --test-id mxnet_dependency_check ``` The test summary displays a `PASSED` result for `mldependencies`.   ## Install TensorFlow dependencies IDT qualification tests for this framework have the following dependencies: + Python 3.6 or Python 3.7. **Note** If you're using Python 3.6, you must create a symbolic link from Python 3.7 to Python 3.6 binaries. This configures your device to meet the Python requirement for AWS IoT Greengrass. For example: ``` sudo ln -s path-to-python-3.6/python3.6 path-to-python-3.7/python3.7 ``` + TensorFlow 1.x. ### Installing TensorFlow Follow the instructions in the TensorFlow documentation to install TensorFlow 1.x [with pip](https://www.tensorflow.org/install/pip) or [from source](https://www.tensorflow.org/install/source). **Note** If Python 2.x and Python 3.x are both installed on your device, use Python 3.x in the commands that you run to install the dependencies. ### Validating the TensorFlow installation Choose one of the following options to validate the TensorFlow installation. #### Option 1: SSH into your device and run a script 1. SSH into your device. 1. Run the following script to verify that the dependency is correctly installed. ``` sudo python3.7 -c "import tensorflow; print(tensorflow.__version__)" ``` The output prints the version number and the script should exit without error. #### Option 2: Run the IDT dependency test 1. Make sure that `device.json` is configured for ML qualification. For more information, see [Configure device.json for ML qualification](set-config.md#device-json-ml-qualification). 1. Run the dependencies test for the framework. ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id mldependencies --test-id tensorflow_dependency_check ``` The test summary displays a `PASSED` result for `mldependencies`.   ## Install Amazon SageMaker AI Neo Deep Learning Runtime (DLR) dependencies IDT qualification tests for this framework have the following dependencies: + Python 3.6 or Python 3.7. **Note** If you're using Python 3.6, you must create a symbolic link from Python 3.7 to Python 3.6 binaries. This configures your device to meet the Python requirement for AWS IoT Greengrass. For example: ``` sudo ln -s path-to-python-3.6/python3.6 path-to-python-3.7/python3.7 ``` + SageMaker AI Neo DLR. + numpy. After you install the DLR test dependencies, you must [compile the model](#ml-qualification-dlr-compile-model). ### Installing DLR Follow the instructions in the DLR documentation to [install the Neo DLR](https://neo-ai-dlr.readthedocs.io/en/latest/install.html#building-on-linux). **Note** If Python 2.x and Python 3.x are both installed on your device, use Python 3.x in the commands that you run to install the dependencies. ### Validating the DLR installation Choose one of the following options to validate the DLR installation. #### Option 1: SSH into your device and run scripts 1. SSH into your device. 1. Run the following scripts to verify that the dependencies are correctly installed. ``` sudo python3.7 -c "import dlr; print(dlr.__version__)" ``` ``` sudo python3.7 -c "import numpy; print(numpy.__version__)" ``` The output prints the version number and the script should exit without error. #### Option 2: Run the IDT dependency test 1. Make sure that `device.json` is configured for ML qualification. For more information, see [Configure device.json for ML qualification](set-config.md#device-json-ml-qualification). 1. Run the dependencies test for the framework. ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id mldependencies --test-id dlr_dependency_check ``` The test summary displays a `PASSED` result for `mldependencies`. ## Compile the DLR model You must compile the DLR model before you can use it for ML qualification tests. For steps, choose one of the following options. ### Option 1: Use Amazon SageMaker AI to compile the model Follow these steps to use SageMaker AI to compile the ML model provided by IDT. This model is pretrained with Apache MXNet. 1. Verify that your device type is supported by SageMaker AI. For more information, see the [target device options](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_OutputConfig.html#sagemaker-Type-OutputConfig-TargetDevice) the *Amazon SageMaker AI API Reference*. If your device type is not currently supported by SageMaker AI, follow the steps in [Option 2: Use TVM to compile the DLR model](#ml-qualification-compile-dlr-option-2). **Note** Running the DLR test with a model compiled by SageMaker AI might take 4 or 5 minutes. Don’t stop IDT during this time. 1. Download the tarball file that contains the uncompiled, pretrained MXNet model for DLR: + [dlr-noncompiled-model-1.0.tar.gz](https://docs.aws.amazon.com/greengrass/latest/developerguide/download-dlr-noncompiled-model-1.0.html) 1. Decompress the tarball. This command generates the following directory structure. ![\[The resnet18 directory contains three files.\]](http://docs.aws.amazon.com/greengrass/v1/developerguide/images/idt/idt-ml-qualification-dlr-uncompiled.png) 1. Move `synset.txt` out of the `resnet18` directory. Make a note of the new location. You copy this file to compiled model directory later. 1. Compress the contents of the `resnet18` directory. ``` tar cvfz model.tar.gz resnet18v1-symbol.json resnet18v1-0000.params ``` 1. Upload the compressed file to an Amazon S3 bucket in your AWS account, and then follow the steps in [Compile a Model (Console)](https://docs.aws.amazon.com/sagemaker/latest/dg/neo-job-compilation-console.html) to create a compilation job. 1. For **Input configuration**, use the following values: + For **Data input configuration**, enter `{"data": [1, 3, 224, 224]}`. + For **Machine learning framework**, choose `MXNet`. 1. For **Output configuration**, use the following values: + For **S3 Output location**, enter the path to the Amazon S3 bucket or folder where you want to store the compiled model. + For **Target device**, choose your device type. 1. Download the compiled model from the output location you specified, and then unzip the file. 1. Copy `synset.txt` into the compiled model directory. 1. Change the name of the compiled model directory to `resnet18`. Your compiled model directory must have the following directory structure. ![\[The resnet18 compiled model directory contains four files.\]](http://docs.aws.amazon.com/greengrass/v1/developerguide/images/idt/idt-ml-qualification-dlr-compiled-sm.png) ### Option 2: Use TVM to compile the DLR model Follow these steps to use TVM to compile the ML model provided by IDT. This model is pretrained with Apache MXNet, so you must install MXNet on the computer or device where you compile the model. To install MXNet, follow the instructions in the [ MXNet documentation](https://mxnet.apache.org/get_started/?platform=linux&language=python&processor=cpu&environ=pip&). **Note** We recommend that you compile the model on your target device. This practice is optional, but it can help ensure compatibility and mitigate potential issues.   1. Download the tarball file that contains the uncompiled, pretrained MXNet model for DLR: + [dlr-noncompiled-model-1.0.tar.gz](https://docs.aws.amazon.com/greengrass/latest/developerguide/download-dlr-noncompiled-model-1.0.html) 1. Decompress the tarball. This command generates the following directory structure. ![\[The resnet18 directory contains three files.\]](http://docs.aws.amazon.com/greengrass/v1/developerguide/images/idt/idt-ml-qualification-dlr-uncompiled.png) 1. Follow the instructions in the TVM documentation to [build and install TVM from source for your platform](https://docs.tvm.ai/install/from_source.html). 1. After TVM is built, run the TVM compilation for the resnet18 model. The following steps are based on [ Quick Start Tutorial for Compiling Deep Learning Models](https://tvm.apache.org/docs/tutorial/relay_quick_start.html#sphx-glr-tutorials-get-started-relay-quick-start-py) in the TVM documentation. 1. Open the `relay_quick_start.py` file from the cloned TVM repository. 1. Update the code that [defines a neural network in relay](https://tvm.apache.org/docs/tutorial/relay_quick_start.html#define-neural-network-in-relay). You can use one of following options: + Option 1: Use `mxnet.gluon.model_zoo.vision.get_model` to get the relay module and parameters: ``` from mxnet.gluon.model_zoo.vision import get_model block = get_model('resnet18_v1', pretrained=True) mod, params = relay.frontend.from_mxnet(block, {"data": data_shape}) ``` + Option 2: From the uncompiled model that you downloaded in step 1, copy the following files to the same directory as the `relay_quick_start.py` file. These files contain the relay module and parameters. + `resnet18v1-symbol.json` + `resnet18v1-0000.params` 1. Update the code that [saves and loads the compiled module](https://tvm.apache.org/docs/tutorial/relay_quick_start.html#save-and-load-compiled-module) to use the following code. ``` from tvm.contrib import util path_lib = "deploy_lib.so" # Export the model library based on your device architecture lib.export_library("deploy_lib.so", cc="aarch64-linux-gnu-g++") with open("deploy_graph.json", "w") as fo: fo.write(graph) with open("deploy_param.params", "wb") as fo: fo.write(relay.save_param_dict(params)) ``` 1. Build the model: ``` python3 tutorials/relay_quick_start.py --build-dir ./model ``` This command generates the following files. + `deploy_graph.json` + `deploy_lib.so` + `deploy_param.params` 1. Copy the generated model files into a directory named `resnet18`. This is your compiled model directory. 1. Copy the compiled model directory to your host computer. Then copy `synset.txt` from the uncompiled model that you downloaded in step 1 into the compiled model directory. Your compiled model directory must have the following directory structure. ![\[The resnet18 compiled model directory contains four files.\]](http://docs.aws.amazon.com/greengrass/v1/developerguide/images/idt/idt-ml-qualification-dlr-compiled-tvm.png) Next, [configure your AWS credentials and `device.json` file](set-config.md). # Configure IDT settings to run the AWS IoT Greengrass qualification suite Before you run tests, you must configure settings for AWS credentials and devices on your host computer. ## Configure your AWS credentials You must configure your IAM user credentials in the ` /configs/config.json` file. Use the credentials for the IDT for AWS IoT Greengrass user created in [Create and configure an AWS account](dev-tst-prereqs.md#config-aws-account-for-idt). You can specify your credentials in one of two ways: + Credentials file + Environment variables ### Configure AWS credentials with a credentials file IDT uses the same credentials file as the AWS CLI. For more information, see [Configuration and credential files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html). The location of the credentials file varies, depending on the operating system you are using: + macOS, Linux: `~/.aws/credentials` + Windows: `C:\Users\UserName\.aws\credentials` Add your AWS credentials to the `credentials` file in the following format: ``` [default] aws_access_key_id = aws_secret_access_key = ``` To configure IDT for AWS IoT Greengrass to use AWS credentials from your `credentials` file, edit your `config.json` file as follows: ``` { "awsRegion": "us-west-2", "auth": { "method": "file", "credentials": { "profile": "default" } } } ``` **Note** If you do not use the `default` AWS profile, be sure to change the profile name in your `config.json` file. For more information, see [Named profiles](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html). ### Configure AWS credentials with environment variables Environment variables are variables maintained by the operating system and used by system commands. They are not saved if you close the SSH session. IDT for AWS IoT Greengrass can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to store your AWS credentials. To set these variables on Linux, macOS, or Unix, use **export**: ``` export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= ``` To set these variables on Windows, use **set**: ``` set AWS_ACCESS_KEY_ID= set AWS_SECRET_ACCESS_KEY= ``` To configure IDT to use the environment variables, edit the `auth` section in your `config.json` file. Here is an example: ``` { "awsRegion": "us-west-2", "auth": { "method": "environment" } } ``` ## Configure device.json In addition to AWS credentials, IDT for AWS IoT Greengrass needs information about the devices that tests are run on (for example, IP address, login information, operating system, and CPU architecture). You must provide this information using the `device.json` template located in ` /configs/device.json`: ------ #### [ Physical device ] ``` [ { "id": "", "sku": "", "features": [ { "name": "os", "value": "linux | ubuntu | openwrt" }, { "name": "arch", "value": "x86_64 | armv6l | armv7l | aarch64" }, { "name": "container", "value": "yes | no" }, { "name": "docker", "value": "yes | no" }, { "name": "streamManagement", "value": "yes | no" }, { "name": "hsi", "value": "yes | no" }, { "name": "ml", "value": "mxnet | tensorflow | dlr | mxnet,dlr,tensorflow | no" }, *********** Remove the section below if the device is not qualifying for ML **************, { "name": "mlLambdaContainerizationMode", "value": "container | process | both" }, { "name": "processor", "value": "cpu | gpu" }, ****************************************************************************************** ], *********** Remove the section below if the device is not qualifying for HSI *************** "hsm": { "p11Provider": "/path/to/pkcs11ProviderLibrary", "slotLabel": "", "slotUserPin": "", "privateKeyLabel": "", "openSSLEngine": "/path/to/openssl/engine" }, ******************************************************************************************** *********** Remove the section below if the device is not qualifying for ML **************** "machineLearning": { "dlrModelPath": "/path/to/compiled/dlr/model", "environmentVariables": [ { "key": "", "value": "" } ], "deviceResources": [ { "name": "", "path": "", "type": "device | volume" } ] }, ****************************************************************************************** "kernelConfigLocation": "", "greengrassLocation": "", "devices": [ { "id": "", "connectivity": { "protocol": "ssh", "ip": "", "port": 22, "auth": { "method": "pki | password", "credentials": { "user": "", "privKeyPath": "/path/to/private/key", "password": "" } } } } ] } ] ``` **Note** Specify `privKeyPath` only if `method` is set to `pki`. Specify `password` only if `method` is set to `password`. ------ #### [ Docker container ] ``` [ { "id": "", "sku": "", "features": [ { "name": "os", "value": "linux | ubuntu | openwrt" }, { "name": "arch", "value": "x86_64" }, { "name": "container", "value": "no" }, { "name": "docker", "value": "no" }, { "name": "streamManagement", "value": "yes | no" }, { "name": "hsi", "value": "no" }, { "name": "ml", "value": "mxnet | tensorflow | dlr | mxnet,dlr,tensorflow | no" }, *********** Remove the section below if the device is not qualifying for ML **************, { "name": "mlLambdaContainerizationMode", "value": "process" }, { "name": "processor", "value": "cpu | gpu" }, ****************************************************************************************** ], *********** Remove the section below if the device is not qualifying for ML **************** "machineLearning": { "dlrModelPath": "/path/to/compiled/dlr/model", "environmentVariables": [ { "key": "", "value": "" } ], "deviceResources": [ { "name": "", "path": "", "type": "device | volume" } ] }, ****************************************************************************************** "kernelConfigLocation": "", "greengrassLocation": "", "devices": [ { "id": "", "connectivity": { "protocol": "docker", "containerId": "", "containerUser": "" } } ] } ] ``` ------ All fields that contain values are required as described here: `id` A user-defined alphanumeric ID that uniquely identifies a collection of devices called a *device pool*. Devices that belong to a pool must have identical hardware. When you run a suite of tests, devices in the pool are used to parallelize the workload. Multiple devices are used to run different tests. `sku` An alphanumeric value that uniquely identifies the device under test. The SKU is used to track qualified boards. If you want to list your board in the AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process. `features` An array that contains the device's supported features. All features are required. `os` and `arch` Supported operating system (OS) and architecture combinations: + `linux`, `x86_64` + `linux`, `armv6l` + `linux`, `armv7l` + `linux`, `aarch64` + `ubuntu`, `x86_64` + `openwrt`, `armv7l` + `openwrt`, `aarch64` If you use IDT to test AWS IoT Greengrass running in a Docker container, only the x86\$164 Docker architecture is supported. `container` Validates that the device meets all of the software and hardware requirements to run Lambda functions in container mode on a Greengrass core. The valid value is `yes` or `no`. `docker` Validates that the device meets all the required technical dependencies to use the Greengrass Docker application deployment connector to run containers The valid value is `yes` or `no`. `streamManagement` Validates that the device meets all of the required technical dependencies to run AWS IoT Greengrass stream manager. The valid value is `yes` or `no`. `hsi` Verifies that the provided HSI shared library can interface with the hardware security module (HSM) and implements the required PKCS\$111 APIs correctly. The HSM and shared library must be able to sign a CSR, perform TLS operations, and provide the correct key lengths and public key algorithm. The valid value is `yes` or `no`. `ml` Validates that the device meets all of the required technical dependencies to perform ML inference locally. The valid value can be any combination of `mxnet`, `tensorflow`, `dlr`, and `no` (for example, `mxnet`, `mxnet,tensorflow`, `mxnet,tensorflow,dlr`, or `no`). `mlLambdaContainerizationMode` Validates that the device meets all of the required technical dependencies to perform ML inference in container mode on a Greengrass device. The valid value is `container`, `process`, or `both`. `processor` Validates that the device meets all of the hardware requirements for the specified processor type. The valid value is `cpu` or `gpu`. If you don't want to use the `container`, `docker`, `streamManager`, `hsi`, or `ml` feature, you can set the corresponding `value` to `no`. Docker only supports feature qualification for `streamManagement` and `ml`. `machineLearning` Optional. Configuration information for ML qualification tests. For more information, see [Configure device.json for ML qualification](#device-json-ml-qualification). `hsm` Optional. Configuration information for testing with an AWS IoT Greengrass Hardware Security Module (HSM). Otherwise, the `hsm` property should be omitted. For more information, see [Hardware security integration](hardware-security.md). This property applies only if `connectivity.protocol` is set to `ssh`. `hsm.p11Provider` The absolute path to the PKCS\$111 implementation's libdl-loadable library. `hsm.slotLabel` The slot label used to identify the hardware module. `hsm.slotUserPin` The user PIN used to authenticate the AWS IoT Greengrass core to the module. `hsm.privateKeyLabel` The label used to identify the key in the hardware module. `hsm.openSSLEngine` The absolute path to the OpenSSL engine's `.so` file that enables PKCS\$111 support on OpenSSL. Used by the AWS IoT Greengrass OTA update agent. `devices.id` A user-defined unique identifier for the device being tested. `connectivity.protocol` The communication protocol used to communicate with this device. Currently, the only supported values are `ssh` for physical devices and `docker` for Docker containers. `connectivity.ip` The IP address of the device being tested. This property applies only if `connectivity.protocol` is set to `ssh`. `connectivity.containerId` The container ID or name of the Docker container being tested. This property applies only if `connectivity.protocol` is set to `docker`. `connectivity.auth` Authentication information for the connection. This property applies only if `connectivity.protocol` is set to `ssh`. `connectivity.auth.method` The authentication method used to access a device over the given connectivity protocol. Supported values are: + `pki` + `password` `connectivity.auth.credentials` The credentials used for authentication. `connectivity.auth.credentials.password` The password used for signing in to the device being tested. This value applies only if `connectivity.auth.method` is set to `password`. `connectivity.auth.credentials.privKeyPath` The full path to the private key used to sign in to the device under test. This value applies only if `connectivity.auth.method` is set to `pki`. `connectivity.auth.credentials.user` The user name for signing in to the device being tested. `connectivity.auth.credentials.privKeyPath` The full path to the private key used to sign in to the device being tested. `connectivity.port` Optional. The port number to use for SSH connections. The default value is 22. This property only applies if `connectivity.protocol` is set to `ssh`. `greengrassLocation` The location of AWS IoT Greengrass Core software on your devices. For physical devices, this value is only used when you use an existing installation of AWS IoT Greengrass. Use this attribute to tell IDT to use the version of the AWS IoT Greengrass Core software installed on your devices. When running tests in a Docker container from Docker image or Dockerfile provided by AWS IoT Greengrass, set this value to `/greengrass`. `kernelConfigLocation` Optional. The path to the kernel configuration file. AWS IoT Device Tester uses this file to check if the devices have the required kernel features enabled. If not specified, IDT uses the following paths to search for the kernel configuration file: `/proc/config.gz` and `/boot/config-`. AWS IoT Device Tester uses the first path it finds. ## Configure device.json for ML qualification This section describes the optional properties in the device configuration file that apply to ML qualification. If you plan to run tests for ML qualification, you must define the properties that apply to your use case. You can use the `device-ml.json` template to define the configuration settings for your device. This template contains the optional ML properties. You can also use `device.json` and add the ML qualification properties. These files are located in `/configs` and includes ML qualification properties. If you use `device-ml.json`, you must rename the file to `device.json` before you run IDT tests. For information about device configuration properties that don't apply to ML qualification, see [Configure device.json](#device-config).   `ml` in the `features` array The ML frameworks that your board supports. This property requires IDT v3.1.0 or later. + If your board supports only one framework, specify the framework. For example: ``` { "name": "ml", "value": "mxnet" } ``` + If your board supports multiple frameworks, specify the frameworks as a comma-separated list. For example: ``` { "name": "ml", "value": "mxnet,tensorflow" } ``` `mlLambdaContainerizationMode` in the `features` array The [containerization mode](lambda-group-config.md#lambda-containerization-considerations) that you want to test with. This property requires IDT v3.1.0 or later. + Choose `process` to run ML inference code with a non-containerized Lambda function. This option requires AWS IoT Greengrass v1.10.x or later. + Choose `container` to run ML inference code with a containerized Lambda function. + Choose `both` to run ML inference code with both modes. This option requires AWS IoT Greengrass v1.10.x or later. `processor` in the `features` array Indicates the hardware accelerator that your board supports. This property requires IDT v3.1.0 or later. + Choose `cpu` if your board uses a CPU as the processor. + Choose `gpu` if your board uses a GPU as the processor. `machineLearning` Optional. Configuration information for ML qualification tests. This property requires IDT v3.1.0 or later. `dlrModelPath` Required to use the `dlr` framework. The absolute path to your DLR compiled model directory, which must be named `resnet18`. For more information, see [Compile the DLR model](idt-ml-qualification.md#ml-qualification-dlr-compile-model). The following is an example path on macOS: `/Users//Downloads/resnet18`. `environmentVariables` An array of key-value pairs that can dynamically pass settings to ML inference tests. Optional for CPU devices. You can use this section to add framework-specific environment variables required by your device type. For information about these requirements, see the official website of the framework or the device. For example, to run MXNet inference tests on some devices, the following environment variables might be required. ``` "environmentVariables": [ ... { "key": "PYTHONPATH", "value": "$MXNET_HOME/python:$PYTHONPATH" }, { "key": "MXNET_HOME", "value": "$HOME/mxnet/" }, ... ] ``` The `value` field might vary based on your MXNet installation. If you're testing Lambda functions that run with [containerization](lambda-group-config.md#lambda-containerization-considerations) on GPU devices, add environment variables for the GPU library. This makes it possible for the GPU to perform computations. To use different GPU libraries, see the official documentation for the library or device. Configure the following keys if the `mlLambdaContainerizationMode` feature is set to `container` or `both`. ``` "environmentVariables": [ { "key": "PATH", "value": ":$PATH" }, { "key": "LD_LIBRARY_PATH", "value": "" }, ... ] ``` `deviceResources` Required by GPU devices. Contains [local resources](access-local-resources.md#lra-resource-types) that can be accessed by Lambda functions. Use this section to add local device and volume resources. + For device resources, specify `"type": "device"`. For GPU devices, device resources should be GPU-related device files under `/dev`. **Note** The `/dev/shm` directory is an exception. It can be configured as a volume resource only. + For volume resources, specify `"type": "volume"`. # Run the AWS IoT Greengrass qualification suite After you [set the required configuration](set-config.md), you can start the tests. The runtime of the full test suite depends on your hardware. For reference, it takes approximately 30 minutes to complete the full test suite on a Raspberry Pi 3B. The following example `run-suite` commands show you how to run the qualification tests for a device pool. A device pool is a set of identical devices. ------ #### [ IDT v3.0.0 and later ] Run all test groups in a specified test suite. ``` devicetester_[linux | mac | win_x86-64] run-suite --suite-id GGQ_1.0.0 --pool-id ``` Use the `list-suites` command to list the test suites that are in the `tests` folder. Run a specific test group in a test suite. ``` devicetester_[linux | mac | win_x86-64] run-suite --suite-id GGQ_1.0.0 --group-id --pool-id ``` Use the `list-groups` command to list the test groups in a test suite. Run a specific test case in a test group. ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id --test-id ``` Run multiple test cases in a test group. ``` devicetester_[linux | mac | win_x86-64] run-suite --group-id --test-id , ``` List the test cases in a test group. ``` devicetester_[linux | mac | win_x86-64] list-test-cases --group-id ``` The options for the `run-suite` command are optional. For example, you can omit `pool-id` if you have only one device pool defined in your `device.json` file. Or, you can omit `suite-id` if you want to run the latest test suite version in the `tests` folder. **Note** IDT prompts you if a newer test suite version is available online. For more information, see [Set the default update behavior](#idt-update-behavior). For more information about `run-suite` and other IDT commands, see [IDT for AWS IoT Greengrass commands](#bk-cli). ------ #### [ IDT v2.3.0 and earlier ] Run all test groups in a specified suite. ``` devicetester_[linux | mac | win_x86-64] run-suite --suite-id GGQ_1 --pool-id ``` Run a specific test group. ``` devicetester_[linux | mac | win_x86-64] run-suite --suite-id GGQ_1 --group-id --pool-id ``` `suite-id` and `pool-id` are optional if you are running a single test suite on a single device pool. This means that you have only one device pool defined in your `device.json` file. ------ ## Check for Greengrass dependencies We recommend that you run the dependency checker test group to make sure all Greengrass dependencies are installed before you run related test groups. For example: + Run `ggcdependencies` before running core qualification test groups. + Run `containerdependencies` before running container-specific test groups. + Run `dockerdependencies` before running Docker-specific test groups. + Run `ggcstreammanagementdependencies` before running stream manager-specific test groups. ## Set the default update behavior When you start a test run, IDT checks online for a newer test suite version. If one is available, IDT prompts you to update to the latest available version. You can set the `upgrade-test-suite` (or `u`) flag to control the default update behavior. Valid values are: + `y`. IDT downloads and uses the latest available version. + `n` (default). IDT uses the version specified in the `suite-id` option. If `suite-id` is not specified, IDT uses the latest version in the `tests` folder. If you don't include the `upgrade-test-suite` flag, IDT prompts you when an update is available and waits 30 seconds for your input (`y` or `n`). If no input is entered, it defaults to `n` and continues running the tests. The following examples show common use cases for this feature: **Automatically use the latest tests available for a test group.** ``` devicetester_linux run-suite -u y --group-id mqtt --pool-id DevicePool1 ``` **Run tests in a specific test suite version.** ``` devicetester_linux run-suite -u n --suite-id GGQ_1.0.0 --group-id mqtt --pool-id DevicePool1 ``` **Prompt for updates at runtime.** ``` devicetester_linux run-suite --pool-id DevicePool1 ``` ## IDT for AWS IoT Greengrass commands The IDT commands are located in the `/bin` directory. Use them for the following operations: ------ #### [ IDT v3.0.0 and later ] `help` Lists information about the specified command. `list-groups` Lists the groups in a given test suite. `list-suites` Lists the available test suites. `list-supported-products` Lists the supported products, in this case AWS IoT Greengrass versions, and test suite versions for the current IDT version. `list-test-cases` Lists the test cases in a given test group. The following option is supported: + `group-id`. The test group to search for. This option is required and must specify a single group. `run-suite` Runs a suite of tests on a pool of devices. The following are some supported options: + `suite-id`. The test suite version to run. If not specified, IDT uses the latest version in the `tests` folder. + `group-id`. The test groups to run, as a comma-separated list. If not specified, IDT runs all test groups in the test suite. + `test-id`. The test cases to run, as a comma-separated list. When specified, `group-id` must specify a single group. + `pool-id`. The device pool to test. You must specify a pool if you have multiple device pools defined in your `device.json` file. + `upgrade-test-suite`. Controls how test suite version updates are handled. Starting in IDT v3.0.0, IDT checks online for updated test suite versions. For more information, see [Test suite versions](idt-gg-qualification.md#idt-test-suite-versions). + `stop-on-first-failure`. Configures IDT to stop execution on the first failure. This option should be used with `group-id` to debug the specified test groups. Do not use this option when running a full test-suite to generate a qualification report. + `update-idt`. Sets the response for the prompt to update IDT. `Y` as input stops the test execution if IDT detects there is a newer version. `N` as input continues the test execution. + `update-managed-policy`. `Y` as input stops the test execution if IDT detects that the user's managed policy isn’t updated. `N` as input continues the test execution. For more information about `run-suite` options, use the `help` option: ``` devicetester_[linux | mac | win_x86-64] run-suite -h ``` ------ #### [ IDT v2.3.0 and earlier ] `help` Lists information about the specified command. `list-groups` Lists the groups in a given test suite. `list-suites` Lists the available test suites. `run-suite` Runs a suite of tests on a pool of devices. For more information about `run-suite` options, use the `help` option: ``` devicetester_[linux | mac | win_x86-64] run-suite -h ``` ------ # Understanding results and logs This section describes how to view and interpret IDT result reports and logs. ## Viewing results While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it generates two test reports. These reports can be found in `/results//`. Both reports capture the results from the qualification test suite execution. The `awsiotdevicetester_report.xml` is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements: + The IDT version. + The AWS IoT Greengrass version that was tested. + The SKU and the device pool name specified in the `device.json` file. + The features of the device pool specified in the `device.json` file. + The aggregate summary of test results. + A breakdown of test results by libraries that were tested based on the device features (for example, local resource access, shadow, MQTT, and so on). The `GGQ_Result.xml` report is in [JUnit XML format](https://llg.cubic.org/docs/junit/). You can integrate it into continuous integration and deployment platforms like [Jenkins](https://jenkins.io/), [Bamboo](https://www.atlassian.com/software/bamboo), and so on. The report contains the following elements: + Aggregate summary of test results. + Breakdown of test results by the AWS IoT Greengrass functionality that was tested. ## Interpreting IDT reports The report section in `awsiotdevicetester_report.xml` or `awsiotdevicetester_report.xml` lists the tests that were run and the results. The first XML tag `` contains the summary of the test execution. For example: ``` ```Attributes used in the `` tag `name` The name of the test suite. `time` The time, in seconds, it took to run the qualification suite. `tests` The number of tests executed. `failures` The number of tests that were run, but did not pass. `errors` The number of tests that IDT couldn't execute. `disabled` This attribute is not used and can be ignored. The `awsiotdevicetester_report.xml` file contains an `` tag that contains information about the product being tested and the product features that were validated after running a suite of tests.Attributes used in the `` tag `name` The name of the product being tested. `version` The version of the product being tested. `features` The features validated. Features marked as `required` are required to submit your board for qualification. The following snippet shows how this information appears in the `awsiotdevicetester_report.xml` file. ``` ``` Features marked as `optional` are not required for qualification. The following snippets show optional features. ``` 
 ``` If there are no test failures or errors for the required features, your device meets the technical requirements to run AWS IoT Greengrass and can interoperate with AWS IoT services. If you want to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence. In the event of test failures or errors, you can identify the test that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag show the test result summary for a test group. For example: ``` ``` The format is similar to the `` tag, but with a `skipped` attribute that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each executed test for a test group. For example: ``` > ```Attributes used in the `` tag `name` The name of the test. `attempts` The number of times IDT executed the test case. When a test fails or an error occurs, `` or `` tags are added to the `` tag with information for troubleshooting. For example: ``` Reason for the test failure Reason for the test execution error ``` ## Viewing logs IDT generates logs from test execution in `/results//logs`. Two sets of logs are generated: `test_manager.log` Logs generated from the Test Manager component of AWS IoT Device Tester (for example, logs related to configuration, test sequencing, and report generation). `.log (for example, ota.log)` Logs of the test group, including logs from the device under test. When a test fails, a tar.gz file that contains the logs of the device under test for the test is created (for example, `ota_prod_test_1_ggc_logs.tar.gz`). For more information, see [IDT for AWS IoT Greengrass troubleshooting](idt-troubleshooting.md).