# Use IDT to run the AWS IoT Greengrass qualification suite
You can use AWS IoT Device Tester for AWS IoT Greengrass V2 to verify that the AWS IoT Greengrass Core software runs on your hardware and can communicate with the AWS Cloud. It also performs end-to-end tests with AWS IoT Core. For example, it verifies that your device can deploy components and upgrade them.
In addition to testing devices, IDT for AWS IoT Greengrass V2 creates resources (for example, AWS IoT things, groups, and so on) in your AWS account to facilitate the qualification process.
To create these resources, IDT for AWS IoT Greengrass V2 uses the AWS credentials configured in the `config.json` file to make API calls on your behalf. These resources are provisioned at various times during a test.
When you use IDT for AWS IoT Greengrass V2 to run the AWS IoT Greengrass qualification suite, it performs the following steps:
1. Loads and validates your device and credentials configuration.
1. Performs selected tests with the required local and cloud resources.
1. Cleans up local and cloud resources.
1. Generates tests reports that indicate if your board passed the tests required for qualification.
## Test suite versions
IDT for AWS IoT Greengrass V2 organizes tests into test suites and test groups.
+ A test suite is the set of test groups used to verify that a device works with particular versions of AWS IoT Greengrass.
+ A test group is the set of individual tests related to a particular feature, such as component deployments.
Test suites are versioned using a `major.minor.patch` format, for example `GGV2Q_1.0.0`. When you download IDT, the package includes the latest Greengrass qualification suite version.
**Important**
Tests from unsupported test suite versions are not valid for device qualification. IDT doesn't print qualification reports for unsupported versions. For more information, see [Support policy for AWS IoT Device Tester for AWS IoT Greengrass](idt-support-policy.md).
You can run `list-supported-products` to list the versions of AWS IoT Greengrass and test suites that are supported by your current version of IDT.
## Test group descriptions
**Required Test Groups for Core Qualification**
These test groups are required to qualify your AWS IoT Greengrass V2 device for the AWS Partner Device Catalog.
Core Dependencies
Validates that the device meets all software and hardware requirements for the AWS IoT Greengrass Core software. This test group includes the following test case:
Java Version
Checks that the required Java version is installed on the device under test. AWS IoT Greengrass requires Java 8 or later.
PreTest Validation
Checks that the device meets the software requirements to run tests.
+ For Linux-based devices, this test checks if the device can run the following Linux commands:
`chmod`, `cp`, `echo`, `grep`, `kill`, `ln`, `mkinfo`, `ps`, `rm`, `sh`, `uname`
+ For Windows-based devices, this test checks if the device has the following Microsoft software installed:
[Powershell](https://learn.microsoft.com/en-us/powershell/?view=powershell-7.1) v5.1 or later, [.NET](https://learn.microsoft.com/en-us/dotnet/) v4.6.1 or later, [Visual C\$1\$1](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist?view=msvc-170) 2017 or later, [PsExec utility](https://learn.microsoft.com/en-us/sysinternals/downloads/psexec)
Version Checker
Checks that the version of AWS IoT Greengrass provided is compatible with the AWS IoT Device Tester version you are using.
Component
Validates that the device can deploy components and upgrade them. This test group includes the following tests:
Cloud Component
Validates device capability for cloud components.
Local Component
Validates device capability for local components.
Lambda
This test is not applicable for Windows-based devices.
Validates that the device can deploy Lambda function components that use the Java runtime, and that the Lambda functions can use AWS IoT Core MQTT topics as event sources for work messages.
MQTT
Validates that the device can subscribe and publish to AWS IoT Core MQTT topics.
**Optional Test Groups**
These test groups are optional, and used only for qualifying Linux-based Greengrass core devices. If you choose to qualify for optional tests, your device is listed with additional capabilities in the AWS Partner Device Catalog.
Docker dependencies
Validates that the device meets all required technical dependencies to use the AWS-provided Docker application manager (`aws.greengrass.DockerApplicationManager`) component.
Docker Application Manager Qualification
Validates that the device can download a Docker container image from Amazon ECR .
Machine Learning Dependencies
The machine learning optional test group is supported only in IDT v4.9.3.
Validates that the device meets all of the required technical dependencies to use the AWS-provided machine learning (ML) components.
Machine Learning Inference Tests
The machine learning optional test group is supported only in IDT v4.9.3.
Validates that the device can perform ML inference using the [Deep Learning Runtime](https://github.com/neo-ai/neo-ai-dlr) and [TensorFlow Lite](https://www.tensorflow.org/lite/guide/python) ML frameworks.
Stream Manager Dependencies
The stream manager optional test group is supported only in IDT v4.9.3.
Validates that the device can download, install, and run the [AWS IoT Greengrass stream manager](manage-data-streams.md).
Hardware Security Integration (HSI)
This test is available in IDT v4.9.3 and later for Linux-based devices only. AWS IoT Greengrass doesn't currently support hardware security integration for Windows devices.
Validates that the device can authenticate connections to the AWS IoT and AWS IoT Greengrass services using a private key and certificate that are stored in a hardware security module (HSM). This test also verifies that the AWS-provided [PKCS\$111 provider component](pkcs11-provider-component.md) can interface with the HSM using a vendor-provided PKCS\$111 library. For more information, see [Hardware security integration](hardware-security.md).
# Prerequisites for running the AWS IoT Greengrass qualification suite
This section describes the prerequisites for using AWS IoT Device Tester (IDT) for AWS IoT Greengrass.
## Download the latest version of AWS IoT Device Tester for AWS IoT Greengrass
Download the [latest version](idt-programmatic-download.md) of IDT and extract the software into a location (**) on your file system where you have read/write permissions.
**Note**
IDT does not support being run by multiple users from a shared location, such as an NFS directory or a Windows network shared folder. We recommend that you extract the IDT package to a local drive and run the IDT binary on your local workstation.
Windows has a path length limitation of 260 characters. If you are using Windows, extract IDT to a root directory like `C:\ ` or `D:\` to keep your paths under the 260 character limit.
## Download the AWS IoT Greengrass software
IDT for AWS IoT Greengrass V2 tests your device for compatibility with a specific version of AWS IoT Greengrass. Run the following command to download the AWS IoT Greengrass Core software to a file named `aws.greengrass.nucleus.zip`. Replace *version* with a [supported nucleus component version](dev-test-versions.md) for your IDT version.
------
#### [ Linux or Unix ]
```
curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-version.zip > aws.greengrass.nucleus.zip
```
------
#### [ Windows Command Prompt (CMD) ]
```
curl -s https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-version.zip > aws.greengrass.nucleus.zip
```
------
#### [ PowerShell ]
```
iwr -Uri https://d2s8p88vqu9w66.cloudfront.net/releases/greengrass-version.zip -OutFile aws.greengrass.nucleus.zip
```
------
Place the downloaded `aws.greengrass.nucleus.zip` file in the `/products/` folder.
**Note**
Do not place multiple files in this directory for the same operating system and architecture.
## Create and configure an AWS account
Before you can use AWS IoT Device Tester for AWS IoT Greengrass V2, you must perform the following steps:
1. [Set up an AWS account.](#create-aws-account-for-idt) If you already have an AWS account, skip to step 2.
1. [Configure permissions for IDT.](#configure-idt-permissions)
These account permissions allow IDT to access AWS services and create AWS resources, such as AWS IoT things and AWS IoT Greengrass components, on your behalf.
To create these resources, IDT for AWS IoT Greengrass V2 uses the AWS credentials configured in the `config.json` file to make API calls on your behalf. These resources are provisioned at various times during a test.
**Note**
Although most tests qualify for [AWS Free Tier](https://aws.amazon.com/free), you must provide a credit card when you sign up for an AWS account. For more information, see [ Why do I need a payment method if my account is covered by the Free Tier?](https://aws.amazon.com/premiumsupport/knowledge-center/free-tier-payment-method/).
### Step 1: Set up an AWS account
In this step, create and configure an AWS account. If you already have an AWS account, skip to [Step 2: Configure permissions for IDT](#configure-idt-permissions).
If you do not have an AWS account, complete the following steps to create one.
**To sign up for an AWS account**
1. Open [https://portal.aws.amazon.com/billing/signup](https://portal.aws.amazon.com/billing/signup).
1. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call or text message and entering a verification code on the phone keypad.
When you sign up for an AWS account, an *AWS account root user* is created. The root user has access to all AWS services and resources in the account. As a security best practice, assign administrative access to a user, and use only the root user to perform [tasks that require root user access](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html#root-user-tasks).
To create an administrator user, choose one of the following options.
****
| Choose one way to manage your administrator | To | By | You can also |
| --- | --- | --- | --- |
| In IAM Identity Center (Recommended) | Use short-term credentials to access AWS.This aligns with the security best practices. For information about best practices, see [Security best practices in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#bp-users-federation-idp) in the *IAM User Guide*. | Following the instructions in [Getting started](https://docs.aws.amazon.com//singlesignon/latest/userguide/getting-started.html) in the AWS IAM Identity Center User Guide. | Configure programmatic access by [Configuring the AWS CLI to use AWS IAM Identity Center](https://docs.aws.amazon.com//cli/latest/userguide/cli-configure-sso.html) in the AWS Command Line Interface User Guide. |
| In IAM (Not recommended) | Use long-term credentials to access AWS. | Following the instructions in [ Create an IAM user for emergency access](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started-emergency-iam-user.html) in the IAM User Guide. | Configure programmatic access by [Manage access keys for IAM users](https://docs.aws.amazon.com//IAM/latest/UserGuide/id_credentials_access-keys.html) in the IAM User Guide. |
### Step 2: Configure permissions for IDT
In this step, configure the permissions that IDT for AWS IoT Greengrass V2 uses to run tests and collect IDT usage data. You can use the [AWS Management Console](#configure-idt-permissions-console) or [AWS Command Line Interface (AWS CLI)](#configure-idt-permissions-cli) to create an IAM policy and a test user for IDT, and then attach policies to the user. If you already created a test user for IDT, skip to [Configure your device to run IDT tests](device-config-setup.md).
#### To configure permissions for IDT (console)
1. Sign in to the [IAM console](https://console.aws.amazon.com/iam).
1. Create a customer managed policy that grants permissions to create roles with specific permissions.
1. In the navigation pane, choose **Policies**, and then choose **Create policy**.
1. If you are not using PreInstalled, on the **JSON** tab, replace the placeholder content with the following policy. If you are using PreInstalled, proceed to the following step.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"passRoleForResources",
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"arn:aws:iam::*:role/idt-*",
"Condition":{
"StringEquals":{
"iam:PassedToService":[
"iot.amazonaws.com",
"lambda.amazonaws.com",
"greengrass.amazonaws.com"
]
}
}
},
{
"Sid":"lambdaResources",
"Effect":"Allow",
"Action":[
"lambda:CreateFunction",
"lambda:PublishVersion",
"lambda:DeleteFunction",
"lambda:GetFunction"
],
"Resource":[
"arn:aws:lambda:*:*:function:idt-*"
]
},
{
"Sid":"iotResources",
"Effect":"Allow",
"Action":[
"iot:CreateThing",
"iot:DeleteThing",
"iot:DescribeThing",
"iot:CreateThingGroup",
"iot:DeleteThingGroup",
"iot:DescribeThingGroup",
"iot:AddThingToThingGroup",
"iot:RemoveThingFromThingGroup",
"iot:AttachThingPrincipal",
"iot:DetachThingPrincipal",
"iot:UpdateCertificate",
"iot:DeleteCertificate",
"iot:CreatePolicy",
"iot:AttachPolicy",
"iot:DetachPolicy",
"iot:DeletePolicy",
"iot:GetPolicy",
"iot:Publish",
"iot:TagResource",
"iot:ListThingPrincipals",
"iot:ListAttachedPolicies",
"iot:ListTargetsForPolicy",
"iot:ListThingGroupsForThing",
"iot:ListThingsInThingGroup",
"iot:CreateJob",
"iot:DescribeJob",
"iot:DescribeJobExecution",
"iot:CancelJob"
],
"Resource":[
"arn:aws:iot:*:*:thing/idt-*",
"arn:aws:iot:*:*:thinggroup/idt-*",
"arn:aws:iot:*:*:policy/idt-*",
"arn:aws:iot:*:*:cert/*",
"arn:aws:iot:*:*:topic/idt-*",
"arn:aws:iot:*:*:job/*"
]
},
{
"Sid":"s3Resources",
"Effect":"Allow",
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject",
"s3:CreateBucket",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:DeleteBucket",
"s3:PutObjectTagging",
"s3:PutBucketTagging"
],
"Resource":"arn:aws:s3::*:idt-*"
},
{
"Sid":"roleAliasResources",
"Effect":"Allow",
"Action":[
"iot:CreateRoleAlias",
"iot:DescribeRoleAlias",
"iot:DeleteRoleAlias",
"iot:TagResource",
"iam:GetRole"
],
"Resource":[
"arn:aws:iot:*:*:rolealias/idt-*",
"arn:aws:iam::*:role/idt-*"
]
},
{
"Sid":"idtExecuteAndCollectMetrics",
"Effect":"Allow",
"Action":[
"iot-device-tester:SendMetrics",
"iot-device-tester:SupportedVersion",
"iot-device-tester:LatestIdt",
"iot-device-tester:CheckVersion",
"iot-device-tester:DownloadTestSuite"
],
"Resource":"*"
},
{
"Sid":"genericResources",
"Effect":"Allow",
"Action":[
"greengrass:*",
"iot:GetThingShadow",
"iot:UpdateThingShadow",
"iot:ListThings",
"iot:DescribeEndpoint",
"iot:CreateKeysAndCertificate"
],
"Resource":"*"
},
{
"Sid":"iamResourcesUpdate",
"Effect":"Allow",
"Action":[
"iam:CreateRole",
"iam:DeleteRole",
"iam:CreatePolicy",
"iam:DeletePolicy",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:TagRole",
"iam:TagPolicy",
"iam:GetPolicy",
"iam:ListAttachedRolePolicies",
"iam:ListEntitiesForPolicy"
],
"Resource":[
"arn:aws:iam::*:role/idt-*",
"arn:aws:iam::*:policy/idt-*"
]
}
]
}
```
1. If you are using PreInstalled, on the **JSON** tab, replace the placeholder content with the following policy. Make sure you:
+ Replace *thingName* and *thingGroup* in the `iotResources` statement with the thing name and thing group that were created during the Greengrass installation on your device under test (DUT) to add permissions.
+ Replace the *passRole* and *roleAlias* in the `roleAliasResources` statement and the `passRoleForResources` statement with the roles that were created during the Greengrass installation on your DUT.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"passRoleForResources",
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"arn:aws:iam::*:role/passRole",
"Condition":{
"StringEquals":{
"iam:PassedToService":[
"iot.amazonaws.com",
"lambda.amazonaws.com",
"greengrass.amazonaws.com"
]
}
}
},
{
"Sid":"lambdaResources",
"Effect":"Allow",
"Action":[
"lambda:CreateFunction",
"lambda:PublishVersion",
"lambda:DeleteFunction",
"lambda:GetFunction"
],
"Resource":[
"arn:aws:lambda:*:*:function:idt-*"
]
},
{
"Sid":"iotResources",
"Effect":"Allow",
"Action":[
"iot:CreateThing",
"iot:DeleteThing",
"iot:DescribeThing",
"iot:CreateThingGroup",
"iot:DeleteThingGroup",
"iot:DescribeThingGroup",
"iot:AddThingToThingGroup",
"iot:RemoveThingFromThingGroup",
"iot:AttachThingPrincipal",
"iot:DetachThingPrincipal",
"iot:UpdateCertificate",
"iot:DeleteCertificate",
"iot:CreatePolicy",
"iot:AttachPolicy",
"iot:DetachPolicy",
"iot:DeletePolicy",
"iot:GetPolicy",
"iot:Publish",
"iot:TagResource",
"iot:ListThingPrincipals",
"iot:ListAttachedPolicies",
"iot:ListTargetsForPolicy",
"iot:ListThingGroupsForThing",
"iot:ListThingsInThingGroup",
"iot:CreateJob",
"iot:DescribeJob",
"iot:DescribeJobExecution",
"iot:CancelJob"
],
"Resource":[
"arn:aws:iot:*:*:thing/thingName",
"arn:aws:iot:*:*:thinggroup/thingGroup",
"arn:aws:iot:*:*:policy/idt-*",
"arn:aws:iot:*:*:cert/*",
"arn:aws:iot:*:*:topic/idt-*",
"arn:aws:iot:*:*:job/*"
]
},
{
"Sid":"s3Resources",
"Effect":"Allow",
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject",
"s3:CreateBucket",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:DeleteBucket",
"s3:PutObjectTagging",
"s3:PutBucketTagging"
],
"Resource":"arn:aws:s3::*:idt-*"
},
{
"Sid":"roleAliasResources",
"Effect":"Allow",
"Action":[
"iot:CreateRoleAlias",
"iot:DescribeRoleAlias",
"iot:DeleteRoleAlias",
"iot:TagResource",
"iam:GetRole"
],
"Resource":[
"arn:aws:iot:*:*:rolealias/roleAlias",
"arn:aws:iam::*:role/idt-*"
]
},
{
"Sid":"idtExecuteAndCollectMetrics",
"Effect":"Allow",
"Action":[
"iot-device-tester:SendMetrics",
"iot-device-tester:SupportedVersion",
"iot-device-tester:LatestIdt",
"iot-device-tester:CheckVersion",
"iot-device-tester:DownloadTestSuite"
],
"Resource":"*"
},
{
"Sid":"genericResources",
"Effect":"Allow",
"Action":[
"greengrass:*",
"iot:GetThingShadow",
"iot:UpdateThingShadow",
"iot:ListThings",
"iot:DescribeEndpoint",
"iot:CreateKeysAndCertificate"
],
"Resource":"*"
},
{
"Sid":"iamResourcesUpdate",
"Effect":"Allow",
"Action":[
"iam:CreateRole",
"iam:DeleteRole",
"iam:CreatePolicy",
"iam:DeletePolicy",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:TagRole",
"iam:TagPolicy",
"iam:GetPolicy",
"iam:ListAttachedRolePolicies",
"iam:ListEntitiesForPolicy"
],
"Resource":[
"arn:aws:iam::*:role/idt-*",
"arn:aws:iam::*:policy/idt-*"
]
}
]
}
```
**Note**
If you want to use a [custom IAM role as the token exchange role](set-config.md#custom-token-exchange-role-idt) for your device under test, make sure you update the `roleAliasResources` statement and the `passRoleForResources` statement in your policy to allow your custom IAM role resource.
1. Choose **Review policy**.
1. For **Name**, enter **IDTGreengrassIAMPermissions**. Under **Summary**, review the permissions granted by your policy.
1. Choose **Create policy**.
1. Create an IAM user and attach the permissions required by IDT for AWS IoT Greengrass.
1. Create an IAM user. Follow steps 1 through 5 in [Creating IAM users (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) in the *IAM User Guide*.
1. Attach the permissions to your IAM user:
1. On the **Set permissions ** page, choose **Attach existing policies to user directly**.
1. Search for the **IDTGreengrassIAMPermissions** policy that you created in the previous step. Select the check box.
1. Choose **Next: Tags**.
1. Choose **Next: Review** to view a summary of your choices.
1. Choose **Create user**.
1. To view the user's access keys (access key IDs and secret access keys), choose **Show** next to the password and access key. To save the access keys, choose **Download.csv** and save the file to a secure location. You use this information later to configure your AWS credentials file.
1. Next step: Configure your [physical device](device-config-setup.md).
#### To configure permissions for IDT (AWS CLI)
1. On your computer, install and configure the AWS CLI if it's not already installed. Follow the steps in [ Installing the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) in the *AWS Command Line Interface User Guide*.
**Note**
The AWS CLI is an open source tool that you can use to interact with AWS services from your command-line shell.
1. Create a customer managed policy that grants permissions to manage IDT and AWS IoT Greengrass roles.
1. If you are not using PreInstalled, open a text editor and save the following policy contents in a JSON file. If you are using PreInstalled, proceed to the following step.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"passRoleForResources",
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"arn:aws:iam::*:role/idt-*",
"Condition":{
"StringEquals":{
"iam:PassedToService":[
"iot.amazonaws.com",
"lambda.amazonaws.com",
"greengrass.amazonaws.com"
]
}
}
},
{
"Sid":"lambdaResources",
"Effect":"Allow",
"Action":[
"lambda:CreateFunction",
"lambda:PublishVersion",
"lambda:DeleteFunction",
"lambda:GetFunction"
],
"Resource":[
"arn:aws:lambda:*:*:function:idt-*"
]
},
{
"Sid":"iotResources",
"Effect":"Allow",
"Action":[
"iot:CreateThing",
"iot:DeleteThing",
"iot:DescribeThing",
"iot:CreateThingGroup",
"iot:DeleteThingGroup",
"iot:DescribeThingGroup",
"iot:AddThingToThingGroup",
"iot:RemoveThingFromThingGroup",
"iot:AttachThingPrincipal",
"iot:DetachThingPrincipal",
"iot:UpdateCertificate",
"iot:DeleteCertificate",
"iot:CreatePolicy",
"iot:AttachPolicy",
"iot:DetachPolicy",
"iot:DeletePolicy",
"iot:GetPolicy",
"iot:Publish",
"iot:TagResource",
"iot:ListThingPrincipals",
"iot:ListAttachedPolicies",
"iot:ListTargetsForPolicy",
"iot:ListThingGroupsForThing",
"iot:ListThingsInThingGroup",
"iot:CreateJob",
"iot:DescribeJob",
"iot:DescribeJobExecution",
"iot:CancelJob"
],
"Resource":[
"arn:aws:iot:*:*:thing/idt-*",
"arn:aws:iot:*:*:thinggroup/idt-*",
"arn:aws:iot:*:*:policy/idt-*",
"arn:aws:iot:*:*:cert/*",
"arn:aws:iot:*:*:topic/idt-*",
"arn:aws:iot:*:*:job/*"
]
},
{
"Sid":"s3Resources",
"Effect":"Allow",
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject",
"s3:CreateBucket",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:DeleteBucket",
"s3:PutObjectTagging",
"s3:PutBucketTagging"
],
"Resource":"arn:aws:s3::*:idt-*"
},
{
"Sid":"roleAliasResources",
"Effect":"Allow",
"Action":[
"iot:CreateRoleAlias",
"iot:DescribeRoleAlias",
"iot:DeleteRoleAlias",
"iot:TagResource",
"iam:GetRole"
],
"Resource":[
"arn:aws:iot:*:*:rolealias/idt-*",
"arn:aws:iam::*:role/idt-*"
]
},
{
"Sid":"idtExecuteAndCollectMetrics",
"Effect":"Allow",
"Action":[
"iot-device-tester:SendMetrics",
"iot-device-tester:SupportedVersion",
"iot-device-tester:LatestIdt",
"iot-device-tester:CheckVersion",
"iot-device-tester:DownloadTestSuite"
],
"Resource":"*"
},
{
"Sid":"genericResources",
"Effect":"Allow",
"Action":[
"greengrass:*",
"iot:GetThingShadow",
"iot:UpdateThingShadow",
"iot:ListThings",
"iot:DescribeEndpoint",
"iot:CreateKeysAndCertificate"
],
"Resource":"*"
},
{
"Sid":"iamResourcesUpdate",
"Effect":"Allow",
"Action":[
"iam:CreateRole",
"iam:DeleteRole",
"iam:CreatePolicy",
"iam:DeletePolicy",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:TagRole",
"iam:TagPolicy",
"iam:GetPolicy",
"iam:ListAttachedRolePolicies",
"iam:ListEntitiesForPolicy"
],
"Resource":[
"arn:aws:iam::*:role/idt-*",
"arn:aws:iam::*:policy/idt-*"
]
}
]
}
```
1. If you are using PreInstalled, open a text editor and save the following policy contents in a JSON file. Make sure you:
+ Replace *thingName* and *thingGroup* in the `iotResources` statement that were created during the Greengrass installation on your device under test (DUT) to add permissions.
+ Replace the *passRole* and *roleAlias* in the `roleAliasResources` statement and the `passRoleForResources` statement with the roles that were created during the Greengrass installation on your DUT.
```
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"passRoleForResources",
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"arn:aws:iam::*:role/passRole",
"Condition":{
"StringEquals":{
"iam:PassedToService":[
"iot.amazonaws.com",
"lambda.amazonaws.com",
"greengrass.amazonaws.com"
]
}
}
},
{
"Sid":"lambdaResources",
"Effect":"Allow",
"Action":[
"lambda:CreateFunction",
"lambda:PublishVersion",
"lambda:DeleteFunction",
"lambda:GetFunction"
],
"Resource":[
"arn:aws:lambda:*:*:function:idt-*"
]
},
{
"Sid":"iotResources",
"Effect":"Allow",
"Action":[
"iot:CreateThing",
"iot:DeleteThing",
"iot:DescribeThing",
"iot:CreateThingGroup",
"iot:DeleteThingGroup",
"iot:DescribeThingGroup",
"iot:AddThingToThingGroup",
"iot:RemoveThingFromThingGroup",
"iot:AttachThingPrincipal",
"iot:DetachThingPrincipal",
"iot:UpdateCertificate",
"iot:DeleteCertificate",
"iot:CreatePolicy",
"iot:AttachPolicy",
"iot:DetachPolicy",
"iot:DeletePolicy",
"iot:GetPolicy",
"iot:Publish",
"iot:TagResource",
"iot:ListThingPrincipals",
"iot:ListAttachedPolicies",
"iot:ListTargetsForPolicy",
"iot:ListThingGroupsForThing",
"iot:ListThingsInThingGroup",
"iot:CreateJob",
"iot:DescribeJob",
"iot:DescribeJobExecution",
"iot:CancelJob"
],
"Resource":[
"arn:aws:iot:*:*:thing/thingName",
"arn:aws:iot:*:*:thinggroup/thingGroup",
"arn:aws:iot:*:*:policy/idt-*",
"arn:aws:iot:*:*:cert/*",
"arn:aws:iot:*:*:topic/idt-*",
"arn:aws:iot:*:*:job/*"
]
},
{
"Sid":"s3Resources",
"Effect":"Allow",
"Action":[
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObjectVersion",
"s3:DeleteObject",
"s3:CreateBucket",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:DeleteBucket",
"s3:PutObjectTagging",
"s3:PutBucketTagging"
],
"Resource":"arn:aws:s3::*:idt-*"
},
{
"Sid":"roleAliasResources",
"Effect":"Allow",
"Action":[
"iot:CreateRoleAlias",
"iot:DescribeRoleAlias",
"iot:DeleteRoleAlias",
"iot:TagResource",
"iam:GetRole"
],
"Resource":[
"arn:aws:iot:*:*:rolealias/roleAlias",
"arn:aws:iam::*:role/idt-*"
]
},
{
"Sid":"idtExecuteAndCollectMetrics",
"Effect":"Allow",
"Action":[
"iot-device-tester:SendMetrics",
"iot-device-tester:SupportedVersion",
"iot-device-tester:LatestIdt",
"iot-device-tester:CheckVersion",
"iot-device-tester:DownloadTestSuite"
],
"Resource":"*"
},
{
"Sid":"genericResources",
"Effect":"Allow",
"Action":[
"greengrass:*",
"iot:GetThingShadow",
"iot:UpdateThingShadow",
"iot:ListThings",
"iot:DescribeEndpoint",
"iot:CreateKeysAndCertificate"
],
"Resource":"*"
},
{
"Sid":"iamResourcesUpdate",
"Effect":"Allow",
"Action":[
"iam:CreateRole",
"iam:DeleteRole",
"iam:CreatePolicy",
"iam:DeletePolicy",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:TagRole",
"iam:TagPolicy",
"iam:GetPolicy",
"iam:ListAttachedRolePolicies",
"iam:ListEntitiesForPolicy"
],
"Resource":[
"arn:aws:iam::*:role/idt-*",
"arn:aws:iam::*:policy/idt-*"
]
}
]
}
```
**Note**
If you want to use a [custom IAM role as the token exchange role](set-config.md#custom-token-exchange-role-idt) for your device under test, make sure you update the `roleAliasResources` statement and the `passRoleForResources` statement in your policy to allow your custom IAM role resource.
1. Run the following command to create a customer managed policy named `IDTGreengrassIAMPermissions`. Replace `policy.json` with the full path to the JSON file that you created in the previous step.
```
aws iam create-policy --policy-name IDTGreengrassIAMPermissions --policy-document file://policy.json
```
1. Create an IAM user and attach the permissions required by IDT for AWS IoT Greengrass.
1. Create an IAM user. In this example setup, the user is named `IDTGreengrassUser`.
```
aws iam create-user --user-name IDTGreengrassUser
```
1. Attach the `IDTGreengrassIAMPermissions` policy you created in step 2 to your IAM user. Replace ** in the command with the ID of your AWS account.
```
aws iam attach-user-policy --user-name IDTGreengrassUser --policy-arn arn:aws:iam:::policy/IDTGreengrassIAMPermissions
```
1. Create a secret access key for the user.
```
aws iam create-access-key --user-name IDTGreengrassUser
```
Store the output in a secure location. You use this information later to configure your AWS credentials file.
1. Next step: Configure your [physical device](device-config-setup.md).
### AWS IoT Device Tester permissions
The following policies describe AWS IoT Device Tester permissions.
AWS IoT Device Tester requires these permissions for version checking and auto-update features.
+ `iot-device-tester:SupportedVersion`
Grants AWS IoT Device Tester permission to fetch the list of supported products, test suites and IDT versions.
+ `iot-device-tester:LatestIdt`
Grants AWS IoT Device Tester permission to fetch the latest IDT version available for download.
+ `iot-device-tester:CheckVersion`
Grants AWS IoT Device Tester permission to check version compatibility for IDT, test suites and products.
+ `iot-device-tester:DownloadTestSuite`
Grants AWS IoT Device Tester permission to download test suites updates.
AWS IoT Device Tester also uses the following permission for optional metrics reporting:
+ `iot-device-tester:SendMetrics`
Grants permission to AWS to collect metrics about AWS IoT Device Tester internal usage. If this permission is omitted, these metrics will not be collected.
# Configure your device to run IDT tests
To enable IDT to run tests for device qualification, you must configure your host computer to access your device, and configure user permissions on your device.
## Install Java on the host computer
Starting with IDT v4.2.0, the optional qualification tests for AWS IoT Greengrass require Java to run.
You can use Java version 8 or greater. We recommend that you use [Amazon Corretto](https://aws.amazon.com/corretto/) or [OpenJDK](https://openjdk.java.net/) long-term support versions. Version 8 or higher is required..
## Configure your host computer to access your device under test
IDT runs on your host computer and must be able to use SSH to connect to your device. There are two options to allow IDT to gain SSH access to your devices under test:
1. Follow the instructions here to create an SSH key pair and authorize your key to sign in to your device under test without specifying a password.
1. Provide a user name and password for each device in the `device.json` file. For more information, see [Configure device.json](set-config.md#device-config).
You can use any SSL implementation to create an SSH key. The following instructions show you how to use [SSH-KEYGEN](https://www.ssh.com/ssh/keygen/) or [ PuTTYgen](https://www.ssh.com/ssh/putty/windows/puttygen) (for Windows). If you are using another SSL implementation, refer to the documentation for that implementation.
IDT uses SSH keys to authenticate with your device under test.
**To create an SSH key with SSH-KEYGEN**
1. Create an SSH key.
You can use the Open SSH **ssh-keygen** command to create an SSH key pair. If you already have an SSH key pair on your host computer, it is a best practice to create a SSH key pair specifically for IDT. This way, after you have completed testing, your host computer can no longer connect to your device without entering a password. It also allows you to restrict access to the remote device to only those who need it.
**Note**
Windows does not have an installed SSH client. For information about installing an SSH client on Windows, see [Download SSH Client Software](https://www.ssh.com/ssh/#sec-Download-client-software).
The **ssh-keygen** command prompts you for a name and path to store the key pair. By default, the key pair files are named `id_rsa` (private key) and `id_rsa.pub` (public key). On macOS and Linux, the default location of these files is `~/.ssh/`. On Windows, the default location is `C:\Users\\.ssh`.
When prompted, enter a key phrase to protect your SSH key. For more information, see [Generate a New SSH key](https://www.ssh.com/ssh/keygen/).
1. Add authorized SSH keys to your device under test.
IDT must use your SSH private key to sign in to your device under test. To authorize your SSH private key to sign in to your device under test, use the **ssh-copy-id** command from your host computer. This command adds your public key into the `~/.ssh/authorized_keys` file on your device under test. For example:
**\$1 ssh-copy-id **@****
Where *remote-ssh-user* is the user name used to sign in to your device under test and *remote-device-ip* is the IP address of the device under test to run tests against. For example:
**ssh-copy-id pi@192.168.1.5**
When prompted, enter the password for the user name you specified in the **ssh-copy-id** command.
**ssh-copy-id** assumes the public key is named `id_rsa.pub` and is stored the default location (on macOS and Linux, `~/.ssh/` and on Windows, `C:\Users\\.ssh`). If you gave the public key a different name or stored it in a different location, you must specify the fully qualified path to your SSH public key using the **-i** option to **ssh-copy-id** (for example, **ssh-copy-id -i \$1/my/path/myKey.pub**). For more information about creating SSH keys and copying public keys, see [SSH-COPY-ID](https://www.ssh.com/ssh/copy-id).
**To create an SSH key using PuTTYgen (Windows only)**
1. Make sure you have the OpenSSH server and client installed on your device under test. For more information, see [OpenSSH](https://www.openssh.com/).
1. Install [PuTTYgen](https://www.puttygen.com/) on your device under test.
1. Open PuTTYgen.
1. Choose **Generate** and move your mouse cursor inside the box to generate a private key.
1. From the **Conversions** menu, choose **Export OpenSSH key**, and save the private key with a `.pem` file extension.
1. Add the public key to the `/home//.ssh/authorized_keys` file on device under test.
1. Copy the public key text from the PuTTYgen window.
1. Use PuTTY to create a session on your device under test.
1. From a command prompt or Windows Powershell window, run the following command:
**C:/**/putty.exe -ssh **@** **
1. When prompted, enter your device's password.
1. Use vi or another text editor to append the public key to the `/home//.ssh/authorized_keys` file on your device under test.
1. Update your `device.json` file with your user name, the IP address, and path to the private key file that you just saved on your host computer for each device under test. For more information, see [Configure device.json](set-config.md#device-config). Make sure you provide the full path and file name to the private key and use forward slashes ('/'). For example, for the Windows path `C:\DT\privatekey.pem`, use `C:/DT/privatekey.pem` in the `device.json` file.
## Configure user credentials for Windows devices
To qualify a Windows-based device, you must configure user credentials in the LocalSystem account on the device under test for the following users:
+ The default Greengrass user (`ggc_user`).
+ The user that you use to connect to the device under test. You configure this user in the [`device.json` file](set-config.md#device-config).
You must create each user in the LocalSystem account on the device under test, and then store the user name and password for the user in the Credential Manager instance for the LocalSystem account.
**To configure users on Windows devices**
1. Open the Windows Command Prompt (`cmd.exe`) as an administrator.
1. Create the users in the LocalSystem account on the Windows device. Run the following command for each user that you want to create. For the default Greengrass user, replace *user-name* with `ggc_user`. Replace *password* with a secure password.
```
net user /add user-name password
```
1. Download and install the [PsExec utility](https://docs.microsoft.com/en-us/sysinternals/downloads/psexec) from Microsoft on the device.
1. Use the PsExec utility to store the user name and password for the default user in the Credential Manager instance for the LocalSystem account.
Run the following command for each user that you want to configure in Credential Manager. For the default Greengrass user, replace *user-name* with `ggc_user`. Replace *password* with the user's password that you set earlier.
```
psexec -s cmd /c cmdkey /generic:user-name /user:user-name /pass:password
```
If the **PsExec License Agreement** opens, choose **Accept** to agree to the license and run the command.
**Note**
On Windows devices, the LocalSystem account runs the Greengrass nucleus, and you must use the PsExec utility to store user information in the LocalSystem account. Using the Credential Manager application stores this information in the Windows account of the currently logged on user, instead of the LocalSystem account.
## Configure user permissions on your device
IDT performs operations on various directories and files in a device under test. Some of these operations require elevated permissions (using **sudo**). To automate these operations, IDT for AWS IoT Greengrass V2 must be able to run commands with sudo without being prompted for a password.
Follow these steps on the device under test to allow sudo access without being prompted for a password.
**Note**
`username` refers to the SSH user used by IDT to access the device under test.
**To add the user to the sudo group**
1. On the device under test, run `sudo usermod -aG sudo `.
1. Sign out and then sign back in for changes to take effect.
1. To verify your user name was added successfully, run **sudo echo test**. If you are not prompted for a password, your user is configured correctly.
1. Open the `/etc/sudoers` file and add the following line to the end of the file:
` ALL=(ALL) NOPASSWD: ALL`
## Configure a custom token exchange role
You can choose to use a custom IAM role as the token exchange role that the device under test assumes to interact with AWS resources. For information about creating an IAM role, see [Creating IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html) in the *IAM User Guide*.
You must meet the following requirements to allow IDT to use your custom IAM role. We strongly recommend that you add only the minimum required policy actions to this role.
+ The [userdata.json](set-config.md#custom-token-exchange-role-idt) configuration file must be updated to set the `GreengrassV2TokenExchangeRole` parameter to `true`.
+ The custom IAM role must be configured with the following minimum trust policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{
"Service":[
"credentials.iot.amazonaws.com",
"lambda.amazonaws.com",
"sagemaker.amazonaws.com"
]
},
"Action":"sts:AssumeRole"
}
]
}
```
------
+ The custom IAM role must be configured with the following minimum permissions policy:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"iot:DescribeCertificate",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"iot:Connect",
"iot:Publish",
"iot:Subscribe",
"iot:Receive",
"iot:ListThingPrincipals",
"iot:GetThingShadow",
"iot:UpdateThingShadow",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:PutObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Resource":"*"
}
]
}
```
------
+ The name of the custom IAM role must match the IAM role resource that you specify in the IAM permissions for the test user. By default, the [test user policy](dev-tst-prereqs.md#configure-idt-permissions) allows access to IAM roles that have the `idt-` prefix in their role names. If your IAM role name doesn't use this prefix, add the `arn:aws:iam::*:role/custom-iam-role-name` resource to the `roleAliasResources` statement and the `passRoleForResources` statement in your test user policy, as shown in the following examples:
**Example `passRoleForResources` statement**
```
{
"Sid":"passRoleForResources",
"Effect":"Allow",
"Action":"iam:PassRole",
"Resource":"arn:aws:iam::*:role/custom-iam-role-name",
"Condition":{
"StringEquals":{
"iam:PassedToService":[
"iot.amazonaws.com",
"lambda.amazonaws.com",
"greengrass.amazonaws.com"
]
}
}
}
```
**Example `roleAliasResources` statement**
```
{
"Sid":"roleAliasResources",
"Effect":"Allow",
"Action":[
"iot:CreateRoleAlias",
"iot:DescribeRoleAlias",
"iot:DeleteRoleAlias",
"iot:TagResource",
"iam:GetRole"
],
"Resource":[
"arn:aws:iot:*:*:rolealias/idt-*",
"arn:aws:iam::*:role/custom-iam-role-name"
]
}
```
## Configure your device to test optional features
This section describes the device requirements to run IDT tests for optional Docker and machine learning (ML) features. The ML features are supported only in IDT v4.9.3. You must make sure your device meets these requirements only if you want to test these features. Otherwise, continue to [Configure IDT settings to run the AWS IoT Greengrass qualification suite](set-config.md).
**Topics**
+ [Docker qualification requirements](#idt-config-docker-components)
+ [ML qualification requirements](#idt-config-ml-components)
+ [HSM qualification requirements](#idt-config-hsm-components)
### Docker qualification requirements
IDT for AWS IoT Greengrass V2 provides Docker qualification tests to validate that your devices can use the AWS-provided [Docker application manager](docker-application-manager-component.md) component to download Docker container images that you can run using custom Docker container components. For information about creating custom Docker components, see [Run a Docker container](run-docker-container.md).
To run Docker qualification tests, your devices under test must meet the following requirements to deploy the Docker application manager component.
+ [Docker Engine](https://docs.docker.com/engine/) 1.9.1 or later installed on the Greengrass core device. Version 20.10 is the latest version that is verified to work with the AWS IoT Greengrass Core software. You must install Docker directly on the core device before you deploy components that run Docker containers.
+ The Docker daemon started and running on the core device before you deploy this component.
+ The system user that runs a Docker container component must have root or administrator permissions, or you must configure Docker to run it as a non-root or non-admistrator user.
+ On Linux devices, you can add a user to the `docker` group to call `docker` commands without `sudo`.
+ On Windows devices, you can add a user to the `docker-users` group to call `docker` commands without adminstrator privileges.
------
#### [ Linux or Unix ]
To add `ggc_user`, or the non-root user that you use to run Docker container components, to the `docker` group, run the following command.
```
sudo usermod -aG docker ggc_user
```
For more information, see [Manage Docker as a non-root user](https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user).
------
#### [ Windows Command Prompt (CMD) ]
To add `ggc_user`, or the user that you use to run Docker container components, to the `docker-users` group, run the following command as an administrator.
```
net localgroup docker-users ggc_user /add
```
------
#### [ Windows PowerShell ]
To add `ggc_user`, or the user that you use to run Docker container components, to the `docker-users` group, run the following command as an administrator.
```
Add-LocalGroupMember -Group docker-users -Member ggc_user
```
------
### ML qualification requirements
**Note**
The machine learning feature is supported only in IDT v4.9.3.
IDT for AWS IoT Greengrass V2 provides ML qualification tests to validate that your devices can use the AWS-provided [machine learning components](machine-learning-components.md) to perform ML inference locally using the [Deep Learning Runtime](https://github.com/neo-ai/neo-ai-dlr) or [TensorFlow Lite](https://www.tensorflow.org/lite/guide/python) ML frameworks. For more information about running ML inference on Greengrass devices, see [Perform machine learning inference](perform-machine-learning-inference.md).
To run ML qualification tests, your devices under test must meet the following requirements to deploy the machine learning components.
+ On Greengrass core devices running Amazon Linux 2 or Ubuntu 18.04, [GNU C Library](https://www.gnu.org/software/libc/) (glibc) version 2.27 or later installed on the device.
+ On Armv7l devices, such as Raspberry Pi, dependencies for OpenCV-Python installed on the device. Run the following command to install the dependencies.
```
sudo apt-get install libopenjp2-7 libilmbase23 libopenexr-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libgtk-3-0 libwebp-dev
```
+ Raspberry Pi devices that run Raspberry Pi OS Bullseye must meet the following requirements:
+ NumPy 1.22.4 or later installed on the device. Raspberry Pi OS Bullseye includes an earlier version of NumPy, so you can run the following command to upgrade NumPy on the device.
```
pip3 install --upgrade numpy
```
+ The legacy camera stack enabled on the device. Raspberry Pi OS Bullseye includes a new camera stack that is enabled by default and isn't compatible, so you must enable the legacy camera stack.
**To enable the legacy camera stack**
1. Run the following command to open the Raspberry Pi configuration tool.
```
sudo raspi-config
```
1. Select **Interface Options**.
1. Select **Legacy camera** to enable the legacy camera stack.
1. Reboot the Raspberry Pi.
### HSM qualification requirements
AWS IoT Greengrass provides [PKCS\$111 provider component](pkcs11-provider-component.md) to integrate with the PKCS Hardware Security Module (HSM) on the device. The HSM setup depends on your device and the HSM module that you have chosen. As long as the expected HSM configuration, as documented in the [IDT configuration settings](set-config.md), is provided, IDT will have the information required to run this optional feature qualification test.
# Configure IDT settings to run the AWS IoT Greengrass qualification suite
Before you run tests, you must configure settings for AWS credentials and devices on your host computer.
## Configure AWS credentials in config.json
You must configure your IAM user credentials in the `/configs/config.json` file. Use the credentials for the IDT for AWS IoT Greengrass V2 user created in [Create and configure an AWS account](dev-tst-prereqs.md#config-aws-account-for-idt). You can specify your credentials in one of two ways:
+ In a credentials file
+ As environment variables
### Configure AWS credentials with a credentials file
IDT uses the same credentials file as the AWS CLI. For more information, see [Configuration and credential files](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html).
The location of the credentials file varies, depending on the operating system you are using:
+ macOS, Linux: `~/.aws/credentials`
+ Windows: `C:\Users\UserName\.aws\credentials`
Add your AWS credentials to the `credentials` file in the following format:
```
[default]
aws_access_key_id =
aws_secret_access_key =
```
To configure IDT for AWS IoT Greengrass V2 to use AWS credentials from your `credentials` file, edit your `config.json` file as follows:
```
{
"awsRegion": "region",
"auth": {
"method": "file",
"credentials": {
"profile": "default"
}
}
}
```
**Note**
If you do not use the `default` AWS profile, be sure to change the profile name in your `config.json` file. For more information, see [Named profiles](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html).
### Configure AWS credentials with environment variables
Environment variables are variables maintained by the operating system and used by system commands. They are not saved if you close the SSH session. IDT for AWS IoT Greengrass V2 can use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables to store your AWS credentials.
To set these variables on Linux, macOS, or Unix, use **export**:
```
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
```
To set these variables on Windows, use **set**:
```
set AWS_ACCESS_KEY_ID=
set AWS_SECRET_ACCESS_KEY=
```
To configure IDT to use the environment variables, edit the `auth` section in your `config.json` file. Here is an example:
```
{
"awsRegion": "region",
"auth": {
"method": "environment"
}
}
```
## Configure device.json
**Note**
IDT v4.9.3 supports testing the `ml`, `docker`, and `streamManagement` features. IDT v4.9.4 and later versions support testing `docker`. If you don't want to test these features, set the corresponding value to `no`.
In addition to AWS credentials, IDT for AWS IoT Greengrass V2 needs information about the devices that tests are run on. Example information would be IP address, login information, operating system, and CPU architecture.
You must provide this information using the `device.json` template located in ` /configs/device.json`:
------
#### [ IDT v4.9.3 ]
```
[
{
"id": "",
"sku": "",
"features": [
{
"name": "arch",
"value": "x86_64 | armv6l | armv7l | aarch64"
},
{
"name": "ml",
"value": "dlr | tensorflowlite | dlr,tensorflowlite | no"
},
{
"name": "docker",
"value": "yes | no"
},
{
"name": "streamManagement",
"value": "yes | no"
},
{
"name": "hsi",
"value": "hsm | no"
}
],
"devices": [
{
"id": "",
"operatingSystem": "Linux | Windows",
"connectivity": {
"protocol": "ssh",
"ip": "",
"port": 22,
"publicKeyPath": "",
"auth": {
"method": "pki | password",
"credentials": {
"user": "",
"privKeyPath": "/path/to/private/key",
"password": ""
}
}
}
}
]
}
]
```
**Note**
Specify `privKeyPath` only if `method` is set to `pki`.
Specify `password` only if `method` is set to `password`.
All properties that contain values are required, as described here:
`id`
A user-defined alphanumeric ID that uniquely identifies a collection of devices called a *device pool*. Devices that belong to a pool must have identical hardware. When you run a suite of tests, devices in the pool are used to parallelize the workload. Multiple devices are used to run different tests.
`sku`
An alphanumeric value that uniquely identifies the device under test. The SKU is used to track qualified boards.
If you want to list your device in the AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process.
`features`
An array that contains the device's supported features. All features are required.
`arch`
The supported operating system architectures that the test run validates. Valid values are:
+ `x86_64`
+ `armv6l`
+ `armv7l`
+ `aarch64`
`ml`
Validates that the device meets all of the required technical dependencies to use the AWS-provided machine learning (ML) components.
Enabling this feature also validates that the device can perform ML inference using the [Deep Learning Runtime](https://github.com/neo-ai/neo-ai-dlr) and [TensorFlow Lite](https://www.tensorflow.org/lite/guide/python) ML frameworks. .
Valid values are any combination of `dlr` and `tensorflowlite`, or `no`.
`docker`
Validates that the device meets all required technical dependencies to use the AWS-provided Docker application manager (`aws.greengrass.DockerApplicationManager`) component.
Enabling this feature also validates that the device can download a Docker container image from Amazon ECR . .
Valid values are any combination of `yes` or `no`.
`streamManagement`
Validates that the device can download, install, and run the [AWS IoT Greengrass stream manager](manage-data-streams.md).
Valid values are any combination of `yes` or `no`.
`hsi`
Validates that the device can authenticate connections to the AWS IoT and AWS IoT Greengrass services using a private key and certificate that are stored in a hardware security module (HSM). This test also verifies that the AWS-provided [PKCS\$111 provider component](pkcs11-provider-component.md) can interface with the HSM using a vendor-provided PKCS\$111 library. For more information, see [Hardware security integration](hardware-security.md).
Valid values are `hsm` or `no`.
Testing the `hsi` is available only with IDT v4.9.3 and later versions.
`devices.id`
A user-defined unique identifier for the device being tested.
`devices.operatingSystem`
The device operating system. Supported values are `Linux` and `Windows`.
`connectivity.protocol`
The communication protocol used to communicate with this device. Currently, the only supported value is `ssh` for physical devices.
`connectivity.ip`
The IP address of the device being tested.
This property applies only if `connectivity.protocol` is set to `ssh`.
`connectivity.port`
Optional. The port number to use for SSH connections.
The default value is 22.
This property applies only if `connectivity.protocol` is set to `ssh`.
`connectivity.publicKeyPath`
Optional. The full path to the public key used to authenticate connections to the device under test.
When you specify the `publicKeyPath`, IDT validates the device’s public key when it establishes an SSH connection to the device under test. If this value is not specified, IDT creates an SSH connection, but doesn’t validate the device’s public key.
We strongly recommend that you specify the path to the public key, and that you use a secure method to fetch this public key. For standard command line-based SSH clients, the public key is provided in the `known_hosts` file. If you specify a separate public key file, this file must use the same format as the `known_hosts` file, that is, ` ip-address key-type public-key`. If there are multiple entries with the same ip-address, the entry for the key-type used by IDT must be before the other entries in the file.
`connectivity.auth`
Authentication information for the connection.
This property applies only if `connectivity.protocol` is set to `ssh`.
`connectivity.auth.method`
The authentication method used to access a device over the given connectivity protocol.
Supported values are:
+ `pki`
+ `password`
`connectivity.auth.credentials`
The credentials used for authentication.
`connectivity.auth.credentials.password`
The password used for signing in to the device being tested.
This value applies only if `connectivity.auth.method` is set to `password`.
`connectivity.auth.credentials.privKeyPath`
The full path to the private key used to sign in to the device under test.
This value applies only if `connectivity.auth.method` is set to `pki`.
`connectivity.auth.credentials.user`
The user name for signing in to the device being tested.
------
#### [ IDT v4.9.4 ]
```
[
{
"id": "",
"sku": "",
"features": [
{
"name": "arch",
"value": "x86_64 | armv6l | armv7l | aarch64"
},
{
"name": "docker",
"value": "yes | no"
},
{
"name": "hsi",
"value": "hsm | no"
}
],
"devices": [
{
"id": "",
"operatingSystem": "Linux | Windows",
"connectivity": {
"protocol": "ssh",
"ip": "",
"port": 22,
"publicKeyPath": "",
"auth": {
"method": "pki | password",
"credentials": {
"user": "",
"privKeyPath": "/path/to/private/key",
"password": ""
}
}
}
}
]
}
]
```
**Note**
Specify `privKeyPath` only if `method` is set to `pki`.
Specify `password` only if `method` is set to `password`.
All properties that contain values are required, as described here:
`id`
A user-defined alphanumeric ID that uniquely identifies a collection of devices called a *device pool*. Devices that belong to a pool must have identical hardware. When you run a suite of tests, devices in the pool are used to parallelize the workload. Multiple devices are used to run different tests.
`sku`
An alphanumeric value that uniquely identifies the device under test. The SKU is used to track qualified boards.
If you want to list your device in the AWS Partner Device Catalog, the SKU you specify here must match the SKU that you use in the listing process.
`features`
An array that contains the device's supported features. All features are required.
`arch`
The supported operating system architectures that the test run validates. Valid values are:
+ `x86_64`
+ `armv6l`
+ `armv7l`
+ `aarch64`
`docker`
Validates that the device meets all required technical dependencies to use the AWS-provided Docker application manager (`aws.greengrass.DockerApplicationManager`) component.
Enabling this feature also validates that the device can download a Docker container image from Amazon ECR . .
Valid values are any combination of `yes` or `no`.
`hsi`
Validates that the device can authenticate connections to the AWS IoT and AWS IoT Greengrass services using a private key and certificate that are stored in a hardware security module (HSM). This test also verifies that the AWS-provided [PKCS\$111 provider component](pkcs11-provider-component.md) can interface with the HSM using a vendor-provided PKCS\$111 library. For more information, see [Hardware security integration](hardware-security.md).
Valid values are `hsm` or `no`.
Testing the `hsi` is available only with IDT v4.9.3 and later versions.
`devices.id`
A user-defined unique identifier for the device being tested.
`devices.operatingSystem`
The device operating system. Supported values are `Linux` and `Windows`.
`connectivity.protocol`
The communication protocol used to communicate with this device. Currently, the only supported value is `ssh` for physical devices.
`connectivity.ip`
The IP address of the device being tested.
This property applies only if `connectivity.protocol` is set to `ssh`.
`connectivity.port`
Optional. The port number to use for SSH connections.
The default value is 22.
This property applies only if `connectivity.protocol` is set to `ssh`.
`connectivity.publicKeyPath`
Optional. The full path to the public key used to authenticate connections to the device under test.
When you specify the `publicKeyPath`, IDT validates the device’s public key when it establishes an SSH connection to the device under test. If this value is not specified, IDT creates an SSH connection, but doesn’t validate the device’s public key.
We strongly recommend that you specify the path to the public key, and that you use a secure method to fetch this public key. For standard command line-based SSH clients, the public key is provided in the `known_hosts` file. If you specify a separate public key file, this file must use the same format as the `known_hosts` file, that is, ` ip-address key-type public-key`. If there are multiple entries with the same ip-address, the entry for the key-type used by IDT must be before the other entries in the file.
`connectivity.auth`
Authentication information for the connection.
This property applies only if `connectivity.protocol` is set to `ssh`.
`connectivity.auth.method`
The authentication method used to access a device over the given connectivity protocol.
Supported values are:
+ `pki`
+ `password`
`connectivity.auth.credentials`
The credentials used for authentication.
`connectivity.auth.credentials.password`
The password used for signing in to the device being tested.
This value applies only if `connectivity.auth.method` is set to `password`.
`connectivity.auth.credentials.privKeyPath`
The full path to the private key used to sign in to the device under test.
This value applies only if `connectivity.auth.method` is set to `pki`.
`connectivity.auth.credentials.user`
The user name for signing in to the device being tested.
------
## Configure userdata.json
IDT for AWS IoT Greengrass V2 also needs additional information about the location of test artifacts and AWS IoT Greengrass software.
You must provide this information using the `userdata.json` template located in ` /configs/userdata.json`:
```
{
"TempResourcesDirOnDevice": "/path/to/temp/folder",
"InstallationDirRootOnDevice": "/path/to/installation/folder",
"GreengrassNucleusZip": "/path/to/aws.greengrass.nucleus.zip",
"PreInstalled": "yes/no",
"GreengrassV2TokenExchangeRole": "custom-iam-role-name",
"hsm": {
"greengrassPkcsPluginJar": "/path/to/aws.greengrass.crypto.Pkcs11Provider-latest.jar",
"pkcs11ProviderLibrary": "/path/to/pkcs11-vendor-library",
"slotId": "slot-id",
"slotLabel": "slot-label",
"slotUserPin": "slot-pin",
"keyLabel": "key-label",
"preloadedCertificateArn": "certificate-arn"
"rootCA": "path/to/root-ca"
}
}
```
All properties that contain values are required as described here:
`TempResourcesDirOnDevice`
The full path to a temporary folder on the device under test in which to store test artifacts. Make sure that sudo permissions are not required to write to this directory.
IDT deletes the contents of this folder when it finishes running a test.
`InstallationDirRootOnDevice`
The full path to a folder on the device in which to install AWS IoT Greengrass. For PreInstalled Greengrass, this is the path to the Greengrass installation directory.
You must set the required file permissions for this folder. Run the following command for each folder in the installation path.
```
sudo chmod 755 folder-name
```
`GreengrassNucleusZip`
The full path to the Greengrass nucleus ZIP (`greengrass-nucleus-latest.zip`) file on your host computer. This field is not required for testing with PreInstalled Greengrass.
For information about the supported versions of the Greengrass nucleus for IDT for AWS IoT Greengrass, see [Latest IDT version for AWS IoT Greengrass V2](dev-test-versions.md#idt-latest-version). To download the latest Greengrass software, see [Download the AWS IoT Greengrass software](https://docs.aws.amazon.com/greengrass/v2/developerguide/dev-tst-prereqs.html#config-gg).
`PreInstalled`
This feature is available for IDT v4.5.8 and later versions on Linux devices only.
(Optional) When the value is *yes*, IDT will assume the path mentioned in `InstallationDirRootOnDevice` to be the directory where Greengrass is installed.
For more information about how to install Greengrass on your device, see [Install AWS IoT Greengrass Core software with automatic resource provisioning](quick-installation.md). If [installing with manual provisioning](https://docs.aws.amazon.com/greengrass/v2/developerguide/manual-installation.html), include the “Add the AWS IoT thing to a new or existing thing group” step when creating an [AWS IoT thing](https://docs.aws.amazon.com/greengrass/v2/developerguide/manual-installation.html#create-iot-thing) manually. IDT assumes that the thing and thing group are created during installation setup. Make sure that these values are reflected in the `effectiveConfig.yaml` file. IDT checks for the file `effectiveConfig.yaml` under `/config/effectiveConfig.yaml`.
For running tests with HSM, make sure that the `aws.greengrass.crypto.Pkcs11Provider` field is updated in `effectiveConfig.yaml`.
`GreengrassV2TokenExchangeRole`
(Optional) The custom IAM role that you want to use as the token exchange role that the device under test assumes to interact with AWS resources.
IDT uses this custom IAM role instead of creating the default token exchange role during the test run. If you use a custom role, you can update the [IAM permissions for the test user](dev-tst-prereqs.md#configure-idt-permissions) to exclude the `iamResourcesUpdate` statement that allows the user to create and delete IAM roles and policies.
For more information about creating a custom IAM role as your token exchange role, see [Configure a custom token exchange role](device-config-setup.md#configure-custom-tes-role-for-idt).
`hsm`
This feature is available for IDT v4.5.1 and later.
(Optional) The configuration information for testing with an AWS IoT Greengrass Hardware Security Module (HSM). Otherwise, the `hsm` property should be omitted. For more information, see [Hardware security integration](hardware-security.md).
This property applies only if `connectivity.protocol` is set to `ssh`.
The HSM configuration may be considered sensitive data if the hardware security module is shared between IDT and another system. In this situation, you may avoid securing these configuration values in plaintext by storing them in an AWS Parameter Store SecureString parameter and configuring IDT to fetch them during test execution. For more information, see [Fetch configuration from AWS Parameter Store](#fetch-config)
`hsm.greengrassPkcsPluginJar`
The full path to the [PKCS\$111 provider component](pkcs11-provider-component.md) that you download to the IDT host machine. AWS IoT Greengrass provides this component as JAR file that you can download to specify as a provisioning plugin during installation. You can download the latest version of the component's JAR file as the following URL: [https://d2s8p88vqu9w66.cloudfront.net/releases/Pkcs11Provider/aws.greengrass.crypto.Pkcs11Provider-latest.jar](https://d2s8p88vqu9w66.cloudfront.net/releases/Pkcs11Provider/aws.greengrass.crypto.Pkcs11Provider-latest.jar).
`hsm.pkcs11ProviderLibrary`
The full path to the PKCS\$111 library that is provided by the hardware security module (HSM) vendor to interact with the HSM.
`hsm.slotId`
The slot ID that is used to identify the HSM slot to which you load the key and certificate.
`hsm.slotLabel`
The slot label that is used to identify the HSM slot to which you load the key and certificate.
`hsm.slotUserPin`
The user PIN that IDT uses to authenticate AWS IoT Greengrass Core software to the HSM.
As a security best practice, don't use the same user PIN on production devices.
`hsm.keyLabel`
The label used to identify the key in the hardware module. Both the key and the certificate must use the same key label.
`hsm.preloadedCertificateArn`
The Amazon Resource Name (ARN) of the uploaded device certificate in the AWS IoT cloud.
You must have previously generated this certificate using the key in the HSM, imported it into your HSM, and uploaded it to the AWS IoT cloud. For information about generating and importing the certificate, see the documentation for your HSM.
You must upload the certificate to the same account and Region that you provide in [config.json.](#cfg-aws-gg). For more information about uploading your certificate to AWS IoT, see [Register a client certificate manually](https://docs.aws.amazon.com/iot/latest/developerguide/manual-cert-registration.html) in the *AWS IoT Developer Guide*.
`hsm.rootCAPath`
(Optional) The full path on the IDT host machine to the root certificate authority (CA) that signed your certificate. This is required if the certificate in your HSM created is not signed by the Amazon root CA.
## Fetch configuration from AWS Parameter Store
AWS IoT Device Tester (IDT) includes an optional feature to fetch configuration values from the [AWS Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html). AWS Parameter Store allows for secure and encrypted storage of configurations. When configured, IDT can fetch parameters from AWS Parameter Store in place of storing parameters in plaintext inside the `userdata.json` file. This is useful for any sensitive data that should be stored encrypted, such as: passwords, pins, and other secrets.
1. To use this feature, you must update the permissions used in creating your [IDT user](https://docs.aws.amazon.com/greengrass/v2/developerguide/dev-tst-prereqs.html) to allow the GetParameter action on the parameters that IDT is configured to use. The below is an example of a permission statement that can be added to the IDT user. For more information, see [AWS Systems Manager userguide](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-access.html).
```
{
"Sid":"parameterStoreResources",
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:*:*:parameter/IDT*"
}
```
The above permission is configured to allow fetching all parameters with a name beginning with `IDT`, by using the wildcard character `*`. You should customize this to your needs so IDT has access to fetch any configured parameters based on the naming of the parameters you are using.
1. You need to store your configuration values inside AWS Paramater Store. This can be done from the AWS console or from the AWS CLI. AWS Parameter Store allows you to choose encrypted or unencrypted storage. For storage of sensitive values like secrets, passwords, and pins, you should use the encrypted option which is a parameter type of SecureString. To upload a parameter using the AWS CLI, you can use the following command:
```
aws ssm put-parameter --name IDT-example-name --value IDT-example-value --type SecureString
```
You can verify that a parameter is stored using the following command. (Optional) Use the `--with-decryption` flag to fetch a decrypted SecureString parameter.
```
aws ssm get-parameter --name IDT-example-name
```
Using the AWS CLI will upload the parameter in the AWS region of the current CLI user and IDT will fetch parameters from the region configured in `config.json`. To check your region from the AWS CLI, use the following:
```
aws configure get region
```
1. Once you have a configuration value in the AWS Cloud, you can update any value inside the IDT configuration to fetch from the AWS Cloud. To do so, you use a placeholder in your IDT configuration of the form `{{AWS.Parameter.parameter_name}}` to fetch the parameter by that name from the AWS Parameter Store.
For example, suppose you want to use the `IDT-example-name` parameter from Step 2 as the HSM keyLabel in your HSM configuration. To do this, you can update your `userdata.json` as follows:
```
"hsm": {
"keyLabel": "{{AWS.Parameter.IDT-example-name}}",
[...]
}
```
IDT will fetch the value of this parameter at runtime that was set to `IDT-example-value` in Step 2. This configuration is similar to setting `"keyLabel": "IDT-example-value"` but, instead, that value is store as encrypted in the AWS Cloud.
# Run the AWS IoT Greengrass qualification suite
After you [set the required configuration](set-config.md), you can start the tests. The runtime of the full test suite depends on your hardware. For reference, it takes approximately 30 minutes to complete the full test suite on a Raspberry Pi 3B.
Use the following `run-suite` command to run a suite of tests.
```
devicetester_[linux | mac | win]_x86-64 run-suite \\
--suite-id suite-id \\
--group-id group-id \\
--pool-id your-device-pool \\
--test-id test-id \\
--update-idt y|n \\
--userdata userdata.json
```
All options are optional. For example, you can omit `pool-id` if you have only one device pool, which is a set of identical devices, defined in your `device.json` file. Or, you can omit `suite-id` if you want to run the latest test suite version in the `tests` folder.
**Note**
IDT prompts you if a newer test suite version is available online. For more information, see [Test suite versions](idt-greengrass-qualification.md#idt-test-suite-versions).
## Example commands to run the qualification suite
The following command line examples show you how to run the qualification tests for a device pool. For more information about `run-suite` and other IDT commands, see [IDT for AWS IoT Greengrass V2 commands](#bk-cli).
Use the following command to run all test groups in a specified test suite. The `list-suites` command lists the test suites that are in the `tests` folder.
```
devicetester_[linux | mac | win]_x86-64 run-suite \
--suite-id GGV2Q_1.0.0 \
--pool-id \
--userdata userdata.json
```
Use the following command to run a specific test group in a test suite. The `list-groups` command lists the test groups in a test suite.
```
devicetester_[linux | mac | win]_x86-64 run-suite \
--suite-id GGV2Q_1.0.0 \
--group-id \
--pool-id \
--userdata userdata.json
```
Use the following command to run a specific test case in a test group.
```
devicetester_[linux | mac | win]_x86-64 run-suite \
--group-id \
--test-id \
--userdata userdata.json
```
Use the following command to run multiple test cases in a test group.
```
devicetester_[linux | mac | win]_x86-64 run-suite \
--group-id \
--test-id ,
--userdata userdata.json
```
Use the following command to list all of the test cases in a test group.
```
devicetester_[linux | mac | win]_x86-64 list-test-cases --group-id
```
We recommend that you run the full qualification test suite, which runs test group dependencies in the correct order. If you choose to run specific test groups, we recommend that you first run the dependency checker test group to make sure all Greengrass dependencies are installed before you run related test groups. For example:
+ Run `coredependencies` before running core qualification test groups.
## IDT for AWS IoT Greengrass V2 commands
The IDT commands are located in the `/bin` directory. To run a test suite, you provide the command in the following format:
`help`
Lists information about the specified command.
`list-groups`
Lists the groups in a given test suite.
`list-suites`
Lists the available test suites.
`list-supported-products`
Lists the supported products, in this case AWS IoT Greengrass versions, and test suite versions for the current IDT version.
`list-test-cases`
Lists the test cases in a given test group. The following option is supported:
+ `group-id`. The test group to search for. This option is required and must specify a single group.
`run-suite`
Runs a suite of tests on a pool of devices. The following are some supported options:
+ `suite-id`. The test suite version to run. If not specified, IDT uses the latest version in the `tests` folder.
+ `group-id`. The test groups to run, as a comma-separated list. If not specified, IDT runs all appropriate test groups in the test suite depending on the configured settings in `device.json`. IDT doesn't run any test groups that the device doesn't support based on your configured settings, even if those test groups are specified in the `group-id` list.
+ `test-id`. The test cases to run, as a comma-separated list. When specified, `group-id` must specify a single group.
+ `pool-id`. The device pool to test. You must specify a pool if you have multiple device pools defined in your `device.json` file.
+ `stop-on-first-failure`. Configures IDT to stop running on the first failure. Use this option with `group-id` when you want to debug the specified test groups. Do not use this option when running a full test-suite to generate a qualification report.
+ `update-idt`. Sets the response for the prompt to update IDT. The `Y` response stops the test execution if IDT detects there is a newer version. The `N` response continues the test execution.
+ `userdata`. The full path to the `userdata.json` file that contains information about test artifact paths. This option is required for the `run-suite` command. The `userdata.json` file must be located in the *devicetester\$1extract\$1location*/devicetester\$1ggv2\$1*[win\$1mac\$1linux]*/configs/ directory.
For more information about `run-suite` options, use the `help` option:
```
devicetester_[linux | mac | win]_x86-64 run-suite -h
```
# Understanding results and logs
This section describes how to view and interpret IDT result reports and logs.
To troubleshoot errors, see [Troubleshooting IDT for AWS IoT Greengrass V2](idt-troubleshooting.md).
## Viewing results
While running, IDT writes errors to the console, log files, and test reports. After IDT completes the qualification test suite, it generates two test reports. These reports are located in `/results//`. Both reports capture the results from running the qualification test suite.
The `awsiotdevicetester_report.xml` is the qualification test report that you submit to AWS to list your device in the AWS Partner Device Catalog. The report contains the following elements:
+ The IDT version.
+ The AWS IoT Greengrass version that was tested.
+ The SKU and the device pool name specified in the `device.json` file.
+ The features of the device pool specified in the `device.json` file.
+ The aggregate summary of test results.
+ A breakdown of test results by libraries that were tested based on the device features, such as local resource access, shadow, and MQTT.
The `GGV2Q_Result.xml` report is in [JUnit XML format](https://llg.cubic.org/docs/junit/). You can integrate it into continuous integration and deployment platforms like [Jenkins](https://jenkins.io/), [Bamboo](https://www.atlassian.com/software/bamboo), and so on. The report contains the following elements:
+ Aggregate summary of test results.
+ Breakdown of test results by the AWS IoT Greengrass functionality that was tested.
## Interpreting AWS IoT Device Tester results
The report section in `awsiotdevicetester_report.xml` or `awsiotdevicetester_report.xml` lists the tests that were run and the results.
The first XML tag `` contains the summary of the test run. For example:
```
```Attributes used in the `` tag
`name`
The name of the test suite.
`time`
The time, in seconds, that it took to run the qualification suite.
`tests`
The number of tests that were run.
`failures`
The number of tests that were run, but did not pass.
`errors`
The number of tests that IDT couldn't run.
`disabled`
Ignore this attribute. It is not used.
The `awsiotdevicetester_report.xml` file contains an `` tag that contains information about the product being tested and the product features that were validated after running a suite of tests.Attributes used in the `` tag
`name`
The name of the product being tested.
`version`
The version of the product being tested.
`features`
The features validated. Features marked as `required` are required to submit your board for qualification. The following snippet shows how this information appears in the `awsiotdevicetester_report.xml` file.
```
```
If there are no test failures or errors for the required features, your device meets the technical requirements to run AWS IoT Greengrass and can interoperate with AWS IoT services. If you want to list your device in the AWS Partner Device Catalog, you can use this report as qualification evidence.
In the event of test failures or errors, you can identify the test that failed by reviewing the `` XML tags. The `` XML tags inside the `` tag show the test result summary for a test group. For example:
```
```
The format is similar to the `` tag, but with a `skipped` attribute that is not used and can be ignored. Inside each `` XML tag, there are `` tags for each test that was run for a test group. For example:
```
>
```Attributes used in the `` tag
`name`
The name of the test.
`attempts`
The number of times IDT ran the test case.
When a test fails or an error occurs, `` or `` tags are added to the `` tag with information for troubleshooting. For example:
```
Reason for the test failure
Reason for the test execution error
```
## Viewing logs
IDT generates logs from test runs in `/results//logs`. Two sets of logs are generated:
`test_manager.log`
Logs generated from the Test Manager component of AWS IoT Device Tester (for example, logs related to configuration, test sequencing, and report generation).
`.log (for example, lambdaDeploymentTest.log)`
Logs of the test case within the test group, including logs from the device under test. Starting with IDT v4.2.0, IDT groups the test logs for each test case in a separate ** folder within the `/results//logs//` directory.