# Generate a test set for Test Workbench You can create a test set to evaluate the performance of your bot. Generate a test set by uploading a test set that is in a CSV file format or by generating a test set from [ conversation logs](https://docs.aws.amazon.com/lexv2/latest/dg/conversation-logs.html). The test set can contain audio or text input. ![\[Create a test set with the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-create.png) If a test set creates validation errors, remove the test set and replace it with another list of test set data, or edit the data in the CSV file by using a spreadsheet editing program. **To create a test set:** 1. Sign in to the AWS Management Console and open the Amazon Lex console at [https://console.aws.amazon.com/lex/](https://console.aws.amazon.com/lex/). 1. Choose **Test workbench** from the left side panel. 1. Select **Test sets** from the options under Test workbench. 1. Select the **Create test set** button on the console. 1. In the **Details**, enter a test set name and an optional description. 1. Select **Generate a baseline test set**. 1. Select **Generate from conversation logs**. 1. Select **Bot name**, **Bot Alias**, and **Language** from the drop down menus. 1. If you are generating a baseline test from a conversation log, choose **Time range** and **IAM role**, if required. You can create a role with the basic Amazon Lex V2 permissions or use an existing role. 1. Choose a modality of **Audio** or **Text** for the test set you are creating. NOTE: The Test Workbench can import text files up to 50k, and up to 5 hours of audio. 1. Select an Amazon S3 location to store your test results, and add an optional KMS key to encrypt output transcripts. 1. Select **Create**. **To upload an existing test set in a CSV file format, or to update the test set:** 1. Choose **Test workbench** from the left side panel. 1. Select **Test sets** from the options under Test workbench. 1. Select **Upload a file to this test set** on the console. 1. Choose **Upload from Amazon Amazon S3 bucket** or **Upload from your computer**. NOTE: You can upload a CSV file created from a template. Click **CSV template** to download a zip file that contains the templates. 1. Choose **Create a role with basic Amazon Lex permissions** or **Use an existing role for Role ARN**. 1. Choose a modality of **Audio** or **Text** for the test set you are creating. NOTE: The Test Workbench can import text files up to 50k, and up to 5 hours of audio. 1. Select an Amazon S3 location to store your test results, and add an optional KMS key to encrypt output transcripts. 1. Select **Create**. If the operation is successful, the confirmation message will indicate that the test set is ready to test, and the status will display **Ready for testing**. # Tips for creating a successful test set + You can create an IAM role for the Test Workbench in the console, or you can configure your IAM role step-by-step. For more information, see [ Create an IAM role for the Test Workbench](https://docs.aws.amazon.com/lexv2/latest/dg/create-iam-test-set.html). + Before executing a test, validate the test set and the bot definition for any inconsistencies using the **Validate discrepancy** button. If the intent and slot naming conventions used in the test set are consistent with the bot, proceed to execute the test. If any anomalies are identified, revise the test set, update the test set, and choose **Validate discrepancy**. Repeat this sequence again until no inconsistencies are noted, then execute the test. + The Test Workbench can test with different slot value formats in the **Expected Output Slot** column. For any built-in slot, you can choose the value provided in the user input (for example, Date = tomorrow), or provide its absolute resolved value (for example, Date = 2023-03-21). For more information around built-in slots and their absolute values, see [ Built-in slots](https://docs.aws.amazon.com/lexv2/latest/dg/howitworks-builtins-slots.html). + For consistency and readability in the **Expected Output Slot** columns, follow the convention of "SlotName = SlotValue" (for example, AppointmentType = cleaning) with a space before and after the equal sign. + If the bot includes composite slots, in **Expected Output Slot** define subslots to the slot name, separated by a period (for example, “Car.Color”). No other syntax and punctuation will work. + If the bot includes multi-value slots, in **Expected Output Slot** provide multiple slot values, separated by a comma ("FlowerType = roses, lilies"). No other syntax and punctuation will work. + Make sure that the test set is created from valid conversation logs. + Slot:slot value will be in the same column after the intent columns in the CSV format. + DTMF input from a User turn is interpreted as an expected transcription and does not list an Amazon S3 location. # Creating a test case within a test set using Test Workbench The Test Workbench results are dependent on the bot definition and its corresponding test set. You can generate a test set with the information from the bot definition to pinpoint areas that need improvement. Create a test dataset with examples that you suspect (or know) will be challenging for the bot to interpret correctly considering the current bot design and your knowledge of your customer conversations. Review your intents based on learnings from your production bot on a regular basis. Continue to add to and adjust the bot’s sample utterances and slot values. Consider improving slot resolution by using the available options, such as runtime hints. The design and development of your bot is an iterative process that is a continuous cycle. Here are some other tips for optimizing your test set: + Select the most common use cases with frequently used intents and slots in the test set. + Explore different ways a customer could refer to your intents and slots. This can include user inputs in the forms of statements, questions, and commands that vary in length from minimal to extended. + Include user inputs with a varied number of slots. + Include commonly used synonyms or abbreviations of custom slot values supported by your bot (for example, “root canal”, “canal”, or “RC”). + Include variations of built-in slot values (for example, “tomorrow”, “asap”, or "the next day"). + Examine the bot robustness for spoken modality by collecting user inputs that can be misinterpreted (for example, “ink”, “ankle”, or "anchor"). # Creating a test set from a CSV file for Test Workbench You can create a test set from the CSV file template provided in the Amazon Lex V2 console by entering the values directly by using a CSV spreadsheet editor. The test set is a comma-separated value (CSV) file consisting of single user utterances and multi-turn conversations recorded in the following columns: + **Line \$1** – this column is an incremental counter that keeps track of the total filled rows to test. + **Conversation \$1** – this column tracks the number of turns in a conversations. For single inputs, this column can be left empty, filled with "-" or "N/A". For conversations, each turn within a conversations will be assigned the same conversation number. + **Source** – this column is set to "User" or "Agent". For single inputs, it will be always set to "User". + **Input** – this column includes the user utterance or the bot prompts. + **Expected Output Intent** – this column captures the intent fulfilled in the input. + **Intent Expected Output Slot 1** – this column captures the first slot elicited in the user input. The test set should include a column called Expected Output Slot X for each slot in the user input. Example of a test set with single inputs: | Line \$1 | Conversation \$1 | Source | Input | Expected Output Intent | Expected Output Slot 1 | Expected Output Slot 2 | | --- | --- | --- | --- | --- | --- | --- | | 1 | | User | book a cleaning appointment tomorrow | MakeAppointment | AppointmentType = cleaning | Date = tomorrow | | 2 | N/A | User | book a cleaning appointment on April 15th | MakeAppointment | AppointmentType = cleaning | Date = 4/15/23 | | 3 | N/A | User | book appointment for December first | MakeAppointment | Date = December first | | | 4 | N/A | User | book a cleaning appointment | MakeAppointment | AppointmentType = cleaning | | | 1 | | User | Can you help me book an appointment? | MakeAppointment | | | Example of a test set with conversations | Line \$1 | Conversation \$1 | Source | Input | Expected Output Intent | Expected Output Slot 1 | Expected Output Slot 2 | Expected Output Slot 3 | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 1 | User | book an appointment | MakeAppointment | | | | | 2 | 1 | Agent | What type of appointment would you like to schedule? | MakeAppointment | | | | | 3 | 1 | User | cleaning | MakeAppointment | AppointmentType = cleaning | | | | 4 | 1 | Agent | When should I schedule your appointment? | MakeAppointment | | | | | 5 | 1 | User | tomorrow | MakeAppointment | | Date = tomorrow | | | 6 | 2 | User | book a root canal appointment today | MakeAppointment | AppointmentType = root canal | Date = today | | | 7 | 2 | Agent | At what time should I schedule your appointment? | MakeAppointment | | | | | 8 | 2 | User | eleven a.m. | MakeAppointment | | | Time = eleven a.m. | # Create an IAM role for the Test Workbench **To create an IAM role for the Test Workbench** 1. Follow the steps at [ Create an IAM user](https://docs.aws.amazon.com/lexv2/latest/dg/gs-account.html#gs-account-user) to create an IAM user which can be used to access test-workbench console. 1. Select the **Create role** button. ![\[The roles screen in the IAM console.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-iam1.png) 1. Select the option for **Custom trust policy**. ![\[Select trusted entity\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-iam2.png) 1. Enter the trust policy below and click **Next**. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "sid4", "Effect": "Allow", "Principal": { "Service": "lexv2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` ------ 1. Select the **Create policy** button. 1. A new tab will open in your browser where you can enter the below policy and click on **Next: Tags** button. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "logs:FilterLogEvents" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "lex:*" ], "Resource": "*" } ] } ``` ------ 1. Enter a policy name, for example ‘LexTestWorkbenchPolicy’ and then click on the **Create Policy**. 1. Return to the previous tab in your browser and Refresh list of policies by clicking the **Refresh** button as shown below. ![\[Refresh the screen to see the new policy.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-iam3.png) 1. Search in list of policies by entering policy name that you used in the 6th step and choose the policy. 1. Select the **Next** button. 1. Enter role name and then click the **Create Role** button. 1. Choose your new IAM role when prompted in the Amazon Lex V2 console for Test Workbench. # Create an IAM role for the Test Workbench - Advanced Features **Permission setup for Test workbench IAM role** This section shows several example AWS Identity and Access Management (IAM) identity-based policies to implement least-privilege access controls for Test Workbench permissions. 1. **Policy for Test Workbench to read audio files in S3** – This policy enables Test Workbench to read audio files being used in the test sets. The below policy should be accordingly modified to update *S3BucketName* and *S3Path* to point them to an Amazon S3 location of the audio files in a test set. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "TestWorkbenchS3AudioFilesReadOnly", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::S3BucketName/S3Path/*" ] } ] } ``` ------ 1. **Policy for Test Workbench to read and write test sets and results into an Amazon S3 bucket** – This policy enables Test Workbench to store the test set inputs and results. The below policy should be modified to update *S3BucketName* to the Amazon S3 Bucket where test-set data will be stored. Test Workbench stores these data exclusively in your Amazon S3 bucket and not in the Lex Service infrastructure. Therefore For this reason, Test Workbench requires access to your Amazon S3 bucket to function properly. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "TestSetDataUploadWithEncryptionOnly", "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set/*", "arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_execution/*", "arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set_discrepancy_report/*" ], "Condition": { "StringEquals": { "s3:x-amz-server-side-encryption": "aws:kms" } } }, { "Sid": "TestSetDataGetObject", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": [ "arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set/*", "arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_execution/*", "arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set_discrepancy_report/*" ] }, { "Sid": "TestSetListS3Objects", "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::S3BucketName" ] } ] } ``` ------ 1. **Policy for Test Workbench to read CloudWatch Logs** – This policy enables Test Workbench to generate test-sets from Lex Conversation Text Logs stored in Amazon CloudWatch Logs. The below policy should be modified to update *Region*, *AwsAccountId*, *LogGroupName*. 1. **Policy for Test Workbench to call Lex Runtime** – This policy enables Test Workbench to execute a test set against Lex bots. The below policy should be modified to update *Region*, *AwsAccountId*, *BotId*. Since Test Workbench can test any bot in your Lex environment, you can replace the resource with "arn:aws:lex:*Region*:*AwsAccountId*:bot-alias/\$1" to allow Test Workbench access to all Amazon Lex V2 bots in an account. 1. **(Optional) Policy for Test Workbench to encrypt and decrypt test set data** – If Test Workbench is configured to store test-set inputs and results in Amazon S3 buckets using a customer managed KMS key, Test Workbench will need both encryption and decryption permission to the KMS key. The below policy should be modified to update *Region*, *AwsAccountId*, and *KmsKeyId* where *KmsKeyId* is the ID of the customer managed KMS key. 1. **(Optional) Policy for Test Workbench to decrypt audio files** – If Audio files are stored in the S3 bucket using customer managed KMS key, Test Workbench will need decryption permission to the KMS keys. The below policy should be modified to update *Region*, *AwsAccountId*, and *KmsKeyId* where *KmsKeyId* is the ID of the customer managed KMS key used to encrypt the audio files.