# Monitoring bot performance in Lex V2
Monitoring is important for maintaining the reliability, availability, and performance of your Amazon Lex V2 chatbots. This topic describes using conversation logs to monitor conversations between your users and your chatbots, using utterance statistics to determine the utterances that your bots detect and miss, and how to use Amazon CloudWatch Logs and AWS CloudTrail to monitor Amazon Lex V2. It also describes the Amazon Lex V2 runtime and channel association metrics.
Use these tools and metrics to understand what directions and actions you can take to improve your bots' performance.
**Topics**
+ [Measuring business performance with Analytics](analytics.md)
+ [Enabling conversation logs for your Lex V2 bots](enabling-logs.md)
+ [Logging errors with error logs in Lex V2](error-logs.md)
+ [Monitoring operational metrics in Lex V2](monitoring-operational-metrics.md)
+ [Evaluating Lex V2 bot performance with the Test Workbench](test-workbench.md)
# Measuring business performance with Analytics
With Analytics, you can evaluate the performance of your bot with metrics that are related to success and failure rates of your bots’ interactions with customers. You can also visualize patterns of conversation flows between your bot and customers. Analytics streamlines your experience by summarizing these metrics in graphs and charts. Analytics provides tools to help you filter results to identify issues and problems involving intents, slots, utterances, and conversations. You can use this data to iterate and improve upon your bot to create a better customer experience.
**Note**
For a user to access Analytics, either the [AWS managed policy: AmazonLexFullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonLexFullAccess) or a custom policy that includes analytics API permissions must be attached to their IAM role. See [Managing access permissions for analytics](analytics-permissions.md) for details on how to handle user permissions with a custom policy. If the [AWS managed policy: AmazonLexReadOnly](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonLexReadOnly) is attached to a customer's IAM role, an error displays the missing permissions that you need to add to the user's IAM role for them to be able to access the Analytics dashboards.
**To access Analytics**
1. Sign in to the AWS Management Console and open the Amazon Lex V2 console at [https://console.aws.amazon.com/lexv2/home](https://console.aws.amazon.com/lexv2/home).
1. In the navigation pane under **Bots**, select the bot you want to view in analytics.
1. Select the section under **Analytics** that you want to view.
**Topics**
+ [Key definitions](analytics-key-definitions.md)
+ [Filtering results](analytics-filter.md)
+ [Overview: a summary of your bot performance](analytics-overview.md)
+ [Conversation dashboard: a summary of your bot conversations](conversation-dashboard.md)
+ [Performance dashboard: a summary of your bot's intent and utterance metrics](performance-dashboard.md)
+ [Using APIs for analytics](analytics-api.md)
+ [Managing access permissions for analytics](analytics-permissions.md)
# Key definitions
This topic provides key definitions that will help you interpret your bot analytics. These definitions are related to the performance of your bot in four contexts: **Intents**, **Slots**, **Conversations**, and **Utterances**. The following fields are relevant to many of the performance metrics:
+ The [`state` field of the `Intent`](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_Intent.html#lexv2-Type-runtime_Intent-state) object.
+ The [`type` field of the `dialogAction` object](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_DialogAction.html#lexv2-Type-runtime_DialogAction-type) within the [SessionState](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_SessionState.html) object.
## Intents
Amazon Lex V2 categorizes intents in the following ways:
+ **Success** – The bot successfully fulfilled the intent. One of the following situations is true:
+ The intent `state` is `ReadyForFulfillment` and the `type` of `dialogAction` is `Close`.
+ The intent `state` is `Fulfilled` and the `type` of `dialogAction` is `Close`.
+ **Failed** – The bot failed to fulfill the intent. The intent state. One of the following situations is true:
+ The intent `state` is `Failed` and the `type` of `dialogAction` is `Close` (for example, the user declined the confirmation prompt).
+ The bot switches to the `AMAZON.FallbackIntent` before the intent is completed.
+ **Switched** – The bot recognizes a different intent and switches to that intent instead, before the original intent is categorized as a *success* or *failed*.
+ **Dropped** – The customer doesn't respond before the intent is categorized as a *success* or *failed*.
## Slots
Amazon Lex V2 categorizes slots in the following ways:
+ **Success** – The bot filled the slot and successfully transitioned to another slot or the confirmation step.
+ **Failed** – The bot wasn't able to fill the slot, even after reaching the maximum number of retries.
+ **Dropped** – The customer doesn't respond or switches to another intent before the slot is categorized as a *success* or *failed*.
## Conversations
When a customer makes a runtime call to Amazon Lex V2, they provide a [https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_PutSession.html#lexv2-runtime_PutSession-request-sessionId](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_PutSession.html#lexv2-runtime_PutSession-request-sessionId) and Amazon Lex V2 generates an [https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_SessionState.html#lexv2-Type-runtime_SessionState-originatingRequestId](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_SessionState.html#lexv2-Type-runtime_SessionState-originatingRequestId). If the customer doesn't respond within the Session timeout ([https://docs.aws.amazon.com/lexv2/latest/APIReference/API_CreateBot.html#lexv2-CreateBot-request-idleSessionTTLInSeconds](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_CreateBot.html#lexv2-CreateBot-request-idleSessionTTLInSeconds)) that you set for the bot, the session expires. If a customer returns to the session by using the same `sessionId`, Amazon Lex V2 generates a new `originatingRequestId`.
For analytics, a *conversation* is a unique combination of a `sessionId` and an `originatingRequestId`. Amazon Lex V2 categorizes conversations in the following ways:
+ **Success** – The final intent in the conversation is categorized as a *success*.
+ **Failed** – The final intent in the conversation is *failed*. The conversation is also *failed* if Amazon Lex V2 defaults to the [AMAZON.FallbackIntent](built-in-intent-fallback.md).
+ **Dropped** – The customer doesn't respond before the conversation is categorized as a *success* or *failed*.
## Utterances
Amazon Lex V2 categorizes utterances in the following ways:
+ **Detected** – Amazon Lex V2 recognizes the utterance as an attempt to invoke an intent configured for a bot.
+ **Missed** – Amazon Lex V2 doesn't recognize the utterance.
# Filtering results
At the top of each page, you can filter the results for your bot analytics.
You can filter by the following parameters:
+ **Time** – You can filter results by a relative or absolute time range. When you select a start and end time, Amazon Lex V2 retrieves conversations that began *after* the start time and ended *before* the end time.
+ **Relative range** – Select **1d** to see results from the past day, **1w** for the past week, or **1m** for the past month.
For more options, select **Custom** and choose a duration in the **Relative range** menu. For more control over the duration, select **Custom range**, enter a number in the **Duration** field, and choose a **Unit of time** from the dropdown menu.
+ **Absolute range** – Select **Custom** and choose the **Absolute range** menu to filter for conversations within a time range that you specify. You can choose a start and end date on the calendar or enter it in YYYY/MM/DD format.
**Note**
The analytics time range has the following restrictions:
The start date must be within the last 365 days from the current date.
The end date must not be more than 1 month after the start date.
+ **Bot filters** – To filter by locale, alias, and version of your bot, select the dropdown menus labeled **All locales**, **All aliases**, and **All versions**.
+ **Modality** – Select the gear icon and choose the **Modality** dropdown menu to choose whether to display results for **Speech** or **Text**.
+ **Channel** – Select the gear icon and choose the **Channel** dropdown menu to choose the channel for which you want to display results. For more information about channel integration, see [Integrating an Amazon Lex V2 bot with a messaging platform](deploying-messaging-platform.md) and [Amazon Connect contact centers](https://docs.aws.amazon.com/connect/latest/adminguide/amazon-connect-contact-centers.html)
# Overview: a summary of your bot performance
The overview page summarizes your bot’s performance on conversations, utterance recognition, and intent usage. The overview consists of the following sections:
+ [Conversation performance](#conversations-performance)
+ [Utterance recognition rate](#utterance-recognition-rate)
+ [Conversation performance history](#conversation-performance-history)
+ [Top 5 used intents](#top-five-used-intents)
+ [Top 5 failed intents](#top-five-failed-intents)
## Conversation performance
Use this chart to track the number and percentage of conversations that are categorized as a *success*, *failed*, and *dropped*. To access a list of conversations, select **View all conversations** to reveal a dropdown menu. You can choose to view a list of all user conversations with the bot or filter for conversations with a specific result (*success*, *failed*, or *dropped*). These links take you to the **Conversations** subsection of the **Conversation dashboard**. For more information, see [Conversations](conversation-dashboard.md#conversations).
To reveal a box with the count and percentage of conversations with that result, hover over a segment of the chart, as in the following image.
![\[A donut chart that visualizes the performance of your conversations.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/conversation-performance.png)
## Utterance recognition rate
Use this chart to track the number and percentage of utterances that were **detected** and **missed** by your bot. To access a list of utterances, select **View utterances** to reveal a dropdown menu. You can choose to view a list of all user utterances or filter for utterances with a specific result (*missed* or *detected*). These links take you to the **Utterance recognition** subsection of the **Performance dashboard**. For more information, see **View Utterances** to navigate to [Utterance recognition](performance-dashboard.md#utterance-recognition).
To reveal a box with the count and percentage of utterances, hover over a segment of the chart, as seen in the following image.
![\[A donut chart that visualizes the performance of your bot's utterance recognition.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/utterance-recognition-rate.png)
## Conversation performance history
Use this graph to track the percentage of conversations categorized as a *success*, *failed*, and *dropped* over the time range that you set in the filters. To see the percentage of conversations with a specific result in an interval of time, hover over that interval, as in the following image.
![\[A line graph that tracks the bot's conversation performance over time.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/conversation-performance-history.png)
## Top 5 used intents
Use this chart to identify the top five intents that customers used with your bot. Hover over a bar to see the number of times that your bot recognized that intent, as in the following image.
![\[A bar graph that shows the top five intents that customers used with your bot.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/top-five-intents.png)
Select **View all intents** to navigate to the **Intents performance** subsection of the **Performance dashboard**, where you can view metrics for your bot’s performance in fulfilling intents. For more information, see [Intent performance](performance-dashboard.md#intent-performance).
## Top 5 failed intents
Use this chart to identify the top five intents that your bot failed to fulfill (see [Intents](analytics-key-definitions.md#analytics-key-definitions-intents) for the definition of a failed intent). Hover over a bar to see the number of times that your bot failed to fulfill that intent, as in the following image.
![\[A bar graph that shows the top five intents that your bot failed to fulfill.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/top-five-failed-intents.png)
Select **View failed intents** to navigate to the **Intents performance** subsection of the **Performance dashboard**, where you can view metrics for the intents that your bot failed to fulfill. For more information, see [Intent performance](performance-dashboard.md#intent-performance).
# Conversation dashboard: a summary of your bot conversations
The conversation dashboard visualizes metrics for customers’ conversations (see [Conversations](analytics-key-definitions.md#analytics-key-definitions-conversations) for the definition of a *conversation*) with your bot.
The **Summary** contains the following information about user conversations with your bot. The numbers are calculated based on the filter settings.
+ **Total conversations** – The total number of conversations with the bot.
+ **Average conversation duration** – The average time of user conversations with the bot in minutes and seconds. The format is mm:ss.
+ **Average turns per conversation** – The average number of turns that a conversation lasts.
The **Conversation count** and **Message count** sections each contain a graph that shows the number of conversations and messages, respectively, over the time range that you specify in your filters. Hover over a segment of time to see the number of conversations or messages in that segment. The size of the time segment depends on the time range you specify:
+ Less than 1 week – The count is displayed for each hour.
+ 1 week or more – The count is displayed for each day.
See an example of the hovering behavior in the following image.
![\[A line graph that shows the number of conversations at each hour in a given time grame.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/conversation-count.png)
The **Time of conversations** section presents the number of conversations that took place between your bot and customers over each two hour interval on each day of the week, within the time range that you specify in the filters. More darkly shaded cells indicate times at which more conversations took place. Hover over a cell to display the number of conversations in the 2 hours beginning from that time slot. For example, the action in the following image shows the number of conversations occurring between 4:00PM and 6:00PM UTC.
![\[A heatmap that visualizes the number of conversations in each two-hour interval over each day of the week.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/time-of-conversations.png)
The **Conversation dashboard** contains two tools, **Conversation flows** and **Conversations**. Access a tool by selecting it under **Conversation dashboard** in the left navigation pane.
## Conversation flows
Use **Conversation flows** to visualize the orders of intents that customers take in conversations with your bot. Below each intent is the percentage and count of conversations that invoked that intent at that point in the conversation. You can toggle the percentage and count off by selecting **Conversation percentage** and **Conversation count** at the top. By default, the five most common intents at that point in the conversation are shown in descending order of frequency. Select **\$1 Others** to display all the intents.
Choose an intent to expand to a new column of branches that shows a list of intents taken at that point in the conversation, sorted in descending frequency.
When you select a node in the conversation flow, you can expand the window below to display a list of conversations that followed that order of intents. Choose the **Session ID** corresponding to a conversation to view details about that conversation. The following image shows a conversation flow and an expanded **Conversations** window at the bottom.
![\[A flowchart that visualizes the different orders of intents that customers take in conversations with your bot.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/conversation-flow.png)
## Conversations
The **Conversations** tool displays a list of conversations for your bot. You can select a column to sort by that column in ascending or descending order.
To filter the conversations by result, select **All results** and choose **Success**, **Failed**, or **Dropped**.
**To filter the conversations by duration**
1. Select the search bar marked **Filter conversations by duration**
1. Define the filter in one of the following ways:
+ Use the predefined options.
1. Select **Duration**.
1. Choose between the = (equals), > (greater than), and < (less than) operators.
1. Choose a length of time.
+ Enter an input in the format "Duration \$1operator\$1 \$1number\$1 sec" For example, to search for all conversations lasting longer than 30 seconds, enter **Duration > 30 sec**. Specify the length of time in seconds.
To view detailed information about the session, including metadata, intent usage, and a transcript, select the **Session ID** of a conversation.
**Note**
Because a conversation is a unique combination of a `sessionId` and `originatingRequestId`, the same `sessionId` might appear multiple times in the table.
The **Details** section contains the following metadata:
+ **Timestamp** – Specifies the date and start time of the conversation. The time is in hh:mm:ss format.
+ **Duration** – Specifies how long the conversation lasted in mm:ss format. The duration doesn't include the Session timeout duration (`idleSessionTTLInSeconds`).
+ **Result** – Specifies whether the conversation was categorized as a *success*, *failed*, or *dropped*. See [Conversations](analytics-key-definitions.md#analytics-key-definitions-conversations) for more details about these results.
+ **Mode** – Specifies whether the conversation was `Speech`, `Text`, or `DTMF` (touch tone keypad presses). A conversation consisting of multiple modes is `Multimode`.
+ **Channel** – Specifies the channel that the conversation took place on, if applicable. See [Integrating an Amazon Lex V2 bot with a messaging platform](deploying-messaging-platform.md).
+ **Language** – Specifies the language of the bot.
The intents that the bot elicited in the conversation are shown below **Details**. Select **Go to Intent** to go to that intent in the intent editor. Select **Snap to transcript** to automatically scroll the **Transcript** to the first instance in which the bot elicited the intent.
Select the right arrow next to the intent name to view details about the slots elicited for the intent, including the names of the slots, the value that the bot elicited for each slot, and the number of times the bot attempted to elicit each slot.
The **Transcript** lets you review the conversation utterances and your bot’s behavior in eliciting intents and slots. User utterances are displayed on the left and bot utterances are displayed on the right. Use the search bar marked **Filter transcripts in this session** to find text in the transcript. Next to **Showing:** there are three pieces of information shown under each conversation turn that you can select to display or not:
+ **Timestamp** – Specifies the time of the utterance.
+ **Intent state** – Specifies the intent that the bot is eliciting during an utterance and the result of the intent, if applicable. The following intent states are possible:
+ Invoked intent: *intent name* – The bot has identified an intent that the customer is invoking.
+ Switched intent: *intent name* – The bot has switched to a different intent based on the utterance.
+ *intent name*: Success – The bot has fulfilled the intent.
+ **Slot state** – Specifies the slot that the bot is eliciting during an utterance, if applicable, and the value that the customer provides.
# Performance dashboard: a summary of your bot's intent and utterance metrics
In the performance dashboard, you can view details about the performance of your bot's intent fulfillment and utterance recognition.
The **Intent performance breakdown** section displays the total number of times your bot invoked an intent and breaks down the number and percentage of times that the intents were categorized as a *success*, *failed*, *dropped*, and *switched*. See [Intents](analytics-key-definitions.md#analytics-key-definitions-intents) for an explanation of these definitions. Hover over a segment of the chart to show a box with the count and percentage of conversations with that result, as in the following image.
![\[A donut chart that shows the different results of intent fulfillment for a bot.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/intent-performance-breakdowns.png)
Select **View all intents** to reveal a dropdown menu, from which you can choose to view a list of intents that the bot elicited. You can also choose to view intents with a specific result (*success*, *failed*, *dropped*, or *switched*). These links take you to the **Intent performance** subsection of the **Performance dashboard**. For more information, see [Intent performance](#intent-performance).
The **Utterance recognition** section summarizes the number of utterances that were *missed* and *detected*. Select **View details** to navigate to a list of utterances for the bot. Choose the number under **Missed utterances** to see a list of missed utterances and the number under **Detected utterances** to see a list of detected utterances for the bot. For more information, see [Utterance recognition](#utterance-recognition).
Select **Intents performance** and **Utterance recognition** under the **Performance dashboard** in the left sidebar to view details about intents and utterances in your bot.
## Intent performance
This dashboard summarizes the performance of intents used with your bot in descending order of frequency. The bar next to each intent visualizes the number of times the intent was categorized as a *success*, *failed*, *dropped*, and *switched*. See [Intents](analytics-key-definitions.md#analytics-key-definitions-intents) for an explanation of these definitions. Hover over a segment of the bar to see the number of conversations using that intent with that result, as in the following image:
![\[A bar chart that shows the different results of intent fulfillment for a bot.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/analytics/intent-performance.png)
**Note**
The dashboard shows the top 1,000 results for a set of filter settings. To get more targeted results, configure granular filter settings.
At the top of the chart, you can toggle the intent statuses that you want to view with the **Success**, **Failed**, **Dropped**, and **Switched** checkboxes.
Select the dropdown menus to the right of **Showing** to adjust the number of intents to display and whether to display intents in ascending or descending order of frequency.
Select an intent name to navigate to a page that shows three charts: **Intent performance breakdown**, **Slot performance**, and **Intent switches**.
The **Intent performance breakdown** section displays the total number of times that the bot used the intent and breaks down the number and percentage of times intent fulfillment was categorized as a *success*, *failed*, *dropped*, and *switched*. See [Intents](analytics-key-definitions.md#analytics-key-definitions-intents) for an explanation of these definitions. Hover over a segment of the chart to see the count and percentage of times that intent fulfillment yielded that result.
The **Slot performance** section displays metrics for the slots that belong to the current intent. To sort by a column, select that column once to sort in ascending order and twice to sort it in descending order. You can use the search bar to find a specific slot or use the page number buttons to navigate through the slots.
**Note**
The dashboard shows the top 1,000 results for a set of filter settings. To get more targeted results, configure granular filter settings..
The **Intent switches** section lists out the instances in which the bot switched from the current intent to another one with the following information:
+ **Stage** – The stage of the conversation at which the bot switched the intent.
+ **Intent switched to** – The intent that the bot switched the current intent to.
+ **Session count** – The number of sessions in which the **Stage** and **Intent switched to** combination occurred.
**Note**
The dashboard shows the top 1,000 results for a set of filter settings. To get more targeted results, configure granular filter settings..
## Utterance recognition
This page lists out all the utterances that were *missed* and *detected* by your bot and provides tools for you to add sample utterances to intents to help train your bot. See [Utterances](analytics-key-definitions.md#analytics-key-definitions-utterances) for an explanation of these definitions. Use the tabs at the top to switch between a list of **Missed utterances** and of **Detected utterances**.
**Note**
The dashboard shows the top 1,000 results for a set of filter settings. To get more targeted results, configure granular filter settings.
To add utterances to an intent:
1. Choose the checkbox next to the utterances that you want to add as sample utterances for an intent.
1. Select **Add to intent** and choose the intent to which you want to add the utterances in the dropdown menu under **Intent**.
1. Select **Add**.
# Using APIs for analytics
This section describes the API operations that you use to retrieve analytics for a bot.
**Note**
To use the [ListUtteranceMetrics](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListUtteranceMetrics.html) and [ListUtteranceAnalyticsData](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListUtteranceAnalyticsData.html), your IAM role must have permissions to perform the [ListAggregatedUtterances](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListAggregatedUtterances.html) operation, which provides access to utterance-related analytics. See [Viewing utterance statistics from Lex V2 conversations](#monitoring-utterances) for details and the IAM policy to apply to the IAM role.
+ The following API operations retrieve summary metrics for a bot:
+ [ListSessionMetrics](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListSessionMetrics.html)
+ [ListIntentMetrics](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListIntentMetrics.html)
+ [ListIntentStageMetrics](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListIntentStageMetrics.html)
+ [ListUtteranceMetrics](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListUtteranceMetrics.html)
+ The following API operations retrieve a list of metadata for sessions and utterances:
+ [ListSessionAnalyticsData](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListSessionAnalyticsData.html)
+ [ListUtteranceAnalyticsData](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListUtteranceAnalyticsData.html)
+ The [ListIntentPaths](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListIntentPaths.html) operation retrieves metrics about an order of intents that customers take in conversations with a bot.
## Filtering results
The Analytics API requests require you to specify the `startTime` and `endTime`. The API returns sessions, intents, intent stages, or utterances that began *after* the `startTime` and ended *before* the `endTime`.
`filters` is an optional field in the Analytics API requests. It maps to a list of [AnalyticsSessionFilter](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsSessionFilter.html), [AnalyticsIntentFilter](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentFilter.html), [AnalyticsIntentStageFilter](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentStageFilter.html), or [AnalyticsUtteranceFilter](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsUtteranceFilter.html) objects. In each object, use the fields to create an expression to filter by. For example, if you add the following filter to the list, the bot searches for conversations that are longer than 30 seconds.
```
{
"name": "Duration",
"operator": "GT",
"value": "30 sec",
}
```
## Retrieving metrics for a bot
Use the `ListSessionMetrics`, `ListIntentMetrics`, `ListIntentStageMetrics`, and `ListUtteranceMetrics` operations to retrieve summary metrics for *sessions*, *intents*, *intent stages*, and *utterances*.
For these operations, fill in the following required fields:
+ Provide a `startTime` and `endTime` to define a time range for which you want to retrieve results.
+ Specify the metrics you want to calculate in `metrics`, a list of [AnalyticsSessionMetric](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsSessionMetric.html), [AnalyticsIntentMetric](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentMetric.html), [AnalyticsIntentStageMetric](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentStageMetric.html), or [AnalyticsUtteranceMetric](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsUtteranceMetric.html) objects. In each object, use the `name` field to specify the metric to calculate the `statistic` field to specify whether to calculate the `Sum`, `Average`, or `Max` number, and the `order` field to specify whether to sort the results in `Ascending` or `Descending` order.
**Note**
Both the `metrics` and `binBy` objects contain an `order` field. You can specify the sorting `order` in only one of the two objects.
The remaining fields in the request are optional. You can filter and organize the results in the following ways:
+ **Filtering results** – Use the `filters` field to filter the results. See [Filtering results](#analytics-api-filters) for more details.
+ **Grouping results by category** – Specify the `groupBy` field, a list containing a single [AnalyticsSessionResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsSessionResult.html), [AnalyticsIntentResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentResult.html), [AnalyticsIntentStageResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentStageResult.html), or [AnalyticsUtteranceResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsUtteranceResult.html) object. In the object, specify the `name` field with the category by which you want to group the results.
If you specify a `groupBy` field in the request, the `results` object in the response contains `groupByKeys`, a list of [AnalyticsSessionGroupByKey](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsSessionGroupByKey.html), [AnalyticsIntentGroupByKey](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentGroupByKey.html), [AnalyticsIntentStageGroupByKey](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentStageGroupByKey.html), or [AnalyticsUtteranceGroupByKey](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsUtteranceGroupByKey.html) objects, each with the `name` that you specified in the request and a member of that category in the `value` field.
+ **Binning results by time** – Specify the `binBy` field, a list containing a single [AnalyticsBinBySpecification](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsBinBySpecification.html) object. In the object, specify the `name` field with `ConversationStartTime` to bin the results by when the conversation began or `UtteranceTimestamp` to bin the results by when the utterance took place. Specify the interval of time by which you want to bin the results in the `interval` field, and whether to sort in `Ascending` or `Descending` order of time in the `order` field.
If you specify a `binBy` field in the request, the `results` object in the response contains `binKeys`, a list of [AnalyticsBinKey](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsBinKey.html) objects, each with the `name` that you specified in the request and the interval of time that defines that bin in the `value` field.
**Note**
Both the `metrics` and `binBy` objects contain an `order` field. You can specify the sorting `order` in only one of the two objects.
Use the following fields to handle the display of the response:
+ Specify a number between 1 and 1,000 in the `maxResults` field to limit the number of results to return in a single response.
+ If the number of results is greater than the number you specify in the `maxResults` field, the response contains a `nextToken`. Make the request again, but use this value in the `nextToken` field to return the next batch of results.
If you are using `ListUtteranceMetrics`, you can specify attributes to return in the `attributes` field. This field maps to a list containing a single [AnalyticsUtteranceAttribute](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsUtteranceAttribute.html) object. Specify `LastUsedIntent` in the `name` field to return the intent that Amazon Lex V2 is using at the time of the utterance.
In the response, the `results` field maps to a list of [AnalyticsSessionResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsSessionResult.html), [AnalyticsIntentResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentResult.html), [AnalyticsIntentStageResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsIntentStageResult.html), or [AnalyticsUtteranceResult](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_AnalyticsUtteranceResult.html) objects. Each object contains a `metrics` field which returns the value of a summary statistic for a metric that you requested, in addition to any bins or groups created from the methods you specified.
## Retrieving metadata for sessions and utterances in a bot
Use the [ListSessionAnalyticsData](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListSessionAnalyticsData.html) and [ListUtteranceAnalyticsData](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListUtteranceAnalyticsData.html) operations to retrieve metadata about individual sessions and utterances.
Fill in the required `startTime` and `endTime` fields to define a time range for which you want to retrieve results.
The remaining fields in the request are optional. To filter and sort results:
+ **Filtering results** – Use the `filters` field to filter the results. See [Filtering results](#analytics-api-filters) for more details.
+ **Sorting results** – Sort the results with the `sortBy` field, which contains a [SessionDataSortBy](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_SessionDataSortBy.html) or [UtteranceDataSortBy](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_UtteranceDataSortBy.html) object. Specify the value you want to sort by in the `name` field and whether to sort in `Ascending` or `Descending` order in the `order` field.
Use the following fields to handle the display of the response:
+ Specify a number between 1 and 1,000 in the `maxResults` field to limit the number of results to return in a single response.
+ If the number of results is greater than the number you specify in the `maxResults` field, the response contains a `nextToken`. Make the request again, but use this value in the `nextToken` field to return the next batch of results.
In the response, the `sessions` or `utterances` field maps to a list of [SessionSpecification](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_SessionSpecification.html) or [UtteranceSpecification](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_UtteranceSpecification.html) objects. Each object contains metadata for a single session or utterance.
## Retrieving intent path analytics data
Use the [ListIntentPaths](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_ListIntentPaths.html) operation to retrieve metrics about an order of intents that customers take in conversation with a bot.
For this operation, fill in the following required fields:
+ Provide a `startTime` and `endTime` to define a time range for which you want to retrieve results.
+ Provide an `intentPath` to define an order of intents for which you want to retrieve metrics. Separate intents in the path with a forward slash. For example, populate the `intentPath` field with **/BookCar/BookHotel** to see details about how many times users invoked the `BookCar` and `BookHotel` intents in that order.
Use the optional `filters` field to filter the results. For more details, see [Filtering results](#analytics-api-filters).
## Viewing utterance statistics from Lex V2 conversations
You can use utterance statistics to determine the utterances that your users are sending to your bot. You can see both the utterances that Amazon Lex V2 successfully detects and the utterances that it doesn't. You can use this information to help tune your bot.
For example, if you find that your users are sending an utterance that Amazon Lex V2 is missing, you can add the utterance to an intent. The Draft version of the intent is updated with the new utterance and you can test it before deploying it to your bot.
An utterance is detected when Amazon Lex V2 recognizes the utterance as an attempt to invoke an intent configured for a bot. An utterance is missed when Amazon Lex V2 doesn't recognize the utterance and invokes the `AMAZON.FallbackIntent` instead.
Utterance statistics can be viewed using the `ListUtteranceMetrics` API and the `ListAggregatedUtterance` API.
Utterance statistics are not generated using `ListUtteranceMetrics` API under the following conditions:
+ The Child Online Privacy Protection Act setting was set to **Yes** when the bot was created with the console, or the `childDirected` field was set to true when the bot was created withe the `CreateBot` operation.
The `ListUtteranceMetrics` API provides additonal features including:
+ More information available, such as mapped intent for detected utterances.
+ More filtering capability (including channel and mode).
+ Longer retention date range (30 days).
+ You can use the API even if you have opted out of data storage. The console functionality for missed and detected utterances will rely on `ListUtteranceMetrics` API.
Utterance statistics are not generated using `ListAggregatedUtterance` API under the following conditions:
+ The Child Online Privacy Protection Act setting was set to **Yes** when the bot was created with the console, or the `childDirected` field was set to true when the bot was created withe the `CreateBot` operation.
+ You are using slot obfuscation with one or more slots.
+ You opted out of participating in improving Amazon Lex.
The `ListAggregatedUtterance` API provides features including:
+ Less detailed information available (no mapped intent for utterances).
+ Limited filtering capability (not including channel and mode).
+ Short retention date range (15 days).
Using utterance statistics, you can see whether a specific utterance was detected or missed, alongside the last time that the utterance was used in a bot interaction.
Amazon Lex V2 stores utterances continuously while users interact with your bot. You can query the statistics using the console or the `ListAggregatedUtterances` operation. It has a data retention of 15 days and it is not available if the user has opted out of data storage. You can delete utterances using the `DeleteUtterances` operation or by opting out of data storage. All utterances are deleted if you close your AWS account. Stored utterances are encrypted with a server-managed key.
When you delete a bot version, utterance statistics are available for the version for up to 30 days with `ListUtteranceMetrics`, and 15 days using `ListAggregatedUtterances`. You can't see statistics for deleted version in the Amazon Lex V2 console. To see the statistics for deleted versions, you can use both `ListAggregatedUtterances` and `ListUtteranceMetrics`operations.
With both the `ListAggregatedUtterances` and `ListUtteranceMetrics` APIs, utterances are aggregated by the text of the utterance. For example, all instances where the customer used the phrase "I want to order a pizza" are aggregated into the same line in a response. When you use the [RecognizeUtterance](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_RecognizeUtterance.html) operation, the text used is the input transcript.
To use the `ListAggregatedUtterances` and `ListUtteranceMetrics` APIs, apply the following policy to a role.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "ListAggregatedUtterancesPolicy",
"Effect": "Allow",
"Action": "lex:ListAggregatedUtterances",
"Resource": "*"
}
]
}
```
------
# Managing access permissions for analytics
To provide a user access to analytics, attach a policy to an IAM role that permits the role to call the API operations for analytics. You can attach the [AWS managed policy: AmazonLexFullAccess](security-iam-awsmanpol.md#security-iam-awsmanpol-AmazonLexFullAccess) to the IAM role to provide full access to Amazon Lex API operations, or you can create a custom policy allowing only permissions to analytics and attach it to an IAM role.
**To create a custom policy containing permissions for analytics**
1. If you need to first create an IAM role, follow the steps at [Creating a role to delegate permissions to an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user.html).
1. Follow the steps at [Creating IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) to create a policy using the following JSON object. To enable analytics access to specific bots for the IAM role, add the ARN of each bot to the `Resource` field. Replace the *region*, *account-id*, and *BOTID* with the values corresponding to the bots. You can also replace the statement identifier, *AnalyticsActions*, with a name of your choice.
1. Attach the policy you created to the role that you want to grant analytics permissions by following the steps at [Adding and removing IAM identity permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html).
1. The role should now have permissions to view analytics for the bots you specified.
# Enabling conversation logs for your Lex V2 bots
Use conversation logs to store user conversations with your bot. Review these logs to identify issues with your bot's interactions with users and modify your bot's behavior with these insights. This section also describes how to obfuscate slot values to protect the privacy of users.
**Topics**
+ [Logging conversations with conversation logs in Lex V2](conversation-logs.md)
+ [Obscuring slot values in conversation logs from Lex V2](monitoring-obfuscate.md)
+ [Selective conversation log capture in Lex V2](monitoring-selective-logging.md)
# Logging conversations with conversation logs in Lex V2
You enable *conversation logs* to store bot interactions. You can use these logs to review the performance of your bot and to troubleshoot issues with conversations. You can log text for the [RecognizeText](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_RecognizeText.html) operation. You can log both text and audio for the [RecognizeUtterance](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_RecognizeUtterance.html) operation. By enabling conversation logs, you get a detailed view of conversations that users have with your bot.
For example, a session with your bot has a session ID. You can use this ID to get the transcript of the conversation including user utterances and the corresponding bot responses. You also get metadata such as intent name and slot values for an utterance.
**Note**
You can't use conversation logs with a bot subject to the Children's Online Privacy Protection Act (COPPA).
Conversation logs are configured for an alias. Each alias can have different settings for their text and audio logs. You can enable text logs, audio logs, or both for each alias. Text logs store text input, transcripts of audio input, and associated metadata in CloudWatch Logs. Audio logs store audio input in Amazon S3. You can enable encryption of text and audio logs using AWS KMS customer managed CMKs.
To configure logging, use the console or the [CreateBotAlias](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_CreateBotAlias.html) or [UpdateBotAlias](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_UpdateBotAlias.html) operation. After enabling conversation logs for an alias, using the [RecognizeText](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_RecognizeText.html) or [RecognizeUtterance](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_RecognizeUtterance.html) operation for that alias logs the text or audio utterances in the configured CloudWatch Logs log group or S3 bucket.
**Topics**
+ [IAM Policies for Conversation Logs](conversation-logs-policies.md)
+ [Configuring conversation logs for your Lex V2 bot](conversation-logs-configure.md)
+ [Viewing text logs in Amazon CloudWatch Logs from Lex V2](conversation-logs-cw.md)
+ [Accessing audio logs in Amazon S3](conversation-logs-s3.md)
+ [Monitoring conversation log status with CloudWatch metrics](conversation-logs-monitoring.md)
# IAM Policies for Conversation Logs
Depending on the type of logging that you select, Amazon Lex V2 requires permission to use Amazon CloudWatch Logs and Amazon Simple Storage Service (S3) buckets to store your logs. You must create AWS Identity and Access Management roles and permissions to enable Amazon Lex V2 to access these resources.
## Creating an IAM Role and Policies for Conversation Logs
To enable conversation logs, you must grant write permission for CloudWatch Logs and Amazon S3. If you enable object encryption for your S3 objects, you need to grant access permission to the AWS KMS keys used to encrypt the objects.
You can use the IAM console, the IAM API, or the AWS Command Line Interface to create the role and policies. These instructions use the AWS CLI to create the role and policies.
**Note**
The following code is formatted for Linux and MacOS. For Windows, replace the Linux line continuation character (\$1) with a caret (^).
**To create an IAM role for conversation logs**
1. Create a document in the current directory called **LexConversationLogsAssumeRolePolicyDocument.json**, add the following code to it, and save it. This policy document adds Amazon Lex V2 as a trusted entity to the role. This allows Amazon Lex V2 to assume the role to deliver logs to the resources configured for conversation logs.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lexv2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
1. In the AWS CLI, run the following command to create the IAM role for conversation logs.
```
aws iam create-role \
--role-name role-name \
--assume-role-policy-document file://LexConversationLogsAssumeRolePolicyDocument.json
```
Next, create and attach a policy to the role that enables Amazon Lex V2 to write to CloudWatch Logs.
**To create an IAM policy for logging conversation text to CloudWatch Logs**
1. Create a document in the current directory called **LexConversationLogsCloudWatchLogsPolicy.json**, add the following IAM policy to it, and save it.
1. In the AWS CLI, create the IAM policy that grants write permission to the CloudWatch Logs log group.
```
aws iam create-policy \
--policy-name cloudwatch-policy-name \
--policy-document file://LexConversationLogsCloudWatchLogsPolicy.json
```
1. Attach the policy to the IAM role that you created for conversation logs.
```
aws iam attach-role-policy \
--policy-arn arn:aws:iam::account-id:policy/cloudwatch-policy-name \
--role-name role-name
```
If you are logging audio to an S3 bucket, create a policy that enables Amazon Lex V2 to write to the bucket.
**To create an IAM policy for audio logging to an S3 bucket**
1. Create a document in the current directory called **LexConversationLogsS3Policy.json**, add the following policy to it, and save it.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::bucket-name/*"
}
]
}
```
------
1. In the AWS CLI, create the IAM policy that grants write permission to your S3 bucket.
```
aws iam create-policy \
--policy-name s3-policy-name \
--policy-document file://LexConversationLogsS3Policy.json
```
1. Attach the policy to the role that you created for conversation logs.
```
aws iam attach-role-policy \
--policy-arn arn:aws:iam::account-id:policy/s3-policy-name \
--role-name role-name
```
## Granting Permission to Pass an IAM Role
When you use the console, the AWS Command Line Interface, or an AWS SDK to specify an IAM role to use for conversation logs, the user specifying the conversation logs IAM role must have permission to pass the role to Amazon Lex V2. To allow the user to pass the role to Amazon Lex V2, you must grant `PassRole` permission to the user's IAM user, role, or group.
The following policy defines the permission to grant to the user, role, or group. You can use the `iam:AssociatedResourceArn` and `iam:PassedToService` condition keys to limit the scope of the permission. For more information, see [ Granting a User Permissions to Pass a Role to an AWS Service ](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html) and [ IAM and AWS STS Condition Context Keys ](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html) in the *AWS Identity and Access Management User Guide*.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::111122223333:role/role-name",
"Condition": {
"StringEquals": {
"iam:PassedToService": "lexv2.amazonaws.com"
},
"StringLike": {
"iam:AssociatedResourceARN": "arn:aws:lex:region:123456789012:bot:bot-name:bot-alias"
}
}
}
]
}
```
------
# Configuring conversation logs for your Lex V2 bot
You enable and disable conversation logs using the console or the `conversationLogSettings` field of the `CreateBotAlias` or `UpdateBotAlias` operation. You can turn on or turn off audio logs, text logs, or both. Logging starts on new bot sessions. Changes to log settings aren't reflected for active sessions.
To store text logs, use an Amazon CloudWatch Logs log group in your AWS account. You can use any valid log group. The log group must be in the same region as the Amazon Lex V2 bot. For more information about creating a CloudWatch Logs log group, see [ Working with Log Groups and Log Streams ](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html) in the *Amazon CloudWatch Logs User Guide*.
To store audio logs, use an Amazon S3 bucket in your AWS account. You can use any valid S3 bucket. The bucket must be in the same region as the Amazon Lex V2 bot. For more information about creating an S3 bucket, see [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html) in the *Amazon Simple Storage Service Getting Started Guide*.
When you manage conversation logs using the console, the console updates your service role so that it has access to the log group and S3 bucket.
If you are not using the console, you must provide an IAM role with policies that enable Amazon Lex V2 to write to the configured log group or bucket. If you create a service-linked role using the AWS Command Line Interface, you must add a custom suffix to the role using the `custom-suffix` option as in the following example. For more information, see [Creating an IAM Role and Policies for Conversation Logs](conversation-logs-policies.md#conversation-logs-role-and-policy).
```
aws iam create-service-linked-role \
--aws-service-name lexv2.amazon.aws.com \
--custom-suffix suffix
```
The IAM role that you use to enable conversation logs must have the `iam:PassRole` permission. The following policy should be attached to the role:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::111122223333:role/role"
}
]
}
```
------
## Enabling conversation logs
**To turn on logs using the console**
1. Open the Amazon Lex V2 console [https://console.aws.amazon.com/lexv2](https://console.aws.amazon.com/lexv2).
1. From the list, choose a bot.
1. From the left menu, choose **Aliases**.
1. In the list of aliases, choose the alias for which you want to configure conversation logs.
1. In the **Conversation logs** section, choose **Manage conversation logs**.
1. For text logs, choose **Enable** then enter the Amazon CloudWatch Logs log group name.
1. For audio logs, choose **Enable** then enter the S3 bucket information.
1. Optional. To encrypt audio logs, choose the AWS KMS key to use for encryption.
1. Choose **Save** to start logging conversations. If necessary, Amazon Lex V2 will update your service role with permissions to access the CloudWatch Logs log group and selected S3 bucket.
## Disabling conversation logs in Lex V2
**To turn off logs using the console**
1. Open the Amazon Lex V2 console [https://console.aws.amazon.com/lexv2](https://console.aws.amazon.com/lexv2).
1. From the list, choose a bot.
1. From the left menu, choose **Aliases**.
1. In the list of aliases, choose the alias for which you want to configure conversation logs.
1. In the **Conversation logs** section, choose **Manage conversation logs**.
1. Disable text logging, audio logging, or both to turn off logging.
1. Choose **Save** to stop logging conversations.
# Viewing text logs in Amazon CloudWatch Logs from Lex V2
Amazon Lex V2 stores text logs for your conversations in Amazon CloudWatch Logs. To view the logs, use the CloudWatch Logs console or API. For more information, see [ Search Log Data Using Filter Patterns](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SearchDataFilterPattern.html) and [CloudWatch Logs Insights Query Syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) in the *Amazon CloudWatch Logs User Guide*.
**To view logs using the Amazon Lex V2 console**
1. Open the Amazon Lex V2 console [https://console.aws.amazon.com/lexv2](https://console.aws.amazon.com/lexv2).
1. From the list, choose a bot.
1. From the left menu, choose **Analytics** and then choose **CloudWatch metrics**.
1. View metrics for your bot on the **CloudWatch metrics** page.
You can also use the CloudWatch console or API to view your log entries. To find the log entries, navigate to the log group that you configured for the alias. You can find the log stream prefix for your logs in the Amazon Lex V2 console or by using the [DescribeBotAlias](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_DescribeBotAlias.html) operation.
Log entries for a user utterance are found in multiple log streams. An utterance in the conversation has an entry in one of the log streams with the specified prefix. An entry in the log stream contains the following information:
message-version
The message schema version.
bot
Details about the bot that the customer is interacting with.
messages
The response that the bot sent back to the user.
utteranceContext
Information about processing this utterance.
+ `runtimeHints`—runtime context used to transcribe and interpret the user's input. For more information, see [Improving recognition of slot values with runtime hints in the conversation](using-hints.md).
+ `slotElicitationStyle`—Slot elicitation style used to interpret user input. For more information, see [Capturing slot values with spelling styles during the conversation](spelling-styles.md).
sessionState
The current state of the conversation between the user and the bot. For more information, see [Understanding Amazon Lex V2 bot conversations](managing-conversations.md).
interpretations
A list of intents that Amazon Lex V2 determined could satisfy the user's utterance. [Using confidence scores to improve conversation accuracy](confidence-scores.md).
interpretationSource
Indicates whether a slot is resolved by Amazon Lex V2 or Amazon Bedrock. Values: Lex \$1 Bedrock
sessionId
The identifier of the user session that is having the conversation.
inputTranscript
A transcription of the input from the user.
+ For text input, this is the text that the user typed. For DTMF input, this is the key that the user input.
+ For speech input, this is the text to which Amazon Lex V2 converts the user utterance in order to invoke an intent or fill a slot.
rawInputTranscript
The raw transcript of the user input before any text processing is applied. Note: Text processing is only for en-US and en-GB locales.
transcriptions
A list of potential transcriptions of the user's input. For more information, see [Using voice transcription confidence scores to improve conversations with your Lex V2 bot](using-transcript-confidence-scores.md).
rawTranscription
Using voice transcription confidence scores. For more information, see [Using voice transcription confidence scores to improve conversations with your Lex V2 bot](using-transcript-confidence-scores.md).
missedUtterance
Indicates whether Amazon Lex V2 was able to recognize the user's utterance.
requestId
Amazon Lex V2 generated request ID for the user input.
timestamp
The timestamp of the user's input.
developerOverride
Indicates whether the conversation flow was updated using a dialog code hook. For more information on using a dialog code hook, see [Integrating an AWS Lambda function into your Amazon Lex V2 bot](lambda.md).
inputMode
Indicates the type of input. Can be audio, DTMF, or text.
requestAttributes
The request attributes used when processing the user's input.
audioProperties
If audio conversation logs are enabled and the user input was in audio format, includes the total duration of the audio input, the duration of voice and the duration of silence in the audio. It also includes a link to the audio file.
bargeIn
Indicates whether the user input interrupted the previous bot response.
responseReason
The reason a response was generated. Can be one of:
+ `UtteranceResponse` – response to user input
+ `StartTimeout` – server generated response when the user didn't provide input
+ `StillWaitingResponse` – server generated response when the user requests the bot wait
+ `FulfillmentInitiated` – server generated response that fulfillment is about to be initiated
+ `FulfillmentStartedResponse` – server generated response that fulfillment has begun
+ `FulfillmentUpdateResponse` – periodic server generated response while fulfillment is in progress
+ `FulfillmentCompletedResponse` – server generated response when fulfillment is complete.
operationName
The API used to interact with the bot. Can be one of `PutSession`, `RecognizeText`, `RecognizeUtterance`, or `StartConversation`.
```
{
"message-version": "2.0",
"bot": {
"id": "string",
"name": "string",
"aliasId": "string",
"aliasName": "string",
"localeId": "string",
"version": "string"
},
"messages": [
{
"contentType": "PlainText | SSML | CustomPayload | ImageResponseCard",
"content": "string",
"imageResponseCard": {
"title": "string",
"subtitle": "string",
"imageUrl": "string",
"buttonsList": [
{
"text": "string",
"value": "string"
}
]
}
}
],
"utteranceContext": {
"activeRuntimeHints": {
"slotHints": {
"string": {
"string": {
"runtimeHintValues": [
{
"phrase": "string"
},
{
"phrase": "string"
}
]
}
}
}
},
"slotElicitationStyle": "string"
},
"sessionState": {
"dialogAction": {
"type": "Close | ConfirmIntent | Delegate | ElicitIntent | ElicitSlot",
"slotToElicit": "string"
},
"intent": {
"name": "string",
"slots": {
"string": {
"value": {
"interpretedValue": "string",
"originalValue": "string",
"resolvedValues": [ "string" ]
}
},
"string": {
"shape": "List",
"value": {
"originalValue": "string",
"interpretedValue": "string",
"resolvedValues": [ "string" ]
},
"values": [
{
"shape": "Scalar",
"value": {
"originalValue": "string",
"interpretedValue": "string",
"resolvedValues": [ "string" ]
}
},
{
"shape": "Scalar",
"value": {
"originalValue": "string",
"interpretedValue": "string",
"resolvedValues": [ "string" ]
}
}
]
}
},
"kendraResponse": {
// Only present when intent is KendraSearchIntent. For details, see
// https://docs.aws.amazon.com/kendra/latest/dg/API_Query.html#API_Query_ResponseSyntax
},
"state": "InProgress | ReadyForFulfillment | Fulfilled | Failed",
"confirmationState": "Confirmed | Denied | None"
},
"originatingRequestId": "string",
"sessionAttributes": {
"string": "string"
},
"runtimeHints": {
"slotHints": {
"string": {
"string": {
"runtimeHintValues": [
{
"phrase": "string"
},
{
"phrase": "string"
}
]
}
}
}
}
},
"dialogEventLogs": [
{
// only for conditional
"conditionalEvaluationResult":[
// all the branches until true
{
"conditionalBranchName": "string",
"expressionString": "string",
"evaluatedExpression": "string",
"evaluationResult": "true | false"
}
],
"dialogCodeHookInvocationLabel": "string",
"response": "string",
"nextStep": {
"dialogAction": {
"type": "Close | ConfirmIntent | Delegate | ElicitIntent | ElicitSlot",
"slotToElicit": "string"
},
"intent": {
"name": "string",
"slots": {
}
}
}
]
"interpretations": [
{
"interpretationSource": "Bedrock | Lex",
"nluConfidence": "string",
"intent": {
"name": "string",
"slots": {
"string": {
"value": {
"originalValue": "string",
"interpretedValue": "string",
"resolvedValues": [ "string" ]
}
},
"string": {
"shape": "List",
"value": {
"interpretedValue": "string",
"originalValue": "string",
"resolvedValues": [ "string" ]
},
"values": [
{
"shape": "Scalar",
"value": {
"interpretedValue": "string",
"originalValue": "string",
"resolvedValues": [ "string" ]
}
},
{
"shape": "Scalar",
"value": {
"interpretedValue": "string",
"originalValue": "string",
"resolvedValues": [ "string" ]
}
}
]
}
},
"kendraResponse": {
// Only present when intent is KendraSearchIntent. For details, see
// https://docs.aws.amazon.com/kendra/latest/dg/API_Query.html#API_Query_ResponseSyntax
},
"state": "InProgress | ReadyForFulfillment | Fulfilled | Failed",
"confirmationState": "Confirmed | Denied | None"
},
"sentimentResponse": {
"sentiment": "string",
"sentimentScore": {
"positive": "string",
"negative": "string",
"neutral": "string",
"mixed": "string"
}
}
}
],
"sessionId": "string",
"inputTranscript": "string",
"rawInputTranscript": "string",
"transcriptions": [
{
"transcription": "string",
"rawTranscription": "string",
"transcriptionConfidence": "number",
},
"resolvedContext": {
"intent": "string"
},
"resolvedSlots": {
"string": {
"name": "slotName",
"shape": "List",
"value": {
"originalValue": "string",
"resolvedValues": [
"string"
]
}
}
}
}
],
"missedUtterance": "bool",
"requestId": "string",
"timestamp": "string",
"developerOverride": "bool",
"inputMode": "DTMF | Speech | Text",
"requestAttributes": {
"string": "string"
},
"audioProperties": {
"contentType": "string",
"s3Path": "string",
"duration": {
"total": "integer",
"voice": "integer",
"silence": "integer"
}
},
"bargeIn": "string",
"responseReason": "string",
"operationName": "string"
}
```
The contents of the log entry depend on the result of a transaction and the configuration of the bot and request.
+ The `intent`, `slots`, and `slotToElicit` fields don't appear in an entry if the `missedUtterance` field is `true`.
+ The `s3PathForAudio` field doesn't appear if audio logs are disabled or if the `inputDialogMode`field is `Text`.
+ The `responseCard` field only appears when you have defined a response card for the bot.
+ The `requestAttributes` map only appears if you have specified request attributes in the request.
+ The `kendraResponse` field is only present when the `AMAZON.KendraSearchIntent` makes a request to search an Amazon Kendra index.
+ The `developerOverride` field is true when an alternative intent was specified in the bot's Lambda function.
+ The `sessionAttributes` map only appears if you have specified session attributes in the request.
+ The `sentimentResponse` map only appears if you configure the bot to return sentiment values.
**Note**
The input format may change without a corresponding change in the `messageVersion`. Your code should not throw an error if new fields are present.
# Accessing audio logs in Amazon S3
Amazon Lex V2 stores audio logs for your conversations in an S3 bucket.
You can use the Amazon S3 console or API to access audio logs. You can see the S3 object key prefix of the audio files in the Amazon Lex V2 console, or in the `conversationLogSettings` field in the `DescribeBotAlias` operation response.
# Monitoring conversation log status with CloudWatch metrics
Use Amazon CloudWatch to monitor delivery metrics of your conversation logs. You can set alarms on metrics so that you are aware of issues with logging if they should occur.
Amazon Lex V2 provides four metrics in the `AWS/Lex` namespace for conversation logs:
+ `ConversationLogsAudioDeliverySuccess`
+ `ConversationLogsAudioDeliveryFailure`
+ `ConversationLogsTextDeliverySuccess`
+ `ConversationLogsTextDeliveryFailure`
The success metrics show that Amazon Lex V2 has successfully written your audio or text logs to their destinations.
The failure metrics show that Amazon Lex V2 couldn't deliver audio or text logs to the specified destination. Typically, this is a configuration error. When your failure metrics are above zero, check the following:
+ Make sure that Amazon Lex V2 is a trusted entity for the IAM role.
+ For text logging, make sure that the CloudWatch Logs log group exists. For audio logging, make sure that the S3 bucket exists.
+ Make sure that the IAM role that Amazon Lex V2 uses to access the CloudWatch Logs log group or S3 bucket has write permission for the log group or bucket.
+ Make sure that the S3 bucket exists in the same region as the Amazon Lex V2 bot and belongs to your account.
# Obscuring slot values in conversation logs from Lex V2
Amazon Lex V2 enables you to obfuscate, or hide, the contents of slots so that the content is not visible. To protect sensitive data captured as slot values, you can enable slot obfuscation to mask those values for logging.
When you choose to obfuscate slot values, Amazon Lex V2 replaces the value of the slot with the name of the slot in conversation logs. For a slot called `full_name`, the value of the slot would be obfuscated as follows:
```
Before:
My name is John Stiles
After:
My name is {full_name}
```
If an utterance contains bracket characters (\$1\$1) Amazon Lex V2 escapes the bracket characters with two back slashes (\$1\$1). For example, the text `{John Stiles}` is obfuscated as follows:
```
Before:
My name is {John Stiles}
After:
My name is \\{{full_name}\\}
```
Slot values are obfuscated in conversation logs. The slot values are still available in the response from the `RecognizeText` and `RecognizeUtterance` operations, and the slot values are available to your validation and fulfillment Lambda functions. If you are using slot values in your prompts or responses, those slot values are not obfuscated in conversation logs.
In the first turn of a conversation, Amazon Lex V2 obfuscates slot values if it recognizes a slot and slot value in the utterance. If no slot value is recognized, Amazon Lex V2 does not obfuscate the utterance.
On the second and later turns, Amazon Lex V2 knows the slot to elicit and if the slot value should be obfuscated. If Amazon Lex V2 recognizes the slot value, the value is obfuscated. If Amazon Lex V2 does not recognize a value, the entire utterance is obfuscated. Any slot values in missed utterances won't be obfuscated.
Amazon Lex V2 also doesn't obfuscate slot values that you store in request or session attributes. If you are storing slot values that should be obfuscated as an attribute, you must encrypt or otherwise obfuscate the value.
Amazon Lex V2 doesn't obfuscate the slot value in audio. It does obfuscate the slot value in the audio transcription.
You can choose which slots to obfuscate by using the console or by using the Amazon Lex V2 API. In the console, choose **Slot obfuscation** in the settings for a slot. If you are using the API, set the `obfuscationSetting` field of the slot to `DEFAULT_OBFUSCATION` when you call the [CreateSlot](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_CreateSlot.html) or [UpdateSlot](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_UpdateSlot.html) operation.
# Selective conversation log capture in Lex V2
The selective conversation log capture allows the user to select how conversation logs are captured with text and audio data from the live conversations.
To enable and capture the output of the selective conversation log capture feature, you must activate the feature in the Amazon Lex V2 console, and enable the required session attributes in the API settings to capture the selected output from the logs.
You can select the following options for the selective conversation log capture:
+ text only
+ audio only
+ text and audio
You can capture specific parts of the conversation, and choose if audio, text, or both are captured for the conversation log.
**Note**
Selective conversation log capture works for Amazon Lex V2 only.
**Topics**
+ [Manage selective conversation log capture](manage-selective-logging.md)
+ [Example of selective conversation log capture](example-selective-logging.md)
# Manage selective conversation log capture
Using the Lex console, you can enable the selective conversation log capture settings and choose which slots you want to enable selective conversation log capture capture for.
**Activate selective conversation log capture in the Amazon Lex V2 console:**
1. Sign in to the AWS Management Console and open the Amazon Lex V2 console at [https://console.aws.amazon.com/lexv2/home](https://console.aws.amazon.com/lexv2/home).
1. Select **Bots** from the left side panels and choose the bot you want to enable the selective conversation log capture. Use an existing bot or create a new one.
1. Choose **Aliases** for your selected bot under the **Deployment** section on the left side panel.
1. Choose your bot’s Alias, then select **Manage conversation logs**.
1. In the **Manage conversation logs** panel, for **Text logs**, choose whether text logs are enabled or disable by selecting the radio button. If you choose **Enabled** for text logs, then you will need to enter a **Log group name** or choose an existing log group name from the drop down menu. Select the check box for **Selectively log utterances** if you are selectively logging text files.
**Note**
Enable text and/or audio logs by selecting the **Selectively log utterances** check box in the **conversation logs settings** (text and/or audio) in build time **BotAlias** settings. You must configure the CloudWatch log group and Amazon S3 bucket to select this option.
1. In the **Audio logs** section, choose whether audio logs are enabled or disable by selecting the radio button. If you choose **Enabled** for audio logs, you need to specify the Amazon S3 bucket location and (optional) the KMS key for encrypting your audio data. Select the check box for **Selectively log utterances** if you are selectively logging audio files.
![\[The panels for managing conversation logs.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/logging/selective-manage-logs.png)
1. Select **Save** in the bottom right corner of the panel to save your selective conversation log capture settings.
**Activate selective conversation log capture in the Amazon Lex V2 console:**
1. Go to **Intents** and select the **Intent name**, **Initial Response**, **Advanced Settings**, **Set Values**, **Session Attributes**.
1. Set the following attributes to based on the intents and slots for which you want to enable selective conversation log capture:
+ `x-amz-lex:enable-audio-logging:intent:slot = "true"`
+ `x-amz-lex:enable-text-logging:intent:slot = "true"`
![\[The panels for adding selective session attributes.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/logging/selective-session-attributes.png)
**Note**
Set `x-amz-lex:enable-audio-logging:intent:slot = "true"` to capture utterances that contain only a specific slot in the conversation. The action to log an utterance depends on the assessment of *intent *:*slot* within the utterance, in comparison to the session attribute expressions, and the corresponding flag value. To log an utterance, at least one expression in the session attribute must allow it, with the enable logging flag set to `true`. The value of *intent* and *slot* can be `"*"` as well. If the slot and/or intent value is `"*"`, it means that any slot and/or intent value of `"*"` will match with it. Similar to `x-amz-lex:enable-audio-logging`, a new session attribute called `x-amz-lex:enable-text-logging` will be used to control text logs.
1. Select **Update options** and build the bot to include the updated settings.
**Note**
Your IAM role must have access permission to allow you to write data to the Amazon S3 bucket and to use a KMS key to encrypt the data. Lex will update your IAM role with Lex permissions to access CloudWatch Logs log group and the selected Amazon S3 bucket.
**Guidelines for using selective conversation log capture:**
You can only enable selective conversation log capture for text and/or audio logs, when you have enabled text and/or audio logs in the **Conversation log settings**. By enabling selective conversation log capture for text and/or audio logs, you disable logging for all intents and slots in the conversation. To generate text and/or audio logs for particular intents and slots, you must set text or/and audio selective conversation log capture session attributes for those intents and slots to "true".
+ If selective conversation log capture is enabled, and no session attributes with the prefix x-amz-lex:enable-audio-logging are present, logging will be disabled by default for all the utterances. This scenario is also true regarding x-amz-lex:enable-text-logging.
+ Utterance logs will be stored exclusively for the segments of text and/or audio conversation if at least one expression in the session attribute allows it.
+ The configurations for selective conversation log capture of text and/or audio, as defined in session attributes, will be effective only when selective conversation log capture for text and/or audio is enabled in Conversation Log Settings within bot alias; otherwise, session attributes will be disregarded.
+ When selective conversation log capture is enabled, any slot values in SessionState, Interpretations, and Transcriptions for which logging is not enabled using session attributes will be obfuscated in the generated text log.
+ The decision to produce audio and/or text logs is evaluated by matching the slot elicited by the bot with the selective conversation log capture session attributes, except for the intent elicitation turn where user can provide slot values along with intent elicitation. In an intent elicitation turn, the slots filled in current turn are matched against the selective conversation log capture session attributes.
+ The slots that are considered filled are derived from the session state at the end of the turn. Therefore, any alterations made by the Dialog Codehook Lambda to the slots in session state will influence the behavior of selective conversation log capture.
+ In an intent elicitation turn, if multiple slot values are given by the user, the text and/or audio log will only get generated if the text/audio session attributes allow logging for all the slots filled in this turn.
+ The recommended operational approach is to set the selective conversation log capture session attribute at the beginning of the session and to refrain from modifying it during the session.
+ If any slots contain sensitive data, you should always enable slot obfuscation.
# Example of selective conversation log capture
Here is an example of a business use case for selective conversation log capture.
**Use case:**
A fintech company utilizes an Amazon Lex V2 bot to support their IVR system, which allows users to make bill payments. In order to meet compliance and auditing requirements, they must retain audio recordings of user-provided authorization consent. However, enabling general audio logs is not feasible as it would make them non-compliant, because it is not possible to obfuscate sensitive slots like CardNumber, CVV, and other information in the audio logs. Instead, they can enable selective conversation log capture for audio logs and set session attribute to only produce audio logs for utterance that has authorization consent.
**BotAlias Settings:**
+ Text Logs Enabled : true
+ Text Logs Selective Logging Enabled : false
+ Audio Logs Enabled : true
+ Audio Logs Selective Logging Enabled : true
**Session Attributes:**
`x-amz-lex:enable-audio-logging:PayBill:AuthorizationConsent = "true"`
**Sample Conversation:**
+ User (Audio Input): "I want to pay my bill with bill number 35XU68."
+ Bot: "What is the due amount in dollars?"
+ User (Audio Input): "235."
+ Bot: "What is your credit card number?"
+ User (Audio Input): "9239829722200348."
+ Bot: "You're paying 235 dollars using your credit card number ending with 0348. Please say 'I authorize to pay 235 dollars.'"
+ User (Audio Input): "I authorize to pay 235 dollars."
+ Bot: "Your bill has been paid."
**Conversation Logs output: **
In this situation, text logs will be produced for all turns. However, audio logs will only be recorded for the particular turn when the **AuthorizationConsent** slot within the **PayBill** intent was elicited, and no audio logs will be produced for any other turn.
# Logging errors with error logs in Lex V2
You enable *error logs* to store bot interactions. You can use these error logs to review the performance of your bot and to troubleshoot errors with conversations.
Error logs are configured for an version. Each version can have different settings for their error logs. Text logs store text input in CloudWatch Logs. You can enable encryption of text logs using AWS KMS customer managed CMKs.
## IAM Policies for Error Logs
Depending on the type of logging that you select, Amazon Lex V2 requires permission to use Amazon CloudWatch Logs and Amazon Simple Storage Service (S3) buckets to store your logs. You must create AWS Identity and Access Management roles and permissions to enable Amazon Lex V2 to access these resources.
### Creating an IAM Role and Policies for Error Logs
To enable conversation logs, you must grant write permission for CloudWatch Logs and Amazon S3. If you enable object encryption for your S3 objects, you need to grant access permission to the AWS KMS keys used to encrypt the objects.
You can use the IAM console, the IAM API, or the AWS Command Line Interface to create the role and policies. These instructions use the AWS CLI to create the role and policies.
****To create an IAM role for error logs
The IAM role that you use to enable conversation logs must have the iam:PassRole permission. The following policy should be attached to the role:
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::111122223333:role/role"
}
]
}
```
------
## Enabling Error Logs in Lex V2
To turn on error logs using the Amazon Lex V2 console:
1. Sign in to the AWS Management Console and open the Amazon Lex console at [https://console.aws.amazon.com/lex/](https://console.aws.amazon.com/lex/).
1. From the list of **Bots**, choose the bot you want to enable for error logs.
1. From the left menu, choose **Version**.
1. In the list of **Version**, choose the Version for which you want to configure error logs.
1. In the **Version detail** section, choose **Enable**.
1. Choose **Save** to start logging conversations. If necessary, Amazon Lex V2 will update your service role with permissions to access the CloudWatch Logs log group.
## Disabling Error Logs in Lex V2
To turn off error logs using the Amazon Lex V2 console:
1. Sign in to the AWS Management Console and open the Amazon Lex console at [https://console.aws.amazon.com/lex/](https://console.aws.amazon.com/lex/).
1. From the list of **Bots**, choose the bot you want to enable for error logs.
1. From the left menu, choose **Version**.
1. In the list of **Version**, choose the Version for which you want to configure error logs.
1. In the **Version detail** section, choose **Disable**.
1. Choose **Save** to stop logging conversations.
# Monitoring operational metrics in Lex V2
Amazon CloudWatch and AWS CloudTrail are two AWS services that integrate with Amazon Lex V2 to help you monitor user interactions with your bot. Use these services to record actions, send near real-time data, and set up notifications and automated actions when criteria are met.
**Topics**
+ [Measuring operational metrics with Amazon CloudWatch](monitoring-cloudwatch.md)
+ [Viewing events with AWS CloudTrail for Lex V2](logging-using-cloudtrail.md)
# Measuring operational metrics with Amazon CloudWatch
You can monitor Amazon Lex V2 using CloudWatch, which collects raw data and processes it into readable, near real-time metrics. These statistics are kept for 15 months, so that you can access historical information and gain a better perspective on how your web application or service is performing. You can also set alarms that watch for certain thresholds, and send notifications or take actions when those thresholds are met. For more information, see the [Amazon CloudWatch User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/).
The Amazon Lex V2 service reports the following metrics in the `AWS/Lex` namespace.
| Metric | Description |
| --- | --- |
| `AssistedSlotResolutionModelAccessDeniedErrorCount` | The number of times that Amazon Lex V2 was denied access to Amazon Bedrock Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for `RecognizeText`: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `AssistedSlotResolutionModelInvocationCount` | The number of times that Amazon Bedrock was invoked. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for `RecognizeText`: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `AssistedSlotResolutionModelSystemErrorCount` | The number of times that a 5xx occurred when calling Amazon Bedrock. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for `RecognizeText`: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `AssistedSlotResolutionModelThrottlingErrorCount` | The number of times that Amazon Lex was throttled by Amazon Bedrock. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for `RecognizeText`: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `AssistedSlotResolutionResolvedSlotCount` | The number of times that Amazon Bedrock returned a slot value. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for `RecognizeText`: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `KendraIndexAccessError` | The number of times that Amazon Lex V2 could not access your Amazon Kendra index. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `KendraLatency` | The amount of time that it takes Amazon Kendra to respond to a request from the `AMAZON.KendraSearchIntent`. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Milliseconds |
| `KendraSuccess` | The number of times that Amazon Lex V2 couldn't access your Amazon Kendra index. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `KendraSystemErrors` | The number of times that Amazon Lex V2 couldn't query the Amazon Kendra index. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `KendraThrottledEvents` | The number of times Amazon Kendra throttled requests from the `AMAZON.KendraSearchIntent`. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeConcurrency` | The number of concurrent connections in the specified time period. `RuntimeConcurrency` is reported as a `StatisticSet`. Valid dimensions for the `RecognizeUtterance` or `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for other operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeInvalidLambdaResponses` | The number of invalid AWS Lambda responses in the specified period. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeLambdaErrors` | The number of Lambda runtime errors in the specified time period. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimePollyErrors` | The number of invalid Amazon Polly responses in the specified time period. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeRequestCount` | The number of runtime requests in the specified time period. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for other operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeRequestLength` | Total length of a conversation with an Amazon Lex V2 bot. Only applicable to the [StartConversation](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_runtime_StartConversation.html) operation. Valid dimensions: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: milliseconds |
| `RuntimeSucessfulRequestLatency` This metric is `RuntimeSucessfulRequestLatency` and not `RuntimeSuccessfulRequestLatency`. | The latency for successful requests between the time the request was made and the response was passed back. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for other operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: milliseconds |
| `RuntimeSystemErrors` | The number of system errors in the specified period. The response code range for a system error is 500 to 599. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for other operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeThrottledEvents` | The number of throttled events. Amazon Lex V2 throttles an event when it receives more requests than the limit of transactions per second set for your account. If the limit set for your account is frequently exceeded, you can request a limit increase. To request an increase, see [AWS service limits](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html). Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for other operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
| `RuntimeUserErrors` | The number of user errors in the specified period. The response code range for a user error is 400 to 499. Valid dimensions for the `RecognizeUtterance` and `StartConversation` operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Valid dimensions for other operations: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/monitoring-cloudwatch.html) Unit: Count |
The following dimensions are supported for the Amazon Lex V2 metrics.
| Dimension | Description |
| --- | --- |
| Operation | The name of the Amazon Lex V2 operation – `RecognizeText`, `RecognizeUtterance`, `StartConversation`, `GetSession`, `PutSession`, `DeleteSession` – that generated the entry. |
| BotId | The alphanumeric unique identifier for the bot. |
| BotAliasId | The alphanumeric unique identifier for the bot alias. |
| BotVersion | The numeric version of the bot. |
| InputMode | The type of input to the bot – speech, text, or DTMF. |
| LocaleId | The identifier of the bot's locale, such as en-US or fr-CA. |
| Model | Indicates the model id of the Amazon Bedrock large language model. |
| ModelType | Indicates the type of large language model invoked from Amazon Bedrock. |
# Viewing events with AWS CloudTrail for Lex V2
Amazon Lex V2 is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, role, or an AWS service in Amazon Lex V2. CloudTrail captures API calls for Amazon Lex V2 as events. The calls captured include calls from the Amazon Lex V2 console and code calls to the Amazon Lex V2 API operations. If you create a trail, you can enable continuous delivery of CloudTrail events to an Amazon S3 bucket, including events for Amazon Lex V2. If you don't configure a trail, you can still view the most recent events in the CloudTrail console in **Event history**. Using the information collected by CloudTrail, you can determine the request that was made to Amazon Lex V2, the IP address from which the request was made, who made the request, when it was made, and additional details.
To learn more about CloudTrail, see the [AWS CloudTrail User Guide](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html).
## Amazon Lex V2 information in CloudTrail
CloudTrail is enabled on your AWS account when you create the account. When activity occurs in Amazon Lex V2, that activity is recorded in a CloudTrail event along with other AWS service events in **Event history**. You can view, search, and download recent events in your AWS account. For more information, see [Viewing events with CloudTrail Event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).
For an ongoing record of events in your AWS account, including events for Amazon Lex V2, create a trail. A *trail* enables CloudTrail to deliver log files to an Amazon S3 bucket. By default, when you create a trail in the console, the trail applies to all AWS Regions. The trail logs events from all Regions in the AWS partition and delivers the log files to the Amazon S3 bucket that you specify. Additionally, you can configure other AWS services to further analyze and act upon the event data collected in CloudTrail logs. For more information, see the following:
+ [Overview for creating a trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html)
+ [CloudTrail supported services and integrations](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-aws-service-specific-topics.html)
+ [Configuring Amazon SNS notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-sns-notifications-for-cloudtrail.html)
+ [Receiving CloudTrail log files from multiple regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/receive-cloudtrail-log-files-from-multiple-regions.html) and [Receiving CloudTrail log files from multiple accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html)
Amazon Lex V2 supports logging for all of the actions listed in [Model Building API V2](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_Operations_Amazon_Lex_Model_Building_V2.html).
Every event or log entry contains information about who generated the request. The identity information helps you determine the following:
+ Whether the request was made with root or AWS Identity and Access Management IAM user credentials.
+ Whether the request was made with temporary security credentials for a role or federated user.
+ Whether the request was made by another AWS service.
For more information, see the [CloudTrail userIdentity element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html).
## Understanding Amazon Lex V2 log file entries
A trail is a configuration that enables delivery of events as log files to an Amazon S3 bucket that you specify. CloudTrail log files contain one or more log entries. An event represents a single request from any source and includes information about the requested action, the date and time of the action, request parameters, and so on. CloudTrail log files aren't an ordered stack trace of the public API calls, so they don't appear in any specific order.
The following example shows a CloudTrail log entry that demonstrates the [CreateBotAlias](https://docs.aws.amazon.com/lexv2/latest/APIReference/API_CreateBotAlias.html) action.
```
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "ID of caller:temporary credentials",
"arn": "arn:aws:sts::111122223333:assumed-role/role name/role ARN",
"accountId": "111122223333",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "ID of caller",
"arn": "arn:aws:iam::111122223333:role/role name",
"accountId": "111122223333",
"userName": "role name"
},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "creation date"
}
}
},
"eventTime": "event timestamp",
"eventSource": "lex.amazonaws.com",
"eventName": "CreateBotAlias",
"awsRegion": "Region",
"sourceIPAddress": "192.0.2.0",
"userAgent": "user agent",
"requestParameters": {
"botAliasLocaleSettingsMap": {
"en_US": {
"enabled": true
}
},
"botId": "bot ID",
"botAliasName": "bot aliase name",
"botVersion": "1"
},
"responseElements": {
"botAliasLocaleSettingsMap": {
"en_US": {
"enabled": true
}
},
"botAliasId": "bot alias ID",
"botAliasName": "bot alias name",
"botId": "bot ID",
"botVersion": "1",
"creationDateTime": creation timestamp
},
"requestID": "unique request ID",
"eventID": "unique event ID",
"readOnly": false,
"eventType": "AwsApiCall",
"recipientAccountId": "111122223333"
}
```
# Evaluating Lex V2 bot performance with the Test Workbench
To improve bot performance, you can evaluate the performance of your bots at scale. The results for your test evaluation are displayed in simple tables and charts.
You can use the Test Workbench to create reference test sets that use existing transcription data. You can test bots to evaluate performance before deployment, and view test result breakdowns at scale.
![\[The work flow diagram to improve bot accuracy with the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-workflow.png)
Users can use the Test Workbench to establish baseline performance for bots. This covers intent and slot performance for utterances that are in the form of single-inputs or conversations. Once a test set is successfully loaded, you can run it against your existing pre-production or production bots. The Test Workbench helps you identify opportunities for improved slot filling and intent classification.
**Topics**
+ [Generate a test set for Test Workbench](test-sets.md)
+ [Manage test sets](manage-test-sets.md)
+ [Execute a test](execute-test-set.md)
+ [Test set coverage in Test Workbench](validation-test-set.md)
+ [View test results](test-results-test-set.md)
+ [Test results details in Test Workbench](test-results-details-test-set.md)
# Generate a test set for Test Workbench
You can create a test set to evaluate the performance of your bot. Generate a test set by uploading a test set that is in a CSV file format or by generating a test set from [ conversation logs](https://docs.aws.amazon.com/lexv2/latest/dg/conversation-logs.html). The test set can contain audio or text input.
![\[Create a test set with the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-create.png)
If a test set creates validation errors, remove the test set and replace it with another list of test set data, or edit the data in the CSV file by using a spreadsheet editing program.
**To create a test set:**
1. Sign in to the AWS Management Console and open the Amazon Lex console at [https://console.aws.amazon.com/lex/](https://console.aws.amazon.com/lex/).
1. Choose **Test workbench** from the left side panel.
1. Select **Test sets** from the options under Test workbench.
1. Select the **Create test set** button on the console.
1. In the **Details**, enter a test set name and an optional description.
1. Select **Generate a baseline test set**.
1. Select **Generate from conversation logs**.
1. Select **Bot name**, **Bot Alias**, and **Language** from the drop down menus.
1. If you are generating a baseline test from a conversation log, choose **Time range** and **IAM role**, if required. You can create a role with the basic Amazon Lex V2 permissions or use an existing role.
1. Choose a modality of **Audio** or **Text** for the test set you are creating. NOTE: The Test Workbench can import text files up to 50k, and up to 5 hours of audio.
1. Select an Amazon S3 location to store your test results, and add an optional KMS key to encrypt output transcripts.
1. Select **Create**.
**To upload an existing test set in a CSV file format, or to update the test set:**
1. Choose **Test workbench** from the left side panel.
1. Select **Test sets** from the options under Test workbench.
1. Select **Upload a file to this test set** on the console.
1. Choose **Upload from Amazon Amazon S3 bucket** or **Upload from your computer**. NOTE: You can upload a CSV file created from a template. Click **CSV template** to download a zip file that contains the templates.
1. Choose **Create a role with basic Amazon Lex permissions** or **Use an existing role for Role ARN**.
1. Choose a modality of **Audio** or **Text** for the test set you are creating. NOTE: The Test Workbench can import text files up to 50k, and up to 5 hours of audio.
1. Select an Amazon S3 location to store your test results, and add an optional KMS key to encrypt output transcripts.
1. Select **Create**.
If the operation is successful, the confirmation message will indicate that the test set is ready to test, and the status will display **Ready for testing**.
# Tips for creating a successful test set
+ You can create an IAM role for the Test Workbench in the console, or you can configure your IAM role step-by-step. For more information, see [ Create an IAM role for the Test Workbench](https://docs.aws.amazon.com/lexv2/latest/dg/create-iam-test-set.html).
+ Before executing a test, validate the test set and the bot definition for any inconsistencies using the **Validate discrepancy** button. If the intent and slot naming conventions used in the test set are consistent with the bot, proceed to execute the test. If any anomalies are identified, revise the test set, update the test set, and choose **Validate discrepancy**. Repeat this sequence again until no inconsistencies are noted, then execute the test.
+ The Test Workbench can test with different slot value formats in the **Expected Output Slot** column. For any built-in slot, you can choose the value provided in the user input (for example, Date = tomorrow), or provide its absolute resolved value (for example, Date = 2023-03-21). For more information around built-in slots and their absolute values, see [ Built-in slots](https://docs.aws.amazon.com/lexv2/latest/dg/howitworks-builtins-slots.html).
+ For consistency and readability in the **Expected Output Slot** columns, follow the convention of "SlotName = SlotValue" (for example, AppointmentType = cleaning) with a space before and after the equal sign.
+ If the bot includes composite slots, in **Expected Output Slot** define subslots to the slot name, separated by a period (for example, “Car.Color”). No other syntax and punctuation will work.
+ If the bot includes multi-value slots, in **Expected Output Slot** provide multiple slot values, separated by a comma ("FlowerType = roses, lilies"). No other syntax and punctuation will work.
+ Make sure that the test set is created from valid conversation logs.
+ Slot:slot value will be in the same column after the intent columns in the CSV format.
+ DTMF input from a User turn is interpreted as an expected transcription and does not list an Amazon S3 location.
# Creating a test case within a test set using Test Workbench
The Test Workbench results are dependent on the bot definition and its corresponding test set. You can generate a test set with the information from the bot definition to pinpoint areas that need improvement. Create a test dataset with examples that you suspect (or know) will be challenging for the bot to interpret correctly considering the current bot design and your knowledge of your customer conversations.
Review your intents based on learnings from your production bot on a regular basis. Continue to add to and adjust the bot’s sample utterances and slot values. Consider improving slot resolution by using the available options, such as runtime hints. The design and development of your bot is an iterative process that is a continuous cycle.
Here are some other tips for optimizing your test set:
+ Select the most common use cases with frequently used intents and slots in the test set.
+ Explore different ways a customer could refer to your intents and slots. This can include user inputs in the forms of statements, questions, and commands that vary in length from minimal to extended.
+ Include user inputs with a varied number of slots.
+ Include commonly used synonyms or abbreviations of custom slot values supported by your bot (for example, “root canal”, “canal”, or “RC”).
+ Include variations of built-in slot values (for example, “tomorrow”, “asap”, or "the next day").
+ Examine the bot robustness for spoken modality by collecting user inputs that can be misinterpreted (for example, “ink”, “ankle”, or "anchor").
# Creating a test set from a CSV file for Test Workbench
You can create a test set from the CSV file template provided in the Amazon Lex V2 console by entering the values directly by using a CSV spreadsheet editor. The test set is a comma-separated value (CSV) file consisting of single user utterances and multi-turn conversations recorded in the following columns:
+ **Line \$1** – this column is an incremental counter that keeps track of the total filled rows to test.
+ **Conversation \$1** – this column tracks the number of turns in a conversations. For single inputs, this column can be left empty, filled with "-" or "N/A". For conversations, each turn within a conversations will be assigned the same conversation number.
+ **Source** – this column is set to "User" or "Agent". For single inputs, it will be always set to "User".
+ **Input** – this column includes the user utterance or the bot prompts.
+ **Expected Output Intent** – this column captures the intent fulfilled in the input.
+ **Intent Expected Output Slot 1** – this column captures the first slot elicited in the user input. The test set should include a column called Expected Output Slot X for each slot in the user input.
Example of a test set with single inputs:
| Line \$1 | Conversation \$1 | Source | Input | Expected Output Intent | Expected Output Slot 1 | Expected Output Slot 2 |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | | User | book a cleaning appointment tomorrow | MakeAppointment | AppointmentType = cleaning | Date = tomorrow |
| 2 | N/A | User | book a cleaning appointment on April 15th | MakeAppointment | AppointmentType = cleaning | Date = 4/15/23 |
| 3 | N/A | User | book appointment for December first | MakeAppointment | Date = December first | |
| 4 | N/A | User | book a cleaning appointment | MakeAppointment | AppointmentType = cleaning | |
| 1 | | User | Can you help me book an appointment? | MakeAppointment | | |
Example of a test set with conversations
| Line \$1 | Conversation \$1 | Source | Input | Expected Output Intent | Expected Output Slot 1 | Expected Output Slot 2 | Expected Output Slot 3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 1 | User | book an appointment | MakeAppointment | | | |
| 2 | 1 | Agent | What type of appointment would you like to schedule? | MakeAppointment | | | |
| 3 | 1 | User | cleaning | MakeAppointment | AppointmentType = cleaning | | |
| 4 | 1 | Agent | When should I schedule your appointment? | MakeAppointment | | | |
| 5 | 1 | User | tomorrow | MakeAppointment | | Date = tomorrow | |
| 6 | 2 | User | book a root canal appointment today | MakeAppointment | AppointmentType = root canal | Date = today | |
| 7 | 2 | Agent | At what time should I schedule your appointment? | MakeAppointment | | | |
| 8 | 2 | User | eleven a.m. | MakeAppointment | | | Time = eleven a.m. |
# Create an IAM role for the Test Workbench
**To create an IAM role for the Test Workbench**
1. Follow the steps at [ Create an IAM user](https://docs.aws.amazon.com/lexv2/latest/dg/gs-account.html#gs-account-user) to create an IAM user which can be used to access test-workbench console.
1. Select the **Create role** button.
![\[The roles screen in the IAM console.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-iam1.png)
1. Select the option for **Custom trust policy**.
![\[Select trusted entity\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-iam2.png)
1. Enter the trust policy below and click **Next**.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "sid4",
"Effect": "Allow",
"Principal": {
"Service": "lexv2.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
```
------
1. Select the **Create policy** button.
1. A new tab will open in your browser where you can enter the below policy and click on **Next: Tags** button.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:FilterLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"lex:*"
],
"Resource": "*"
}
]
}
```
------
1. Enter a policy name, for example ‘LexTestWorkbenchPolicy’ and then click on the **Create Policy**.
1. Return to the previous tab in your browser and Refresh list of policies by clicking the **Refresh** button as shown below.
![\[Refresh the screen to see the new policy.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/testworkbench-iam3.png)
1. Search in list of policies by entering policy name that you used in the 6th step and choose the policy.
1. Select the **Next** button.
1. Enter role name and then click the **Create Role** button.
1. Choose your new IAM role when prompted in the Amazon Lex V2 console for Test Workbench.
# Create an IAM role for the Test Workbench - Advanced Features
**Permission setup for Test workbench IAM role**
This section shows several example AWS Identity and Access Management (IAM) identity-based policies to implement least-privilege access controls for Test Workbench permissions.
1. **Policy for Test Workbench to read audio files in S3** – This policy enables Test Workbench to read audio files being used in the test sets. The below policy should be accordingly modified to update *S3BucketName* and *S3Path* to point them to an Amazon S3 location of the audio files in a test set.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "TestWorkbenchS3AudioFilesReadOnly",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::S3BucketName/S3Path/*"
]
}
]
}
```
------
1. **Policy for Test Workbench to read and write test sets and results into an Amazon S3 bucket** – This policy enables Test Workbench to store the test set inputs and results. The below policy should be modified to update *S3BucketName* to the Amazon S3 Bucket where test-set data will be stored. Test Workbench stores these data exclusively in your Amazon S3 bucket and not in the Lex Service infrastructure. Therefore For this reason, Test Workbench requires access to your Amazon S3 bucket to function properly.
------
#### [ JSON ]
****
```
{
"Version":"2012-10-17",
"Statement": [
{
"Sid": "TestSetDataUploadWithEncryptionOnly",
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set/*",
"arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_execution/*",
"arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set_discrepancy_report/*"
],
"Condition": {
"StringEquals": {
"s3:x-amz-server-side-encryption": "aws:kms"
}
}
},
{
"Sid": "TestSetDataGetObject",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set/*",
"arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_execution/*",
"arn:aws:s3:::S3BucketName/*/lex_testworkbench/test_set_discrepancy_report/*"
]
},
{
"Sid": "TestSetListS3Objects",
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::S3BucketName"
]
}
]
}
```
------
1. **Policy for Test Workbench to read CloudWatch Logs** – This policy enables Test Workbench to generate test-sets from Lex Conversation Text Logs stored in Amazon CloudWatch Logs. The below policy should be modified to update *Region*, *AwsAccountId*, *LogGroupName*.
1. **Policy for Test Workbench to call Lex Runtime** – This policy enables Test Workbench to execute a test set against Lex bots. The below policy should be modified to update *Region*, *AwsAccountId*, *BotId*. Since Test Workbench can test any bot in your Lex environment, you can replace the resource with "arn:aws:lex:*Region*:*AwsAccountId*:bot-alias/\$1" to allow Test Workbench access to all Amazon Lex V2 bots in an account.
1. **(Optional) Policy for Test Workbench to encrypt and decrypt test set data** – If Test Workbench is configured to store test-set inputs and results in Amazon S3 buckets using a customer managed KMS key, Test Workbench will need both encryption and decryption permission to the KMS key. The below policy should be modified to update *Region*, *AwsAccountId*, and *KmsKeyId* where *KmsKeyId* is the ID of the customer managed KMS key.
1. **(Optional) Policy for Test Workbench to decrypt audio files** – If Audio files are stored in the S3 bucket using customer managed KMS key, Test Workbench will need decryption permission to the KMS keys. The below policy should be modified to update *Region*, *AwsAccountId*, and *KmsKeyId* where *KmsKeyId* is the ID of the customer managed KMS key used to encrypt the audio files.
# Manage test sets
You can download, update, and delete test sets from the test set window. Or you can use the list of available test sets to edit or manually annotate your test set file. Then, upload it again to retry validation, due to errors or other input issues.
**To download the test set file from test set record:**
1. Select the name of the test set from the list of test sets.
1. In the test set record window, select the **Download** button on the right side of the screen in the **Test Inputs** section.
1. if there are any validation error details at the top of the window regarding the test set, select the **Download** button. The file will be saved to your Downloads folder. You can fix the validation errors in the test set from the error messages in the test set CSV file. Find the error identified in the validation step, fix the line or remove it, and upload the file to retry the validation step.
1. if you successfully download the test set, a green banner messages will appear.
**To download a test set from the list of test sets:**
1. From the list of test sets, select the radio button next to the test set item you want to download.
1. From the Action menu at the top right, choose **Download**.
1. A green banner message will indicate if you successfully have downloaded the test set. The file will be saved to your Downloads folder.
# Test set columns supported by Test Workbench
Below is the complete list of test set columns supported by Test Workbench and the instruction on how to use them with Amazon Lex V2.
| Column Name | Test set type | Value Type | Multiple Columns Support | Description |
| --- | --- | --- | --- | --- |
| Line Number | Text and Audio | Number | No | This is a user column which is ignored by Amazon Lex V2. It is intended to help a test set author to sort and filter the test-set rows. "Line \$1" can be used as an alternative column name. |
| Conversation Number | Text and Audio | Number | No | This column allows you to put rows in a conversation together. "Conversation \$1" can be used as an alternative column name. |
| Source | Text and Audio | Enum ("User", "Agent") | No | The value in this column indicates if the row is for a user or an agent. "Conversation Participant" can be used as an alternative column name. |
| Input | Text | String | No | This column is used to add the transcript for text test set. Text input should be used in User rows. The Agent prompt should be used in Agent rows. |
| Expected Transcription | Audio | String | No | This column is used to add the transcript for the audio test set. Expected transcription of the audio file should be used in User rows with audio input. DTMF input can be used in User rows with DTMF input. The Agent prompt should be used in Agent rows. |
| S3 Audio Location | Audio | String | No | This column is used to add the audio file location and is applicable only to audio test sets. The S3 path should be used in the User rows with the audio input. This field should be left empty in User rows with DTMF input. This field should be left empty in Agent rows. |
| Input Context Tag | Text and Audio | String | Yes | This column is used to provide name of an input context which will be used in input to Lex while executing the row in the test set. This refers to input context in [Setting intent context for your Lex V2 bot](https://docs.aws.amazon.com/lexv2/latest/dg/context-mgmt-active-context.html). Note that Test Workbench only supports name of context. It does not support the parameters in a context. Multiple columns named such as 'Input Context Tag 1', 'Input Context Tag 2', and so on, may be used. |
| Request Attribute | Text and Audio | String | Yes | This column is used to provide a request attribute which will be used in input to Lex while executing the row in the test set. Value in a column should be provided in format ` = `. Spaces can be added around '=' for readability. For example: request-attribute-foo = this is a dummy response request-attribute-foo = 'this is a "dummy response"' request-attribute-foo = "this is a 'dummy response'". Multiple columns named such as 'Request Attribute 1', 'Request Attribute 2', and so on, may be used. |
| Session Attribute | Text and Audio | String | Yes | This column is used to provide a session attribute which will be used in input to Lex while executing the row in the test set. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/file-input-test-sets.html) |
| RunTime Hint | Text and Audio | String | Yes | This column is used to provide a Runtime Hint for a slot within an intent which will be used in input to Lex while executing the row in the test set. Below are examples: [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/file-input-test-sets.html) |
| Barge In | Audio | Boolean | No | This column is used specify if Test Workbench should barge-in when sending audio file to Lex Runtime for the row in the test set. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/file-input-test-sets.html) |
| Expected Output Intent | Text and Audio | String | No | This column is used specify name of an intent expected in output from Lex for the row in the test set. |
| Expected Output Slot | Text and Audio | String | Yes | This column is used to provide a slot value expected in output from Lex while executing the row in the test set. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/file-input-test-sets.html) |
| Expected Output Context Tag | Text and Audio | String | Yes | This column is used to specify name of an output context expected in output from Lex for the row in the test set. [\[See the AWS documentation website for more details\]](http://docs.aws.amazon.com/lexv2/latest/dg/file-input-test-sets.html) |
# View test validation errors in test workbench
You can correct test sets that report validation errors. These validation errors are generated when a test set is not ready to be tested. The Test Workbench can show you which required columns in the test set input CSV file did not have a value in the expected format.
**To view test validation errors:**
1. From the list of test sets, select the name of the test set that reports a Status of **Validation Error** that you want to view. The names of the test sets are active links that take you to details regarding the test set.
1. The test set record displays validation error details at the top of the screen. Choose **View Details** to see the report on Validation Errors.
1. From the error report window, review the Line \$1 and Error Type to see where the error occurs. For a lengthy list of errors, you can choose to **Download** the error report.
1. Compare the errors listed in your test set input CSV file to your original test file to correct any issues and upload the test set again.
The following table lists the input CSV validation error messages with scenarios.
| Scenario | Error message | Notes |
| --- | --- | --- |
| Test Set File Size Exceeds | Test Set file size is larger than 200 MB. Provide smaller file and try your request again. | |
| Test set exceeds max records | Input file had records more than supported maximum number of 200,000. | |
| Upload Empty Test set | Imported test set is empty. Provide non-empty test set and try your request again. | |
| Empty column header name | Column Headers Row: found empty column name in column number 5. | |
| Unrecognized column header name | Column Headers Row: could not recognize column name 'dummy' in column number 2. | |
| Duplicate column header name | Column Headers Row: found multiple columns 'S3 audio link' and 'S3 audio link' that are same or equivalent. Remove or rename one of those columns. | |
| Multi value column name exceeded the limit | Column Headers Row: count of columns for 'Expected Output Slot' exceeded maximum supported count: 6. Remove some columns for 'Expected Output Slot' and try again. | Maximum Number of columns supported for multi value column is 6. |
| Text or Audio related column header not present | Could not find columns for text or audio conversations. For text conversations, use \$1'Text input'\$1 columns. For audio conversations, use \$1'S3 audio link', 'Expected transcription'\$1 columns. | Audio Mandatory Columns: \$1'S3 audio link', 'Expected transcription'\$1Text Mandatory Columns: \$1'Text input'\$1 |
| Both Text and Audio related column header exist | Found columns for both text and audio conversations. You can either use \$1'Text input'\$1 columns for text conversations, or \$1'S3 audio link', 'Expected transcription'\$1 columns for audio conversations. | Audio Mandatory Columns: \$1'S3 audio link', 'Expected transcription'\$1Text Mandatory Columns: \$1'Text input'\$1 |
| Mandatory column is missing | Could not find mandatory columns ["Expected Output Intent"]. | Mandatory Columns:\$1"Line \$1", "Source", "Expected Output Intent"\$1 |
| Found a data in column with no header | Found data in column number 8 for row number 6, but corresponding column did not have a column header. | |
| Data not found for mandatory columns | Row=12: no values found for mandatory columns: \$1"Source", "Expected Output Intent"\$1 | |
| Duplicate conversation id found | conversation number '19' was seen for previous conversation at row number 39." Make sure that same conversation number has not been provided for two conversations, you can do this by ensuring that all rows for a conversation number are grouped together. | |
| Invalid conversation id provided | Found invalid value 'test\$1conversation' in 'Conversation \$1' column. Value for this column must be either numeric or N/A (i.e. Not Applicable) for a user row. | |
| Non numeric value provided for line number | Found non-numeric value 'test\$1line' in 'Line \$1' column. Its value must be numeric. | |
| Conversation id not found in agent row | No value found for 'Conversation \$1' column. It must be provided for an agent row. | |
| Non numeric conversation id found in agent row | Found non-numeric value 'test\$1conversation' in 'Conversation \$1' column. Its value must be numeric for an agent row. | |
| Invalid S3 location | Invalid value 'bucket/folder' was provided. Valid format is S3:///. | |
| Invalid S3 bucket name | Invalid s3 bucket name 'test\$1bucket' was provided. Check the bucket name. | |
| S3 audio location is folder | Provided audio location 'S3://bucket/folder' is invalid. It points to an S3 folder. | |
| Invalid intent name | Invalid characters were present in intent 'intent@name'. Check the intent name. | Regex check: ^([0-9a-zA-Z][\$1-]?)\$1\$1 |
| Invalid slot name | Invalid characters were present in slot 'Slot@Name'. Check the slot name. | Regex: ^([0-9a-zA-Z][\$1-]?)\$1\$1It should not start or end with dot(.) |
| Slot value provided for parent slot | Slot values were provided for subslot 'Address.City' as well as parent slot 'Address'. Values should be only provided for the subslot. | Parent slot in CST should not have slot value |
| Invalid character in context name | Invalid characters were present in context name 'context@1'. Check the context name. | Regex: ^([A-Za-z]\$1?)\$1\$1 |
| Invalid slot spelling style | Invalid value 'test' was provided. Make sure that they are all upper case. Valid values are ["Default", "SpellByLetter", "SpellByWord"]. | Supported values["Default", "SpellByLetter", "SpellByWord" |
| Participant or source has to be either agent or User | Invalid value 'bot' was provided. Valid values are ["Agent", "User"]. | Supported Enums: "Agent", "User" |
| Line Number should not be decimal | Invalid value '10.1' was provided. It should be a valid number without any fractions. | |
| Conversation Number should not be decimal | Invalid value '10.1' was provided. It should be a valid number without any fractions. | |
| Line number should be with in range | Invalid value '92233720368547758071' was provided. It should be greater than or equal to 1 and less than or equal to 9223372036854775807. | |
| Barge-in column only accepts boolean value | Invalid value 'test' was provided. It should be a valid boolean value such as 'true' or 'false'. Alternatively 'yes' and 'no' can be used. | Possible Values:"True", "true", "T", "Yes", "yEs", "Y", "1", "1.0", "False", "false", "F", "No", "no", "N", "0", "0.0" |
| Expected slot, Session Attribute, Request Attribute should be separated by equal to (=) | Value 'slotName:slotValue' does not have '='. Such value should be provided as a key-value pair in format '='. | For example: slotName = slotType |
| Expected slot, Session Attribute, Request Attribute should be have key value pair | '=slotValue' does not have a key before '='. Such value should be provided as a key-value pair in format '='. | For example: slotName = slotType |
| Invalid quote at end | Found incorrect quoting in 'Foo's item'“. It starts with quote character `"` but does not end with same quote character. | For example: `"Foo's item", KFC` |
| Invalid quote at middle | Found incorrect quoting in `"Foo's" Burger, etc.`. It contains quote character `"` inside its content. Values containing single quotes should be wrapped within double quotes and vice-versa. | Correct For example: `"Foo's item", KFC` |
| Required quotes | `key = Foo's item` contains single-quotes or double-quotes but has not been wrapped inside quotes. Values containing single quotes should be wrapped inside double quotes and vice-versa. | |
| Duplicate Key repeated in column | Key `key1` was repeated in two columns: `Session Attribute 3` and `Session Attribute 1`. | |
| Invalid format in Runtime hint | Invalid key `BookFlight.Car."` provided for Runtime Hints. For Runtime Hints, key should be in format .. | If '.' must be present in middle of the key, intent name and slot name cannot be extracted from such key. examples of such incorrect formatting: "BookFlight", ".BookFlight.Car", "BookFlight.Car." |
| Invalid Intent name in runtime hint key | Found invalid intent `intent@name` for Runtime Hints. Check intent name. | Regex check: ^([0-9a-zA-Z][\$1-]?)\$1\$1 |
| Invalid Slot name in runtime hint key | Found invalid slot name in `Slot@Name` for Runtime Hints. Check slot name. | Regex: ^([0-9a-zA-Z][\$1-]?)\$1\$1It should not start or end with dot(.) |
# Delete a test set in Test Workbench
You can easily delete a test set from your list of test sets.
**To delete a test set:**
1. Go to the list of **Test Sets** from the left side menu to see the list of test sets.
1. From the list of test sets, select the test set you want to delete.
1. Go to the **Actions** drop down menu in the top right, and choose **Delete**.
1. A message confirms that the test set is deleted.
# Edit test set details
You can edit a Test Set name and details in the list of test sets. The name or details can be added or updated later. However, you will have to update your test set before running the test with your bot or transcription data.
**To edit test set details:**
1. Go to the list of test set from the left side menu to see the list of test sets.
1. From the list of test sets, select the check box for the test set you want to edit.
1. Go to the **Actions** drop down menu in the top right, and choose **Edit Details**.
1. A message confirms that the test set is successfully edited.
# Update test set
You can update, correct, modify, or delete items from the test set to optimize your baseline results, or to correct other errors that may have occurred in the test set
You can download a test set and fix the validation errors before uploading the corrected test set. See [View test validation errors](https://docs.aws.amazon.com/lexv2/latest/dg/view-errors-test-sets.html).
**To update a test set:**
1. From the test set record, choose the **Update Test Set** button in the top right.
1. Choose a file to upload from your Amazon S3 account or upload a CSV test file from your computer. NOTE: Updating a test set will overwrite the existing data.
1. Select the **Update** button.
1. A message confirms that the test set is successfully updated. NOTE: This operation can take a few minutes, depending on the complexity and size of the test set.
1. A message confirms that the test set is successfully updated and the **Status** displays **Ready for Testing**.
# Execute a test
To execute a test set, you must choose the appropriate bot to run the test against the test set. You can choose a bot from your AWS account from the drop down menu under Test Set. This operation will test your selected bot against your validated test data to report performance metrics against the baseline data from the test set.
![\[The screen to execute a test in the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-extest.png)
**To execute a test in the Test Workbench**
1. In the test set record page, choose **Execute Test**.
1. Select the test set you want to use in the test.
1. Select the name of the bot to use in the test from the **Bot** drop down menu.
1. Choose a bot alias, if applicable, from the **Bot alias** drop down menu.
1. From the **Languages** selection, choose a version of English.
1. Select **Text** or **Audio** for the Modality type.
1. Choose your Amazon S3 location. (audio only)
1. Select your **Endpoint selection** for your bot. (streaming only)
1. Select the **Validate coverage** button to confirm your test in ready to run. If there are any errors present in the validation step, review the previous parameters and make corrections.
1. Select **Execute** to run the test.
1. A message confirms that the test is successfully executed.
# Test set coverage in Test Workbench
Limited coverage of intents and slots between the test set and the bot can result in expected performance measures. We recommend that you review the test set coverage ahead of running the test.
![\[Review intents in the validation step with the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-discr1.png)
**To review validation coverage**
1. In the test set records, choose the **Validate coverage** button.
1. The message indicates it is validating coverage between the test set and the bot selected.
1. Once the operation is completed, the message indicates **Coverage validation successful**.
1. Choose the **View Details** button at the bottom of the window.
1. View the test set discrepancies for intents and slots by choosing the tab for each. You can download this data into a CSV format by choosing the **Download** button.
1. Review the validation results for your test set data, bot intents, and slots. Identify issues and make changes in your bot test set architecture to improve results. Upload the edited test set and bot to run the test once you have made changes to the CSV file. NOTE: Validation coverage runs against the test set and not against the bot. Intents in the bot but not present in the test set will not be covered.
# View test results
Interpret test results from the Test Workbench to determine where the conversation between your bot and the customer might be failing, or requiring the customer to make multiple attempts to fulfill the intent.
By locating these issues in your test results, you can optimize your bot’s performance by improving intent performance using different training data or utterances that are more consistent with the real time bot transcription values.
You can get a detailed view of intents and slots that had performance discrepancies. Once you have identified intents or slots that have discrepancies, you can further drill down and review the utterances and conversation flow.
![\[List of completed tests using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-testresults.png)
**To review test results:**
1. Go to the list of test sets from the left side menu to select the **Test results** option under Test workbench. NOTE: Test results indicate a **Status** of complete if they were successful.
1. Select the **Test Result ID** for the test results you want to review.
# Test results details in Test Workbench
The test results show the test set details, intents used, and the slots used. It also provides the overall test set input breakdown includes the overall results, conversation results, intent, and slot results.
Test results comprise all testing related information such as:
+ Test details metadata
+ Overall results
+ Conversation results
+ Intent and slot results
+ Detailed results
**Overall results tab:**
![\[The test set input breakdown chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results1.png)
**Test set input breakdown** – This chart shows the breakdown of number of conversations and single input utterances in the test set.
![\[The single input breakdown chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results2.png)
**Single input breakdown** – Displays two charts that included end-to-end conversations and speech transcriptions. The number of passed and failed inputs are indicated on each chart. Note: Speech transcription chart will be visible only for the audio test set.
![\[The conversation breakdown chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results3.png)
**Conversation breakdown** – Displays two charts that included end-to-end conversations and speech transcriptions. The number of passed and failed inputs are indicated on each chart. Note: Speech transcription chart will be visible only for the audio test set.
**Conversation results tab:**
![\[The conversation pass rates chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results4.png)
**Conversation pass rates** – The conversation pass rates table is used to see which intents and slots are used in each conversation in the test set. You can visualize where the conversation has failed by reviewing which intent or slot failed, along with the pass percentage of each intent and slot.
![\[The conversation intent failure metrics chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results5.png)
**Conversation intent failure metrics** – This metric shows the top 5 worst performing intents in the test set. This panel shows a chart of what percent or number of intents were successful or failed based on the bot’s conversation logs or transcription. A successful intent does not mean that the entire conversation was successful. These metrics only apply to the value of the intents, regardless of which intent came before or after.
![\[The Conversation slot failure metrics chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results6.png)
**Conversation slot failure metrics** – This metric shows the top 5 worst performing slots in the test set. Indicated the success rate for each slot in the intent. Bar graph shows both speech transcription and end-to-end conversations for each slot in the intent.
**Intent and slot results tab:**
![\[The Intent recognition metrics chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results7.png)
**Intent recognition metrics** – Shows a table of how many intents were recognized successfully. Displays the pass rate of speech transcription and end-to-end conversations.
![\[The Slot resolution metrics chart in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results8.png)
**Slot resolution metrics** – Shows the intents and slots separately, and the success and failure rate of each slot for each intent used in the conversation or single input. Displays the pass rate of speech transcription and end-to-end conversations.
**Detailed results tab:**
![\[The detailed results in test results using the Test Workbench.\]](http://docs.aws.amazon.com/lexv2/latest/dg/images/testworkbench/test-workbench-results9.png)
**Detailed results** – Shows a detailed table on the conversation log with User and Agent utterances and the expected output and expected transcription for each slot. You can download this report by selecting the **Download** button.
The following table lists the result failure error messages with scenarios.
| Scenario | Error message | Action |
| --- | --- | --- |
| Intent Mismatch | Expected BookFlight intent but it was BookHotel intent. | Skip other turns in the conversation |
| Slot Elicitation mismatch | Expected departureDate slot to be elicited but it was cabinType. | Skip other turns in the conversation |
| Slot value mismatch | Mismatch between expected and actual slot value. | Continue with other turns in the conversations |
| Back-to-back agent prompt is missing | Expected bot to return an agent prompt in this turn but it was not received. | Skip other turns in the conversation |
| Transcription Mismatch | Expected transcription didn't match actual transcription. | Continue with other turns in the conversations |
| Optional slot not elicited | Expected to elicit cabinType slot in next turn, however current intent fulfilled before that. | Skip other turns in the conversation |
| Slot not recognized | Expected departureDate slot was not recognized in this turn. | Skip other turns in the conversation |
| Extra back-to-back agent prompt | Expected a user turn but it was agent prompt | Skip other turns in the conversation |