Monitoring Amazon Quick Suite chat and feedback using CloudWatch Logs

You can use Amazon CloudWatch Logs to deliver user conversations and response feedback in Amazon Quick Suite for you to analyze. These logs can be delivered to multiple destinations, such as CloudWatch, Amazon S3, or (standard rates apply). We recommend that you set up conversation and feedback logging shortly after creating your Amazon Quick Suite chat agent.

The following are examples of tasks you can complete with logs from conversations and response feedback in Amazon Quick Suite:

Supported log destinations

Amazon Quick Suite can deliver logs to the following destinations:

Prerequisites

Before you can enable logging, ensure you have:

Configure logging

To enable logging for Amazon Quick Suite chat and feedback, you need to configure IAM permissions, create a delivery source and destination, and verify that logs are being delivered successfully.

Set up IAM permissions

To set up CloudWatch Logs for Amazon Quick Suite, use the following IAM policy examples to grant the necessary permissions.

{
    "Version": "2012-10-17"		 	 	 ,
    "Statement": [{
        "Sid": "QuicksightLogDeliveryPermissions",
        "Effect": "Allow",
        "Action": "quicksight:AllowVendedLogDeliveryForResource",
        "Resource": "arn:aws:quicksight:region:account-id:account/account-id"
    }]
}

You must also allow the delivery.logs.amazonaws.com service principal in your customer managed AWS KMS key policy.

{
    "Effect": "Allow"		 	 	 ,
    "Principal": {
        "Service": "delivery.logs.amazonaws.com"
    },
    "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "kms:EncryptionContext:SourceArn": "arn:partition:logs:region:account-id:*"
        }
    }
}

Configure log subscription

For example IAM policies with all the required permissions for your specific logging destination, see Enable logging from AWS services in the Amazon CloudWatch Logs User Guide.

Create a delivery source with the PutDeliverySource CloudWatch Logs API operation. Give the delivery source a name and for resourceArn, specify the ARN of your application. For logType, specify CHAT_LOGS or FEEDBACK_LOGS

{
    "logType": "CHAT_LOGS",
    "name": "my-quick-suite-delivery-source",
    "resourceArn": "arn:aws:quicksight:your-region:your-account-id:account/account-id"
}

{
    "logType": "FEEDBACK_LOGS",
    "name": "my-quick-suite-delivery-source",
    "resourceArn": "arn:aws:quicksight:your-region:your-account-id:account/account-id"
}

To enable user conversation logging with the CloudWatch Logs API operations, you call the PutDeliverySource, PutDeliveryDestination, and CreateDelivery API operations.

Verify log delivery

Once configured, verify that logs are being delivered to your destination:

Log schema and format

Amazon Quick Suite logs follow a structured schema with common fields shared across all log types and specific fields for chat and feedback logs.

Common fields

All log events include these common fields:

Chat logs

Chat logs capture conversation interactions and contains below fields:

The following is an example of chat logs:

{
    "conversationId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "systemMessageId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "latency": "10000", 
    "timeToFirstToken": "10000",
    "messageScope": "General Knowledge, ALL etc",
    "userMessageId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "userMessage": "What is the status of my project?",
    "agentId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "flowId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d?",
    "systemTextMessage": "What is the status of my project?",
    "surfaceType": "Slack, WebApp etc.",
    "webSearch": "TRUE"
    "userSelectedResources": [{"resource_type": "Dashboard","resource_id": "146abs-1222-534894"},{"resource_type": "Space","resource_id": "123abs-1234-534894"}],
    "actionConnectors": [{"user_selected_resource_type": "Dashboard","resource_id": "146abs-1222-534894"},{"user_selected_resource_type": "Space","resource_id": "123abs-1234-534894"}]
    "citedResource": [{"cited_resource_name": "Dashboard","cited_resource_id": "146abs-1222-534894","cited_resource_name": "ds1"},{"cited_resource_name": "Space","cited_resource_id": "123abs-1234-534894","cited_resource_name": "space1"}],
    "fileAttachment": [{"file_attachmet_type": "pdf","file_attachment_name": "file1.pdf"},{"file_attachmet_type": "txt","file_attachment_name": "file2.txt"}]
}

Feedback logs

Feedback logs capture user feedback on chat and contains below fields:

The following is an example of feedback logs:

Chat Feedback: 
    "conversationId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "researchId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "systemMessageId": "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "userMessageId" : "a11b2bbc-c123-3abc-a12b-12a34b5c678d",
    "feedback_type" :"thumbsUp,thumbsDown,ease_of_use etc."
    "feedback_reason" : "Too wordy,Issue with sources,Other etc."
    "feedback_details" : "additional text shared by user"
    "rating" : "thumbsUp,thumbsDown,ease_of_use etc."

Security considerations