# OTA troubleshooting The following sections contain information to help you troubleshoot issues with OTA updates. **Topics** + [ # Set up CloudWatch Logs for OTA updates ](ota-logging.md) + [ # Log AWS IoT OTA API calls with AWS CloudTrail ](iot-using-cloudtrail-afr.md) + [ # Get CreateOTAUpdate failure details using the AWS CLI ](ota-create-failure.md) + [ # Get OTA failure codes with the AWS CLI ](ota-failure-codes.md) + [ # Troubleshoot OTA updates of multiple devices ](ota-troubleshooting-multi-thing.md) + [ # Troubleshoot OTA updates with the Texas Instruments CC3220SF Launchpad ](ota-troubleshooting-ti.md) # Set up CloudWatch Logs for OTA updates The OTA Update service supports logging with Amazon CloudWatch. You can use the AWS IoT console to enable and configure Amazon CloudWatch logging for OTA updates. For more information, see [Cloudwatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html). To enable logging, you must create an IAM role and configure OTA update logging. **Note** Before you enable OTA update logging, make sure you understand the CloudWatch Logs access permissions. Users with access to CloudWatch Logs can see your debugging information. For information, see [Authentication and Access Control for Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/auth-and-access-control-cwl.html). ## Create a logging role and enable logging Use the [AWS IoT console](https://console.aws.amazon.com/iot/home) to create a logging role and enable logging. 1. From the navigation pane, choose **Settings**. 1. Under **Logs**, choose **Edit**. 1. Under **Level of verbosity**, choose **Debug**. 1. Under **Set role**, choose **Create new** to create an IAM role for logging. 1. Under **Name**, enter a unique name for your role. Your role will be created with all required permissions. 1. Choose **Update**. ## OTA update logs The OTA Update service publishes logs to your account when one of the following occurs: + An OTA update is created. + An OTA update is completed. + A code-signing job is created. + A code-signing job is completed. + An AWS IoT job is created. + An AWS IoT job is completed. + A stream is created. You can view your logs in the [CloudWatch console.](https://console.aws.amazon.com/cloudwatch/home) **To view an OTA update in CloudWatch Logs** 1. From the navigation pane, choose **Logs**. 1. In **Log Groups**, choose **AWSIoTLogsV2**. OTA update logs can contain the following properties: **accountId** The AWS account ID in which the log was generated. **actionType** The action that generated the log. This can be set to one of the following values: + `CreateOTAUpdate`: An OTA update was created. + `DeleteOTAUpdate`: An OTA update was deleted. + `StartCodeSigning`: A code-signing job was started. + `CreateAWSJob`: An AWS IoT job was created. + `CreateStream`: A stream was created. + `GetStream`: A request for a stream was sent to the AWS IoT MQTT-based file delivery feature. + `DescribeStream`: A request for information about a stream was sent to the AWS IoT MQTT-based file delivery feature. **awsJobId** The AWS IoT job ID that generated the log. **clientId** The MQTT client ID that made the request that generated the log. **clientToken** The client token associated with the request that generated the log. **details** More information about the operation that generated the log. **logLevel** The logging level of the log. For OTA update logs, this is always set to `DEBUG`. **otaUpdateId** The ID of the OTA update that generated the log. **protocol** The protocol used to make the request that generated the log. **status** The status of the operation that generated the log. Valid values are: + Success + Failure **streamId** The AWS IoT stream ID that generated the log. **timestamp** The time when the log was generated. **topicName** An MQTT topic used to make the request that generated the log. ### Example logs The following is an example log generated when a code-signing job is started: ``` { "timestamp": "2018-07-23 22:59:44.955", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "StartCodeSigning", "otaUpdateId": "08957b03-eea3-448a-87fe-743e6891ca3a", "details": "Start code signing job. The request status is SUCCESS." } ``` The following is an example log generated when an AWS IoT job is created: ``` { "timestamp": "2018-07-23 22:59:45.363", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "CreateAWSJob", "otaUpdateId": "08957b03-eea3-448a-87fe-743e6891ca3a", "awsJobId": "08957b03-eea3-448a-87fe-743e6891ca3a", "details": "Create AWS Job The request status is SUCCESS." } ``` The following is an example log generated when an OTA update is created: ``` { "timestamp": "2018-07-23 22:59:45.413", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "CreateOTAUpdate", "otaUpdateId": "08957b03-eea3-448a-87fe-743e6891ca3a", "details": "OTAUpdate creation complete. The request status is SUCCESS." } ``` The following is an example log generated when a stream is created: ``` { "timestamp": "2018-07-23 23:00:26.391", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "CreateStream", "otaUpdateId": "3d3dc5f7-3d6d-47ac-9252-45821ac7cfb0", "streamId": "6be2303d-3637-48f0-ace9-0b87b1b9a824", "details": "Create stream. The request status is SUCCESS." } ``` The following is an example log generated when an OTA update is deleted: ``` { "timestamp": "2018-07-23 23:03:09.505", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "DeleteOTAUpdate", "otaUpdateId": "9bdd78fb-f113-4001-9675-1b595982292f", "details": "Delete OTA Update. The request status is SUCCESS." } ``` The following is an example log generated when a device requests a stream from the MQTT-based file delivery feature: ``` { "timestamp": "2018-07-25 22:09:02.678", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "GetStream", "protocol": "MQTT", "clientId": "b9d2e49c-94fe-4ed1-9b07-286afed7e4c8", "topicName": "$aws/things/b9d2e49c-94fe-4ed1-9b07-286afed7e4c8/streams/1e51e9a8-9a4c-4c50-b005-d38452a956af/get/json", "streamId": "1e51e9a8-9a4c-4c50-b005-d38452a956af", "details": "The request status is SUCCESS." } ``` The following is an example log generated when a device calls the `DescribeStream` API: ``` { "timestamp": "2018-07-25 22:10:12.690", "logLevel": "DEBUG", "accountId": "123456789012", "status": "Success", "actionType": "DescribeStream", "protocol": "MQTT", "clientId": "581075e0-4639-48ee-8b94-2cf304168e43", "topicName": "$aws/things/581075e0-4639-48ee-8b94-2cf304168e43/streams/71c101a8-bcc5-4929-9fe2-af563af0c139/describe/json", "streamId": "71c101a8-bcc5-4929-9fe2-af563af0c139", "clientToken": "clientToken", "details": "The request status is SUCCESS." } ``` # Log AWS IoT OTA API calls with AWS CloudTrail FreeRTOS is integrated with CloudTrail, a service that captures AWS IoT OTA API calls and delivers the log files to an Amazon S3 bucket that you specify. CloudTrail captures API calls from your code to the AWS IoT OTA APIs. Using the information collected by CloudTrail, you can determine the request that was made to AWS IoT OTA, the source IP address from which the request was made, who made the request, when it was made, and so on. For more information about CloudTrail, including how to configure and enable it, see the [https://docs.aws.amazon.com/awscloudtrail/latest/userguide/](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/). ## FreeRTOS information in CloudTrail When CloudTrail logging is enabled in your AWS account, API calls made to AWS IoT OTA actions are tracked in CloudTrail log files where they are written with other AWS service records. CloudTrail determines when to create and write to a new file based on a time period and file size. The following AWS IoT OTA control plane actions are logged by CloudTrail: + [CreateStream](https://docs.aws.amazon.com/iot/latest/apireference/API_CreateStream.html) + [DescribeStream](https://docs.aws.amazon.com/iot/latest/apireference/API_DescribeStream.html) + [ListStreams](https://docs.aws.amazon.com/iot/latest/apireference/API_ListStreams.html) + [UpdateStream](https://docs.aws.amazon.com/iot/latest/apireference/API_UpdateStream.html) + [DeleteStream](https://docs.aws.amazon.com/iot/latest/apireference/API_DeleteStream.html) + [CreateOTAUpdate](https://docs.aws.amazon.com/iot/latest/apireference/API_CreateOTAUpdate.html) + [GetOTAUpdate](https://docs.aws.amazon.com/iot/latest/apireference/API_GetOTAUpdate.html) + [ListOTAUpdates](https://docs.aws.amazon.com/iot/latest/apireference/API_ListOTAUpdates.html) + [DeleteOTAUpdate](https://docs.aws.amazon.com/iot/latest/apireference/API_DeleteOTAUpdate.html) **Note** AWS IoT OTA data plane actions (device side) are not logged by CloudTrail. Use CloudWatch to monitor these. Every log entry contains information about who generated the request. The user identity information in the log entry helps you determine the following: + Whether the request was made with root or IAM user credentials. + Whether the request was made with temporary security credentials for a role or federated user. + Whether the request was made by another AWS service. For more information, see the [CloudTrail userIdentity Element](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-user-identity.html). AWS IoT OTA actions are documented in the [AWS IoT OTA API Reference](https://docs.aws.amazon.com/iot/latest/apireference). You can store your log files in your Amazon S3 bucket for as long as you want, but you can also define Amazon S3 lifecycle rules to archive or delete log files automatically. By default, your log files are encrypted with Amazon S3 server-side encryption (SSE). If you want to be notified when log files are delivered, you can configure CloudTrail to publish Amazon SNS notifications. For more information, see [ Configuring Amazon SNS Notifications for CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/getting_notifications_top_level.html). You can also aggregate AWS IoT OTA log files from multiple AWS Regions and multiple AWS accounts into a single Amazon S3 bucket. For more information, see [ Receiving CloudTrail Log Files from Multiple Regions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html) and [Receiving CloudTrail Log Files from Multiple Accounts](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-receive-logs-from-multiple-accounts.html). ## Understanding FreeRTOS log file entries CloudTrail log files can contain one or more log entries. Each entry lists multiple JSON-formatted events. A log entry represents a single request from any source and includes information about the requested action, the date and time of the action, request parameters, and so on. Log entries are not an ordered stack trace of the public API calls, so they do not appear in any specific order. The following example shows a CloudTrail log entry that demonstrates the log from a call to `CreateOTAUpdate` action. ``` { "eventVersion": "1.05", "userIdentity": { "type": "IAMUser", "principalId": "EXAMPLE", "arn": "arn:aws:iam::your_aws_account:user/your_user_id", "accountId": "your_aws_account", "accessKeyId": "your_access_key_id", "userName": "your_username", "sessionContext": { "attributes": { "mfaAuthenticated": "false", "creationDate": "2018-08-23T17:27:08Z" } }, "invokedBy": "apigateway.amazonaws.com" }, "eventTime": "2018-08-23T17:27:19Z", "eventSource": "iot.amazonaws.com", "eventName": "CreateOTAUpdate", "awsRegion": "your_aws_region", "sourceIPAddress": "apigateway.amazonaws.com", "userAgent": "apigateway.amazonaws.com", "requestParameters": { "targets": [ "arn:aws:iot:your_aws_region:your_aws_account:thing/Thing_CMH" ], "roleArn": "arn:aws:iam::your_aws_account:role/Role_FreeRTOSJob", "files": [ { "fileName": "/sys/mcuflashimg.bin", "fileSource": { "fileId": 0, "streamId": "your_stream_id" }, "codeSigning": { "awsSignerJobId": "your_signer_job_id" } } ], "targetSelection": "SNAPSHOT", "otaUpdateId": "FreeRTOSJob_CMH-23-1535045232806-92" }, "responseElements": { "otaUpdateArn": "arn:aws:iot:your_aws_region:your_aws_account:otaupdate/FreeRTOSJob_CMH-23-1535045232806-92", "otaUpdateStatus": "CREATE_PENDING", "otaUpdateId": "FreeRTOSJob_CMH-23-1535045232806-92" }, "requestID": "c9649630-a6f9-11e8-8f9c-e1cf2d0c9d8e", "eventID": "ce9bf4d9-5770-4cee-acf4-0e5649b845c0", "eventType": "AwsApiCall", "recipientAccountId": "recipient_aws_account" } ``` # Get CreateOTAUpdate failure details using the AWS CLI If the process of creating an OTA update job fails, there may be actions you can take to remedy the problem. When you create an OTA update job, the OTA manager service creates an IoT job and schedules it for the target devices, and this process also creates or uses other types of AWS resources in your account (a code-signing job, an AWS IoT stream, an Amazon S3 object). Any error encountered may cause the process to fail without creating an AWS IoT job. In this troubleshooting section we give instructions on how to retrieve the details of the failure. 1. Install and configure the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html). 1. Run `aws configure` and enter the following information. ``` $ aws configure AWS Access Key ID [None]: AccessID AWS Secret Access Key [None]: AccessKey Default region name [None]: Region Default output format [None]: json ``` For more information, see [ Quick configuration with `aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config). 1. Run: ``` aws iot get-ota-update --ota-update-id ota_update_job_001 ``` Where *ota\$1update\$1job\$1001* is the ID you gave the OTA update when you created it. 1. The output will look like this: ``` { "otaUpdateInfo": { "otaUpdateId": "ota_update_job_001", "otaUpdateArn": "arn:aws:iot:region:account_id:otaupdate/ota_update_job_001", "creationDate": 1584646864.534, "lastModifiedDate": 1584646865.913, "targets": [ "arn:aws:iot:region:account_id:thing/thing_001" ], "protocols": [ "MQTT" ], "awsJobExecutionsRolloutConfig": {}, "awsJobPresignedUrlConfig": {}, "targetSelection": "SNAPSHOT", "otaUpdateFiles": [ { "fileName": "/12ds", "fileLocation": { "s3Location": { "bucket": "bucket_name", "key": "demo.bin", "version": "Z7X.TWSAS7JSi4rybc02nMdcE41W1tV3" } }, "codeSigning": { "startSigningJobParameter": { "signingProfileParameter": {}, "signingProfileName": "signing_profile_name", "destination": { "s3Destination": { "bucket": "bucket_name", "prefix": "SignedImages/" } } }, "customCodeSigning": {} } } ], "otaUpdateStatus": "CREATE_FAILED", "errorInfo": { "code": "AccessDeniedException", "message": "S3 object demo.bin not accessible. Please check your permissions (Service: AWSSigner; Status Code: 403; Error Code: AccessDeniedException; Request ID: 01d8e7a1-8c7c-4d85-9fd7-dcde975fdd2d)" } } } ``` If the create failed, the `otaUpdateStatus` field in the command output will contain `CREATE_FAILED` and the `errorInfo` field will contain the details of the failure. # Get OTA failure codes with the AWS CLI 1. Install and configure the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html). 1. Run `aws configure` and enter following information. ``` $ aws configure AWS Access Key ID [None]: AccessID AWS Secret Access Key [None]: AccessKey Default region name [None]: Region Default output format [None]: json ``` For more information, see [ Quick configuration with `aws configure`](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config). 1. Run: ``` aws iot describe-job-execution --job-id JobID --thing-name ThingName ``` Where *JobID* is the complete job ID string for the job whose status we want to get (it was associated with the OTA update job when it was created) and *ThingName* is the AWS IoT thing name that the device is registered as in AWS IoT 1. The output will look like this: ``` { "execution": { "jobId": "AFR_OTA-****************", "status": "FAILED", "statusDetails": { "detailsMap": { "reason": "0xEEEEEEEE: 0xffffffff" } }, "thingArn": "arn:aws:iot:Region:AccountID:thing/ThingName", "queuedAt": 1569519049.9, "startedAt": 1569519052.226, "lastUpdatedAt": 1569519052.226, "executionNumber": 1, "versionNumber": 2 } } ``` In this example output, the "`reason`" in the "`detailsmap`" has two fields: the field shown as "0xEEEEEEEE" contains the generic error code from the OTA Agent; the field shown as "0xffffffff" contains the sub-code. The generic error codes are listed in [ https://docs.aws.amazon.com/freertos/latest/lib-ref/html1/aws\$1\$1ota\$1\$1agent\$18h.html](https://docs.aws.amazon.com/freertos/latest/lib-ref/html1/aws__ota__agent_8h.html). See error codes with the prefix "`kOTA_Err_`". The sub-code can be a platform specific code or provide more details about the generic error. # Troubleshoot OTA updates of multiple devices To perform OTAs on multiple devices (things) using the same firmware image, implement a function (for example `getThingName()`) that retrieves `clientcredentialIOT_THING_NAME` from non-volatile memory. Make sure that this function reads the thing name from a part of non-volatile memory that is not overwritten by the OTA, and that the thing name is provisioned before running the first job. If you are using the JITP flow, you can read the thing name out of your device certificate's common name. # Troubleshoot OTA updates with the Texas Instruments CC3220SF Launchpad The CC3220SF Launchpad platform provides a software tamper-detection mechanism. It uses a security alert counter that is incremented whenever there is an integrity violation. The device is locked when the security alert counter reaches a predetermined threshold (the default is 15) and the host receives the `SL_ERROR_DEVICE_LOCKED_SECURITY_ALERT` asynchronous event. The locked device then has limited accessibility. To recover the device, you can reprogram it or use the restore-to-factory process to revert to the factory image. You should program the desired behavior by updating the asynchronous event handler in `network_if.c`.