# Debugging a failed model training You might encounter errors during model training. Amazon Rekognition Custom Labels reports training errors in the console and in the response from [DescribeProjectVersions](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjectVersions). Errors are either terminal (training can't continue), or they are non-terminal (training can continue). For errors that relate to the contents of the training and testing datasets, you can download the validation results ( a [manifest summary](tm-debugging-summary.md) and [training and testing validation manifests](tm-debugging-scope-json-line.md)). Use the error codes in the validation results to find further information in this section. This section also provides information for manifest file errors (terminal errors that happen before the manifest file contents are validated). **Note** A manifest is the file used to store the contents of a dataset. You can fix some errors by using the Amazon Rekognition Custom Labels console. Other errors might require you to make updates to the training or testing manifest files. You might need to make other changes, such as IAM permissions. For more information, see the documentation for individual errors. ## Terminal errors Terminal errors stop the training of a model. There are 3 categories of terminal training errors – service errors, manifest file errors, and manifest content errors. In the console, Amazon Rekognition Custom Labels shows terminal errors for a model in the **Status message** column of the projects page. The project management dashboard showing list of projects with name, versions, date created, model performance, and status message indicating model state such as training completed or failed ![\[A screenshot of the Project management dashboard.\]](http://docs.aws.amazon.com/rekognition/latest/customlabels-dg/images/terminal-errors.png) If you using the AWS SDK, you can find out if a terminal manifest file error or a terminal manifest content error has occured by checking the response from [DescribeProjectVersions](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjectVersions). In this case, the `Status` value is `TRAINING_FAILED` and `StatusMessage` field contains the error. ### Service errors Terminal service errors occur when Amazon Rekognition experiences a service issue and can't continue training. For example, the failure of another service that Amazon Rekognition Custom Labels depends upon. Amazon Rekognition Custom Labels reports service errors in the console as *Amazon Rekognition experienced a service issue*. If you use the AWS SDK, service errors that occur during training are raised as an `InternalServerError` exception by [CreateProjectVersion](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_CreateProjectVersion) and [DescribeProjectVersions](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjectVersions). If a service error occurs, retry training of the model. If training continues to fail, contact *[AWS Support](https://aws.amazon.com/premiumsupport/)* and include any error information reported with the service error. ### List of terminal manifest file errors Manifest file errors are terminal errors, in the training and testing datasets, that happen at the file level, or across multiple files. Manifest file errors are detected before the contents of the training and testing datasets are validated. Manifest file errors prevent the reporting of [non-terminal validation errors](#tm-error-category-non-terminal-errors). For example, an empty training manifest file generates an *The manifest file is empty* error. Since the file is empty, no non-terminal JSON Line validation errors can be reported. The manifest summary is also not created. You must fix manifest file errors before you can train your model. The following lists the manifest file errors. + [The manifest file extension or contents are invalid.](tm-terminal-errors-reference.md#tm-error-message-ERROR_MANIFEST_INACCESSIBLE_OR_UNSUPPORTED_FORMAT) + [The manifest file is empty.](tm-terminal-errors-reference.md#tm-error-message-ERROR_EMPTY_MANIFEST) + [The manifest file size exceeds the maximum supported size.](tm-terminal-errors-reference.md#tm-error-message-ERROR_MANIFEST_SIZE_TOO_LARGE) + [Unable to write to output S3 bucket.](tm-terminal-errors-reference.md#tm-error-message-ERROR_CANNOT_WRITE_OUTPUT_S3_BUCKET) + [The S3 bucket permissions are incorrect.](tm-terminal-errors-reference.md#tm-error-message-ERROR_INVALID_PERMISSIONS_MANIFEST_S3_BUCKET) ### List of terminal manifest content errors Manifest content errors are terminal errors that relate to the content within a manifest. For example, if you get the error [The manifest file contains insufficient labeled images per label to perform auto-split](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_INSUFFICIENT_IMAGES_PER_LABEL_FOR_AUTOSPLIT), training can't finish as there aren't enough labeled images in the training dataset to create a testing dataset. As well as being reported in the console and in the response from `DescribeProjectVersions`, the error is reported in the manifest summary along with any other terminal manifest content errors. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). Non terminal JSON Line errors are also reported in separate training and testing validation results manifests. The non-terminal JSON Line errors found by Amazon Rekognition Custom Labels are not necessarily related to the manifest content error(s) that stop training. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). You must fix manifest content errors before you can train your model. The following are the error messages for manifest content errors. + [The manifest file contains too many invalid rows.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_TOO_MANY_INVALID_ROWS_IN_MANIFEST) + [The manifest file contains images from multiple S3 buckets.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_IMAGES_IN_MULTIPLE_S3_BUCKETS) + [Invalid owner id for images S3 bucket.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_INVALID_IMAGES_S3_BUCKET_OWNER) + [The manifest file contains insufficient labeled images per label to perform auto-split.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_INSUFFICIENT_IMAGES_PER_LABEL_FOR_AUTOSPLIT) + [The manifest file has too few labels.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_MANIFEST_TOO_FEW_LABELS) + [The manifest file has too many labels.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_MANIFEST_TOO_MANY_LABELS) + [Less than \$1\$1% label overlap between the training and testing manifest files.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_INSUFFICIENT_LABEL_OVERLAP) + [The manifest file has too few usable labels.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_MANIFEST_TOO_FEW_USABLE_LABELS) + [Less than \$1\$1% usable label overlap between the training and testing manifest files.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_INSUFFICIENT_USABLE_LABEL_OVERLAP) + [Failed to copy images from S3 bucket.](tm-debugging-aggregate-errors.md#tm-error-message-ERROR_FAILED_IMAGES_S3_COPY) ## List of non-terminal JSON line validation errors JSON Line validation errors are non-terminal errors that don't require Amazon Rekognition Custom Labels to stop training a model. JSON Line validation errors are not shown in the console. In the training and testing datasets, a JSON Line represents the training or testing information for a single image. Validation errors in a JSON Line, such as an invalid image, are reported in the training and testing validation manifests. Amazon Rekognition Custom Labels completes training using the other, valid, JSON Lines that are in the manifest. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). For information about validation rules, see [Validation rules for manifest files](md-create-manifest-file-validation-rules.md). **Note** Training fails if there are too many JSON Line errors. We recommend that you also fix non-terminal JSON Line errors errors as they can potentially cause future errors or impact your model training. Amazon Rekognition Custom Labels can generate the following non-terminal JSON Line validation errors. + [The source-ref key is missing.](tm-debugging-json-line-errors.md#tm-error-ERROR_MISSING_SOURCE_REF) + [The format of the source-ref value is invalid. ](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_SOURCE_REF_FORMAT) + [No label attributes found.](tm-debugging-json-line-errors.md#tm-error-ERROR_NO_LABEL_ATTRIBUTES) + [The format of the label attribute \$1\$1 is invalid.](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_LABEL_ATTRIBUTE_FORMAT) + [The format of the label attributemetadata is invalid.](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_LABEL_ATTRIBUTE_METADATA_FORMAT) + [No valid label attributes found.](tm-debugging-json-line-errors.md#tm-error-ERROR_NO_VALID_LABEL_ATTRIBUTES) + [One or more bounding boxes has a missing confidence value.](tm-debugging-json-line-errors.md#tm-error-ERROR_MISSING_BOUNDING_BOX_CONFIDENCE) + [One of more class ids is missing from the class map.](tm-debugging-json-line-errors.md#tm-error-ERROR_MISSING_CLASS_MAP_ID) + [The JSON Line has an invalid format.](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_JSON_LINE) + [The image is invalid. Check S3 path and/or image properties.](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_IMAGE) + [The bounding box has off frame values.](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_BOUNDING_BOX) + [The height and width of the bounding box is too small.](tm-debugging-json-line-errors.md#tm-error-ERROR_BOUNDING_BOX_TOO_SMALL) + [There are more bounding boxes than the allowed maximum.](tm-debugging-json-line-errors.md#tm-error-ERROR_TOO_MANY_BOUNDING_BOXES) + [No valid annotations found.](tm-debugging-json-line-errors.md#tm-error-ERROR_NO_VALID_ANNOTATIONS) # Understanding the manifest summary The manifest summary contains the following information. + Error information about [List of terminal manifest content errors](tm-debugging.md#tm-error-category-combined-terminal) encountered during validation. + Error location information for [List of non-terminal JSON line validation errors](tm-debugging.md#tm-error-category-non-terminal-errors) in the training and testing datasets. + Error statistics such as the total number of invalid JSON Lines found in the training and testing datasets. The manifest summary is created during training if there are no [List of terminal manifest file errors](tm-debugging.md#tm-error-category-terminal). To get the location of the manifest summary file (*manifest\$1summary.json*), see [Getting the validation results](tm-debugging-getting-validation-data.md). **Note** [Service errors](tm-debugging.md#tm-error-category-service) and [manifest file errors](tm-debugging.md#tm-error-category-terminal) are not reported in the manifest summary. For more information, see [Terminal errors](tm-debugging.md#tm-error-categories-terminal). For information about specific manifest content errors, see [Terminal manifest content errors](tm-debugging-aggregate-errors.md). ## Manifest summary file format A manifest file has 2 sections, `statistics` and `errors`. ### statistics `statistics` contains information about the errors in the training and testing datasets. + `training` – statistics and errors found in the training dataset. + `testing` – statistics and errors found in the testing dataset. Objects in the `errors` array contain the error code and message for manifest content errors. `` The `error_line_indices` array contains the line numbers for each JSON Line in the training or test manifest that has an error. For more information, see [Fixing training errors](tm-debugging-fixing-validation-errors.md). ### errors Errors spanning both the training and testing dataset. For example, an [ERROR\$1INSUFFICIENT\$1USABLE\$1LABEL\$1OVERLAP](tm-debugging-aggregate-errors.md#tm-error-ERROR_INSUFFICIENT_USABLE_LABEL_OVERLAP) occurs when there is isn't enough usable labels that overlap the training and testing datasets. ``` { "statistics": { "training": { "use_case": String, # Possible values are IMAGE_LEVEL_LABELS, OBJECT_LOCALIZATION and NOT_DETERMINED "total_json_lines": Number, # Total number json lines (images) in the training manifest. "valid_json_lines": Number, # Total number of JSON Lines (images) that can be used for training. "invalid_json_lines": Number, # Total number of invalid JSON Lines. They are not used for training. "ignored_json_lines": Number, # JSON Lines that have a valid schema but have no annotations. The aren't used for training and aren't counted as invalid. "error_json_line_indices": List[int], # Contains a list of line numbers for JSON line errors in the training dataset. "errors": [ { "code": String, # Error code for a training manifest content error. "message": String # Description for a training manifest content error. } ] }, "testing": { "use_case": String, # Possible values are IMAGE_LEVEL_LABELS, OBJECT_LOCALIZATION and NOT_DETERMINED "total_json_lines": Number, # Total number json lines (images) in the manifest. "valid_json_lines": Number, # Total number of JSON Lines (images) that can be used for testing. "invalid_json_lines": Number, # Total number of invalid JSON Lines. They are not used for testing. "ignored_json_lines": Number, # JSON Lines that have a valid schema but have no annotations. They aren't used for testing and aren't counted as invalid. "error_json_line_indices": List[int], # contains a list of error record line numbers in testing dataset. "errors": [ { "code": String, # # Error code for a testing manifest content error. "message": String # Description for a testing manifest content error. } ] } }, "errors": [ { "code": String, # # Error code for errors that span the training and testing datasets. "message": String # Description of the error. } ] } ``` ## Example manifest summary The following example is a partial manifest summary that shows a terminal manifest content error ([ERROR\$1TOO\$1MANY\$1INVALID\$1ROWS\$1IN\$1MANIFEST](tm-debugging-aggregate-errors.md#tm-error-ERROR_TOO_MANY_INVALID_ROWS_IN_MANIFEST)). The `error_json_line_indices` array contains the line numbers of non-terminal JSON Line errors in the corresponding training or testing validation manifest. ``` { "errors": [], "statistics": { "training": { "use_case": "NOT_DETERMINED", "total_json_lines": 301, "valid_json_lines": 146, "invalid_json_lines": 155, "ignored_json_lines": 0, "errors": [ { "code": "ERROR_TOO_MANY_INVALID_ROWS_IN_MANIFEST", "message": "The manifest file contains too many invalid rows." } ], "error_json_line_indices": [ 15, 16, 17, 22, 23, 24, . . . . 300 ] }, "testing": { "use_case": "NOT_DETERMINED", "total_json_lines": 15, "valid_json_lines": 13, "invalid_json_lines": 2, "ignored_json_lines": 0, "errors": [], "error_json_line_indices": [ 13, 15 ] } } } ``` # Understanding training and testing validation result manifests During training, Amazon Rekognition Custom Labels creates validation result manifests to hold non-terminal JSON Line errors. The validation results manifests are copies of the training and testing datasets with error information added. You can access the validation manifests after training completes. For more information, see [Getting the validation results](tm-debugging-getting-validation-data.md). Amazon Rekognition Custom Labels also creates a manifest summary that includes overview information for JSON Line errors, such as error locations and JSON Line error counts. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). **Note** Validation results (Training and Testing Validation Result Manifests and Manifest Summary) are only created if there are no [List of terminal manifest file errors](tm-debugging.md#tm-error-category-terminal). A manifest contains JSON Lines for each image in the dataset. Within the validation results manifests, JSON Line error information is added to the JSON Lines where errors occur. A JSON Line error is a non-terminal error related to a single image. A non-terminal validation error can invalidate the entire JSON Line or just a portion. For example, if the image referenced in a JSON Line is not in PNG or JPG format, an [ERROR\$1INVALID\$1IMAGE](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_IMAGE) error occurs and the entire JSON Line is excluded from training. Training continues with other valid JSON Lines. Within a JSON Line, an error might mean the JSON Line can stil be used for training. For example, if the left value for one of four bounding boxes associated with a label is negative, the model is still trained using the other valid bounding boxes. JSON Line error information is returned for the invalid bounding box ([ERROR\$1INVALID\$1BOUNDING\$1BOX](tm-debugging-json-line-errors.md#tm-error-ERROR_INVALID_BOUNDING_BOX)). In this example, the error information is added to the `annotation` object where the error occurs. Warning errors, such as [WARNING\$1NO\$1ANNOTATIONS](tm-debugging-json-line-errors.md#tm-warning-WARNING_NO_ANNOTATIONS), aren't used for training and count as ignored JSON lines (`ignored_json_lines`) in the manifest summary. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). Additionally, ignored JSON Lines don't count towards the 20% error threshold for training and testing. For information about specific non-terminal data validation errors, see [Non-Terminal JSON Line Validation Errors](tm-debugging-json-line-errors.md). **Note** If there are too many data validation errors, training is stopped and a [ERROR\$1TOO\$1MANY\$1INVALID\$1ROWS\$1IN\$1MANIFEST](tm-debugging-aggregate-errors.md#tm-error-ERROR_TOO_MANY_INVALID_ROWS_IN_MANIFEST) terminal error is reported in the manifest summary. For information about correcting JSON Line errors, see [Fixing training errors](tm-debugging-fixing-validation-errors.md). ## JSON line error format Amazon Rekognition Custom Labels adds non-terminal validation error information to image level and object localization format JSON Lines. For more information, see [Creating a manifest file](md-create-manifest-file.md). ### Image Level Errors The following example shows the `Error` arrays in an image level JSON Line. There are two sets of errors. Errors related to label attribute metadata (in this example, sport-metadata) and errors related to the image. An error includes an error code (code), error message (message). For more information, see [Importing image-level labels in manifest files](md-create-manifest-file-classification.md). ``` { "source-ref": String, "sport": Number, "sport-metadata": { "class-name": String, "confidence": Float, "type": String, "job-name": String, "human-annotated": String, "creation-date": String, "errors": [ { "code": String, # error codes for label "message": String # Description and additional contextual details of the error } ] }, "errors": [ { "code": String, # error codes for image "message": String # Description and additional contextual details of the error } ] } ``` ### Object localization errors The following example show the error arrays in an object localization JSON Line. The JSON Line contains an `Errors` array information for fields in the following JSON Line sections. Each `Error` object includes the error code and the error message. + *label attribute* – Errors for the label attribute fields. See `bounding-box` in the example. + *annotations* – Annotation errors (bounding boxes) are stored in the `annotations` array inside the label attribute. + *label attribute-metadata* – Errors for the label attribute metadata. See `bounding-box-metadata` in the example. + *image* – Errors not related to the label attribute, annotation, and label attribute metadata fields. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). ``` { "source-ref": String, "bounding-box": { "image_size": [ { "width": Int, "height": Int, "depth":Int, } ], "annotations": [ { "class_id": Int, "left": Int, "top": Int, "width": Int, "height": Int, "errors": [ # annotation field errors { "code": String, # annotation field error code "message": String # Description and additional contextual details of the error } ] } ], "errors": [ #label attribute field errors { "code": String, # error code "message": String # Description and additional contextual details of the error } ] }, "bounding-box-metadata": { "objects": [ { "confidence": Float } ], "class-map": { String: String }, "type": String, "human-annotated": String, "creation-date": String, "job-name": String, "errors": [ #metadata field errors { "code": String, # error code "message": String # Description and additional contextual details of the error } ] }, "errors": [ # image errors { "code": String, # error code "message": String # Description and additional contextual details of the error } ] } ``` ## Example JSON line error The following object localization JSON Line (formatted for readability) shows an [ERROR\$1BOUNDING\$1BOX\$1TOO\$1SMALL](tm-debugging-json-line-errors.md#tm-error-ERROR_BOUNDING_BOX_TOO_SMALL) error. In this example, the bounding box dimensions (height and width) aren't greater than 1 x 1. ``` { "source-ref": "s3://bucket/Manifests/images/199940-1791.jpg", "bounding-box": { "image_size": [ { "width": 3000, "height": 3000, "depth": 3 } ], "annotations": [ { "class_id": 1, "top": 0, "left": 0, "width": 1, "height": 1, "errors": [ { "code": "ERROR_BOUNDING_BOX_TOO_SMALL", "message": "The height and width of the bounding box is too small." } ] }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 } ] }, "bounding-box-metadata": { "objects": [ { "confidence": 1 }, { "confidence": 1 } ], "class-map": { "0": "Echo", "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2019-11-20T02:57:28.288286", "job-name": "my job" } } ``` # Getting the validation results The validation results contain error information for [List of terminal manifest content errors](tm-debugging.md#tm-error-category-combined-terminal) and [List of non-terminal JSON line validation errors](tm-debugging.md#tm-error-category-non-terminal-errors). There are three validation results files. + *training\$1manifest\$1with\$1validation.json* – A copy of the training dataset manifest file with JSON Line error information added. + *testing\$1manifest\$1with\$1validation.json* – A copy of the testing dataset manifest file with JSON Line error error information added. + *manifest\$1summary.json* – A summary of manifest content errors and JSON Line errors found in the training and testing datasets. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). For information about the contents of the training and testing validation manifests, see [Debugging a failed model training](tm-debugging.md). **Note** The validation results are created only if no [List of terminal manifest file errors](tm-debugging.md#tm-error-category-terminal) are generated during training. If a [service error](tm-debugging.md#tm-error-category-service) occurs after the training and testing manifest are validated, the validation results are created, but the response from [DescribeProjectVersions](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjectVersions) doesn't include the validation results file locations. After training completes or fails, you can download the validation results by using the Amazon Rekognition Custom Labels console or get the Amazon S3 bucket location by calling [DescribeProjectVersions](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjectVersions) API. ## Getting validation results (Console) If you are using the console to train your model, you can download the validation results from a project's list of models, as shown in the following diagram. The Models panel shows model training and validation results with option to download validation results. ![\[Interface showing model training and validation results with option to download validation results.\]](http://docs.aws.amazon.com/rekognition/latest/customlabels-dg/images/models-validation-results.jpg) You can also access download the validation results from a model's details page. The details page shows the dataset details with status, training and test datasets, and download links for manifest summary, training validation manifest, and testing validation manifest. ![\[Screenshot of the dataset details panel with status, links to training and test datasets, and download links for manifest items.\]](http://docs.aws.amazon.com/rekognition/latest/customlabels-dg/images/model-validation-results.jpg) For more information, see [Training a model (Console)](training-model.md#tm-console). ## Getting validation results (SDK) After model training completes, Amazon Rekognition Custom Labels stores the validation results in the Amazon S3 bucket specified during training. You can get the S3 bucket location by calling the [DescribeProjectVersions](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeProjectVersions) API, after training completes. To train a model, see [Training a model (SDK)](training-model.md#tm-sdk). A [ValidationData](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_ValidationData) object is returned for the training dataset ([TrainingDataResult](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_TrainingDataResult)) and the testing dataset ([TestingDataResult](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_TestingDataResult)). The manifest summary is returned in `ManifestSummary`. After you get the Amazon S3 bucket location, you can download the validation results. For more information, see [How do I download an object from an S3 bucket?](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/download-objects.html). You can also use the [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/dev/GettingObjectsUsingAPIs.html) operation. **To get validation data (SDK)** 1. If you haven't already done so, install and configure the AWS CLI and the AWS SDKs. For more information, see [Step 4: Set up the AWS CLI and AWS SDKs](su-awscli-sdk.md). 1. Use the following example to get the location of the validation results. ------ #### [ Python ] Replace `project_arn` with the Amazon Resource Name (ARN) of the project that contains the model. For more information, see [Managing an Amazon Rekognition Custom Labels project](managing-project.md). Replace `version_name` with the name of the model version. For more information, see [Training a model (SDK)](training-model.md#tm-sdk). ``` import boto3 import io from io import BytesIO import sys import json def describe_model(project_arn, version_name): client=boto3.client('rekognition') response=client.describe_project_versions(ProjectArn=project_arn, VersionNames=[version_name]) for model in response['ProjectVersionDescriptions']: print(json.dumps(model,indent=4,default=str)) def main(): project_arn='project_arn' version_name='version_name' describe_model(project_arn, version_name) if __name__ == "__main__": main() ``` ------ 1. In the program output, note the `Validation` field within the `TestingDataResult` and `TrainingDataResult` objects. The manifest summary is in `ManifestSummary`. # Fixing training errors You use the manifest summary to identify [List of terminal manifest content errors](tm-debugging.md#tm-error-category-combined-terminal) and [List of non-terminal JSON line validation errors](tm-debugging.md#tm-error-category-non-terminal-errors) encountered during training. You must fix manifest content errors. We recommend that you also fix non-terminal JSON Line errors. For information about specific errors, see [Non-Terminal JSON Line Validation Errors](tm-debugging-json-line-errors.md) and [Terminal manifest content errors](tm-debugging-aggregate-errors.md). You can makes fixes to the training or testing dataset used for training. Alternatively, you can make the fixes in the training and testing validation manifest files and use them to train the model. After you make your fixes, you need to import the updated manifests(s) and retrain the model. For more information, see [Creating a manifest file](md-create-manifest-file.md). The following procedure shows you how to use the manifest summary to fix terminal manifest content errors. The procedure also shows you how to locate and fix JSON Line errors in the training and testing validation manifests. **To fix Amazon Rekognition Custom Labels training errors** 1. Download the validation results files. The file names are *training\$1manifest\$1with\$1validation.json*, *testing\$1manifest\$1with\$1validation.json* and *manifest\$1summary.json*. For more information, see [Getting the validation results](tm-debugging-getting-validation-data.md). 1. Open the manifest summary file (*manifest\$1summary.json*). 1. Fix any errors in the manifest summary. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). 1. In the manifest summary, iterate through the `error_line_indices` array in `training` and fix the errors in `training_manifest_with_validation.json` at the corresponding JSON Line numbers. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). 1. Iterate through the `error_line_indices` array in `testing` and fix the errors in `testing_manifest_with_validation.json` at the corresponding JSON Line numbers. 1. Retrain the model using the validation manifest files as the training and testing datasets. For more information, see [Training an Amazon Rekognition Custom Labels model](training-model.md). If you are using the AWS SDK and choose to fix the errors in the training or the test validation data manifest files, use the location of the validation data manifest files in the [TrainingData](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_TrainingData) and [TestingData](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_TestingData) input parameters to [CreateProjectVersion](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_CreateProjectVersion). For more information, see [Training a model (SDK)](training-model.md#tm-sdk). ## JSON line error precedence The following JSON Line errors are detected first. If any of these errors occur, validation of JSON Line errors is stopped. You must fix these errors before you can fix any of the other JSON Line errors + MISSING\$1SOURCE\$1REF + ERROR\$1INVALID\$1SOURCE\$1REF\$1FORMAT + ERROR\$1NO\$1LABEL\$1ATTRIBUTES + ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1FORMAT + ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1METADATA\$1FORMAT + ERROR\$1MISSING\$1BOUNDING\$1BOX\$1CONFIDENCE + ERROR\$1MISSING\$1CLASS\$1MAP\$1ID + ERROR\$1INVALID\$1JSON\$1LINE # Terminal manifest file errors This topic describes the [List of terminal manifest file errors](tm-debugging.md#tm-error-category-terminal). Manifest file errors do not have an associated error code. The validation results manifests are not created when a terminal manifest file error occurs. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). Terminal manifest errors prevent the reporting of [Non-Terminal JSON Line Validation Errors](tm-debugging-json-line-errors.md). ## The manifest file extension or contents are invalid. The training or testing manifest file doesn't have a file extension or its contents are invalid. **To fix error *The manifest file extension or contents are invalid.*** + Check the following possible causes in both the training and testing manifest files. + The manifest file is missing a file extension. By convention the file extension is `.manifest`. + The Amazon S3 bucket or key for the manifest file couldn't be found. ## The manifest file is empty. The training or testing manifest file used for training exists, but it is empty. The manifest file needs a JSON Line for each image that you use for training and testing. **To fix error *The manifest file is empty.*** 1. Check which of the training or testing manifests are empty. 1. Add JSON Lines to the empty manifest file. For more information, see [Creating a manifest file](md-create-manifest-file.md). Alternatively, create a new dataset with the console. For more information, see [Creating training and test datasets with images](md-create-dataset.md). ## The manifest file size exceeds the maximum supported size. The training or testing manifest file size (in bytes) is too large. For more information, see [Guidelines and quotas in Amazon Rekognition Custom Labels](limits.md). A manifest file can have less than the maximum number of JSON Lines and still exceed the maximum file size. You can't use the Amazon Rekognition Custom Labels console to fix error *The manifest file size exceeds the maximum supported size*. **To fix error *The manifest file size exceeds the maximum supported size.*** 1. Check which of the training and testing manifests exceed the maximum file size. 1. Reduce the number of JSON Lines in the manifest files that are too large. For more information, see [Creating a manifest file](md-create-manifest-file.md). ## The S3 bucket permissions are incorrect. Amazon Rekognition Custom Labels doesn't have permissions to one or more of the buckets containing the training and testing manifest files. You can't use the Amazon Rekognition Custom Labels console to fix this error. **To fix error *The S3 bucket permissions are incorrect.*** + Check the permissions for the bucket(s) containing the training and testing manifests. For more information, see [Step 2: Set up Amazon Rekognition Custom Labels console permissions](su-console-policy.md). ## Unable to write to output S3 bucket. The service is unable to generate the training output files. **To fix error *Unable to write to output S3 bucket.*** + Check that the Amazon S3 bucket information in the [OutputConfig](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_OutputConfig) input parameter to [CreateProjectVersion](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_CreateProjectVersion) is correct. You can't use the Amazon Rekognition Custom Labels console to fix this error. # Terminal manifest content errors This topic describes the [List of terminal manifest content errors](tm-debugging.md#tm-error-category-combined-terminal) reported in the manifest summary. The manifest summary includes an error code and message for each detected error. For more information, see [Understanding the manifest summary](tm-debugging-summary.md). Terminal manifest content errors don't stop the reporting of [List of non-terminal JSON line validation errors](tm-debugging.md#tm-error-category-non-terminal-errors). ## ERROR\$1TOO\$1MANY\$1INVALID\$1ROWS\$1IN\$1MANIFEST ### Error message The manifest file contains too many invalid rows. ### More information An `ERROR_TOO_MANY_INVALID_ROWS_IN_MANIFEST` error occurs if there are too many JSON Lines that contain invalid content. You can't use the Amazon Rekognition Custom Labels console to fix an `ERROR_TOO_MANY_INVALID_ROWS_IN_MANIFEST` error. **To fix ERROR\$1TOO\$1MANY\$1INVALID\$1ROWS\$1IN\$1MANIFEST** 1. Check the manifest for JSON Line errors. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). 1. Fix JSON Lines that have errors For more information, see [Non-Terminal JSON Line Validation Errors](tm-debugging-json-line-errors.md). ## ERROR\$1IMAGES\$1IN\$1MULTIPLE\$1S3\$1BUCKETS ### Error message The manifest file contains images from multiple S3 buckets. ### More information A manifest can only reference images stored in a single bucket. Each JSON Line stores the Amazon S3 location of an image location in the value of `source-ref`. In the following example, the bucket name is *my-bucket*. ``` "source-ref": "s3://my-bucket/images/sunrise.png" ``` You can't use the Amazon Rekognition Custom Labels console to fix this error. **To fix `ERROR_IMAGES_IN_MULTIPLE_S3_BUCKETS`** + Ensure that all your images are in the same Amazon S3 bucket and that the value of `source-ref` in every JSON Line references the bucket where your images are stored. Alternatively, choose a preferred Amazon S3 bucket and remove the JSON Lines where `source-ref` doesn't reference your preferred bucket. ## ERROR\$1INVALID\$1PERMISSIONS\$1IMAGES\$1S3\$1BUCKET ### Error message The permissions for the images S3 bucket are invalid. ### More information The permissions on the Amazon S3 bucket that contains the images are incorrect. You can't use the Amazon Rekognition Custom Labels console to fix this error. **To fix `ERROR_INVALID_PERMISSIONS_IMAGES_S3_BUCKET`** + Check the permissions of the bucket containing the images. The value of the `source-ref` for an image contains the bucket location. ## ERROR\$1INVALID\$1IMAGES\$1S3\$1BUCKET\$1OWNER ### Error message Invalid owner id for images S3 bucket. ### More information The owner of the bucket that contains the training or test images is different from the owner of the bucket that contains the training or test manifest. You can use the following command to find the owner of a bucket. ``` aws s3api get-bucket-acl --bucket amzn-s3-demo-bucket ``` The `OWNER` `ID` must match for the buckets that store the images and manifest files. **To fix ERROR\$1INVALID\$1IMAGES\$1S3\$1BUCKET\$1OWNER** 1. Choose the desired owner of the training, testing, output, and image buckets. The owner must have permissions to use Amazon Rekognition Custom Labels. 1. For each bucket not currently owned by the desired owner, create a new Amazon S3 bucket owned by the preferred owner. 1. Copy the old bucket contents to the new bucket. For more information, see [How can I copy objects between Amazon S3 buckets?](https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/). You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1INSUFFICIENT\$1IMAGES\$1PER\$1LABEL\$1FOR\$1AUTOSPLIT ### Error message The manifest file contains insufficient labeled images per label to perform auto-split. ### More information During model training, you can create a testing dataset by using 20% of the images from the training dataset. ERROR\$1INSUFFICIENT\$1IMAGES\$1PER\$1LABEL\$1FOR\$1AUTOSPLIT occurs when there aren't enough images to create an acceptable testing dataset. You can't use the Amazon Rekognition Custom Labels console to fix this error. **To fix ERROR\$1INSUFFICIENT\$1IMAGES\$1PER\$1LABEL\$1FOR\$1AUTOSPLIT** + Add more labeled image to your training dataset. You can add images in the Amazon Rekognition Custom Labels console by adding images to the training dataset, or by adding JSON Lines to your training manifest. For more information, see [Managing datasets](managing-dataset.md). ## ERROR\$1MANIFEST\$1TOO\$1FEW\$1LABELS ### Error message The manifest file has too few labels. ### More information Training and testing datasets have a required minumum number of labels. The minimum depends on if the dataset trains/tests a model to detect image-level labels (classification) or if the model detects object locations. If the training dataset is split to create a testing dataset, the number of labels in the dataset is determined after the training dataset is split. For more information, see [Guidelines and quotas in Amazon Rekognition Custom Labels](limits.md). **To fix ERROR\$1MANIFEST\$1TOO\$1FEW\$1LABELS (console)** 1. Add more new labels to the dataset. For more information, see [Managing labels](md-labels.md). 1. Add the new labels to images in the dataset. If your model detects image-level labels, see [Assigning image-level labels to an image](md-assign-image-level-labels.md). If your model detects object locations, see [Labeling objects with bounding boxes](md-localize-objects.md). **To fix ERROR\$1MANIFEST\$1TOO\$1FEW\$1LABELS (JSON Line)** + Add JSON Lines for new images that have new labels. For more information, see [Creating a manifest file](md-create-manifest-file.md). If your model detects image-level labels, you add new labels names to the `class-name` field. For example, the label for the following image is *Sunrise*. ``` { "source-ref": "s3://bucket/images/sunrise.png", "testdataset-classification_Sunrise": 1, "testdataset-classification_Sunrise-metadata": { "confidence": 1, "job-name": "labeling-job/testdataset-classification_Sunrise", "class-name": "Sunrise", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "type": "groundtruth/image-classification" } } ``` If your model detects object locations, add new labels to the `class-map`, as shown in the following example. ``` { "source-ref": "s3://custom-labels-bucket/images/IMG_1186.png", "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, "bounding-box-metadata": { "objects": [{ "confidence": 1 }, { "confidence": 1 }], "class-map": { "0": "Echo", "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "job-name": "my job" } } ``` You need to map the class map table to the bounding box annotations. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). ## ERROR\$1MANIFEST\$1TOO\$1MANY\$1LABELS ### Error message The manifest file has too many labels. #### More information The number of unique labels in the manifest (dataset) is more than the allowed limit. If the training dataset is split to create a testing dataset, the mumber of labels is determined after the split. **To fix ERROR\$1MANIFEST\$1TOO\$1MANY\$1LABELS (Console)** + Remove labels from the dataset. For more information, see [Managing labels](md-labels.md). The labels are automatically removed from the images and bounding boxes in your dataset. **To fix ERROR\$1MANIFEST\$1TOO\$1MANY\$1LABELS (JSON Line)** + Manifests with image level JSON Lines – If the image has a single label, remove the JSON Lines for images that use the desired label. If the JSON Line contains multiple labels, remove only the JSON object for the desired label. For more information, see [Adding multiple image-level labels to an image](md-create-manifest-file-classification.md#md-dataset-purpose-classification-multiple-labels). Manifests with object location JSON Lines – Remove the bounding box and associated label information for the label that you want to remove. Do this for each JSON Line that contains the desired label. You need to remove the label from the `class-map` array and corresponding objects in the `objects` and `annotations` array. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). ## ERROR\$1INSUFFICIENT\$1LABEL\$1OVERLAP ### Error message Less than \$1\$1% label overlap between the training and testing manifest files. ### More information There is less than 50% overlap between the testing dataset label names and the training dataset label names. **To fix ERROR\$1INSUFFICIENT\$1LABEL\$1OVERLAP (Console)** + Remove labels from the training dataset. Alternatively, add more common labels to your testing dataset. For more information, see [Managing labels](md-labels.md). The labels are automatically removed from the images and bounding boxes in your dataset. **To fix ERROR\$1INSUFFICIENT\$1LABEL\$1OVERLAP by removing labels from the training dataset (JSON Line)** + Manifests with image level JSON Lines – If the image has a single label, remove the JSON Line for the image that use the desired label. If the JSON Line contains multiple labels, remove only the JSON object for the desired label. For more information, see [Adding multiple image-level labels to an image](md-create-manifest-file-classification.md#md-dataset-purpose-classification-multiple-labels). Do this for each JSON Line in the manifest that contains the label that you want to remove. Manifests with object location JSON Lines – Remove the bounding box and associated label information for the label that you want to remove. Do this for each JSON Line that contains the desired label. You need to remove the label from the `class-map` array and corresponding objects in the `objects` and `annotations` array. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). **To fix ERROR\$1INSUFFICIENT\$1LABEL\$1OVERLAP by adding common labels to the testing dataset (JSON Line)** + Add JSON Lines to the testing dataset that include images labeled with labels already in the training dataset. For more information, see [Creating a manifest file](md-create-manifest-file.md). ## ERROR\$1MANIFEST\$1TOO\$1FEW\$1USABLE\$1LABELS ### Error message The manifest file has too few usable labels. ### More information A training manifest can contain JSON Lines in image-level label format and in object location format. Depending on type of JSON Lines found in the training manifest, Amazon Rekognition Custom Labels chooses to create a model that detects image-level labels, or a model that detects object locations. Amazon Rekognition Custom Labels filters out valid JSON records for JSON Lines that are not in the chosen format. ERROR\$1MANIFEST\$1TOO\$1FEW\$1USABLE\$1LABELS occurs when the number of labels in the chosen model type manifest is insufficient to train the model. A minimum of 1 label is required to train a model that detects image-level labels. A minimum of 2 labels is required to train a model that object locations. **To fix ERROR\$1MANIFEST\$1TOO\$1FEW\$1USABLE\$1LABELS (Console)** 1. Check the `use_case` field in the manifest summary. 1. Add more labels to the training dataset for the use case (image level or object localization) that matches the value of `use_case`. For more information, see [Managing labels](md-labels.md). The labels are automatically removed from the images and bounding boxes in your dataset. **To fix ERROR\$1MANIFEST\$1TOO\$1FEW\$1USABLE\$1LABELS (JSON Line)** 1. Check the `use_case` field in the manifest summary. 1. Add more labels to the training dataset for the use case (image level or object localization) that matches the value of `use_case`. For more information, see [Creating a manifest file](md-create-manifest-file.md). ## ERROR\$1INSUFFICIENT\$1USABLE\$1LABEL\$1OVERLAP ### Error message Less than \$1\$1% usable label overlap between the training and testing manifest files. ### More information A training manifest can contain JSON Lines in image-level label format and in object location format. Depending on the formats found in the training manifest, Amazon Rekognition Custom Labels chooses to create a model that detects image-level labels, or a model that detects object locations. Amazon Rekognition Custom Labels doesn't use valid JSON records for JSON Lines that are not in the chosen model format. ERROR\$1INSUFFICIENT\$1USABLE\$1LABEL\$1OVERLAP occurs when there is less than 50% overlap between the testing and training labels that are used. **To fix ERROR\$1INSUFFICIENT\$1USABLE\$1LABEL\$1OVERLAP (Console)** + Remove labels from the training dataset. Alternatively, add more common labels to your testing dataset. For more information, see [Managing labels](md-labels.md). The labels are automatically removed from the images and bounding boxes in your dataset. **To fix ERROR\$1INSUFFICIENT\$1USABLE\$1LABEL\$1OVERLAP by removing labels from the training dataset (JSON Line)** + Datasets used to detect image-level labels – If the image has a single label, remove the JSON Line for the image that use the desired label. If the JSON Line contains multiple labels, remove only the JSON object for the desired label. For more information, see [Adding multiple image-level labels to an image](md-create-manifest-file-classification.md#md-dataset-purpose-classification-multiple-labels). Do this for each JSON Line in the manifest that contains the label that you want to remove. Datasets used to detects object locations – Remove the bounding box and associated label information for the label that you want to remove. Do this for each JSON Line that contains the desired label. You need to remove the label from the `class-map` array and corresponding objects in the `objects` and `annotations` array. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). **To fix ERROR\$1INSUFFICIENT\$1USABLE\$1LABEL\$1OVERLAP by adding common labels to the testing dataset (JSON Line)** + Add JSON Lines to the testing dataset that include images labeled with labels already in the training dataset. For more information, see [Creating a manifest file](md-create-manifest-file.md). ## ERROR\$1FAILED\$1IMAGES\$1S3\$1COPY ### Error message Failed to copy images from S3 bucket. ### More information The service wasn't able to copy any of the images in your your dataset. You can't use the Amazon Rekognition Custom Labels console to fix this error. **To fix ERROR\$1FAILED\$1IMAGES\$1S3\$1COPY** 1. Check the permissions of your images. 1. If you are using AWS KMS, check the bucket policy. For more information, see [Decrypting files encrypted with AWS Key Management Service](su-encrypt-bucket.md#su-kms-encryption). ## The manifest file has too many terminal errors. There are too many JSON lines with terminal content errors. **To fix `ERROR_TOO_MANY_RECORDS_IN_ERROR`** + Reduce the number of JSON Lines (images) with terminal content errors. For more information, see [Terminal manifest content errors](#tm-debugging-aggregate-errors). You can't use the Amazon Rekognition Custom Labels console to fix this error. # Non-Terminal JSON Line Validation Errors This topic lists the non-terminal JSON Line validation errors reported by Amazon Rekognition Custom Labels during training. The errors are reported in the training and testing validation manifest. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). You can fix a non-terminal JSON Line error by updating the JSON Line in the training or test manifest file. You can also remove the JSON Line from the manifest, but doing so might reduce the quality of your model. If there are many non-terminal validation errors, you might find it easier to recreate the manifest file. Validation errors typically occur in manually created manifest files. For more information, see [Creating a manifest file](md-create-manifest-file.md). For information about fixing validation errors, see [Fixing training errors](tm-debugging-fixing-validation-errors.md). Some errors can be fixed by using the Amazon Rekognition Custom Labels console. ## ERROR\$1MISSING\$1SOURCE\$1REF ### Error message The source-ref key is missing. ### More information The JSON Line `source-ref` field provides the Amazon S3 location of an image. This error occurs when the `source-ref` key is missing or is misspelt. This error typically occurs in manually created manifest files. For more information, see [Creating a manifest file](md-create-manifest-file.md). **To fix `ERROR_MISSING_SOURCE_REF`** 1. Check that the `source-ref` key is present and is spelt correctly. A complete `source-ref` key and value is similar to the following. is `"source-ref": "s3://bucket/path/image"`. 1. Update or the `source-ref` key in the JSON Line. Alternatively, remove, the JSON Line from the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1INVALID\$1SOURCE\$1REF\$1FORMAT ### Error message The format of the source-ref value is invalid. ### More information The `source-ref` key is present in the JSON Line, but the schema of the Amazon S3 path is incorrect. For example, the path is `https://....` instead of `S3://....`. An ERROR\$1INVALID\$1SOURCE\$1REF\$1FORMAT error typically occurs in manually created manifest files. For more information, see [Creating a manifest file](md-create-manifest-file.md). **To fix `ERROR_INVALID_SOURCE_REF_FORMAT`** 1. Check that the schema is `"source-ref": "s3://bucket/path/image"`. For example, `"source-ref": "s3://custom-labels-console-us-east-1-1111111111/images/000000242287.jpg"`. 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this `ERROR_INVALID_SOURCE_REF_FORMAT`. ## ERROR\$1NO\$1LABEL\$1ATTRIBUTES ### Error message No label attributes found. ### More information The label attribute or the label attribute `-metadata` key name (or both) is invalid or missing. In the following example, `ERROR_NO_LABEL_ATTRIBUTES` occurs whenever the `bounding-box` or `bounding-box-metadata` key (or both) is missing. For more information, see [Creating a manifest file](md-create-manifest-file.md). ``` { "source-ref": "s3://custom-labels-bucket/images/IMG_1186.png", "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, "bounding-box-metadata": { "objects": [{ "confidence": 1 }, { "confidence": 1 }], "class-map": { "0": "Echo", "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "job-name": "my job" } } ``` A `ERROR_NO_LABEL_ATTRIBUTES` error typically occurs in a manually created manifest file. For more information, see [Creating a manifest file](md-create-manifest-file.md). **To fix `ERROR_NO_LABEL_ATTRIBUTES`** 1. Check that label attribute identifier and label attribute identifer `-metadata` keys are present and that the key names are spelt correctly. 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix `ERROR_NO_LABEL_ATTRIBUTES` . ## ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1FORMAT ### Error message The format of the label attribute \$1\$1 is invalid. ### More information The schema for the label attribute key is missing or invalid. An ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1FORMAT error typically occurs in manually created manifest files. for more information, see [Creating a manifest file](md-create-manifest-file.md). **To fix `ERROR_INVALID_LABEL_ATTRIBUTE_FORMAT`** 1. Check that the JSON Line section for the label attribute key is correct. In the following example object location example, the `image_size` and `annotations` objects must be correct. The label attribute key is named `bounding-box`. ``` "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, ``` 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1METADATA\$1FORMAT ### Error message The format of the label attribute metadata is invalid. ### More information The schema for the label attribute metadata key is missing or invalid. An ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1METADATA\$1FORMAT error typically occurs in manually created manifest files. For more information, see [Creating a manifest file](md-create-manifest-file.md). **To fix `ERROR_INVALID_LABEL_ATTRIBUTE_FORMAT`** 1. Check that the JSON Line schema for the label attribute metadata key is similar to the following example. The label attribute metadata key is named `bounding-box-metadata`. ``` "bounding-box-metadata": { "objects": [{ "confidence": 1 }, { "confidence": 1 }], "class-map": { "0": "Echo", "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "job-name": "my job" } ``` 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1NO\$1VALID\$1LABEL\$1ATTRIBUTES ### Error message No valid label attributes found. ### More information No valid label attributes were found in the JSON Line. Amazon Rekognition Custom Labels checks both the label attribute and the label attribute identifier. An ERROR\$1INVALID\$1LABEL\$1ATTRIBUTE\$1FORMAT error typically occurs in manually created manifest files. for more information, see [Creating a manifest file](md-create-manifest-file.md). If a JSON Line isn't in a supported SageMaker AI manifest format, Amazon Rekognition Custom Labels marks the JSON Line as invalid and an `ERROR_NO_VALID_LABEL_ATTRIBUTES` error is reported. Currently, Amazon Rekognition Custom Labels supports classification job and bounding box formats. For more information, see [Creating a manifest file](md-create-manifest-file.md). **To fix `ERROR_NO_VALID_LABEL_ATTRIBUTES`** 1. Check that the JSON for the label attribute key and label attribute metadata is correct. 1. Update, or remove, the JSON Line in the manifest file. For more information, see [Creating a manifest file](md-create-manifest-file.md). You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1MISSING\$1BOUNDING\$1BOX\$1CONFIDENCE ### Error message One or more bounding boxes has a missing confidence value. ### More information The confidence key is missing for one or more object location bounding boxes. The confidence key for a bounding box is in the label attribute metadata, as shown in the following example. A ERROR\$1MISSING\$1BOUNDING\$1BOX\$1CONFIDENCE error typically occurs in manually created manifest files. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). ``` "bounding-box-metadata": { "objects": [{ "confidence": 1 }, { "confidence": 1 }], ``` **To fix `ERROR_MISSING_BOUNDING_BOX_CONFIDENCE`** 1. Check that the `objects` array in the label attribute contains the same number of confidence keys as there are objects in the label attribute `annotations` array. 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1MISSING\$1CLASS\$1MAP\$1ID ### Error message One of more class ids is missing from the class map. ### More information The `class_id` in an annotation (bounding box) object doesn't have a matching entry in the label attribute metadata class map (`class-map`). For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). A ERROR\$1MISSING\$1CLASS\$1MAP\$1ID error typically occurs in manually created manifest files. **To fix ERROR\$1MISSING\$1CLASS\$1MAP\$1ID** 1. Check that the `class_id` value in each annotation (bounding box) object has a corresponding value in the `class-map` array, as shown in the following example. The `annotations` array and `class_map` array should have the same number of elements. ``` { "source-ref": "s3://custom-labels-bucket/images/IMG_1186.png", "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, "bounding-box-metadata": { "objects": [{ "confidence": 1 }, { "confidence": 1 }], "class-map": { "0": "Echo", "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "job-name": "my job" } } ``` 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1INVALID\$1JSON\$1LINE ### Error message The JSON Line has an invalid format. ### More information An unexpected character was found in the JSON Line. The JSON Line is replaced with a new JSON Line that contains only the error information. An ERROR\$1INVALID\$1JSON\$1LINE error typically occurs in manually created manifest files. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). You can't use the Amazon Rekognition Custom Labels console to fix this error. **To fix `ERROR_INVALID_JSON_LINE`** 1. Open the manifest file and navigate to the JSON Line where the ERROR\$1INVALID\$1JSON\$1LINE error occurs. 1. Check that the JSON Line doesn't contain invalid characters and that required `;` or `,` characters are not missing. 1. Update, or remove, the JSON Line in the manifest file. ## ERROR\$1INVALID\$1IMAGE ### Error message The image is invalid. Check S3 path and/or image properties. ### More information The file referenced by `source-ref` is not a valid image. Potential causes include the image aspect ratio, the size of the image, and the image format. For more information, see [Guidelines and quotas in Amazon Rekognition Custom Labels](limits.md). **To fix `ERROR_INVALID_IMAGE`** 1. Check the following. + The aspect ratio of the image is less than 20:1. + The size of the image is greater than 15 MB + The image is in PNG or JPEG format. + The path to the image in `source-ref` is correct. + The minimum image dimension of the image is greater 64 pixels x 64 pixels. + The maximum image dimension of the image is less than 4096 pixels x 4096 pixels. 1. Update, or remove, the JSON Line in the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1INVALID\$1IMAGE\$1DIMENSION ### Error message The image dimension(s) do not conform to allowed dimensions. ### More information The image referenced by `source-ref` doesn't conform to the allowed image dimensions. The minimum dimension is 64 pixels. The maximum dimension is 4096 pixels. `ERROR_INVALID_IMAGE_DIMENSION` is reported for images with bounding boxes. For more information, see [Guidelines and quotas in Amazon Rekognition Custom Labels](limits.md). **To fix `ERROR_INVALID_IMAGE_DIMENSION` (Console)** 1. Update the image in the Amazon S3 bucket with dimensions that Amazon Rekognition Custom Labels can process. 1. In the Amazon Rekognition Custom Labels console, do the following: 1. Remove the existing bounding boxes from the image. 1. Re-add the bounding boxes to the image. 1. Save your changes. For more information, [Labeling objects with bounding boxes](md-localize-objects.md). **To fix `ERROR_INVALID_IMAGE_DIMENSION` (SDK)** 1. Update the image in the Amazon S3 bucket with dimensions that Amazon Rekognition Custom Labels can process. 1. Get the existing JSON Line for the image by calling [ListDatasetEntries](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_ListDatasetEntries). For the `SourceRefContains` input parameter specify the Amazon S3 location and filename of the image. 1. Call [UpdateDatasetEntries](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_UpdateDatasetEntries) and provide the JSON line for the image. Make sure the value of `source-ref` matches the image location in the Amazon S3 bucket. Update the bounding box annotations to match the bounding box dimensions needed for the updated image. ``` { "source-ref": "s3://custom-labels-bucket/images/IMG_1186.png", "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, "bounding-box-metadata": { "objects": [{ "confidence": 1 }, { "confidence": 1 }], "class-map": { "0": "Echo", "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2013-11-18T02:53:27", "job-name": "my job" } } ``` ## ERROR\$1INVALID\$1BOUNDING\$1BOX ### Error message The bounding box has off frame values. ### More information The bounding box information specifies an image that is either off the image frame or contains negative values. For more information, see [Guidelines and quotas in Amazon Rekognition Custom Labels](limits.md). **To fix `ERROR_INVALID_BOUNDING_BOX`** 1. Check the values of the bounding boxes in the `annotations` array. ``` "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }] }, ``` 1. Update, or alternatively remove, the JSON Line from the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1NO\$1VALID\$1ANNOTATIONS ### Error message No valid annotations found. ### More information None of the annotation objects in the JSON Line contain valid bounding box information. **To fix `ERROR_NO_VALID_ANNOTATIONS`** 1. Update the `annotations` array to include valid bounding box objects. Also, check that corresponding bounding box information (`confidence` and `class_map`) in the label attribute metadata is correct. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). ``` { "source-ref": "s3://custom-labels-bucket/images/IMG_1186.png", "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [ { "class_id": 1, #annotation object "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, "bounding-box-metadata": { "objects": [ >{ "confidence": 1 #confidence object }, { "confidence": 1 }], "class-map": { "0": "Echo", #label "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "job-name": "my job" } } ``` 1. Update, or alternatively remove, the JSON Line from the manifest file. You can't use the Amazon Rekognition Custom Labels console to fix this error. ## ERROR\$1BOUNDING\$1BOX\$1TOO\$1SMALL ### Error message The height and width of the bounding box is too small. ### More information The bounding box dimensions (height and width) have to be greater than 1 x 1 pixels. During training, Amazon Rekognition Custom Labels resizes an image if any of its dimensions are greater than 1280 pixels (the source images aren't affected). The resulting bounding box heights and widths must be greater than 1 x 1 pixels. A bounding box location is stored in the `annotations` array of an object location JSON Line. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md) ``` "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [{ "class_id": 1, "top": 251, "left": 399, "width": 155, "height": 101 }] }, ``` The error information is added to the annotation object. **To fix ERROR\$1BOUNDING\$1BOX\$1TOO\$1SMALL** + Choose one of the following options. + Increase the size of bounding boxes that are too small. + Remove bounding boxes that are too small. For information about removing a bounding box, see [ERROR\$1TOO\$1MANY\$1BOUNDING\$1BOXES](#tm-error-ERROR_TOO_MANY_BOUNDING_BOXES). + Remove the image (JSON Line) from the manifest. ## ERROR\$1TOO\$1MANY\$1BOUNDING\$1BOXES ### Error message There are more bounding boxes than the allowed maximum. ### More information There are more bounding boxes than the allowed limit (50). You can remove excess bounding boxes in the Amazon Rekognition Custom Labels console, or you can remove them from the JSON Line. **To fix `ERROR_TOO_MANY_BOUNDING_BOXES` (Console).** 1. Decide which bounding boxes to remove. 1. Open the Amazon Rekognition console at [https://console.aws.amazon.com/rekognition/](https://console.aws.amazon.com/rekognition/). 1. Choose **Use Custom Labels**. 1. Choose **Get started**. 1. In the left navigation pane, choose the project that contains the dataset that you want to use. 1. In the **Datasets** section, choose the dataset that you want to use. 1. In the dataset gallery page, choose **Start labeling** to enter labeling mode. 1. Choose the image that you want to remove bounding boxes from. 1. Choose **Draw bounding box**. 1. In the drawing tool, choose the bounding box that you want to delete. 1. Press the delete key on your keyboard to delete the bounding box. 1. Repeat the previous 2 steps until you have deleted enough bounding boxes. 1. Choose **Done** 1. Choose **Save changes** to save your changes. 1. Choose **Exit** to exit labeling mode. **To fix ERROR\$1TOO\$1MANY\$1BOUNDING\$1BOXES (JSON Line).** 1. Open the manifest file and navigate to the JSON Line where the ERROR\$1TOO\$1MANY\$1BOUNDING\$1BOXES error occurs. 1. Remove the following for each bounding box that you want to remove. + Remove the required `annotation` object from `annotations` array. + Remove the corresponding `confidence` object from the `objects` array in the label attribute metadata. + If no longer used by other bounding boxes, remove the label from the `class-map`. Use the following example to identify which items to remove. ``` { "source-ref": "s3://custom-labels-bucket/images/IMG_1186.png", "bounding-box": { "image_size": [{ "width": 640, "height": 480, "depth": 3 }], "annotations": [ { "class_id": 1, #annotation object "top": 251, "left": 399, "width": 155, "height": 101 }, { "class_id": 0, "top": 65, "left": 86, "width": 220, "height": 334 }] }, "bounding-box-metadata": { "objects": [ >{ "confidence": 1 #confidence object }, { "confidence": 1 }], "class-map": { "0": "Echo", #label "1": "Echo Dot" }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2018-10-18T22:18:13.527256", "job-name": "my job" } } ``` ## WARNING\$1UNANNOTATED\$1RECORD ### Warning Message Record is unannotated. ### More information An image added to a dataset by using the Amazon Rekognition Custom Labels console wasn't labeled. The JSON line for the image isn't used for training. ``` { "source-ref": "s3://bucket/images/IMG_1186.png", "warnings": [ { "code": "WARNING_UNANNOTATED_RECORD", "message": "Record is unannotated." } ] } ``` **To fix WARNING\$1UNANNOTATED\$1RECORD** + Label the image by using the Amazon Rekognition Custom Labels console. For instructions, see [Assigning image-level labels to an image](md-assign-image-level-labels.md). ## WARNING\$1NO\$1ANNOTATIONS ### Warning Message No annotations provided. ### More information A JSON Line in Object Localization format doesn't contain any bounding box information, despite being annotated by a human (`human-annotated = yes`). The JSON Line is valid, but isn't used for training. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). ``` { "source-ref": "s3://bucket/images/IMG_1186.png", "bounding-box": { "image_size": [ { "width": 640, "height": 480, "depth": 3 } ], "annotations": [ ], "warnings": [ { "code": "WARNING_NO_ATTRIBUTE_ANNOTATIONS", "message": "No attribute annotations were found." } ] }, "bounding-box-metadata": { "objects": [ ], "class-map": { }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2013-11-18 02:53:27", "job-name": "my job" }, "warnings": [ { "code": "WARNING_NO_ANNOTATIONS", "message": "No annotations were found." } ] } ``` **To fix WARNING\$1NO\$1ANNOTATIONS** + Choose one of the following options. + Add the bounding box (`annotations`) information to the JSON Line. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). + Remove the image (JSON Line) from the manifest. ## WARNING\$1NO\$1ATTRIBUTE\$1ANNOTATIONS ### Warning Message No attribute annotations provided. #### More information A JSON Line in Object Localization format doesn't contain any bounding box annotation information, despite being annotated by a human (`human-annotated = yes`). The `annotations` array is not present or is not populuated. The JSON Line is valid, but isn't used for training. For more information, see [Understanding training and testing validation result manifests](tm-debugging-scope-json-line.md). ``` { "source-ref": "s3://bucket/images/IMG_1186.png", "bounding-box": { "image_size": [ { "width": 640, "height": 480, "depth": 3 } ], "annotations": [ ], "warnings": [ { "code": "WARNING_NO_ATTRIBUTE_ANNOTATIONS", "message": "No attribute annotations were found." } ] }, "bounding-box-metadata": { "objects": [ ], "class-map": { }, "type": "groundtruth/object-detection", "human-annotated": "yes", "creation-date": "2013-11-18 02:53:27", "job-name": "my job" }, "warnings": [ { "code": "WARNING_NO_ANNOTATIONS", "message": "No annotations were found." } ] } ``` **To fix WARNING\$1NO\$1ATTRIBUTE\$1ANNOTATIONS** + Choose one of the following options. + Add one or more bounding box `annotation` objects to the JSON Line. For more information, see [Object localization in manifest files](md-create-manifest-file-object-detection.md). + Remove the bounding box attribute. + Remove the image (JSON Line) from the manifest. If other valid bounding box attributes exist in the JSON Line, you can instead remove just the invalid bounding box attribute from the JSON Line. ## ERROR\$1UNSUPPORTED\$1USE\$1CASE\$1TYPE ### Warning Message ### More information The value of the `type `field isn't `groundtruth/image-classification` or `groundtruth/object-detection`. For more information, see [Creating a manifest file](md-create-manifest-file.md). ``` { "source-ref": "s3://bucket/test_normal_8.jpg", "BB": { "annotations": [ { "left": 1768, "top": 1007, "width": 448, "height": 295, "class_id": 0 }, { "left": 1794, "top": 1306, "width": 432, "height": 411, "class_id": 1 }, { "left": 2568, "top": 1346, "width": 710, "height": 305, "class_id": 2 }, { "left": 2571, "top": 1020, "width": 644, "height": 312, "class_id": 3 } ], "image_size": [ { "width": 4000, "height": 2667, "depth": 3 } ] }, "BB-metadata": { "job-name": "labeling-job/BB", "class-map": { "0": "comparator", "1": "pot_resistor", "2": "ir_phototransistor", "3": "ir_led" }, "human-annotated": "yes", "objects": [ { "confidence": 1 }, { "confidence": 1 }, { "confidence": 1 }, { "confidence": 1 } ], "creation-date": "2021-06-22T09:58:34.811Z", "type": "groundtruth/wrongtype", "cl-errors": [ { "code": "ERROR_UNSUPPORTED_USE_CASE_TYPE", "message": "The use case type of the BB-metadata label attribute metadata is unsupported. Check the type field." } ] }, "cl-metadata": { "is_labeled": true }, "cl-errors": [ { "code": "ERROR_NO_VALID_LABEL_ATTRIBUTES", "message": "No valid label attributes found." } ] } ``` **To fix ERROR\$1UNSUPPORTED\$1USE\$1CASE\$1TYPE** + Choose one of the following options: + Change the value of the `type`field to `groundtruth/image-classification` or `groundtruth/object-detection`, depending on the type of model that you want to create. For more information, see [Creating a manifest file](md-create-manifest-file.md). + Remove the image (JSON Line) from the manifest. ## ERROR\$1INVALID\$1LABEL\$1NAME\$1LENGTH ### More information The length of a label name is too long. The maximum length is 256 characters. **To fix ERROR\$1INVALID\$1LABEL\$1NAME\$1LENGTH** + Choose one of the following options: + Reduce the length of the label name to 256 characters or less. + Remove the image (JSON Line) from the manifest.