# Use Cases and Examples Using Amazon A2I Use Cases and Examples You can use Amazon Augmented AI to incorporate a human review into your workflow for *built-in task types*, Amazon Textract and Amazon Rekognition, or your own custom tasks using a *custom task type*. When you create a human review workflow using one of the built-in task types, you can specify conditions, such as confidence thresholds, that initiate a human review. The service (Amazon Rekognition or Amazon Textract) creates a human loop on your behalf when these conditions are met and supplies your input data directly to Amazon A2I to send to human reviewers. To learn more about the built-in task types, use the following: + [Use Amazon Augmented AI with Amazon Textract](a2i-textract-task-type.md) + [Use Amazon Augmented AI with Amazon Rekognition](a2i-rekognition-task-type.md) When you use a custom task type, you create and start a human loop using the Amazon A2I Runtime API. Use the custom task type to incorporate a human review workflow with other AWS services or your own custom ML application. + For more details, see [Use Amazon Augmented AI with Custom Task Types](a2i-task-types-custom.md) The following table outlines a variety of Amazon A2I use cases that you can explore using SageMaker AI Jupyter Notebooks. To get started with a Jupyter Notebook, use the instructions in [Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook](#a2i-task-types-notebook-demo). For more examples, see this [GitHub repository](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks). **** | **Use Case** | **Description** | **Task Type** | | --- | --- | --- | | [Use Amazon A2I with Amazon Textract](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(A2I)%20and%20Textract%20AnalyzeDocument.ipynb) | Have humans review single-page documents to review important form key-value pairs, or have Amazon Textract randomly sample and send documents from your dataset to humans for review. | Built-in | | [Use Amazon A2I with Amazon Rekognition](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(A2I)%20and%20Rekognition%20DetectModerationLabels.ipynb) | Have humans review unsafe images for explicit adult or violent content if Amazon Rekognition returns a low confidence score, or have Amazon Rekognition randomly sample and send images from your dataset to humans for review. | Built-in | | [Use Amazon A2I with Amazon Comprehend](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(A2I)%20and%20Comprehend%20DetectSentiment.ipynb) | Have humans review Amazon Comprehend inferences about text data such as sentiment analysis, text syntax, and entity detection. | Custom | | [Use Amazon A2I with Amazon Transcribe](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/A2I-Video-Transcription-with-Amazon-Transcribe.ipynb) | Have humans review Amazon Transcribe transcriptions of video or audio files. Use the results of transcription human review loops to create a custom vocabulary and improve future transcriptions of similar video or audio content. | Custom | | [Use Amazon A2I with Amazon Translate](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(A2I)%20and%20Amazon%20Translate.ipynb) | Have humans review low-confidence translations returned from Amazon Translate. | Custom | | [Use Amazon A2I to review real time ML inferences](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20A2I%20with%20Amazon%20SageMaker%20for%20object%20detection%20and%20model%20retraining.ipynb) | Use Amazon A2I to review real-time, low-confidence inferences made by a model deployed to a SageMaker AI hosted endpoint and incrementally train your model using Amazon A2I output data. | Custom | | [Use Amazon A2I to review tabular data](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(Amazon%20A2I)%20Integration%20with%20tabular%20data.ipynb) | Use Amazon A2I to integrate a human review loop into an ML application that uses tabular data. | Custom | **Topics** + [ ## Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook ](#a2i-task-types-notebook-demo) + [ # Use Amazon Augmented AI with Amazon Textract ](a2i-textract-task-type.md) + [ # Use Amazon Augmented AI with Amazon Rekognition ](a2i-rekognition-task-type.md) + [ # Use Amazon Augmented AI with Custom Task Types ](a2i-task-types-custom.md) ## Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook For an end-to-end example that demonstrates how to integrate an Amazon A2I human review loop into a machine learning workflow, you can use a Jupyter Notebook from this [GitHub Repository](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks) in a SageMaker notebook instance. **To use an Amazon A2I custom task type sample notebook in an Amazon SageMaker notebook instance:** 1. If you do not have an active SageMaker notebook instance, create one by following the instructions in [Create an Amazon SageMaker Notebook Instance for the tutorial](gs-setup-working-env.md). 1. When your notebook instance is active, choose **Open JupyterLab** to the right of the notebook instance's name. It may take a few moments for JupyterLab to load. 1. Choose the add Github repository icon (![\[Diagonal arrow icon with N and 1 symbols, representing a one-to-many relationship.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/icons/Git_squip_add_repo.png)) to clone a GitHub repository into your workspace. 1. Enter the [amazon-a2i-sample-jupyter-notebooks](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks) repository HTTPS URL. 1. Choose **CLONE**. 1. Open the notebook that you would like to run. 1. Follow the instructions in the notebook to configure your human review workflow and human loop and run the cells. 1. To avoid incurring unnecessary charges, when you are done with the demo, stop and delete your notebook instance in addition to any Amazon S3 buckets, IAM roles, and CloudWatch Events resources created during the walkthrough. # Use Amazon Augmented AI with Amazon Textract Use with Amazon Textract Amazon Textract enables you to add document text detection and analysis to your applications. Amazon Augmented AI (Amazon A2I) directly integrates with Amazon Textract's `AnalyzeDocument` API operation. You can use `AnalyzeDocument` to analyze a document for relationships between detected items. When you add an Amazon A2I human review loop to an `AnalyzeDocument` request, Amazon A2I monitors the Amazon Textract results and sends a document to one or more human workers for review when the conditions specified in your flow definition are met. For example, if you want a human to review a specific key like `Full name:` and their associated input values, you can create an activation condition that starts a human review any time the `Full name:` key is detected or when the inference confidence for that key falls within a range that you specify. The following image depicts the Amazon A2I built-in workflow with Amazon Textract. On the left, the resources that are required to create an Amazon Textract human review workflow are depicted: and Amazon S3 bucket, activation conditions, a worker task template, and a work team. These resources are used to create a human review workflow, or flow definition. An arrow points right to the next step in the workflow: using Amazon Textract to configure a human loop with the human review workflow. A second arrow points right from this step to the step in which activation conditions specified in the human review workflow are met. This initiates the creation of a human loop. On the right of the image, the human loop is depicted in three steps: 1) the worker UI and tools are generated and the task is made available to workers,2) workers review input data, and finally, 3) results are saved in Amazon S3. ![\[Use Amazon Augmented AI with Amazon Textract\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/a2i/diagrams/product-page-diagram_A21-Components_Textract@2x.png) You can specify when Amazon Textract sends a task to a human worker for review when creating a human review workflow or flow definition by specifying *activation conditions*. You can set the following activation conditions when using the Amazon Textract task type: + Initiate a human review for specific form keys based on the form key confidence score. + Initiate a human review when specific form keys are missing. + Initiate human review for all form keys identified by Amazon Textract with confidence scores in a specified range. + Randomly send a sample of forms to humans for review. When your activation condition depends on form key confidence scores, you can use two types of prediction confidence to initiate human loops: + **Identification confidence** – The confidence score for key-value pairs detected within a form. + **Qualification confidence** – The confidence score for text contained within key and value in a form. In the image in the following section, **Full Name: Jane Doe** is the key-value pair, **Full Name** is the key, and **Jane Doe** is the value. You can set these activation conditions using the Amazon SageMaker AI console when you create a human review workflow, or by creating a JSON for human loop activation conditions and specifying this as input in the `HumanLoopActivationConditions` parameter of `CreateFlowDefinition` API operation. To learn how specify activation conditions in JSON format, see [JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI](a2i-human-fallback-conditions-json-schema.md) and [Use Human Loop Activation Conditions JSON Schema with Amazon Textract](a2i-json-humantaskactivationconditions-textract-example.md). **Note** When using Augmented AI with Amazon Textract, create Augmented AI resources in the same AWS Region you use to call `AnalyzeDocument`. ## Get Started: Integrate a Human Review into an Amazon Textract Analyze Document Job To integrate a human review into an Amazon Textract text detection and analysis job, you need to create a flow definition, and then use the Amazon Textract API to integrate that flow definition into your workflow. To learn how to create a flow definition using the SageMaker AI console or Augmented AI API, see the following topics: + [Create a Human Review Workflow (Console)](a2i-create-flow-definition.md#a2i-create-human-review-console) + [Create a Human Review Workflow (API)](a2i-create-flow-definition.md#a2i-create-human-review-api) After you've created your flow definition, see [Using Augmented AI with Amazon Textract](https://docs.aws.amazon.com/textract/latest/dg/a2i-textract.html) to learn how to integrate your flow definition into your Amazon Textract task. ## End-to-End Example Using Amazon Textract and Amazon A2I For an end-to-end example that demonstrates how to use Amazon Textract with Amazon A2I using the console, see [Tutorial: Get Started in the Amazon A2I Console](a2i-get-started-console.md). To learn how to use the Amazon A2I API to create and start a human review, you can use [Amazon Augmented AI (Amazon A2I) integration with Amazon Textract's Analyze Document [Example]](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(A2I)%20and%20Textract%20AnalyzeDocument.ipynb) in a SageMaker Notebook instance. To get started, see [Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook](a2i-task-types-general.md#a2i-task-types-notebook-demo). ## A2I Textract Worker Console Preview When they're assigned a review task in an Amazon Textract workflow, workers might see a user interface similar to the following: ![\[Example review task in the A2I Textract worker console.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/a2i-textract-example.png) You can customize this interface in the SageMaker AI console when you create your human review definition, or by creating and using a custom template. To learn more, see [Create and Manage Worker Task Templates](a2i-instructions-overview.md). # Use Amazon Augmented AI with Amazon Rekognition Use with Amazon Rekognition Amazon Rekognition makes it easy to add image analysis to your applications. The Amazon Rekognition `DetectModerationLabels` API operation is directly integrated with Amazon A2I so that you can easily create a human loop to review unsafe images, such as explicit adult or violent content. You can use `DetectModerationLabels` to configure a human loop using a flow definition ARN. This enables Amazon A2I to analyze predictions made by Amazon Rekognition and send results to a human for review to ensure they meet the conditions set in your flow definition. The following image depicts the Amazon A2I built-in workflow with Amazon Rekognition. On the left, the resources that are required to create an Amazon Rekognition human review workflow are depicted: and Amazon S3 bucket, activation conditions, a worker task template, and a work team. These resources are used to create a human review workflow, or flow definition. An arrow points right to the next step in the workflow: using Amazon Rekognition to configure a human loop with the human review workflow. A second arrow points right from this step to the step in which activation conditions specified in the human review workflow are met. This initiates the creation of a human loop. On the right of the image, the human loop is depicted in three steps: 1) the worker UI and tools are generated and the task is made available to workers, 2) workers review input data, and finally, 3) results are saved in Amazon S3. ![\[Use Amazon Augmented AI with Amazon Rekognition\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/a2i/diagrams/product-page-diagram_A2I-Components_Rekognition@2x.png) You can set the following activation conditions when using the Amazon Rekognition task type: + Initiate human review for labels identified by Amazon Rekognition based on the label confidence score. + Randomly send a sample of images to humans for review. You can set these activation conditions using the Amazon SageMaker AI console when you create a human review workflow, or by creating a JSON for human loop activation conditions and specifying this as input in the `HumanLoopActivationConditions` parameter of the `CreateFlowDefinition` API operation. To learn how specify activation conditions in JSON format, see [JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI](a2i-human-fallback-conditions-json-schema.md) and [Use Human Loop Activation Conditions JSON Schema with Amazon Rekognition](a2i-json-humantaskactivationconditions-rekognition-example.md). **Note** When using Augmented AI with Amazon Rekognition, create Augmented AI resources in the same AWS Region you use to call `DetectModerationLabels`. ## Get Started: Integrate a Human Review into an Amazon Rekognition Image Moderation Job To integrate a human review into an Amazon Rekognition, see the following topics: + [Create a Human Review Workflow (Console)](a2i-create-flow-definition.md#a2i-create-human-review-console) + [Create a Human Review Workflow (API)](a2i-create-flow-definition.md#a2i-create-human-review-api) After you've created your flow definition, see [Using Augmented AI with Amazon Rekognition](https://docs.aws.amazon.com/rekognition/latest/dg/a2i-rekognition.html) to learn how to integrate your flow definition into your Amazon Rekognition task. ## End-to-end Demo Using Amazon Rekognition and Amazon A2I For an end-to-end example that demonstrates how to use Amazon Rekognition with Amazon A2I using the console, see [Tutorial: Get Started in the Amazon A2I Console](a2i-get-started-console.md). To learn how to use the Amazon A2I API to create and start a human review, you can use [Amazon Augmented AI (Amazon A2I) integration with Amazon Rekognition [Example]](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks/blob/master/Amazon%20Augmented%20AI%20(A2I)%20and%20Rekognition%20DetectModerationLabels.ipynb) in a SageMaker notebook instance. To get started, see [Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook](a2i-task-types-general.md#a2i-task-types-notebook-demo). ## A2I Rekognition Worker Console Preview When they're assigned a review task in an Amazon Rekognition workflow, workers might see a user interface similar to the following: ![\[Example image in the A2I Rekognition worker console.\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/a2i-rekognition-example.png) You can customize this interface in the SageMaker AI console when you create your human review definition, or by creating and using a custom template. To learn more, see [Create and Manage Worker Task Templates](a2i-instructions-overview.md). # Use Amazon Augmented AI with Custom Task Types Use With Custom Task Types You can use Amazon Augmented AI (Amazon A2I) to incorporate a human review (human loop) into *any* machine learning workflow using the *custom task type*. This options gives you the most flexibility to customize the conditions under which your data objects are sent to humans for review, as well as the look and feel of your worker user interface. When you use a custom task type, you create a custom human review workflow and specify the conditions under which a data object is sent for human review directly in your application. The following image depicts the Amazon A2I custom workflow. A custom ML model is used to generate predictions. The client application filters these predictions using user-defined criteria and determines if a human review is required. If so, these predictions are sent to Amazon A2I for human review. Amazon A2I collects the results of human review in Amazon S3, which can access by the client application. If the filter determines that no human review is needed, predictions can be fed directly to the client application. ![\[Use Amazon Augmented AI with Custom Task Types\]](http://docs.aws.amazon.com/sagemaker/latest/dg/images/a2i/diagrams/product-page-diagram_A2I-Components_Custom@2x.png) Use the procedures on this page to learn how to integrate Amazon A2I into any machine learning workflow using the custom task type. **Create a human loop using a flow definition, integrate it into your application, and monitor the results** 1. Complete the Amazon A2I [Prerequisites to Using Augmented AI](a2i-getting-started-prerequisites.md). Note the following: + The path to the Amazon Simple Storage Service (Amazon S3) bucket or buckets where you store your input and output data. + The Amazon Resource Name (ARN) of an AWS Identity and Access Management (IAM) role with required permissions attached. + (Optional) The ARN of your private workforce, if you plan to use one. 1. Using HTML elements, create a custom worker template which Amazon A2I uses to generate your worker task UI. To learn how to create a custom template, see [Create Custom Worker Task Templates](a2i-custom-templates.md). 1. Use the custom worker template from step 2 to generate a worker task template in the Amazon SageMaker AI console. To learn how, see [Create a Worker Task Template](a2i-worker-template-console.md#a2i-create-worker-template-console). In the next step, you create a flow definition: + If you want to create a flow definition using the SageMaker API, note the ARN of this worker task template for the next step. + If you are creating a flow definition using the console, your template automatically appears in **Worker task template** section when you choose **Create human review workflow**. 1. When creating your flow definition, provide the path to your S3 buckets, your IAM role ARN, and your worker template. + To learn how to create a flow definition using the SageMaker AI `CreateFlowDefinition` API, see [Create a Human Review Workflow (API)](a2i-create-flow-definition.md#a2i-create-human-review-api). + To learn how to create a flow definition using the SageMaker AI console, see [Create a Human Review Workflow (Console)](a2i-create-flow-definition.md#a2i-create-human-review-console). 1. Configure your human loop using the [Amazon A2I Runtime API](https://docs.aws.amazon.com/augmented-ai/2019-11-07/APIReference/Welcome.html). To learn how, see [Create and Start a Human Loop](a2i-start-human-loop.md). 1. To control when human reviews are initiated in your application, specify conditions under which `StartHumanLoop` is called in your application. Human loop activation conditions, such as confidence thresholds that initiate the human loop, are not available when using Amazon A2I with custom task types. Every `StartHumanLoop` invocation results in a human review. Once you have started a human loop, you can manage and monitor your loops using the Amazon Augmented AI Runtime API and Amazon EventBridge (also known as Amazon CloudWatch Events). To learn more, see [Monitor and Manage Your Human Loop](a2i-monitor-humanloop-results.md). ## End-to-end Tutorial Using Amazon A2I Custom Task Types For an end-to-end examples that demonstrates how to integrate Amazon A2I into a variety of ML workflows, see the table in [Use Cases and Examples Using Amazon A2I](a2i-task-types-general.md). To get started using one of these notebooks, see [Use SageMaker Notebook Instance with Amazon A2I Jupyter Notebook](a2i-task-types-general.md#a2i-task-types-notebook-demo).