# JSON Schema for Human Loop Activation Conditions in Amazon Augmented AI The `HumanLoopActivationConditions` is an input parameter of the [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateFlowDefinition.html) API. This parameter is a JSON-formatted string. The JSON models the conditions under which a human loop is created when those conditions are evaluated against the response from an integrating AI service API (such as `Rekognition.DetectModerationLabels` or `Textract.AnalyzeDocument`). This response is referred to as an *inference*. For example, Amazon Rekognition sends an inference of a moderation label with an associated confidence score. In this example, the inference is the model's best estimate of the appropriate label for an image. For Amazon Textract, inference is made on the association between blocks of text (*key-value pairs*), such as the association between `Name:` and `Sue` in a form as well as content within a block of text, or *word block*, such as 'Name'. The following is the schema for the JSON. At the top level, the `HumanLoopActivationConditions` has a JSON array, `Conditions`. Each member of this array is an independent condition that, if evaluated to `true`, results in Amazon A2I creating a human loop. Each such independent condition can be a simple condition or a complex condition. A simple condition has the following attributes: + `ConditionType`: This attribute identifies the type of condition. Each AWS AI service API that integrates with Amazon A2I defines its own set of allowed `ConditionTypes`. + Rekognition `DetectModerationLabels` – This API supports the `ModerationLabelConfidenceCheck` and `Sampling` `ConditionType` values. + Textract `AnalyzeDocument` – This API supports the `ImportantFormKeyConfidenceCheck`, `MissingImportantFormKey`, and `Sampling` `ConditionType` values. + `ConditionParameters` – This is a JSON object that parameterizes the condition. The set of allowed attributes of this object is dependent on the value of the `ConditionType`. Each `ConditionType` defines its own set of `ConditionParameters`. A member of the `Conditions` array can model a complex condition. This is accomplished by logically connecting simple conditions using the `And` and `Or` logical operators and nesting the underlying simple conditions. Up to two levels of nesting are supported. ``` { "$schema": "http://json-schema.org/draft-07/schema#", "definitions": { "Condition": { "type": "object", "properties": { "ConditionType": { "type": "string" }, "ConditionParameters": { "type": "object" } }, "required": [ "ConditionType" ] }, "OrConditionArray": { "type": "object", "properties": { "Or": { "type": "array", "minItems": 2, "items": { "$ref": "#/definitions/ComplexCondition" } } } }, "AndConditionArray": { "type": "object", "properties": { "And": { "type": "array", "minItems": 2, "items": { "$ref": "#/definitions/ComplexCondition" } } } }, "ComplexCondition": { "anyOf": [ { "$ref": "#/definitions/Condition" }, { "$ref": "#/definitions/OrConditionArray" }, { "$ref": "#/definitions/AndConditionArray" } ] } }, "type": "object", "properties": { "Conditions": { "type": "array", "items": { "$ref": "#/definitions/ComplexCondition" } } } } ``` **Note** Human loop activation conditions aren't available for human review workflows that are integrated with custom task types. The `HumanLoopActivationConditions` parameter is disabled for custom task types. **Topics** + [ # Use Human Loop Activation Conditions JSON Schema with Amazon Textract ](a2i-json-humantaskactivationconditions-textract-example.md) + [ # Use Human Loop Activation Conditions JSON Schema with Amazon Rekognition ](a2i-json-humantaskactivationconditions-rekognition-example.md) # Use Human Loop Activation Conditions JSON Schema with Amazon Textract When used with Amazon A2I, the `AnalyzeDocument` operation supports the following inputs in the `ConditionType` parameter: + `ImportantFormKeyConfidenceCheck` – Use this condition to create a human loop when inference confidence is within a specified range for document form keys and word blocks. A *form key* is any word in a document that is associated with an input. The input is called a *value*. Together, form keys and values are referred to as *key-value pairs*. A *word block* refers to the words that Amazon Textract recognizes inside of a detected block of text. To learn more about Amazon Textract document blocks, see [Documents and Block Objects](https://docs.aws.amazon.com/textract/latest/dg/how-it-works-document-layout.html) in the *Amazon Textract Developer Guide*. + `MissingImportantFormKey` – Use this condition to create a human loop when Amazon Textract did not identify the key or its associated aliases within the document. + `Sampling` – Use this condition to specify a percentage of forms to send to humans for review, regardless of inference confidence scores. Use this condition to do the following: + Audit your ML model by randomly sampling all forms analyzed by your model and sending a specified percentage to humans for review. + Using the `ImportantFormKeyConfidenceCheck` condition, randomly sample a percentage of the inferences that met the conditions specified in `ImportantFormKeyConfidenceCheck` to start a human loop and send only the specified percentage to humans for review. **Note** If you send the same request to `AnalyzeDocument` multiple times, the result of `Sampling` does not change for the inference of that input. For example, if you make an `AnalyzeDocument` request once, and `Sampling` doesn't initiate a human loop, subsequent requests to `AnalyzeDocument` with the same configuration do not initiate a human loop. ## `ImportantFormKeyConfidenceCheck` Inputs and Results The `ImportantFormKeyConfidenceCheck` `ConditionType` supports the following `ConditionParameters`: + `ImportantFormKey` – A string representing a key in a key-value pair detected by Amazon Textract that needs to be reviewed by human workers. If the value of this parameter is the special catch-all value (\$1), then all keys are considered to be matched to the condition. You can use this to model the case where any key-value pair satisfying certain confidence thresholds needs human review. + `ImportantFormKeyAliases` – An array that represents alternate spellings or logical equivalents for the important form key. + `KeyValueBlockConfidenceEquals` + `KeyValueBlockConfidenceLessThan` + `KeyValueBlockConfidenceLessThanEquals` + `KeyValueBlockConfidenceGreaterThan` + `KeyValueBlockConfidenceGreaterThanEquals` + `WordBlockConfidenceEquals` + `WordBlockConfidenceLessThan` + `WordBlockConfidenceLessThanEquals` + `WordBlockConfidenceGreaterThan` + `WordBlockConfidenceGreaterThanEquals` When you use the `ImportantFormKeyConfidenceCheck` `ConditionType`, Amazon A2I sends the key-value block and word block inferences of the key-value blocks and associated aliases that you specified in `ImportantFormKey` and `ImportantFormKeyAliases` for human review. When creating a flow definition, if you use the default worker task template that is provided in the **Human review workflows** section of the Amazon SageMaker AI console, key-value and block inferences sent for human review by this activation condition are included in the worker UI. If you use a custom worker task template, you need to include the `{{ task.input.selectedAiServiceResponse.blocks }}` element to include initial-value input data (inferences) from Amazon Textract. For an example of a custom template that uses this input element, see [Custom Template Example for Amazon Textract](a2i-custom-templates.md#a2i-custom-templates-textract-sample). ## `MissingImportantFormKey` Inputs and Results The `MissingImportantFormKey` `ConditionType` supports the following `ConditionParameters`: + `ImportantFormKey` – A string representing a key in a key-value pair detected by Amazon Textract that needs to be reviewed by human workers. + `ImportantFormKeyAliases` – An array that represents alternate spellings or logical equivalents for the important form key. When you use the `MissingImportantFormKey` `ConditionType`, if the key in `ImportantFormKey` or aliases in `ImportantFormKeyAliases` are not included in the Amazon Textract inference, that form is sent to human for review and no predicted key-value pairs are included. For example, if Amazon Textract only identified `Address` and `Phone` in a form, but was missing the `ImportantFormKey` `Name` (in the `MissingImportantFormKey` condition type) that form would be sent to humans for review without any of the form keys detected (`Address` and `Phone`). If you use the default worker task template that is provided in the SageMaker AI console, a task is created asking workers to identify the key in `ImportantFormKey` and associated value. If you use a custom worker task template, you need to include the `` custom HTML element to configure this task. ## Sampling Inputs and Results The `Sampling` `ConditionType` supports the `RandomSamplingPercentage` `ConditionParameters`. The input for `RandomSamplingPercentage` must be real number between 0.01 and 100. This number represents the percentage of data that qualifies for a human review and is sent to humans for review. If you use the `Sampling` condition without any other conditions, this number represents the percentage of all resulting inferences made by the `AnalyzeDocument` operation from a single request that is sent to humans for review. If you specify the `Sampling` condition without any other condition type, all key-value and block inferences are sent to workers for review. When creating a flow definition, if you use the default worker task template that is provided in the **Human review workflows** section of the SageMaker AI console, all key-value and block inferences sent for human review by this activation condition are included in the worker UI. If you use a custom worker task template, you need to include the `{{ task.input.selectedAiServiceResponse.blocks }}` element to include initial-value input data (inferences) from Amazon Textract. For an example of a custom template that uses this input element, see [Custom Template Example for Amazon Textract](a2i-custom-templates.md#a2i-custom-templates-textract-sample). ## Examples While only one condition needs to evaluate to `true` to initiate a human loop, Amazon A2I evaluates all conditions for each object analyzed by Amazon Textract. The human reviewers are asked to review the important form keys for all the conditions that evaluated to `true`. **Example 1: Detect important form keys with confidence scores in a specified range that initiate a human loop** The following example shows a `HumanLoopActivationConditions` JSON that initiates a human loop if any one of the following three conditions is met: + The Amazon Textract `AnalyzeDocument` API returns a key-value pair whose key is one of `Employee Name`, `Name`, or `EmployeeName`, with the confidence of the key-value block being less than 60 and the confidences of each of the word blocks making up the key and value being less than 85. + The Amazon Textract `AnalyzeDocument` API returns a key-value pair whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with the confidence of the key-value block being less than 65 and the confidences of each of the word blocks making up the key and value being less than 85. + The Amazon Textract `AnalyzeDocument` API returns a key-value pair whose key is one of `Gross Pay`, `GrossPay`, or `GrossAmount`, with the confidence of the key-value block being less than 60 and the confidences of each of the word blocks making up the key and value being less than 85. ``` { "Conditions": [ { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Employee Name", "ImportantFormKeyAliases": [ "Name", "EmployeeName" ], "KeyValueBlockConfidenceLessThan": 60, "WordBlockConfidenceLessThan": 85 } }, { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Pay Date", "ImportantFormKeyAliases": [ "PayDate", "DateOfPay", "pay-date" ], "KeyValueBlockConfidenceLessThan": 65, "WordBlockConfidenceLessThan": 85 } }, { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Gross Pay", "ImportantFormKeyAliases": [ "GrossPay", "GrossAmount" ], "KeyValueBlockConfidenceLessThan": 60, "WordBlockConfidenceLessThan": 85 } } ] } ``` **Example 2: Use `ImportantFormKeyConfidenceCheck`** In the following example, if Amazon Textract detects a key-value pair whose confidence for the key-value block is less than 60 and is less than 90 for any underlying word blocks, it creates a human loop. The human reviewers are asked to review all the form key-value pairs that matched the confidence value comparisons. ``` { "Conditions": [ { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "*", "KeyValueBlockConfidenceLessThan": 60, "WordBlockConfidenceLessThan": 90 } } ] } ``` **Example 3: Use Sampling** In the following example, 5% of inferences resulting from an Amazon Textract `AnalyzeDocument` request are sent to human workers for review. All detected key-value pairs returned by Amazon Textract are sent to workers for review. ``` { "Conditions": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } } ] } ``` **Example 4: Use `MissingImportantFormKey`** In the following example, if `Mailing Address` or its alias, `Mailing Address:`, is missing from keys detected by Amazon Textract, a human review is initiated. When using the default worker task template, the worker UI asks workers to identify the key `Mailing Address` or `Mailing Address:` and its associated value. ``` { "ConditionType": "MissingImportantFormKey", "ConditionParameters": { "ImportantFormKey": "Mailing Address", "ImportantFormKeyAliases": ["Mailing Address:"] } } ``` **Example 5: Use Sampling and `ImportantFormKeyConfidenceCheck` with the `And` operator** In this example, 5% of key-value pairs detected by Amazon Textract whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with the confidence of the key-value block less than 65 and the confidences of each of the word blocks making up the key and value less than 85, are sent to workers for review. ``` { "Conditions": [ { "And": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } }, { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Pay Date", "ImportantFormKeyAliases": [ "PayDate", "DateOfPay", "pay-date" ], "KeyValueBlockConfidenceLessThan": 65, "WordBlockConfidenceLessThan": 85 } } ] } ] } ``` **Example 6: Use Sampling and `ImportantFormKeyConfidenceCheck` with the `And` operator** Use this example to configure your human review workflow to always send low confidence inferences of a specified key-value pair for human review and sample high confidence inference of a key-value pair at a specified rate. In the following example, a human review is initiated in one of the following ways: + Key-value pairs detected whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with key-value and word block confidences less than 60, are sent for human review. Only the `Pay Date` form key (and its aliases) and associated values are sent to workers to review. + 5% of key-value pairs detected whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with key-value and word block confidences greater than 90, are sent for human review. Only the `Pay Date` form key (and its aliases) and associated values are sent to workers to review. ``` { "Conditions": [ { "Or": [ { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Pay Date", "ImportantFormKeyAliases": [ "PayDate", "DateOfPay", "pay-date" ], "KeyValueBlockConfidenceLessThan": 60, "WordBlockConfidenceLessThan": 60 } }, { "And": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } }, { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Pay Date", "ImportantFormKeyAliases": [ "PayDate", "DateOfPay", "pay-date" ], "KeyValueBlockConfidenceLessThan": 90 "WordBlockConfidenceGreaterThan": 90 } } ] } ] } ] } ``` **Example 7: Use Sampling and `ImportantFormKeyConfidenceCheck` with the `Or` operator** In the following example, the Amazon Textract `AnalyzeDocument` operation returns a key-value pair whose key is one of `Pay Date`, `PayDate`, `DateOfPay`, or `pay-date`, with the confidence of the key-value block less than 65 and the confidences of each of the word blocks making up the key and value less than 85. Additionally, 5% of all other forms initiate a human loop. For each form randomly chosen, all key-value pairs detected for that form are sent to humans for review. ``` { "Conditions": [ { "Or": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } }, { "ConditionType": "ImportantFormKeyConfidenceCheck", "ConditionParameters": { "ImportantFormKey": "Pay Date", "ImportantFormKeyAliases": [ "PayDate", "DateOfPay", "pay-date" ], "KeyValueBlockConfidenceLessThan": 65, "WordBlockConfidenceLessThan": 85 } } } ] } ] } ``` # Use Human Loop Activation Conditions JSON Schema with Amazon Rekognition When used with Amazon A2I, the Amazon Rekognition `DetectModerationLabels` operation supports the following inputs in the `ConditionType` parameters: + `ModerationLabelConfidenceCheck` – Use this condition type to create a human loop when inference confidence is low for one or more specified labels. + `Sampling` – Use this condition to specify a percentage of all inferences to send to humans for review. Use this condition to do the following: + Audit your ML model by randomly sampling all of your model's inferences and sending a specified percentage to humans for review. + Using the `ModerationLabelConfidenceCheck` condition, randomly sample a percentage of the inferences that met the conditions specified in `ModerationLabelConfidenceCheck` to start a human loop and send only the specified percentage to humans for review. **Note** If you send the same request to `DetectModerationLabels` multiple times, the result of `Sampling` does not change for the inference of that input. For example, if you make a `DetectModerationLabels` request once, and `Sampling` does not initiate a human loop, subsequent requests to `DetectModerationLabels` with the same configuration don't initiate a human loop. When creating a flow definition, if you use the default worker task template that is provided in the **Human review workflows** section of the Amazon SageMaker AI console, inferences sent for human review by these activation conditions are included in the worker UI when a worker opens your task. If you use a custom worker task template, you need to include the `` custom HTML element to access these inferences. For an example of a custom template that uses this HTML element, see [Custom Template Example for Amazon Rekognition](a2i-custom-templates.md#a2i-custom-templates-rekognition-sample). ## `ModerationLabelConfidenceCheck` Inputs For the `ModerationLabelConfidenceCheck` `ConditionType`, the following `ConditionParameters` are supported: + `ModerationLabelName` – The exact (case-sensitive) name of a [ModerationLabel](https://docs.aws.amazon.com/rekognition/latest/dg/API_ModerationLabel.html) detected by the Amazon Rekognition `DetectModerationLabels` operation. You can specify the special catch-all value (\$1) to denote any moderation label. + `ConfidenceEquals` + `ConfidenceLessThan` + `ConfidenceLessThanEquals` + `ConfidenceGreaterThan` + `ConfidenceGreaterThanEquals` When you use the `ModerationLabelConfidenceCheck` `ConditionType`, Amazon A2I sends label inferences for the labels that you specified in `ModerationLabelName` for human review. ## Sampling Inputs The `Sampling` `ConditionType` supports the `RandomSamplingPercentage` `ConditionParameters`. The input for the `RandomSamplingPercentage` prameter should be real number between 0.01 and 100. This number represents the percentage of inferences that qualifies for a human review that are sent to humans for review. If you use the `Sampling` condition without any other conditions, this number represents the percentage of all inferences that result from a single `DetectModerationLabel` request that are sent to humans for review. ## Examples **Example 1: Use `ModerationLabelConfidenceCheck` with the `And` operator** The following example of a `HumanLoopActivationConditions` condition initiates a human loop when one or more of the following conditions are met: + Amazon Rekognition detects the `Graphic Male Nudity` moderation label with a confidence between 90 and 99. + Amazon Rekognition detects the `Graphic Female Nudity` moderation label with a confidence between 80 and 99. Note the use of the `Or` and `And` logical operators to model this logic. Although only one of the two conditions under the `Or` operator needs to evaluate to `true` for a human loop to be created, Amazon Augmented AI evaluates all conditions. Human reviewers are asked to review the moderation labels for all the conditions that evaluated to `true`. ``` { "Conditions": [{ "Or": [{ "And": [{ "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Male Nudity", "ConfidenceLessThanEquals": 99 } }, { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Male Nudity", "ConfidenceGreaterThanEquals": 90 } } ] }, { "And": [{ "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Female Nudity", "ConfidenceLessThanEquals": 99 } }, { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Female Nudity", "ConfidenceGreaterThanEquals": 80 } } ] } ] }] } ``` **Example 2: Use `ModerationLabelConfidenceCheck` with the catch-all value (\$1) ** In the following example, if any moderation label with a confidence greater than or equal to 75 is detected, a human loop is initiated. Human reviewers are asked to review all moderation labels with confidence scores greater than or equal to 75. ``` { "Conditions": [ { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "*", "ConfidenceGreaterThanEquals": 75 } } ] } ``` **Example 3: Use Sampling** In the following example, 5% of Amazon Rekognition inferences from a `DetectModerationLabels` request are sent to human workers. When using the default worker task template provided in the SageMaker AI console, all moderation labels returned by Amazon Rekognition are sent to workers for review. ``` { "Conditions": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } } ] } ``` **Example 4: Use Sampling and `ModerationLabelConfidenceCheck` with the `And` operator** In this example, 5% of Amazon Rekognition inferences of the `Graphic Male Nudity` moderation label with a confidence greater than 50 are sent workers for review. When using the default worker task template provided in the SageMaker AI console, only the inferences of the `Graphic Male Nudity` label are sent to workers for review. ``` { "Conditions": [ { "And": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } }, { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Male Nudity", "ConfidenceGreaterThan": 50 } } ] } ] } ``` **Example 5: Use Sampling and `ModerationLabelConfidenceCheck` with the `And` operator** Use this example to configure your human review workflow to always send low-confidence inferences of a specified label for human review and sample high-confidence inferences of a label at a specified rate. In the following example, a human review is initiated in one of the following ways: + Inferences for the `Graphic Male Nudity` moderation label the with confidence scores less than 60 are always sent for human review. Only the `Graphic Male Nudity` label is sent to workers to review. + 5% of all inferences for the `Graphic Male Nudity` moderation label the with confidence scores greater than 90 are sent for human review. Only the `Graphic Male Nudity` label is sent to workers to review. ``` { "Conditions": [ { "Or": [ { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Male Nudity", "ConfidenceLessThan": 60 } }, { "And": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } }, { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Male Nudity", "ConfidenceGreaterThan": 90 } } ] } ] } ] } ``` **Example 6: Use Sampling and `ModerationLabelConfidenceCheck` with the `Or` operator** In the following example, a human loop is created if the Amazon Rekognition inference response contains the 'Graphic Male Nudity' label with inference confidence greater than 50. Additionally, 5% of all other inferences initiate a human loop. ``` { "Conditions": [ { "Or": [ { "ConditionType": "Sampling", "ConditionParameters": { "RandomSamplingPercentage": 5 } }, { "ConditionType": "ModerationLabelConfidenceCheck", "ConditionParameters": { "ModerationLabelName": "Graphic Male Nudity", "ConfidenceGreaterThan": 50 } } ] } ] } ```