# Review Policies
Using Amazon Mechanical Turk Review Policies you can evaluate Worker submissions against a defined set of criteria. You specify the Review Policy(s) that you want to use when you call the [CreateHIT](ApiReference_CreateHITOperation.md) operation.
There are two types of Review Policies, Assignment-level and HIT-level:
+ An Assignment-level Review Policy is applied as soon as a Worker submits an assignment. For more information, see [Assignment Review Policies](ApiReference_AssignmentReviewPolicies.md).
+ A HIT-level Review Policy is applied when a HIT becomes reviewable. For more information, see [HIT Review Policies](ApiReference_HITReviewPolicies.md).
You can select from a set of pre-defined Review Policies. One Review Policy leverages *known answers* or *gold standards* within a Human Intelligence Task (HIT) and has Mechanical Turk calculate a Worker’s performance on these known answers. You can specify an action for Mechanical Turk to take automatically based on Worker performance against the known answer.
Mechanical Turk has Review Policies that calculate consensus/agreement among multiple Workers performing the same HITs. For instance, you can specify a Review Policy that measures agreement on work items within the HIT and authorizes Mechanical Turk to keep asking additional Workers to work on the HIT, until a certain level of agreement is achieved. Once the required level of agreement is achieved, the results are returned to you for immediate use.
Review Policies that track Worker performance on your known answers and agreement with other Workers give you information you can use to manage your Workers. For more information about using Review Policies, see [Review Policy Use Cases](ApiReference_ReviewPolicyUseCases.md).
# How Review Policies Work
You specify the Review Policy(s) that you want Mechanical Turk to apply when you call the [CreateHIT](ApiReference_CreateHITOperation.md) operation. You must specify Review Policies when you create a HIT. You cannot apply a Review Policy to an existing HIT.
As assignments are submitted, Mechanical Turk applies the Review Policy(s) that you specify. You call the ListReviewPolicyResultsForHIT operation to gather the results from the application of the Review Policy.
There are two types of Review Policies, Assignment-level Review Policies that are applied as soon as a Worker submits an assignment and HIT-level Review Policies that are applied when a HIT becomes reviewable. For more information, see [Assignment Review Policies](ApiReference_AssignmentReviewPolicies.md) and [HIT Review Policies](ApiReference_HITReviewPolicies.md).
You can specify one Assignment-level Review Policy and one HIT-level Review Policy when you call `CreateHIT` using the [HIT Review Policy](ApiReference_HITReviewPolicyDataStructureArticle.md) data structure. The Assignment-level Review Policy ScoreMyKnownAnswer/2011-09-01 and the HIT-level Review Policy SimplePlurality/2011-09-01 can be used in the same call to `CreateHIT`.
Once an Assignment-level Review Policy is applied, the Assignment's status is changed to *Submitted* and optionally an event notification can be sent. Assignments with *Submitted* status are returned by the ListAssignmentsForHIT operation and the results of applying the Review Policy are available by using the ListReviewPolicyResultsForHIToperation.
You can use different Review Policies on distinct HITs in a HIT type. For example, you may wish to apply the ScoreMyKnownAnswers/2011-09-01 policy to a small number of HITs that have known answers in them, but apply the SimplePlurality/2011-09-01 policy to all HITs in a group. Workers do not have access on the Worker User Interface to information about whether a Review Policy has been applied to a HIT.
# Assignment Review Policies
Assignment-level Review Policies are applied as soon as a Worker submits an assignment.
## ScoreMyKnownAnswers/2011-09-01
ScoreMyKnownAnswers/2011-09-01 is an Assignment-level Review Policy.
### Description
You can use the ScoreMyKnownAnswers/2011-09-01 Review Policy for QuestionForm (QAP) HITs and for ExternalQuestion (iframe) HITs. You provide an answer key when you call the [CreateHIT](ApiReference_CreateHITOperation.md) operation. The answer key is a collection of QuestionIds, where each QuestionId has a set of zero or more values that represent the correct response for that QuestionId. For more information about QuestionForm and ExternalQuestion HITs, see [QuestionForm](ApiReference_QuestionFormDataStructureArticle.md) and [ExternalQuestion](ApiReference_ExternalQuestionArticle.md).
You can specify if one question in your HIT has a known answer or if many questions in your HIT have known answers. When a Worker submits an assignment Mechanical Turk examines the Worker's answers and compares them against the set of known answers that you provide when you create the HIT. Mechanical Turk then calculates a score, for example, 4 out of 10 known answers were correct.
Based on how the Worker's level of agreement with the known answers compares with various configurable thresholds, Mechanical Turk can automatically take actions you requested to approve the assignment, automatically reject the assignment, or automatically extend the HIT to publish an assignment for another Worker.
A Worker’s performance on known answers within a specific assignment are returned from calling the ListReviewPolicyResultsForHIT operation.
Mechanical Turk evaluates answers and considers the following answers as not matching:
+ The Worker left an empty value set in the answer key.
+ The answer key has an empty value set but the Worker supplied an answer.
+ The Worker provides an answer that is the wrong case or has incorrect punctuation that doesn't match the answer exactly. You can either use structured HTML form elements to restrict the values a Worker can submit, or use JavaScript to validate and normalize the submitted values.
+ The answer key says a question's answer is A and B but the Worker's value is A.
+ The answer key says a question's answer is A and the Worker selected both A and B.
When comparing answers for a match, Mechanical Turk removes any whitespace from before and after the Worker's answer, and from before and after the answer you provide.
### Parameters
The following parameters are specified in the AssignmentReviewPolicy element when calling the `CreateHIT` operation. You must also specify the PolicyName *ScoreYourKnownAnswers/2011-09-01* as part of the AssignmentReviewPolicy element. For an example of how to structure the AssignmentReviewPolicy element, see the [HIT Review Policy](ApiReference_HITReviewPolicyDataStructureArticle.md) data structure.
| Name | Description | Required |
| --- | --- | --- |
| `AnswerKey` | Question IDs and the answers to the questions. Type: MapEntry, see the [HIT Review Policy](ApiReference_HITReviewPolicyDataStructureArticle.md) data structure. Default: None | Yes |
| `ApproveIfKnownAnswerScoreIsAtLeast` | Approve the assignment if the KnownAnswerScore is equal to or greater than this value. If not specified, assignments are left in the submitted state and are not approved or rejected. Type: Integer Constraints: Minimum value 0 (always approve), maximum 101 (never approve) | No |
| `ApproveReason` | A description provided to the Worker about the reason the assignment was approved. If not specified, the reason is left blank. Type: String | No |
| `RejectIfKnownAnswerScoreIsLessThan` | Reject the assignment if the KnownAnswerScore is equal to or less than this value. If not specified, assignments are left in the submitted state and are not approved or rejected. Type: Integer Constraints: Minimum value 0 (never reject), maximum 101 (always reject). | No |
| `RejectReason` | A description provided to the Worker about the reason the assignment was rejected. If not specified, the reason is left blank. Type: String | No |
| `ExtendIfKnownAnswerScoreIsLessThan` | Extend the HIT by one assignment to allow one more Worker to complete it if the known answer score is less than this value. Ordinarily this is done to replace an assignment that is being rejected or that is not usable because the Worker didn't answer the known answer correctly. If omitted the HIT is not extended. Type: String Constraint: Minimum value 0 (never extend), maximum 101 (always extend). | No |
| `ExtendMaximumAssignments` | The maximum number of assignments the HIT can be extended. Note that if you use the CreateAdditionalAssignmentsForHIT operation and specify a maximum assignment count greater than this value, ScoreMyKnownAnswers will not extend the HIT. **Note**: If a HIT is created with fewer than 10 assignments, it will not extend to have 10 or more assignments. Type: Integer Constraint: Minimum value 2, maximum 25. Default: 5 | No |
| `ExtendMinimumTimeInSeconds` | The additional time in seconds to let other Workers complete the extended assignment. Type: Integer Constraints: Minimum of 60 (one minute), Maximum of 31536000 (one year). Default: 0 | No |
# HIT Review Policies
A HIT-level Review Policy is applied when a Human Intelligence Task (HIT) becomes reviewable.
## SimplePlurality/2011-09-01
SimplePlurality/2011-09-01 is a HIT-level Review Policy.
### Description
The SimplePlurality/2011-09-01 policy allows you to automatically compare answers received from multiple Workers and detect if there is a majority or consensus answer. The results can optionally trigger additional actions, such as approving the assignments that matched the majority answer. The results of this comparison are available as a part of the ListReviewPolicyResultsForHIT operation.
Mechanical Turk evaluates answers and considers the following answers as not matching:
+ The Worker provides an answer that is the wrong case or incorrect punctuation that doesn't match the answer exactly to another Worker. You can either use structured HTML form elements to restrict the values a Worker can submit, or use JavaScript to validate and normalize the submitted values.
+ One Worker's answer is A and B, but another Worker's value is A.
+ One Worker's answer is A, but another Worker selected both A and B.
When comparing answers for a match, Mechanical Turk removes any whitespace from before and after the Worker's answer.
**Note**
Answers that are longer than 256 characters are not used in the computation of HIT review policies.
### Parameters
The following parameters are specified in the HITReviewPolicy element when calling the [CreateHIT](ApiReference_CreateHITOperation.md) operation. You must also specify the PolicyName *SimplePlurality/2011-09-01* as part of the HitReviewPolicy element. For an example, see [HIT Review Policy](ApiReference_HITReviewPolicyDataStructureArticle.md) data structure.
| Name | Description | Required |
| --- | --- | --- |
| `QuestionIds` | A comma-separated list of questionIds used to determine agreement. Type: String Constraints: none | Yes |
| `QuestionAgreementThreshold` | If the Question Agreement Score is greater than this value, the questionId is considered to have an *agreed answer*. Type: Integer Constraints: none | Yes |
| `DisregardAssignmentIfRejected` | Excludes rejected assignments from agreement calculation. Type: Boolean Constraints: T or F | Yes |
| `DisregardAssignmentIfKnownAnswerScoreIsLessThan` | Excludes answers from agreement calculation if the KnownAnswerScore is present and less than the provided value. Type: Integer Constraints: none | No |
| `ExtendIfHITAgreementScoreIsLessThan ` | If the HIT Agreement Score is less than this value, extend the HIT to another Worker to complete. If omitted, extending on failure is disabled. Type: Integer Constraints: 1-100 | No |
| `ExtendMaximumAssignments` | If the ExtendIfHITAgreementScoreIsLessThan is provided, this sets the total maximum number of assignments for the HIT. If you use ExtendHIT operation and specify the maximum assignment count greater than this value, ScoreMyKnownAnswers will not extend the HIT. **Note**: If a HIT is created with fewer than 10 assignments, it will not extend to have 10 or more assignments. Type: Integer Constraints: none Conditions: Required if ExtendIfHITAgreementScoreIsLessThan is provided. | Conditional |
| `ExtendMinimumTimeInSeconds` | If the ExtendIfHITAgreementScoreIsLessThan is provided, this sets the additional time that the HIT will be extended by. Type: Integer Constraints: Minimum of 60 (one minute), Maximum of 31536000 (365 days) Conditions: Required if ExtendIfHITAgreementScoreIsLessThan is provided. | Conditional |
| `ApproveIfWorkerAgreementScoreIsAtLeast` | If the Worker Agreement Score is not less than this value, approve the Worker's assignment. If omitted, assignment will not be approved or rejected. Type: Integer Constraints: none | No |
| `RejectIfWorkerAgreementScoreIsLessThan` | If the Worker Agreement Score is less than this value, reject the Worker's assignment. If omitted, assignment will not be approved or rejected. Type: Integer Constraints: none | No |
| `RejectReason` | If the RejectIfWorkerAgreementIsScoreLessThan value is provided, this value sets the reason for any automated rejections. Type: String Constraints: none | Optional |
### Scores
The following scores are calculated data from the SimplePlurality/2011-09-01 policy. Based on the value of these scores, Mechanical Turk can take various actions that you specify in the `CreateHIT` operation. It is important to understand how these scores are calculated so you can specify the appropriate actions to take, including approving or rejecting assignments, or extending HITs. The following chart describes how the scores are calculated.
| Score | Description |
| --- | --- |
| Question Agreement Score | Percentage of Workers who provided the agreed-upon answer for a HIT. Note: Answer values are not normalized for case, whitespace, or punctuation before comparison. Answers can contain multiple values (such as in a set of check boxes); two answers agree with each other if they have the same values present and absent. We don't recommend using free format answers because values are not normalized. |
| HIT Agreement Score | Percentage of questions within the HIT with an agreed-upon answer. The number of questions within the HIT with an agreed-upon answer, divided by the number of questions evaluated. |
| Worker Agreement Score | The percentage of questions to which a Worker's answer agreed with other Workers' answers in the same HIT. If a question does not have an agreed upon answer the question is disregarded in this calculation. |
The example chart below describes how the Answer Agreement Score and Worker Agreement Score is calculated for a HIT with 4 questions and answers from 3 Workers.
| QuestionId | Worker1's answers | Worker2's answers | Worker3's answers | Has Agreed-upon value? | Agreed-upon value | Question Agreement Score |
| --- | --- | --- | --- | --- | --- | --- |
| A | coat | sweater | coat | Yes | coat | 66% |
| B | blue | blue | green | Yes | blue | 66% |
| C | large | large | large | Yes | large | 100% |
| D | Furry | fur | furr | No | n/a | n/a |
| Worker Agreement Score | 100% | 66% | 66% | | | |
The Question Agreement Score for questions A and B are 66% because two Workers agreed on the same answer. The HIT Agreement Score for this HIT is 75%. The HIT had four questions, and three of them had an agreed-upon answer for a percentage of 75%. The Worker Agreement Score for Worker 1 is 100% because this Worker agreed with the other Workers for each answer, except Question D where there was no conclusive answer.
# Review Policy Use Cases
The following use cases show you how to apply ScoreYourKnownAnswers and SimplePlurality policies when you call the [CreateHIT](ApiReference_CreateHITOperation.md) operation.
## Photo Moderation Use Case – Single Worker with Known Answers
In this scenario, you want Workers to moderate photos and screen the photos for inappropriate content. You place 20 photos in a single HIT and 5 of the 20 photos are your known answers. You are using Master Workers and have created the HIT with only one initial assignment. You want to use the answers based on the Worker getting at least 4 of the 5 known answers (80% Answer Agreement Score) correct. If the first Worker does not meet the Answer Agreement score of 80%, then you want to extend the HIT to another Worker. But, in this scenario, you only want to extend the HIT to a maximum of three Workers.
### Elements and Parameters
The following is a list of elements and parameters you need to specify in the [CreateHIT](ApiReference_CreateHITOperation.md) operation to execute the above scenario and allow Mechanical Turk to automatically calculate the known answer score. Note that this `CreateHIT` example assumes you have already created a HIT Type.
| Element | Parameter | Value |
| --- | --- | --- |
| `AssignmentReviewPolicy` | PolicyName | ScoreMyKnownAnswers/2011/09/01 |
| `AssignmentReviewPolicy` | AnswerKey | List of questionIDs and answers. |
| `AssignmentReviewPolicy` | ExtendIfKnownAnswerScoreIsLessThan | 80 |
| `AssignmentReviewPolicy` | ExtendMaximumAssignments | 3 |
### Examples
The following example shows how to use the above elements and parameters with the `CreateHIT` operation.
#### Sample CreateHIT Request
The following example shows a `CreateHIT` request.
```
1.
2. T100CN9P324W00EXAMPLE
3. [CDATA block or XML Entity encoded]
4. 604800
5.
6. ScoreMyKnownAnswers/2011-09-01
7.
8. AnswerKey
9.
10. QuestionId3
11. B
12.
14. QuestionId7
15. A
16.
17.
18. QuestionId15
19. F
20.
21.
22. QuestionId17
23. C
24.
25.
26. QuestionId18
27. A
28.
29.
30.
31. ExtendIfKnownAnswerScoreIsLessThan
32. 80
33.
34.
35. ExtendMaximumAssignments
36. 3
37.
38.
39.
```
## Photo Moderation Use Case – Multiple Workers with Agreement
In this scenario, you want Workers to moderate photos and screen the photos for inappropriate content. You place 20 photos in a single HIT and 5 of the 20 photos are your known answers. You want to approve the assignment if the Worker completes at least 4 of the 5 known answers correct (at least 80% Answer Agreement Score).
You want 3 Workers to complete each HIT and you want to calculate the HIT Agreement Score for the 15 photos you don’t know the answer to. Also, you want to disregard the Worker's answer in the Agreement Score if they don't get 4 of 5 of the known answers correct.
### Elements and Parameters
The following is a list of elements and parameters you need to specify in the [CreateHIT](ApiReference_CreateHITOperation.md) operation to execute the above scenario and allow Mechanical Turk to automatically approve the assignments. Note that this CreateHIT example assumes you have already created a HIT Type.
| Element | Parameter | Value |
| --- | --- | --- |
| `AssignmentReviewPolicy` | PolicyName | ScoreMyKnownAnswers/2011/09/01 |
| `AssignmentReviewPolicy` | Answer | List of questionIDs and answers. |
| `AssignmentReviewPolicy` | ApproveIfKnownAnswerScoreIsAtLeast | 80 |
| `AssignmentReviewPolicy` | ExtendIfKnownAnswerScoreIsLessThan | 80 |
| `AssignmentReviewPolicy` | ExtendMaximumAssignments | 3 |
| `HITReviewPolicy` | PolicyName | SimplePlurality/2011-09-01 |
| `HITReviewPolicy` | QuestionIDs | Your list of 15 question IDs. |
| `HITReviewPolicy` | QuestionAgreementThreshold | 100 |
| `HITReviewPolicy` | DisregardAssignmentIfKnownAnswerScoreIsLessThan | 80 |
### Examples
The following example shows how to use the above elements and parameters with the `CreateHIT` operation.
#### Sample CreateHIT Request
The following example shows a `CreateHIT` request.
```
1.
2. T100CN9P324W00EXAMPLE
3. [CDATA block or XML Entity encoded]
4. 604800
5.
6. ScoreMyKnownAnswers/2011-09-01
7.
8. AnswerKey
9.
10. QuestionId3
11. B
12.
14. QuestionId4
15. A
16.
17.
18. QuestionId13
19. F
20.
21.
22. QuestionId14
23. C
24.
25.
26. QuestionId19
27. A
28.
29.
30.
31. ApproveIfKnownAnswerScoreIsAtLeast
32. 80
33.
34.
35. ExtendIfKnownAnswerScoreIsLessThan
36. 80
37.
38.
39. ExtendMaximumAssignments
40. 3
41.
42.
43.
44. SimplePlurality/2011-09-01
45.
46. QuestionIDs
47. questionid1
48. questionid2
49. questionid5
50. questionid6
51. questionid7
52. .....
53.
54.
55. QuestionAgreementThreshold
56. 100
57.
58.
59. DisregardAssignmentIfKnownAnswerScoreIsLessThan
60. 80
61.
62.
63.
```
## Categorization and Tagging Use Case – Multiple Workers
In this scenario, you want Workers to categorize a product and provide multiple tags for the product in a HIT. You also want the Workers to be able to comment on your HIT and give you feedback.
You want to calculate the Answer Agreement Score for only the categorization question. If two Workers do not agree on the product categorization question, you want to extend the HIT to a third Worker. Also, you want to extend the assignment by an hour so the third Worker has time to work on the assignment.
### Elements and Parameters
The following is a list of elements and parameters you need to specify in the [CreateHIT](ApiReference_CreateHITOperation.md) operation to execute the above scenario and allow Mechanical Turk to automatically calculate agreement and approve or reject the assignments. Note that this CreateHIT example assumes you have already created a HIT Type.
| Element | Parameter | Value |
| --- | --- | --- |
| `HITReviewPolicy` | PolicyName | SimplePlurality/2011-09-01 |
| `HITReviewPolicy` | QuestionIDs | questionID1 |
| `HITReviewPolicy` | QuestionAgreementThreshold | 100 |
| `HITReviewPolicy` | ExtendMinimumTimeInSeconds | 3600 |
| `HITReviewPolicy` | ExtendMaximumAssignments | 3 |
### Examples
The following example shows how to use the above elements and parameters with the `CreateHIT` operation.
#### Sample CreateHIT Request
The following example shows a CreateHIT request.
```
1.
2. T100CN9P324W00EXAMPLE
3. [CDATA block or XML Entity encoded]
4. 604800
5.
6. SimplePlurality/2011-09-01
7.
8. QuestionIDs
9. questionID1
10.
11.
12. QuestionAgreementThreshold
13. 100
14.
15.
16. ExtendMaximumAssignments
17. 3
18.
19.
20. ExtendMinimumTimeInSeconds
21. 3600
22.
23.
24.
```