get-automated-reasoning-policy-test-result¶
Description¶
Retrieves the test result for a specific Automated Reasoning policy test. Returns detailed validation findings and execution status.
See also: AWS API Documentation
Synopsis¶
get-automated-reasoning-policy-test-result
--policy-arn <value>
--build-workflow-id <value>
--test-case-id <value>
[--cli-input-json | --cli-input-yaml]
[--generate-cli-skeleton <value>]
[--debug]
[--endpoint-url <value>]
[--no-verify-ssl]
[--no-paginate]
[--output <value>]
[--query <value>]
[--profile <value>]
[--region <value>]
[--version <value>]
[--color <value>]
[--no-sign-request]
[--ca-bundle <value>]
[--cli-read-timeout <value>]
[--cli-connect-timeout <value>]
[--cli-binary-format <value>]
[--no-cli-pager]
[--cli-auto-prompt]
[--no-cli-auto-prompt]
Options¶
--policy-arn (string)
The Amazon Resource Name (ARN) of the Automated Reasoning policy.
--build-workflow-id (string)
The build workflow identifier. The build workflow must display aCOMPLETEDstatus to get results.
--test-case-id (string)
The unique identifier of the test for which to retrieve results.
--cli-input-json | --cli-input-yaml (string)
Reads arguments from the JSON string provided. The JSON string follows the format provided by --generate-cli-skeleton. If other arguments are provided on the command line, those values will override the JSON-provided values. It is not possible to pass arbitrary binary values using a JSON-provided value as the string will be taken literally. This may not be specified along with --cli-input-yaml.
--generate-cli-skeleton (string)
Prints a JSON skeleton to standard output without sending an API request. If provided with no value or the value input, prints a sample input JSON that can be used as an argument for --cli-input-json. Similarly, if provided yaml-input it will print a sample input YAML that can be used with --cli-input-yaml. If provided with the value output, it validates the command inputs and returns a sample output JSON for that command. The generated JSON skeleton is not stable between versions of the AWS CLI and there are no backwards compatibility guarantees in the JSON skeleton generated.
Global Options¶
--debug (boolean)
Turn on debug logging.
--endpoint-url (string)
Override command’s default URL with the given URL.
--no-verify-ssl (boolean)
By default, the AWS CLI uses SSL when communicating with AWS services. For each SSL connection, the AWS CLI will verify SSL certificates. This option overrides the default behavior of verifying SSL certificates.
--no-paginate (boolean)
Disable automatic pagination. If automatic pagination is disabled, the AWS CLI will only make one call, for the first page of results.
--output (string)
The formatting style for command output.
- json
- text
- table
- yaml
- yaml-stream
--query (string)
A JMESPath query to use in filtering the response data.
--profile (string)
Use a specific profile from your credential file.
--region (string)
The region to use. Overrides config/env settings.
--version (string)
Display the version of this tool.
--color (string)
Turn on/off color output.
- on
- off
- auto
--no-sign-request (boolean)
Do not sign requests. Credentials will not be loaded if this argument is provided.
--ca-bundle (string)
The CA certificate bundle to use when verifying SSL certificates. Overrides config/env settings.
--cli-read-timeout (int)
The maximum socket read time in seconds. If the value is set to 0, the socket read will be blocking and not timeout. The default value is 60 seconds.
--cli-connect-timeout (int)
The maximum socket connect time in seconds. If the value is set to 0, the socket connect will be blocking and not timeout. The default value is 60 seconds.
--cli-binary-format (string)
The formatting style to be used for binary blobs. The default format is base64. The base64 format expects binary blobs to be provided as a base64 encoded string. The raw-in-base64-out format preserves compatibility with AWS CLI V1 behavior and binary values must be passed literally. When providing contents from a file that map to a binary blob fileb:// will always be treated as binary and use the file contents directly regardless of the cli-binary-format setting. When using file:// the file contents will need to properly formatted for the configured cli-binary-format.
- base64
- raw-in-base64-out
--no-cli-pager (boolean)
Disable cli pager for output.
--cli-auto-prompt (boolean)
Automatically prompt for CLI input parameters.
--no-cli-auto-prompt (boolean)
Disable automatically prompt for CLI input parameters.
Output¶
testResult -> (structure)
The test result containing validation findings, execution status, and detailed analysis.
testCase -> (structure)
Represents a test for validating an Automated Reasoning policy. tests contain sample inputs and expected outcomes to verify policy behavior.
testCaseId -> (string)
The unique identifier of the test.guardContent -> (string)
The output content to be validated by the policy, typically representing a foundation model response.queryContent -> (string)
The input query or prompt that generated the content. This provides context for the validation.expectedAggregatedFindingsResult -> (string)
The expected result of the Automated Reasoning check for this test.createdAt -> (timestamp)
The timestamp when the test was created.updatedAt -> (timestamp)
The timestamp when the test was last updated.confidenceThreshold -> (double)
The minimum confidence level for logic validation. Content meeting this threshold is considered high-confidence and can be validated.policyArn -> (string)
The Amazon Resource Name (ARN) of the Automated Reasoning policy that was tested.testRunStatus -> (string)
The overall status of the test run (e.g., COMPLETED, FAILED, IN_PROGRESS).testFindings -> (list)
Detailed findings from the test run, including any issues, violations, or unexpected behaviors discovered.
(tagged union structure)
Represents the result of an Automated Reasoning validation check, indicating whether the content is logically valid, invalid, or falls into other categories based on the policy rules.
Note
This is a Tagged Union structure. Only one of the following top level keys can be set:valid,invalid,satisfiable,impossible,translationAmbiguous,tooComplex,noTranslations.valid -> (structure)
Indicates that the claims are true. The claims are implied by the premises and the Automated Reasoning policy. Given the Automated Reasoning policy and premises, it is not possible for these claims to be false.
translation -> (structure)
The logical translation of the input that this finding validates.
premises -> (list)
The logical statements that serve as the foundation or assumptions for the claims.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are being validated against the premises and policy rules.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.untranslatedPremises -> (list)
References to portions of the original input text that correspond to the premises but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.untranslatedClaims -> (list)
References to portions of the original input text that correspond to the claims but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.confidence -> (double)
A confidence score between 0 and 1 indicating how certain the system is about the logical translation.claimsTrueScenario -> (structure)
An example scenario demonstrating how the claims are logically true.
statements -> (list)
List of logical assignments and statements that define this scenario.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.supportingRules -> (list)
The automated reasoning policy rules that support why this result is considered valid.
(structure)
References a specific automated reasoning policy rule that was applied during evaluation.
id -> (string)
The unique identifier of the automated reasoning rule.policyVersionArn -> (string)
The ARN of the automated reasoning policy version that contains this rule.logicWarning -> (structure)
Indication of a logic issue with the translation without needing to consider the automated reasoning policy rules.
type -> (string)
The category of the detected logical issue, such as statements that are always true or always false.premises -> (list)
The logical statements that serve as premises under which the claims are validated.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are validated while assuming the policy and premises.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.invalid -> (structure)
Indicates that the claims are false. The claims are not implied by the premises and Automated Reasoning policy. Furthermore, there exist different claims that are consistent with the premises and Automated Reasoning policy.
translation -> (structure)
The logical translation of the input that this finding invalidates.
premises -> (list)
The logical statements that serve as the foundation or assumptions for the claims.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are being validated against the premises and policy rules.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.untranslatedPremises -> (list)
References to portions of the original input text that correspond to the premises but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.untranslatedClaims -> (list)
References to portions of the original input text that correspond to the claims but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.confidence -> (double)
A confidence score between 0 and 1 indicating how certain the system is about the logical translation.contradictingRules -> (list)
The automated reasoning policy rules that contradict the claims in the input.
(structure)
References a specific automated reasoning policy rule that was applied during evaluation.
id -> (string)
The unique identifier of the automated reasoning rule.policyVersionArn -> (string)
The ARN of the automated reasoning policy version that contains this rule.logicWarning -> (structure)
Indication of a logic issue with the translation without needing to consider the automated reasoning policy rules.
type -> (string)
The category of the detected logical issue, such as statements that are always true or always false.premises -> (list)
The logical statements that serve as premises under which the claims are validated.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are validated while assuming the policy and premises.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.satisfiable -> (structure)
Indicates that the claims can be true or false. It depends on what assumptions are made for the claim to be implied from the premises and Automated Reasoning policy rules. In this situation, different assumptions can make input claims false and alternative claims true.
translation -> (structure)
The logical translation of the input that this finding evaluates.
premises -> (list)
The logical statements that serve as the foundation or assumptions for the claims.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are being validated against the premises and policy rules.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.untranslatedPremises -> (list)
References to portions of the original input text that correspond to the premises but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.untranslatedClaims -> (list)
References to portions of the original input text that correspond to the claims but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.confidence -> (double)
A confidence score between 0 and 1 indicating how certain the system is about the logical translation.claimsTrueScenario -> (structure)
An example scenario demonstrating how the claims could be logically true.
statements -> (list)
List of logical assignments and statements that define this scenario.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claimsFalseScenario -> (structure)
An example scenario demonstrating how the claims could be logically false.
statements -> (list)
List of logical assignments and statements that define this scenario.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.logicWarning -> (structure)
Indication of a logic issue with the translation without needing to consider the automated reasoning policy rules.
type -> (string)
The category of the detected logical issue, such as statements that are always true or always false.premises -> (list)
The logical statements that serve as premises under which the claims are validated.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are validated while assuming the policy and premises.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.impossible -> (structure)
Indicates that Automated Reasoning cannot make a statement about the claims. This can happen if the premises are logically incorrect, or if there is a conflict within the Automated Reasoning policy itself.
translation -> (structure)
The logical translation of the input that this finding evaluates.
premises -> (list)
The logical statements that serve as the foundation or assumptions for the claims.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are being validated against the premises and policy rules.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.untranslatedPremises -> (list)
References to portions of the original input text that correspond to the premises but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.untranslatedClaims -> (list)
References to portions of the original input text that correspond to the claims but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.confidence -> (double)
A confidence score between 0 and 1 indicating how certain the system is about the logical translation.contradictingRules -> (list)
The automated reasoning policy rules that contradict the claims and/or premises in the input.
(structure)
References a specific automated reasoning policy rule that was applied during evaluation.
id -> (string)
The unique identifier of the automated reasoning rule.policyVersionArn -> (string)
The ARN of the automated reasoning policy version that contains this rule.logicWarning -> (structure)
Indication of a logic issue with the translation without needing to consider the automated reasoning policy rules.
type -> (string)
The category of the detected logical issue, such as statements that are always true or always false.premises -> (list)
The logical statements that serve as premises under which the claims are validated.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are validated while assuming the policy and premises.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.translationAmbiguous -> (structure)
Indicates that an ambiguity was detected in the translation, making it unsound to continue with validity checking. Additional context or follow-up questions might be needed to get translation to succeed.
options -> (list)
Different logical interpretations that were detected during translation of the input.
(structure)
Represents one possible logical interpretation of ambiguous input content.
translations -> (list)
Different logical interpretations that were detected during translation of the input.
(structure)
Contains the logical translation of natural language input into formal logical statements, including premises, claims, and confidence scores.
premises -> (list)
The logical statements that serve as the foundation or assumptions for the claims.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.claims -> (list)
The logical statements that are being validated against the premises and policy rules.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.untranslatedPremises -> (list)
References to portions of the original input text that correspond to the premises but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.untranslatedClaims -> (list)
References to portions of the original input text that correspond to the claims but could not be fully translated.
(structure)
References a portion of the original input text that corresponds to logical elements.
text -> (string)
The specific text from the original input that this reference points to.confidence -> (double)
A confidence score between 0 and 1 indicating how certain the system is about the logical translation.differenceScenarios -> (list)
Scenarios showing how the different translation options differ in meaning.
(structure)
Represents a logical scenario where claims can be evaluated as true or false, containing specific logical assignments.
statements -> (list)
List of logical assignments and statements that define this scenario.
(structure)
Represents a logical statement that can be expressed both in formal logic notation and natural language, providing dual representations for better understanding and validation.
logic -> (string)
The formal logic representation of the statement using mathematical notation and logical operators.naturalLanguage -> (string)
The natural language representation of the logical statement, providing a human-readable interpretation of the formal logic.tooComplex -> (structure)
Indicates that the input contains too much information for Automated Reasoning to process within its latency limits.noTranslations -> (structure)
Identifies that some or all of the input prompt wasn’t translated into logic. This can happen if the input isn’t relevant to the Automated Reasoning policy, or if the policy doesn’t have variables to model relevant input.testRunResult -> (string)
The overall result of the test run, indicating whether the policy passed or failed validation.aggregatedTestFindingsResult -> (string)
A summary of all test findings, aggregated to provide an overall assessment of policy quality and correctness.updatedAt -> (timestamp)
The timestamp when the test results were last updated.