Improve accuracy by adding Automated Reasoning checks in Amazon Bedrock Guardrails

Automated Reasoning checks in Amazon Bedrock Guardrails mathematically verify natural language content against your defined policies, ensuring strict compliance with your guardrails. These checks can help to systematically block harmful or non-compliant content before it reaches your users. Unlike pattern-matching approaches, Automated Reasoning delivers higher accuracy with fewer false positives, particularly for complex policy requirements. For customers prioritizing precision, policy rules can be customized to enhance guardrail effectiveness through clear logic statements.

A key challenge with large language models (LLMs) is ensuring the accuracy of their responses. Without validation, LLMs can sometimes produce hallucinations or inaccurate information that undermines trust.

Automated Reasoning checks in Amazon Bedrock Guardrails solve this problem by using mathematical techniques to:

This feature is especially valuable when you need to demonstrate the factual basis for an LLM's response in:

Automated Reasoning checks in Amazon Bedrock Guardrails will not protect against prompt injection attacks. These checks validate exactly what you send them - if malicious or manipulated content is provided as input, the validation will be performed on that content as-is (garbage-in, garbage-out). To detect and block prompt injection attacks, use Content filters in combination with Automated Reasoning checks.

Automated Reasoning only analyzes and detects text that is relevant to the Automated Reasoning policy. It will ignore the rest of the content and cannot tell developers whether the answer is off-topic or not. If you need to detect off-topic responses, use other guardrail components such as topic policies.

Limitations and considerations

Before implementing Automated Reasoning checks, be aware of these important limitations:

Best practices

Follow these best practices to maximize the effectiveness of your Automated Reasoning policies:

Pricing

Automated Reasoning checks in Amazon Bedrock Guardrails are charged based on the number of validation requests processed. For current pricing information, see the Amazon Bedrock pricing page.

Charges are incurred for each validation request, regardless of the result (for example, VALID, INVALID, TRANSLATION_AMBIGUOUS). To optimize costs:

Cross-region inference for policy operations

Automated Reasoning utilizes cross-region inference to optimize the performance and availability of policy creation and testing operations. Specific API operations automatically distribute processing across AWS Regions within your geographic boundary to ensure reliable service delivery.

The following Automated Reasoning API operations employ cross-region inference:

These operations invoke large language models to extract formal logic rules from source documents and translate natural language constructs into structured logical representations. To ensure optimal performance and availability, request processing is distributed according to the following geographic routing:

Cross-region inference operates transparently without requiring customer configuration. API functionality remains consistent regardless of the specific Region that processes the request.

To use Automated Reasoning checks in Amazon Bedrock Guardrails, follow these steps:

The workflow can be visualized as:

Source Document → Extracted Policy → Testing → Deployment → Runtime Validation
    (rules)        (formal logic)    (verify)   (activate)    (check responses)
  

Policies

Automated Reasoning policies are the foundation of accuracy validation, containing logic rules and variables that are automatically extracted from your source document. These policies serve as the mathematical representation of your business rules, enabling the system to systematically validate whether foundation model responses comply with your defined constraints. To perform Automated Reasoning checks in your application, you configure your guardrail to use a policy that matches your specific domain and validation requirements.

Rules

Rules are logic that Automated Reasoning extracts from your source document. These might be written as if-then statements. Here are some examples of the rule format:

if <premise>, then <claim>

<premise> is true

Variables

Variables represent concepts in your Automated Reasoning policy that can have values assigned to them when translating natural language into formal logic. Your policy rules define constraints on what are valid or invalid values for these variables.

Variables have a name, a type, and a description. For example, a policy about employee benefits might have a variable with the name "employee_age", the type "integer", and the description "The age of the employee in years". This variable could be assigned a value like 25 based on natural language in a prompt to an application.

For example, an is_full_time variable in an HR policy might have a description that states "Employees that work more than 20 hours per week," which is a direct quote from the source document. When using a chatbot, users are more likely to say "I'm full-time," and not, "I work more than 20 hours per week."

The accuracy of the translation from natural language to formal logic is highly dependent on the quality of variable descriptions. While the reasoning process is sound once the translation is complete, clear and comprehensive variable descriptions ensure that user prompts are correctly interpreted. Without complete variable descriptions, Automated Reasoning might return NO_DATA because it can't translate the input natural language into its formal logic representation.

It's important for a variable description to account for scenarios like this. A comprehensive variable description might state, "Employees who work more than 20 hours per week are full-time. Users will say full-time to set this value to true, or part-time to set it to false."

Predefined variable types

The following table describes the predefined types of variables that your policy can have.