Amazon Q Business will no longer be open to new customers starting on July 31, 2026. If you would like to use the service, please sign up prior to July 30. For capabilities similar to Q Business, explore Amazon Quick. [Learn more](https://docs.aws.amazon.com/amazonq/latest/qbusiness-ug/qbusiness-availability-change.html).

# Configuring a `robots.txt` file for Amazon Q Business Web Crawler
<a name="stop-web-crawler"></a>

The `robots.txt` file is a standard used to implement the Robots Exclusion Protocol, allowing website owners to specify which parts of their site visiting web crawlers and robots can access. Amazon Q Business Web Crawler adheres to the rules set in your website’s `robots.txt` file, which determines the areas it is allowed or not allowed to visit. Amazon Q Business Web Crawler respects standard robots.txt directives like `Allow` and `Disallow`. To control how Amazon Q Business Web Crawler interacts with your website, you can simply adjust these rules in your robots.txt file.

**Topics**
+ [Configuring how Amazon Q Web Crawler accesses your website](#configure-web-crawler-website-access)
+ [Stopping Amazon Q Web Crawler from crawling your website](#stop-web-crawler-access)

## Configuring how Amazon Q Web Crawler accesses your website
<a name="configure-web-crawler-website-access"></a>

You can control how the Amazon Q Web Crawler indexes your website using `Allow` and `Disallow` directives. You can also control which web pages are indexed and which web pages are not crawled.

**To allow Amazon Q Web Crawler to crawl all web pages except disallowed web pages, use the following directive:**

```
User-agent: amazon-QBusiness    # Amazon Q Web Crawler
Disallow: /credential-pages/ # disallow access to specific pages
```

**To allow Amazon Q Web Crawler to crawl only specific web pages, use the following directive:**

```
User-agent: amazon-QBusiness   # Amazon Q Web Crawler
Allow: /pages/ # allow access to specific pages
```

**To allow Amazon Q Web Crawler to crawl all website content and disallow crawling for any other robots, use the following directive:**

```
User-agent: amazon-QBusiness # Amazon Q Web Crawler
Allow: / # allow access to all pages
User-agent: * # any (other) robot
Disallow: / # disallow access to any pages
```

## Stopping Amazon Q Web Crawler from crawling your website
<a name="stop-web-crawler-access"></a>

You can stop Amazon Q Web Crawler from indexing your website using the `Disallow` directive. You can also control which web pages are crawled and which aren't.

**To stop Amazon Q Web Crawler from crawling the website, use the following directive:**

```
User-agent: amazon-QBusiness # Amazon Q Web Crawler
Disallow: / # disallow access to any pages
```

If you have any questions or concerns about Amazon Q Web Crawler, you can reach out to the [AWS support team](https://aws.amazon.com/contact-us/?nc1=f_m).