Web Crawler

The Web Crawler connects to and crawls URLs you specify for use in your managed knowledge base. The Web Crawler traverses HTML pages starting from your seed URLs, following child links according to your crawl scope and limits. You can also provide sitemap URLs as starting points. The Web Crawler respects robots.txt in accordance with RFC 9309.

Supported features

Authentication methods

The Web Crawler supports four authentication methods. Choose the method that matches how the target site authenticates users. For public sites with no sign-in, use NO_AUTH.

Prerequisites

For the website you want to crawl, make sure you:

In your AWS account, make sure you:

How to set up a Web Crawler data source

Setting up a Web Crawler data source involves the following steps:

Create the data source

Console

To connect the Web Crawler to your managed knowledge base

1. Under Data source, provide a name for your data source.

2. Select Web Crawler from the data source dropdown.

3. Under Source, choose Source URLs (up to 10 starting-point URLs) or Source sitemaps (up to 3 sitemap URLs).

4. Enter your URLs in the Add URLs text area, one per line.

5. Under Authentication, select No authentication, Basic authentication, Form authentication, or SAML authentication. For any method other than No authentication, select or create an AWS Secrets Manager secret to store your credentials.

6. (Optional) Expand Sync scope to set the crawl depth (0–10), maximum links per URL (1–1000), maximum URLs crawled per minute (1–300), and the crawl scope: Default (same host and same initial URL path as the seed URL), Host only (same host, any path), or Subdomains (same primary domain, including subdomains).

7. (Optional) Expand URL filter patterns to add regular expressions that include or exclude specific URLs.

API

To create a Web Crawler data source, send a CreateDataSource request with an Agents for Amazon Bedrock build-time endpoint. The following AWS Command Line Interface example creates a data source that crawls a public site without authentication. For a description of each field, see the connector parameters reference that follows.

aws bedrock-agent create-data-source \
 --name "WebCrawler-connector" \
 --knowledge-base-id "your-knowledge-base-id" \
 --data-source-configuration file://webcrawler-managed-connector.json

The webcrawler-managed-connector.json file contains the following:

{
    "type": "MANAGED_KNOWLEDGE_BASE_CONNECTOR",
    "managedKnowledgeBaseConnectorConfiguration": {
        "connectorParameters": {
            "type": "WEB",
            "version": "1",
            "connectionConfiguration": {
                "seedUrls": [
                    "https://docs.example.com"
                ],
                "authType": "NO_AUTH"
            },
            "crawlConfiguration": {
                "crawlDepth": 3,
                "maxLinksPerUrl": 100,
                "maxCrawledUrlsPerMinute": 50,
                "syncScope": "SUB_DOMAINS",
                "crawlAttachments": true
            },
            "filterConfiguration": {
                "exclusionPatterns": [
                    "https://docs.example.com/private/.*"
                ]
            }
        }
    }
}

For an authenticated site, set authType to BASIC_AUTH, FORM, or SAML, and add a secretArn to connectionConfiguration.

For managed knowledge bases, CreateDataSource is asynchronous: the data source status transitions from CREATING to AVAILABLE when the operation completes.

Connector parameters

The data source configuration uses the following connector parameters. To use the Web Crawler, specify WEB as the connector type in connectorParameters. For the fields that wrap connectorParameters (such as deletionProtectionConfiguration and mediaExtractionConfiguration), see Connect a data source.

Authentication credentials

If your website requires authentication, store your credentials in an AWS Secrets Manager secret. The secret format depends on the authentication type you choose.

Basic authentication (BASIC_AUTH)

{
    "userName": "your-username",
    "password": "your-password",
    "authentication": "BASIC_AUTH"
}

Form authentication (FORM)

For form-based authentication, provide XPath expressions that identify the user name field, password field, and submit button on the login page.

{
    "authentication": "FORM",
    "loginPageUrl": "https://example.com/login",
    "userName": "your-username",
    "password": "your-password",
    "userNameFieldXpath": "//input[@name='username']",
    "passwordFieldXpath": "//input[@name='password']",
    "userNameButtonXpath": "//button[@type='submit']",
    "passwordButtonXpath": "//button[@type='submit']"
}

SAML authentication (SAML)

For SAML authentication, provide the SAML identity provider's login page URL and XPath expressions for the form fields.

{
    "authentication": "SAML",
    "loginPageUrl": "https://your-idp.example.com/login",
    "userName": "your-username",
    "password": "your-password",
    "userNameFieldXpath": "//input[@name='username']",
    "passwordFieldXpath": "//input[@name='password']",
    "userNameButtonXpath": "//button[@type='submit']",
    "passwordButtonXpath": "//button[@type='submit']"
}

Troubleshooting