Step 4: Creating an Amazon Kendra index and ingesting the metadata

Open the Amazon Kendra console at https://console.aws.amazon.com/kendra/.

Choose Create an index.

For Index details on the Specify index details page, do the following:

1. For Index name, enter kendra-index.

2. Keep the Description field blank.

3. For IAM role, choose Create a new role. This role provides access to your Amazon S3 bucket.

4. For Role name, enter kendra-role. The IAM role will have the prefix AmazonKendra-.

5. Keep default settings for Encryption and Tags and choose Next.

For Access control settings on the Configure user access control page, choose No and then choose Next.

For Provisioning editions on the Provisioning details page, choose Developer edition and choose Create.

To create and attach an IAM role for Amazon Kendra that recognizes it as a trusted entity, do the following:

1. Save the following trust policy as a JSON file called kendra-trust-policy.json in a text editor on your local device.

   JSON

   {
       "Version": "2012-10-17",
       "Statement": {
           "Effect": "Allow",
           "Principal": {
               "Service": "kendra.amazonaws.com"
           },
           "Action": "sts:AssumeRole"
       }
   }
   

2. To create an IAM role called kendra-role and attach your saved kendra-trust-policy.json file to it, use the create-role command:

   Linux

   aws iam create-role \
             --role-name kendra-role \
             --assume-role-policy-document file://path/kendra-trust-policy.json

   Where:

   • path/ is the filepath to kendra-trust-policy.json on your local device.

   macOS

   aws iam create-role \
             --role-name kendra-role \
             --assume-role-policy-document file://path/kendra-trust-policy.json

   Where:

   • path/ is the filepath to kendra-trust-policy.json on your local device.

   Windows

   aws iam create-role ^
             --role-name kendra-role ^
             --assume-role-policy-document file://path/kendra-trust-policy.json

   Where:

   • path/ is the filepath to kendra-trust-policy.json on your local device.

3. Copy the Amazon Resource Name (ARN) to your text editor and save it locally as kendra-role-arn.

   Note

   The ARN has a format similar to arn:aws:iam::123456789012:role/kendra-role. You need the ARN you saved as kendra-role-arn to run Amazon Kendra jobs.

Before you create an index, you must provide your kendra-role the permission to write to CloudWatch Logs. To do this, complete the following steps:

1. Save the following trust policy as a JSON file called kendra-cloudwatch-policy.json in a text editor on your local device.

   JSON

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": "cloudwatch:PutMetricData",
               "Resource": "*",
               "Condition": {
                   "StringEquals": {
                       "cloudwatch:namespace": "Kendra"
                   }
               }
           },
           {
               "Effect": "Allow",
               "Action": "logs:DescribeLogGroups",
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": "logs:CreateLogGroup",
               "Resource": "arn:aws:logs:aws-region:aws-account-id:log-group:/aws/kendra/*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "logs:DescribeLogStreams",
                   "logs:CreateLogStream",
                   "logs:PutLogEvents"
               ],
               "Resource": "arn:aws:logs:aws-region:aws-account-id:log-group:/aws/kendra/*:log-stream:*"
           }
       ]
   }
   

   Replace aws-region with your AWS region, and aws-account-id with your 12-digit AWS account ID.

2. To create an IAM policy to access CloudWatch Logs, use the create-policy command:

   Linux

   aws iam create-policy \
             --policy-name kendra-cloudwatch-policy \
             --policy-document file://path/kendra-cloudwatch-policy.json

   Where:

   • path/ is the filepath to kendra-cloudwatch-policy.json on your local device.

   macOS

   aws iam create-policy \
             --policy-name kendra-cloudwatch-policy \
             --policy-document file://path/kendra-cloudwatch-policy.json

   Where:

   • path/ is the filepath to kendra-cloudwatch-policy.json on your local device.

   Windows

   aws iam create-policy ^
             --policy-name kendra-cloudwatch-policy ^
             --policy-document file://path/kendra-cloudwatch-policy.json

   Where:

   • path/ is the filepath to kendra-cloudwatch-policy.json on your local device.

3. Copy the Amazon Resource Name (ARN) to your text editor and save it locally as kendra-cloudwatch-arn.

   Note

   The ARN has a format similar to arn:aws:iam::123456789012:role/kendra-cloudwatch-policy. You need the ARN you saved as kendra-cloudwatch-arn to attach the kendra-cloudwatch-policy to your IAM role.

4. To attach the kendra-cloudwatch-policy to your IAM role, use the attach-role-policy command:

   Linux

   aws iam attach-role-policy \
             --policy-arn policy-arn \
             --role-name kendra-role

   Where:

   • policy-arn is your saved kendra-cloudwatch-arn.

   macOS

   aws iam attach-role-policy \
             --policy-arn policy-arn \
             --role-name kendra-role

   Where:

   • policy-arn is your saved kendra-cloudwatch-arn.

   Windows

   aws iam attach-role-policy ^
             --policy-arn policy-arn ^
             --role-name kendra-role

   Where:

   • policy-arn is your saved kendra-cloudwatch-arn.

To create an index, use the create-index command:

Copy the index Id and save it in a text editor as kendra-index-id. The Id helps you track the status of your index creation.

To track the progress of your index creation job, use the describe-index command:

Open the IAM console at https://console.aws.amazon.com/iam/.

In the left navigation pane, choose Roles and enter kendra-role in the Search box above Role name.

From the suggested options, click on kendra-role.

In Summary, choose Attach policies.

In Attach permissions, in the Search box, enter S3 and select the checkbox next to the AmazonS3ReadOnlyAccess policy from the suggested options.

Choose Attach policy. On the Summary page, you will now see two policies attached to the IAM role.

Return to the Amazon Kendra console at https://console.aws.amazon.com/kendra/ and wait for the status of your index to change from Creating to Active before continuing to the next step.

Save the following text in a JSON file called kendra-S3-access-policy.json in a text editor on your local device.

Replace amzn-s3-demo-bucket with your S3 bucket name, aws-region with your AWS region, aws-account-id with your 12-digit AWS account ID, and kendra-index-id with your saved kendra-index-id.

To create an IAM policy to access your S3 bucket, use the create-policy command:

Copy the Amazon Resource Name (ARN) to your text editor and save it locally as kendra-S3-access-arn.

To attach the kendra-S3-access-policy to your Amazon Kendra IAM role, use the attach-role-policy command:

Open the Amazon Kendra console at https://console.aws.amazon.com/kendra/.

From the Indexes list, click on kendra-index.

From the left navigation panel, under Data management, choose Facet definition.

From the Index fields menu, choose Add field.

In the Add index field dialog box, do the following:

1. In Field name, enter COMMERCIAL_ITEM.

2. In Data type, choose String list.

3. In Usage types, select Facetable, Searchable, and Displayable, and then choose Add.

4. Repeat steps a to c for each Amazon Comprehend entity type: COMMERCIAL_ITEM, DATE, EVENT, LOCATION, ORGANIZATION, OTHER, PERSON, QUANTITY, TITLE.

Save the following text as a JSON file called custom-attributes.json in a text editor on your local device.

[
   {
       "Name": "COMMERCIAL_ITEM",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "DATE",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "EVENT",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "LOCATION",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "ORGANIZATION",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "OTHER",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "PERSON",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "QUANTITY",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   },
   {
       "Name": "TITLE",
       "Type": "STRING_LIST_VALUE",
       "Search": {
           "Facetable": true,
           "Searchable": true,
           "Displayable": true
       }
   }
]

To create custom fields in your index, use the update-index command:

To verify that the custom attributes have been added to your index, use the describe-index command:

Open the Amazon Kendra console at https://console.aws.amazon.com/kendra/.

From the Indexes list, click on kendra-index.

From the left navigation menu, under Data management, choose Data sources.

Under the Select data source connector type section, navigate to Amazon S3, and choose Add connector.

In the Specify data source details page, do the following:

1. Under Name and description, for Data source name, enter S3-data-source.

2. Keep the Description section blank.

3. Keep the default settings for Tags.

4. Choose Next.

On the Configure sync settings page, in the Sync scope section, do the following:

1. In Enter the data source location, choose Browse S3.

2. In Choose resources, select your S3 bucket and then choose Choose.

3. In Metadata files prefix folder location, choose Browse S3.

4. In Choose resources, click on the name of your bucket from the list of buckets.

5. For Objects, select the option box for metadata and choose Choose. The location field should now say metadata/.

6. Keep the default settings for Access control list configuration file location, Select decryption key, and Additional configuration.

For IAM role, on the Configure sync settings page, choose kendra-role.

On the Configure sync settings page, under Sync run schedule, for Frequency, choose Run on demand and then choose Next.

On the Review and create page, review your choices for the data source details and choose Add data source.

Save the following text as a JSON file called S3-data-connector.json in a text editor on your local device.

{
   "S3Configuration":{
      "BucketName":"amzn-s3-demo-bucket",
      "DocumentsMetadataConfiguration":{
         "S3Prefix":"metadata"
      }
   }
}

Replace amzn-s3-demo-bucket with the name of your S3 bucket.

To connect your S3 bucket to your index, use the create-data-source command:

Copy the connector Id and save it in a text editor as S3-connector-id. The Id helps you track the status of the data-connection process.

To ensure that your S3 data source was connected successfully, use the describe-data-source command:

Open the Amazon Kendra console at https://console.aws.amazon.com/kendra/.

From the Indexes list, click on kendra-index.

From the left navigation menu, choose Data sources.

From Data sources, select S3-data-source.

From the top navigation bar, choose Sync now.

To sync your index, use the start-data-source-sync-job command:

To check the status of the index sync, use the list-data-source-sync-jobs command: