Set up centralized logging at enterprise scale by using Terraform

Deploy AWS Control Tower by following the instructions in the AWS Control Tower documentation.

Deploy AFT by following the instructions in the AWS Control Tower documentation.

Configure AWS Command Line Interface (AWS CLI) with management account credentials, which provide administrative permissions to manage AWS Control Tower.

Run the following AWS CLI command in any AWS Region:

aws ram enable-sharing-with-aws-organization

This enables resource sharing within your organization in AWS Organizations across all Regions that support AWS Resource Access Manager (AWS RAM).

Create a folder named Application_account in the root path of the aft-account-customizations repository. This repository is created automatically when you set up AFT (see the previous epic).

Navigate to the root directory of the centralised-logging-at-enterprise-scale-using-terraform repository, copy the contents of the aft/account directory, and then paste them into the Application_account directory that you created in step 1 in the aft-account-customizations repository.

From the root directory of the centralised-logging-at-enterprise-scale-using-terraform repository, copy the contents of the Application_account directory into the Application_account/terraform directory in the aft-account-customizations repository.

In the aft-account-customizations/Application_account/terraform.tfvars file, confirm that all parameters are passed as arguments in the corresponding Terraform configuration files.

Create a folder named Log_archive_account in the root path of the aft-account-customizations repository, This repository is created automatically when you set up AFT.

Navigate to the root directory of the centralised-logging-at-enterprise-scale-using-terraform repository, copy the contents of the aft/account directory, and then paste them into the Log_archive_account directory that you created in the previous step in the aft-account-customizations repository.

From the root directory of the centralised-logging-at-enterprise-scale-using-terraform repository, copy the contents of the Log_archive_account directory into the Log_archive_account/terraform directory in the aft-account-customizations repository.

In the aft-account-customizations/Log_archive_account/terraform.tfvars file, confirm that all parameters are passed as arguments in the corresponding Terraform configuration files.

Clone the GitHub repository and configure the input parameters in the terraform.tfvars file.

Run the following command:

$ terraform init

Preview changes:

$ terraform plan

This command evaluates the Terraform configuration to determine the desired state of the resources and compares it with the current state of your infrastructure.

Apply changes:

$ terraform apply

Review the planned changes and type yes at the prompt to proceed with the application.

In the Application account, open the CloudWatch console.

In the left navigation pane, choose Log groups.

Select each log group (/aws/rds, /aws/eks, /aws/lambda) and choose the Subscription filters tab.

You should see active subscription filters that point to the destination ARN, based on the name that you specified in the Terraform configuration file.

Choose any subscription filter to verify its configuration and status.

In the Log Archive account, open the Firehose console.

In the left navigation pane, choose Firehose streams.

Choose any Firehose streams and verify the following:

In the Log Archive account, open the Amazon S3 console.

Select each central logging bucket.

Navigate through the folder structure: AWSLogs/AccountID/Region/Service.

You should see log files organized by timestamp (YYYY/MM/DD/HH).

Choose any recent log file and verify its format and data integrity.

In the Log Archive account, open the Amazon SQS console.

In the left navigation pane, choose Queues.

Select each configured queue and choose Send and receive messages.

You should see messages that contain S3 event notifications for new log files.

Choose any message to verify that  it contains the correct S3 object information.

Navigate to the aft-account-customizations repository.

Go to the terraform directory.

Delete the following files:

Clear the contents of the main.tf file.

Push your changes to the repository:

# Stage all changes
$ git add *

# Commit cleanup changes
$ git commit -m "Remove AFT customizations"

# Push to repository
$ git push origin main

Initialize the Terraform configuration:

$ terraform init

This command initializes Terraform and ensures access to the current state.

Preview cleanup changes:

$ terraform destroy

This command evaluates which resources will be destroyed and compares the desired state with the current state of your infrastructure.

Run cleanup. When you're prompted, type yes to confirm and run the destruction plan.