Automate the deployment of AWS Supply Chain data lakes in a multi-repository setup by using GitHub Actions, Artifactory, and Terraform

Clone the repository.

To clone this pattern’s repository, run the following command in your local workstation:

git clone https://github.com/aws-samples/sample-automate-aws-supply-chain-deployment.git
cd ASC-Deployment

(Automated option) Verify prerequisites for deployment.

Make sure that the Prerequisites are complete for the automated deployment.

(Manual option) Prepare for deployment of AWS Supply Chain datasets.

To go to the terraform-deployment directory of ASC-Datasets, run the following command:

cd ASC-Datasets/terraform-deployment

To assume the role ARN that was created in the Prerequisites, run the following command:

aws sts assume-role --role-arn <enter AWS user role ARN> --role-session-name <your-session-name>

To configure and export the environment variables, run the following commands:

# Export Environment variables
export REGION=<Enter deployment region>
export REPO_NAME=<Enter Current ASC Datasets dir name>
export PROJECT_NAME="asc-deployment-poc"
export ACCOUNT_ID=<Enter deployment Account ID>
export ENVIRONMENT="dev"
export LAMBDA_LAYER_TEMP_DIR_TERRAFORM="layerOutput"
export LAMBDA_FUNCTION_TEMP_DIR_TERRAFORM="lambdaOutput"
export AWS_USER_ROLE=<Enter user role ARN for AWS Console access and deployment>
export S3_TERRAFORM_ARTIFACTS_BUCKET_NAME="$PROJECT_NAME-$ACCOUNT_ID-$REGION-terraform-artifacts-$ENVIRONMENT"

(Manual option) Prepare for managing AWS Supply Chain integration flows in deployment.

To go to the terraform-deployment directory of ASC-Integration-Flows, run the following command:

cd ASC-Integration-Flows/terraform-deployment

To assume the role ARN that was created earlier, run the following command:

aws sts assume-role --role-arn <enter AWS user role ARN> --role-session-name <your-session-name>

To configure and export the environment variables, run the following commands:

# Export Environment variables
export REGION=<Enter deployment region>
export REPO_NAME=<Enter Current ASC Integration Flows dir name>
export ASC_DATASET_VARS_REPO=<Enter Current ASC Datasets dir name>  #Must be the same directory name used for ASC Datasets deployment
export PROJECT_NAME="asc-deployment-poc"
export ACCOUNT_ID=<Enter deployment Account ID>
export ENVIRONMENT="dev"
export LAMBDA_LAYER_TEMP_DIR_TERRAFORM="layerOutput"
export LAMBDA_FUNCTION_TEMP_DIR_TERRAFORM="lambdaOutput"
export S3_TERRAFORM_ARTIFACTS_BUCKET_NAME="$PROJECT_NAME-$ACCOUNT_ID-$REGION-terraform-artifacts-$ENVIRONMENT"

Copy the ASC-Datasets directory.

To copy the ASC-Datasets directory to a new location, use the following steps:

Set up the ASC-Datasets directory.

To set up ASC-Datasets as a standalone repository in your organization, run the following commands:

git init
git add .
git commit -m "Initial commit: ASC-Datasets standalone repository"
git remote add origin <INSERT_ASC_DATASETS_GITHUB_URL>
git branch -M dev

Configure the branch name in the .github workflow file.

Set up the branch name in the deployment workflow file as shown in the following example:

   on:
     workflow_dispatch:
     push:
       branches:
         - dev     #Change to any other branch preferred for deployment

Set up GitHub environments and configure environment values.

To set up GitHub environments in your GitHub organization, use the instructions in Setup GitHub environments in this pattern’s repository.

To configure environment values in the workflow files, use the instructions in Setup environment values in the workflow files in this pattern’s repository.

Trigger the workflow.

To push your changes to your GitHub organization and trigger the deployment workflow, run the following command:

git push -u origin dev

Copy the ASC-Integration-Flows directory.

To copy the ASC-Integration-Flows directory to a new location, use the following steps:

Set up the ASC-Integration-Flows directory.

To set up the ASC-Integration-Flows directory as a standalone repository in your organization, run the following commands:

git init
git add .
git commit -m "Initial commit: ASC-Integration-Flows standalone repository"
git remote add origin <INSERT_ASC_Integration_Flows_GITHUB_URL>
git branch -M dev

Configure the branch name in the .github workflow file.

Set up the branch name in the deployment workflow file as shown in the following example:

   on:
     workflow_dispatch:
     push:
       branches:
         - dev     #Change to any other branch preferred for deployment

Set up GitHub environments and configure environment values.

To set up GitHub environments in your GitHub organization, use the instructions in Setup GitHub environments in this pattern’s repository.

To configure environment values in the workflow files, use the instructions in Setup environment values in the workflow files in this pattern’s repository.

Trigger the workflow.

To push your changes to your GitHub organization and trigger the deployment workflow, run the following command:

git push -u origin dev

Navigate to the terraform-deployment directory.

To go to the terraform-deployment directory of ASC-Datasets, run the following command:

cd ASC-Datasets/terraform-deployment

Set up the Terraform state Amazon S3 bucket.

To set up the Terraform state Amazon S3 bucket, use the following script:

# Setup terraform bucket
chmod +x ../scripts/setup-terraform.sh
../scripts/setup-terraform.sh

Set up the Terraform artifacts Amazon S3 bucket.

To set up the Terraform artifacts Amazon S3 bucket, use the following script:

# Setup terraform artifacts bucket
chmod +x ../scripts/setup-terraform-artifacts-bucket.sh
../scripts/setup-terraform-artifacts-bucket.sh

Set up the Terraform backend and providers configuration.

To set up the Terraform backend and providers configuration, use the following script:

# Setup terraform backend and providers config if they don't exist
chmod +x ../scripts/generate-terraform-config.sh
../scripts/generate-terraform-config.sh

Generate a deployment plan.

To generate a deployment plan, run the following commands:

# Run terraform init and validate
terraform init
terraform validate

# Run terraform plan
terraform plan \
-var-file="tfInputs/$ENVIRONMENT.tfvars" \
-var="project_name=$PROJECT_NAME" \
-var="environment=$ENVIRONMENT" \
-var="user_role=$AWS_USER_ROLE" \
-var="lambda_temp_dir=$LAMBDA_FUNCTION_TEMP_DIR_TERRAFORM" \
-var="layer_temp_dir=$LAMBDA_LAYER_TEMP_DIR_TERRAFORM" \
-parallelism=40 \
-out='tfplan.out'

Deploy the configurations.

To deploy the configurations, run the following command:

# Run terraform apply
terraform apply tfplan.out

Update other configurations and store outputs.

To update AWS KMS key policies and store the applied configurations outputs in the Terraform artifacts Amazon S3 bucket, run the following commands:

# Update AWS Supply Chain KMS Key policy with the service's requirements
chmod +x ../scripts/update-asc-kms-policy.sh
../scripts/update-asc-kms-policy.sh

# Update AWS KMS Keys' policy with IAM roles
chmod +x ../scripts/update-kms-policy.sh
../scripts/update-kms-policy.sh

# Create terraform outputs file to be used as input variables
terraform output -json > raw_output.json
jq -r 'to_entries | map(
  if .value.type == "string" then
      "\(.key) = \"\(.value.value)\""
  else
      "\(.key) = \(.value.value | tojson)"
  end
) | .[]' raw_output.json > $REPO_NAME-outputs.tfvars

# Upload reformed outputs file to Amazon S3 terraform artifacts bucket (For retrieval from other repositories)
aws s3 cp $REPO_NAME-outputs.tfvars s3://$S3_TERRAFORM_ARTIFACTS_BUCKET_NAME/$REPO_NAME-outputs.tfvars
rm -f raw_output.json
rm -f $REPO_NAME-outputs.tfvars

Navigate to the terraform-deployment directory.

To go to the terraform-deployment directory of ASC-Integration-Flows, run the following command:

cd ASC-Integration-Flows/terraform-deployment

Set up the Terraform backend and providers configuration.

To set up the Terraform backend and provider configurations, use the following script:

# Setup terraform backend and providers config if they don't exist
chmod +x ../scripts/generate-terraform-config.sh
../scripts/generate-terraform-config.sh

Generate a deployment plan.

To generate a deployment plan, run the following commands. These commands initialize your Terraform environment, merge configuration variables from ASC-Datasets with your existing Terraform configurations, and generate a deployment plan.

# Run terraform init and validate
terraform init
terraform validate

# Download and merge ASC DATASET tfvars
chmod +x ../scripts/download-vars-through-s3.sh
../scripts/download-vars-through-s3.sh $ASC_DATASET_VARS_REPO

# Run terraform plan
terraform plan \
-var-file="tfInputs/$ENVIRONMENT.tfvars" \
-var="project_name=$PROJECT_NAME" \
-var="environment=$ENVIRONMENT" \
-var="lambda_temp_dir=$LAMBDA_FUNCTION_TEMP_DIR_TERRAFORM" \
-var="layer_temp_dir=$LAMBDA_LAYER_TEMP_DIR_TERRAFORM" \
-parallelism=40 \
-out='tfplan.out'

Deploy the configurations.

To deploy the configurations, run the following command:

# Run terraform apply
terraform apply tfplan.out

Update other configurations.

To update AWS KMS key policies and store the applied configurations outputs in the Terraform artifacts Amazon S3 bucket, run the following commands:

# Update AWS KMS Keys' policy with IAM roles
chmod +x ../scripts/update-kms-policy-through-s3.sh
../scripts/update-kms-policy-through-s3.sh $ASC_DATASET_VARS_REPO

# Create terraform outputs file to be used as input variables
terraform output -json > raw_output.json
jq -r 'to_entries | map(
  if .value.type == "string" then
      "\(.key) = \"\(.value.value)\""
  else
      "\(.key) = \(.value.value | tojson)"
  end
) | .[]' raw_output.json > $REPO_NAME-outputs.tfvars

# Upload reformed outputs file to Amazon S3 terraform artifacts bucket (For retrieval from other repositories)
aws s3 cp $REPO_NAME-outputs.tfvars s3://$S3_TERRAFORM_ARTIFACTS_BUCKET_NAME/$REPO_NAME-outputs.tfvars
rm -f raw_output.json
rm -f $REPO_NAME-outputs.tfvars

Upload sample CSV files.

To upload sample CSV files for the datasets, use the following steps:

Set up AWS Supply Chain access.

To set up AWS Supply Chain access from the AWS Management Console, use the following steps:

Trigger the destroy workflow for integration flows resources.

Trigger the destroy workflow of ASC-Integration-Flows from your deployment branch in your GitHub organization.

Trigger the destroy workflow for datasets resources.

Trigger the destroy workflow of ASC-Datasets from your deployment branch in your GitHub organization.

Navigate to the terraform-deployment directory.

To go to the terraform-deployment directory of ASC-Integration-Flows, run the following command:

cd ASC-Integration-Flows/terraform-deployment

Set up the Terraform backend and providers configuration.

To set up the Terraform backend and providers configuration, use the following script:

# Setup terraform backend and providers config if they don't exist
chmod +x ../scripts/generate-terraform-config.sh
../scripts/generate-terraform-config.sh

Generate infrastructure destruction plan.

To prepare for the controlled destruction of your AWS infrastructure by generating a detailed teardown plan, run the following commands. The process initializes Terraform, incorporates AWS Supply Chain dataset configurations, and creates a destruction plan that you can review before executing.

# Run terraform init and validate
terraform init
terraform validate

# Download and merge ASC DATASET tfvars
chmod +x ../scripts/download-vars-through-s3.sh
../scripts/download-vars-through-s3.sh $ASC_DATASET_VARS_REPO

# Run terraform plan
terraform plan -destroy\
-var-file="tfInputs/$ENVIRONMENT.tfvars" \
-var="project_name=$PROJECT_NAME" \
-var="environment=$ENVIRONMENT" \
-var="lambda_temp_dir=$LAMBDA_FUNCTION_TEMP_DIR_TERRAFORM" \
-var="layer_temp_dir=$LAMBDA_LAYER_TEMP_DIR_TERRAFORM" \
-parallelism=40 \
-out='tfplan.out'

Execute infrastructure destruction plan.

To execute the planned destruction of your infrastructure, run the following command:

# Run terraform apply
terraform apply tfplan.out

Remove Terraform outputs from Amazon S3 bucket.

To remove the outputs file that was uploaded during the deployment of ASC-Integration-Flows, run the following command:

# Delete the outputs file
aws s3 rm s3://$S3_TERRAFORM_ARTIFACTS_BUCKET_NAME/$REPO_NAME-outputs.tfvars

Navigate to the terraform-deployment directory.

To go to the terraform-deployment directory of ASC-Datasets, run the following command:

cd ASC-Datasets/terraform-deployment

Set up the Terraform backend and providers configuration.

To set up the Terraform backend and providers configuration, use the following script:

# Setup terraform backend and providers config if they don't exist
chmod +x ../scripts/generate-terraform-config.sh
../scripts/generate-terraform-config.sh

Generate infrastructure destruction plan.

To create a plan for destroying AWS Supply Chain dataset resources, run the following commands:

# Run terraform init and validate
terraform init
terraform validate

# Run terraform plan
terraform plan -destroy\
-var-file="tfInputs/$ENVIRONMENT.tfvars" \
-var="project_name=$PROJECT_NAME" \
-var="environment=$ENVIRONMENT" \
-var="user_role=$AWS_USER_ROLE" \
-var="lambda_temp_dir=$LAMBDA_FUNCTION_TEMP_DIR_TERRAFORM" \
-var="layer_temp_dir=$LAMBDA_LAYER_TEMP_DIR_TERRAFORM" \
-parallelism=40 \
-out='tfplan.out'

Empty Amazon S3 buckets.

To empty all Amazon S3 buckets (except the server access logging bucket, which is configured for force-destroy), use the following script:

# Delete S3 buckets excluding server access logging bucket
chmod +x ../scripts/empty-s3-buckets.sh
../scripts/empty-s3-buckets.sh tfplan.out

Execute infrastructure destruction plan.

To execute the planned destruction of your AWS Supply Chain dataset infrastructure using the generated plan, run the following command:

# Run terraform apply
terraform apply tfplan.out

Remove Terraform outputs from the Amazon S3 Terraform artifacts bucket.

To complete the cleanup process, remove the outputs file that was uploaded during the deployment of ASC-Datasets by running the following command:

# Delete the outputs file
aws s3 rm s3://$S3_TERRAFORM_ARTIFACTS_BUCKET_NAME/$REPO_NAME-outputs.tfvars