Secure file transfers by using Transfer Family, Amazon Cognito, and GuardDuty - AWS Prescriptive Guidance
SummaryPrerequisites and limitationsArchitectureToolsBest practicesEpicsTroubleshootingRelated resourcesAdditional information

Secure file transfers by using Transfer Family, Amazon Cognito, and GuardDuty

Manoj Kumar, Amazon Web Services

Summary

This solution helps you securely transfer files through an SFTP server by using AWS Transfer Family. It includes automated malware scanning capabilities through Malware Protection for S3, a feature of Amazon GuardDuty. It is designed for organizations that need to securely exchange files with external parties and validate that all incoming files are scanned for malware before being processed.

The infrastructure as code (IaC) templates provided with this pattern help you deploy the following:

  • A secure SFTP server with Amazon Cognito authentication through AWS Lambda

  • Amazon Simple Storage Service (Amazon S3) buckets for uploads and incoming files that have been scanned for malware

  • A virtual private cloud (VPC)-based architecture with public and private subnets across multiple Availability Zones

  • IP-based access control for both ingress and egress traffic, with configurable allow and deny lists

  • Automated malware scanning through GuardDuty

  • Intelligent file routing based on scan results through Amazon EventBridge and Lambda

  • Real-time notifications for security incidents through Amazon Simple Notification Service (Amazon SNS)

  • Encryption for Amazon S3 buckets and Lambda environment variables through AWS Key Management Service (AWS KMS)

  • Amazon Virtual Private Cloud (Amazon VPC) endpoints for access without internet exposure

  • Comprehensive logging through Amazon CloudWatch integration

Prerequisites and limitations

Prerequisites 

  • An active AWS account

  • Permissions in AWS Identity and Access Management (IAM) to perform the actions described in this pattern, including deploying AWS CloudFormation templates that provision IAM roles

  • GuardDuty, enabled in the target account

  • Malware Protection for S3, enabled in the target account

  • Service quotas allow you to create the following in the target account:

    • One VPC

    • One private subnet

    • One public subnet

    • Three elastic IP addresses

    • Sufficient Lambda concurrency limits

  • A valid email address for security-related notifications

  • (Optional) A list of IP addresses or CIDR ranges that you want to allow or deny

  • (Optional) AWS Command Line Interface (AWS CLI), installed and configured

Limitations 

  • Malware Protection for S3 is subject to quotas, such as maximum file sizes. For more information, see Quotas in Malware Protection for S3 and Supportability of Amazon S3 features in the GuardDuty documentation.

  • This solution uses Amazon Cognito username and password authentication only. Certificate-based or other authentication methods are not supported in this template. By default, this solution does not configure multi-factor authentication (MFA).

  • The solution implements IP-based access control through security groups only.

Architecture

The following architecture diagram shows the resources that are deployed in this pattern. This solution uses Amazon Cognito for user authentication and authorization. An AWS Transfer Family SFTP server is used for file uploads. Files are stored in Amazon S3 buckets, and Amazon GuardDuty scans the files for malware. Amazon SNS sends an email notification if malware is detected.

Using GuardDuty and Cognito to securely transfer files to Amazon S3 buckets.

The diagram shows the following workflow:

  1. A user connects to the SFTP server endpoint in AWS Transfer Family. This initiates the authentication process with the Amazon Cognito user pool.

  2. A Lambda function initiates the authentication and authorization process and validates the user’s credentials with Amazon Cognito.

  3. The Lambda function returns the UploadBucket Amazon S3 bucket as the home directory. The user assumes the IAM role for the Transfer Family server, and the Lambda function notifies the user that they have been successfully authenticated.

  4. The user uploads a file to the Transfer Family SFTP server. The file is stored in the UploadBucket Amazon S3 bucket.

  5. GuardDuty scans the file for malware. The potential scan results are NO_THREATS_FOUND, THREATS_FOUND, UNSUPPORTED, ACCESS_DENIED, and FAILED. For sample results, see S3 object scan result in the GuardDuty documentation.

  6. An EventBridge rule detects the scan result event.

  7. EventBridge initiates the file-routing Lambda function.

  8. The Lambda function processes the event and filters the files based on the scan results as follows:

    • Files that have a NO_THREATS_FOUND scan result are sent to the CleanBucket Amazon S3 bucket.

    • Files that have a THREATS_FOUND scan result are sent to the MalwareBucket Amazon S3 bucket.

    • Files that have an UNSUPPORTED scan result are sent to the ErrorBucket Amazon S3 bucket.

    • Files that have an ACCESS_DENIED scan result are sent to the ErrorBucket Amazon S3 bucket.

    • Files that have a FAILED scan result are sent to the ErrorBucket Amazon S3 bucket.

    All files are encrypted with an AWS KMS key.

  9. If a file was sent to the MalwareBucket Amazon S3 bucket, the Lambda function initiates an Amazon SNS topic. The Amazon SNS topic sends an email notification to an email address that you configure.

Tools

AWS services

  • Amazon CloudWatch helps you monitor the metrics of your AWS resources and the applications you run on AWS in real time.

  • Amazon Cognito provides authentication, authorization, and user management for web and mobile apps.

  • Amazon EventBridge is a serverless event bus service that helps you connect your applications with real-time data from a variety of sources. For example, AWS Lambda functions, HTTP invocation endpoints using API destinations, or event buses in other AWS accounts.

  • Amazon GuardDuty is a continuous security monitoring service that analyzes and processes logs to identify unexpected and potentially unauthorized activity in your AWS environment.

  • AWS Key Management Service (AWS KMS) helps you create and control cryptographic keys to help protect your data.

  • AWS Lambda is a compute service that helps you run code without needing to provision or manage servers. It runs your code only when needed and scales automatically, so you pay only for the compute time that you use.

  • Amazon Simple Notification Service (Amazon SNS) helps you coordinate and manage the exchange of messages between publishers and clients, including web servers and email addresses.

  • Amazon Simple Storage Service (Amazon S3) is a cloud-based object storage service that helps you store, protect, and retrieve any amount of data.

  • AWS Transfer Family helps you transfer files into and out of AWS storage services over the SFTP, FTPS, or FTP protocols.

  • Amazon Virtual Private Cloud (Amazon VPC) helps you launch AWS resources into a virtual network that you’ve defined. This virtual network resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

Code repository

The code for this pattern is available in the GitHub AWS Transfer Family and GuardDuty Malware Scanning Solution repository.

Best practices

The CloudFormation template provided is designed to incorporate many AWS best practices, such as least-privilege permissions for IAM roles and policies, encryption at rest and in transit, and automatic key rotation. For production environments, consider implementing the following additional recommendations:

  • Enable MFA for Amazon Cognito users

  • Implement AWS Shield for distributed denial of service (DDoS) protection

  • Configure AWS Config for continuous compliance monitoring

  • Implement AWS CloudTrail for comprehensive API logging

  • Set up Amazon GuardDuty for threat detection beyond malware scanning

  • Implement AWS Security Hub for centralized security management

  • Use AWS Secrets Manager for credential management

  • Implement network traffic monitoring with Traffic Mirroring

  • Configure Amazon Macie for sensitive data discovery and protection in Amazon S3

  • Implement regular security assessments and penetration testing

  • Establish a formal incident response plan

  • Implement automated patching for all components

  • Conduct regular security training for administrators

  • Set up AWS Organizations for multi-account security management

Epics

TaskDescriptionSkills required

Clone the repository.

Enter the following command to clone the AWS Transfer Family and GuardDuty malware scanning solution repository to your local workstation:

git clone https://github.com/aws-samples/sample-secure-transfer-family-code.git
App developer, DevOps engineer

Create the CloudFormation stack.

  1. Follow the instructions in the CloudFormation documentation to create a stack through the CloudFormation console or through the AWS CLI. Configure the following parameters:

    • UploadBucketName - Upload bucket name

    • CleanBucketName - Clean files bucket name

    • MalwareBucketName - Malware files bucket name

    • ErrorBucketName - Error files bucket name

    • SecurityTeamEmail - Email address for alerts

    • VpcCIDR - CIDR range for VPC

    • PublicSubnet1CIDR - CIDR range for public subnet 1

    • PublicSubnet2CIDR - CIDR range for public subnet 2

    • PrivateSubnet1CIDR - CIDR range for private subnet 1

    • PrivateSubnet2CIDR - CIDR range for private subnet 2

    • AllowedIPAddresses - Allowed IP addresses or CIDR ranges

    • EnableEgressRules – If you want to enable egress rules for security groups, enter true. If you want disable the egress rules, enter false.

  2. Wait for the stack deployment to complete.

  3. In the stack creation outputs, make note of the SFTP server endpoint. You need this value later.

Cloud administrator, DevOps engineer
TaskDescriptionSkills required

Turn on malware protection.

  1. Sign in to the AWS Management Console and open the GuardDuty console.

  2. In the navigation pane, choose Malware Protection for S3.

  3. In the Protected buckets section, choose Enable.

  4. Under Enter S3 bucket details, enter the Amazon S3 bucket name. Alternatively, choose Browse S3 to select a bucket. Choose the UploadBucket Amazon S3 bucket.

  5. Under Prefix, select All the objects in the S3 bucket.

  6. Under Tag scanned objects, select Tag objects.

  7. In the Service access section, choose Create and use a new service role.

  8. Choose Enable.

Cloud administrator, AWS administrator

Add users to the user pool.

Add one or more users to the Amazon Cognito user pool. For instructions, see Managing users in your user pool in the Amazon Cognito documentation.

Cloud administrator, AWS administrator
TaskDescriptionSkills required

Connect to the SFTP server endpoint.

  1. Enter the following command to get the SFTP server endpoint, where <stack-name> is the name of your CloudFormation stack:

    TRANSFER_ENDPOINT=$(aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --query 'Stacks[0].Outputs[?OutputKey==`TransferServerEndpoint`].OutputValue' \
  --output text)

  2. Enter the following command to connect to the SFTP server:

    sftp user@company.com@$TRANSFER_ENDPOINT

  3. Validate that you receive the following prompt:

    sftp>

    This indicates that you can now enter commands to perform file transfers.

App developer, Cloud administrator, Cloud architect, DevOps engineer

Troubleshooting

IssueSolution

User authentication fails

  1. Verify that the user exists in the Amazon Cognito user pool.

  2. Validate that the user is confirmed and enabled.

  3. Make sure the user is providing the correct password.

  4. Check for errors in CloudWatch Logs.

For a list of AWS CLI commands that can help you perform these troubleshooting steps, see Useful commands for troubleshooting in the Additional information section.

SFTP authentication fails

  1. Verify that the user exists in the Amazon Cognito user pool.

  2. Validate that the user is confirmed and enabled.

  3. Make sure the user is providing the correct password.

  4. Verify that the IP address is allowed in the security group for the AWS Transfer Family server.

  5. Check for errors in the logs for the authentication Lambda function.

  6. Verify that the IAM role for AWS Transfer Family has correct permissions.

For a list of AWS CLI commands that can help you perform these troubleshooting steps, see Useful commands for troubleshooting in the Additional information section.

File upload access denied

  1. Verify that the AWS KMS key permissions are properly configured.

  2. Check the Amazon S3 bucket policies.

  3. Confirm that the IAM role permissions are properly configured.

  4. Verify that the VPC endpoints are properly configured

  5. Verify that the IAM role for AWS Transfer Family has correct permissions.

For a list of AWS CLI commands that can help you perform these troubleshooting steps, see Useful commands for troubleshooting in the Additional information section.

No malware scanning

  1. Enable GuardDuty in your account.

  2. Enable Malware Protection for S3.

  3. Verify that the Amazon EventBridge rule is active.

  4. Check for errors in the logs for the file-routing Lambda function.

  5. Verify that notifications are configured correctly in Amazon SNS.

For a list of AWS CLI commands that can help you perform these troubleshooting steps, see Useful commands for troubleshooting in the Additional information section.

Lambda function errors

  1. Check the VPC configuration.

  2. Verify that the VPC endpoints are properly configured.

  3. Verify that the IAM role for AWS Transfer Family has correct permissions.

  4. Check for errors in CloudWatch Logs.

  5. Verify that the AWS KMS key permissions are properly configured.

For a list of AWS CLI commands that can help you perform these troubleshooting steps, see Useful commands for troubleshooting in the Additional information section.

Related resources

Additional information

Useful commands for troubleshooting

Check the status of a CloudFormation stack:

aws cloudformation describe-stacks \
  --stack-name <STACK_NAME>

List all users in an Amazon Cognito user pool:

aws cognito-idp list-users \
  --user-pool-id <USER_POOL_ID>

View logs for Lambda functions:

aws logs describe-log-groups \
  --log-group-name-prefix /aws/lambda/

Check the status of GuardDuty:

aws guardduty list-detectors

Check security group rules:

aws ec2 describe-security-groups \
  --group-ids <SECURITY_GROUP_ID> \
  --output table

Check the status of the AWS Transfer Family server:

aws transfer describe-server \
  --server-id <SERVER_ID>

List all files in an Amazon S3 bucket:

aws s3 ls s3://<BUCKET_NAME>/ \
  --recursive

Check the status of an EventBridge rule:

aws events describe-rule \
  --name <RULE_NAME>