# Working with the AWS FIS scenario library Working with the scenario library Scenarios define events or conditions that customers can apply to test the resiliency of their applications, such as the interruption of compute resources on which the application is running. Scenarios are created and owned by AWS, and minimize undifferentiated heavy lifting by providing you with a group of pre-defined targets and fault actions (e.g., stopping 30% of instances in an autoscaling group) for common application impairments. Scenarios are provided through a console-only scenario library and run using an AWS FIS experiment template. In order to run an experiment using a scenario, you will select the scenario from the library, specify parameters matching your workload details, and save it as an experiment template in your account. **Topics** + [Viewing a scenario](#viewing-a-scenario) + [Using a scenario](#using-a-scenario) + [Exporting a scenario](#exporting-a-scenario) + [Scenarios reference](scenario-library-scenarios.md) ## Viewing a scenario Viewing a scenario To view a scenario using the console: 1. Open the AWS FIS console at [https://console.aws.amazon.com/fis/](https://console.aws.amazon.com/fis/). 1. In the navigation pane, choose **Scenario library**. 1. To view information about a specific scenario, select the scenario card to bring up a split panel. + In the **Description** tab in the split panel at the bottom of the page, you can view a short description of the scenario. You can also find a short summary of pre-requisites containing a summary of the target resources required and any actions you need to take to prepare the resources for use with the scenario. Finally you can also see additional information about the targets and actions in the scenario as well as the anticipated duration when the experiment runs successfully with default settings. + In the **Content** tab in the split panel at the bottom of the page, you can preview a partially populated version of the experiment template that will be created from the scenario. + In the **Details** tab in the split panel at the bottom of the page, you can find a detailed explanation how the scenario is implemented. This may contain detailed information about how individual aspects of the scenario are approximated. Where applicable you can also read about what metrics to use as stop conditions and to provide observability to learn from the experiment. Finally you will find recommendations how to expand the resulting experiment template. ## Using a scenario Using a scenario To use a scenario using the console: 1. Open the AWS FIS console at [https://console.aws.amazon.com/fis/](https://console.aws.amazon.com/fis/). 1. In the navigation pane, choose **Scenario library**. 1. To view information about a specific scenario, select the scenario card to bring up a split panel 1. To use the scenario, select the scenario card and choose **Create template with scenario**. 1. In the **Create experiment template** view fill in any missing items. 1. Some scenarios allow you to edit parameters that are shared across multiple actions or targets. This functionality will be disabled once you make any changes to the scenario, including changes by the shared parameter editing. To use this feature select the **Edit shared parameters** button. Edit parameters in the modal and select the **Save** button. 1. Some experiment templates may have missing action or target parameters, highlighted on each action and target card. Select the ** Edit** button for each card, add the missing information, and select the **Save** button on the card. 1. All templates require a **Service access** execution role. You can choose an existing role or create a new role for this experiment template. 1. We recommend defining one or more optional **Stop conditions** by selecting an existing AWS CloudWatch alarm. Learn more about [Stop conditions for AWS FIS](stop-conditions.md). If you don't have an alarm configured yet, you can follow the instructions at [Using Amazon CloudWatch Alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) and update the experiment template later. 1. We recommend enabling optional experiment **Logs** to Amazon CloudWatch logs or to an Amazon S3 bucket. Learn more about [Experiment logging for AWS FIS](monitoring-logging.md). If you don't have appropriate resources configured yet, you can update the experiment template later. 1. In the **Create experiment template** select **Create experiment template**. 1. From the **Experiment templates** view of the AWS FIS console select **Start experiment**. Learn more about [Managing AWS FIS experiment templatesManaging your AWS FIS experiments](experiments.md). ## Exporting a scenario Exporting a scenario Scenarios are a console-only experience. While similar to experiment templates, scenarios are not complete experiment templates and can not be directly imported into AWS FIS. If you wish to use scenarios as part of your own automation, you can use one of two paths: 1. Follow the steps in [Using a scenario](#using-a-scenario) to create a valid AWS FIS experiment template and export that template. 1. Follow the steps in [Viewing a scenario](#viewing-a-scenario) and in step 3, from the **Content** tab, copy and save the scenario content, then add missing parameters manually to create a valid experiment template. # Scenarios reference Scenarios reference Scenarios included in the scenario library are designed to use [tags](https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/what-are-tags.html) where possible and each scenario describes the required tags in the **Prerequisites** and **How it works** sections of the scenario description. You can tag your resources with those pre-defined tags or you can set your own tags using the shared parameter editing experience (see [Using a scenario](scenario-library.md#using-a-scenario)). This reference describes the common scenarios in the AWS FIS scenario library. You can also list the supported scenarios using the AWS FIS console. For more information, see [Working with the AWS FIS scenario library](scenario-library.md). AWS FIS supports the following Amazon EC2 scenarios. These scenarios target instances using [tags](https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/what-are-tags.html). You can use your own tags or use the default tags included in the scenario. Some of these scenarios [use SSM documents](https://docs.aws.amazon.com//fis/latest/userguide/actions-ssm-agent.html). + **EC2 stress: instance failure** - Explore the effect of instance failure by stopping one or more EC2 instances. Target instances in the current region that have a specific tag attached. In this scenario we will stop those instances and restart them at the end of the action duration, by default 5 min. + **EC2 stress: Disk** - Explore impact of increased disk utilization on your EC2 based application. In this scenario we will target EC2 instances in the current region that have a specific tag attached. In this scenario you can customize an increasing amount disk utilization injected on targeted EC2 instances for the action duration, by default 5 min for each disk stress action. + **EC2 stress: CPU** - Explore impact of increased CPU on your EC2 based application. In this scenario we will target EC2 instances in the current region that have a specific tag attached. In this scenario you can customize an increasing amount of CPU stress injected on targeted EC2 instances for the action duration, by default 5 min for each CPU stress action. + **EC2 stress: Memory** - Explore impact of increased memory utilization on your EC2 based application. In this scenario we will target EC2 instances in the current region that have a specific tag attached. In this scenario you can customize an increasing amount of memory stress injected on targeted EC2 instances for the action duration, by default 5 min for each memory stress action. + **EC2 stress: Network Latency** - Explore impact of increased network latency on your EC2 based application. In this scenario we will target EC2 instances in the current region that have a specific tag attached. In this scenario you can customize an increasing amount of network latency injected on targeted EC2 instances for the action duration, by default 5 min for each latency action. AWS FIS supports the following Amazon EKS scenarios. These scenarios target EKS pods using a Kubernetes application labels. You can use your own labels or use the default labels included in the scenario. For more information about EKS with FIS, see [EKS Pod actions](eks-pod-actions.md). + **EKS stress: Pod Delete** - Explore the effect of EKS pod failure by deleting one or more pods. In this scenario we will target pods in the current region that are associated with an application label. In this scenario we will terminate all matched pods. Re-creation of pods will be controlled by kubernetes configuration. + **EKS stress: CPU** - Explore impact of increased CPU on your EKS based application. In this scenario we will target pods in the current region that are associated with an application label. In this scenario you can customize an increasing amount of CPU stress injected on targeted EKS pods for the action duration, by default 5 min for each CPU stress action. + **EKS stress: Disk** - Explore impact of increased disk utilization on your EKS based application. In this scenario we will target pods in the current region that are associated with an application label. In this scenario you can customize an increasing amount of disk stress injected on targeted EKS pods for the action duration, by default 5 min for each CPU stress action. + **EKS stress: Memory** - Explore impact of increased memory utilization on your EKS based application. In this scenario we will target pods in the current region that are associated with an application label. In this scenario you can customize an increasing amount of memory stress injected on targeted EKS pods for the action duration, by default 5 min for each memory stress action. + **EKS stress: Network latency** - Explore impact of increased network latency on your EKS based application. In this scenario we will target pods in the current region that are associated with an application label. In this scenario you can customize an increasing amount of network latency injected on targeted EKS pods for the action duration, by default 5 min for each latency action. AWS FIS supports the following scenarios for single-AZ, multi-AZ and multi-Region applications. These scenarios target multiple resource types. + **AZ Availability: Power Interruption** - Inject the expected symptoms of a complete interruption of power in an Availability Zone (AZ). Learn more about [AZ Availability: Power Interruption](az-availability-scenario.md). + **AZ: Application Slowdown** - Add latency between resources within a single Availability Zone (AZ) to slow down an application. Learn more about [AZ: Application Slowdown](az-application-slowdown-scenario.md). + **Cross-AZ: Traffic Slowdown** - Inject packet loss to disrupt and slow down traffic between Availability Zones (AZs). Learn more about [Cross-AZ: Traffic Slowdown](cross-az-traffic-slowdown-scenario.md). + **Cross-Region: Connectivity** - Block application network traffic from the experiment Region to the destination Region and pause cross-Region data replication. Learn more about using [Cross-Region: Connectivity](cross-region-scenario.md). AWS FIS supports the following scenarios for Amazon EBS volumes. These scenarios target volumes using tags. You can use your own tags or use the default tags included in the scenario. The target volumes must be in the same Availability Zone. For more information, [Fault testing on Amazon EBS](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-fis.html). + **EBS: Sustained Latency** — Explore impact of persistent I/O latency on your application. In this scenario, we will target volumes in the current Availability Zone that have a specific tag attached. This scenario injects constant latency of 500 ms on 50 percent of read and 100 percent of write operations for a volume, using a single latency action over a 15-minute period. In this scenario, you can customize the amount of latency injected, the percentage of I/O injected, and the duration for the action. + **EBS: Increasing Latency** — Explore impact of increasing I/O latency on your application. In this scenario, we will target volumes in the current Availability Zone that have a specific tag attached. This scenario injects increasing latency of 50 ms, 200 ms, 700 ms, 1 second, and 15 seconds on 10 percent of read and 25 percent of write operations for a volume using five latency actions over a 15-minute period. In this scenario, you can customize the amount of latency injected, the percentage of I/O injected, and the action duration, for each latency action. + **EBS: Intermittent Latency** — Explore impact of intermittent I/O latency spikes on your application. In this scenario, we will target volumes in the current Availability Zone that have a specific tag attached. This scenario injects three sharp intermittent latency spikes of 30 seconds, 10 seconds, and 20 seconds on 0.1 percent of the read and write I/O operations for a volume, using three latency actions, with intervals of recovery in between each spike over a 15-minute period. In this scenario, you can customize the amount of latency injected, the percentage of I/O injected, and the action duration, for each latency action. + **EBS: Decreasing Latency** — Explore impact of decreasing I/O latency on your application. In this scenario, we will target volumes in the current Availability Zone that have a specific tag attached. This scenario injects decreasing latency of 20 seconds, 5 seconds, 900 ms, 300 ms, and 40 ms on 10 percent of read and write operations for a volume, using five latency actions over a 15-minute period. In this scenario, you can customize the amount of latency injected, the percentage of I/O injected, and the action duration, for each latency action. # AZ Availability: Power Interruption AZ Availability: Power Interruption You can use the AZ Availability: Power Interruption scenario to induce the expected symptoms of a complete interruption of power in an Availability Zone (AZ). This scenario can be used to demonstrate that multi-AZ applications operate as expected during a single, complete AZ power interruption. It includes loss of zonal compute (Amazon EC2, EKS, and ECS), no re-scaling of compute in the AZ, subnet connectivity loss, RDS failover, ElastiCache failover, impaired access to S3 Express One Zone directory buckets, and unresponsive EBS volumes. By default, actions for which no targets are found will be skipped. ## Actions Actions Together, the following actions create many of the expected symptoms of a complete power interruption in a single AZ. AZ Availability: Power Interruption only affects services that are expected to see impact during a single AZ power interruption. By default, the scenario injects power interruption symptoms for 30 minutes and then, for an additional 30 minutes, injects symptoms that may occur during recovery. ### Stop-Instances Stop-Instances During an AZ power interruption, EC2 instances in the affected AZ will shut down. After power is restored instances will reboot. AZ Availability: Power Interruption includes [aws:ec2:stop-instances](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#stop-instances) to stop all instances in the affected AZ for the interruption duration. After the duration, the instances are restarted. Stopping EC2 instances managed by Amazon EKS causes dependent EKS pods to be deleted. Stopping EC2 instances managed by Amazon ECS causes dependent ECS tasks to be stopped. This action targets EC2 instances running in the affected AZ. By default, it targets instances with a tag named `AzImpairmentPower` with a value of `StopInstances`. You can add this tag to your instances or replace the default tag with your own tag in the experiment template. By default, if no valid instances are found this action will be skipped. ### Stop-ASG-Instances Stop-ASG-Instances During an AZ power interruption, EC2 instances managed by an Auto Scaling group in the affected AZ will shut down. After power is restored instances will reboot. AZ Availability: Power Interruption includes [aws:ec2:stop-instances](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#stop-instances) to stop all instances, including those managed by Auto Scaling, in the affected AZ for the interruption duration. After the duration, the instances are restarted. This action targets EC2 instances running in the affected AZ. By default, it targets instances with a tag named `AzImpairmentPower` with a value of `IceAsg`. You can add this tag to your instances or replace the default tag with your own tag in the experiment template. By default, if no valid instances are found this action will be skipped. ### Pause Instance Launches Pause Instance Launches During an AZ power interruption, EC2 API calls to provision capacity in the AZ will fail. In particular, the following APIs will be impacted: `ec2:StartInstances`, `ec2:CreateFleet`, and `ec2:RunInstances`. AZ Availability: Power Interruption includes includes [aws:ec2:api-insufficient-instance-capacity-error](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#api-ice) to prevent new instances from being provisioned in the affected AZ. This action targets IAM roles used to provision instances. These must be targeted using an ARN. By default, if no valid IAM roles are found this action will be skipped. ### Pause ASG Scaling Pause ASG Scaling During an AZ power interruption, EC2 API calls made by the Auto Scaling control plane to recover lost capacity in the AZ will fail. In particular, the following APIs will be impacted: `ec2:StartInstances`, `ec2:CreateFleet`, and `ec2:RunInstances`. AZ Availability: Power Interruption includes [aws:ec2:asg-insufficient-instance-capacity-error](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#asg-ice) to prevent new instances from being provisioned in the affected AZ. This also prevents Amazon EKS and Amazon ECS from scaling in the affected AZ. This action targets Auto Scaling groups. By default, it targets Auto Scaling groups with a tag named `AzImpairmentPower` with a value of `IceAsg`. You can add this tag to your Auto Scaling groups or replace the default tag with your own tag in the experiment template. By default, if no valid Auto Scaling groups are found this action will be skipped. ### Pause Network Connectivity Pause Network Connectivity During an AZ power interruption, networking in the AZ will be unavailable. When this happens some AWS services may take up to a few minutes to update DNS to reflect that private endpoints in the affected AZ are not available. During this time, DNS lookups may return inaccessible IP addresses. AZ Availability: Power Interruption includes [aws:network:disrupt-connectivity](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#network-actions-reference) to block all network connectivity for all subnets in the affected AZ for 2 minutes. This will force timeouts and DNS refreshes for most applications. Ending the action after 2 minutes allows for subsequent recovery of regional service DNS while the AZ continues to be unavailable. This action targets subnets. By default, it targets clusters with a tag named `AzImpairmentPower` with a value of `DisruptSubnet`. You can add this tag to your subnets or replace the default tag with your own tag in the experiment template. By default, if no valid subnets are found this action will be skipped. ### Failover RDS Failover RDS During an AZ power interruption, RDS nodes in the affected AZ will shut down. Single AZ RDS nodes in the affected AZ will be fully unavailable. For multi-AZ clusters, the writer node will failover into an unaffected AZ and reader nodes in the affected AZ will be unavailable. For multi-AZ clusters, AZ Availability: Power Interruption includes [aws:rds:failover-db-cluster](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#failover-db-cluster) to failover if the writer is in the affected AZ. This action targets RDS clusters. By default, it targets clusters with a tag named `AzImpairmentPower` with a value of `DisruptRds`. You can add this tag to your clusters or replace the default tag with your own tag in the experiment template. By default, if no valid clusters are found this action will be skipped. ### Pause ElastiCache Replication Group Pause ElastiCache Replication Group During an AZ power interruption, ElastiCache nodes in the AZ are unavailable. AZ Availability: Power Interruption includes [aws:elasticache:replicationgroup-interrupt-az-power](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#interrupt-elasticache) to terminate ElastiCache nodes in the affected AZ. For the duration of the interruption, new instances will not be provisioned in the affected AZ, so the replication group will remain at reduced capacity. This action targets ElastiCache replication groups. By default, it targets replication groups with a tag named `AzImpairmentPower` with a value of `ElasticacheImpact`. You can add this tag to your replication groups or replace the default tag with your own tag in the experiment template. By default, if no valid replication groups are found this action will be skipped. Note that only replication groups with nodes in the affected AZ will be considered valid targets. ### Start ARC Zonal Autoshift Start ARC Zonal Autoshift Five minutes after the AZ power interruption begins, the recovery action `aws:arc:start-zonal-autoshift` automatically shifts resource traffic away from the specified AZ for the remaining 25 minutes of the power interruption. After that duration, traffic shifts back to the original AZ. Note that during a real-world AZ power interruption AWS will detect the impairment and shift resource traffic if autoshift is enabled. While the timing of this shift varies it is estimated to occur five minutes from the impairment commencing. This action targets Amazon Application Recovery Controller (ARC) autoshift-enabled resources. By default, it targets resources with the tag key `AzImpairmentPower` and value `RecoverAutoshiftResources`. You can add this tag to your resources or replace the default tag with your own tag in the experiment template. For example, you may want to use an application-specific tag. By default, if no valid resources are found this action will be skipped. ### Pause EBS I/O Pause EBS I/O After an AZ power interruption, once power is restored a very small percentage of instances may experience unresponsive EBS volumes. AZ Availability: Power Interruption includes [aws:ebs:pause-io](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#ebs-actions-reference) to leave 1 EBS volume in an unresponsive state. By default, only volumes set to persist after the instance is terminated are targeted. This action targets volumes with a tag named `AzImpairmentPower` with a value of `APIPauseVolume`. You can add this tag to your volumes or replace the default tag with your own tag in the experiment template. By default, if no valid volumes are found this action will be skipped. ### Disrupt connectivity to S3 Express One Zone directory buckets Disrupt connectivity to S3 Express One Zone directory buckets During an AZ power interruption, data stored in S3 Express One Zone directory buckets in the AZ is not accessible. AZ Availability: Power Interruption includes [aws:network:disrupt-connectivity](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#network-actions-reference) to disrupt connectivity between subnets and One Zone directory buckets in the affected AZ for the duration of the experiment, resulting in timeouts to Zonal endpoint data plane API operations. Use this action to test disruption when compute is co-located with storage in an AZ. This action targets subnets. By default, it targets subnets with a tag named `AzImpairmentPower` with a value of `DisruptSubnet`. You can add this tag to your subnets or replace the default tag with your own tag in the experiment template. By default, if no valid subnets are found this action will be skipped. ## Limitations Limitations + This scenario does not include [stop conditions](https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html). The correct stop conditions for your application should be added to the experiment template. + In the targeted AZ, Amazon EKS Pods running on EC2 will be terminated with EC2 worker nodes and starting of new EC2 nodes will be blocked. However, Amazon EKS Pods running on AWS Fargate are not supported. + In the targeted AZ, Amazon ECS tasks running on EC2 will be terminated with EC2 worker nodes and starting of new EC2 nodes will be blocked. However, Amazon ECS tasks running on AWS Fargate are not supported. + [Amazon RDS Multi-AZ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html#multi-az-db-clusters-migrating-to-with-read-replica) with two readable standby DB instances is not supported. In this case, the instances will be terminated, RDS will failover, and capacity will immediately be provisioned back in the affected AZ. The readable standby in the affected AZ will remain available. ## Requirements Requirements + Add the required permission to the AWS FIS [experiment role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html). + Resource tags must be applied to resources that are to be targeted by the experiment. These can use your own tagging convention or the default tags defined in the scenario. ## Permissions Permissions ARC zonal autoshift uses an IAM service-linked role `AWSServiceRoleForZonalAutoshiftPracticeRun` to perform zonal shift on your behalf. This role uses the IAM managed policy [https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSZonalAutoshiftPracticeRunSLRPolicy.html](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AWSZonalAutoshiftPracticeRunSLRPolicy.html). You don’t need to create the role manually. When you create an experiment template from the AZ Power Interruption scenario in the AWS Management Console, the AWS CLI, or an AWS SDK, ARC creates the service-linked role for you. For more information, see [Using the service-linked role for zonal autoshift in ARC](https://docs.aws.amazon.com/r53recovery/latest/dg/using-service-linked-roles-zonal-autoshift.html). The following policy grants AWS FIS the necessary permissions to execute an experiment with the AZ Availability: Power Interruption scenario. This policy must be attached to the [experiment role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Sid": "AllowFISExperimentLoggingActionsCloudwatch", "Effect": "Allow", "Action": [ "logs:CreateLogDelivery", "logs:PutResourcePolicy", "logs:DescribeResourcePolicies", "logs:DescribeLogGroups" ], "Resource": "*" }, { "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": "arn:aws:ec2:*:*:network-acl/*", "Condition": { "StringEquals": { "ec2:CreateAction": "CreateNetworkAcl", "aws:RequestTag/managedByFIS": "true" } } }, { "Effect": "Allow", "Action": "ec2:CreateNetworkAcl", "Resource": "arn:aws:ec2:*:*:network-acl/*", "Condition": { "StringEquals": { "aws:RequestTag/managedByFIS": "true" } } }, { "Effect": "Allow", "Action": [ "ec2:CreateNetworkAclEntry", "ec2:DeleteNetworkAcl" ], "Resource": [ "arn:aws:ec2:*:*:network-acl/*", "arn:aws:ec2:*:*:vpc/*" ], "Condition": { "StringEquals": { "ec2:ResourceTag/managedByFIS": "true" } } }, { "Effect": "Allow", "Action": "ec2:CreateNetworkAcl", "Resource": "arn:aws:ec2:*:*:vpc/*" }, { "Effect": "Allow", "Action": [ "ec2:DescribeVpcs", "ec2:DescribeManagedPrefixLists", "ec2:DescribeSubnets", "ec2:DescribeNetworkAcls" ], "Resource": "*" }, { "Effect": "Allow", "Action": "ec2:ReplaceNetworkAclAssociation", "Resource": [ "arn:aws:ec2:*:*:subnet/*", "arn:aws:ec2:*:*:network-acl/*" ] }, { "Effect": "Allow", "Action": [ "rds:FailoverDBCluster" ], "Resource": [ "arn:aws:rds:*:*:cluster:*" ] }, { "Effect": "Allow", "Action": [ "rds:RebootDBInstance" ], "Resource": [ "arn:aws:rds:*:*:db:*" ] }, { "Effect": "Allow", "Action": [ "elasticache:DescribeReplicationGroups", "elasticache:InterruptClusterAzPower" ], "Resource": [ "arn:aws:elasticache:*:*:replicationgroup:*" ] }, { "Sid": "TargetResolutionByTags", "Effect": "Allow", "Action": [ "tag:GetResources" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ec2:StartInstances", "ec2:StopInstances" ], "Resource": "arn:aws:ec2:*:*:instance/*" }, { "Effect": "Allow", "Action": [ "ec2:DescribeInstances" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "kms:CreateGrant" ], "Resource": [ "arn:aws:kms:*:*:key/*" ], "Condition": { "StringLike": { "kms:ViaService": "ec2.*.amazonaws.com" }, "Bool": { "kms:GrantIsForAWSResource": "true" } } }, { "Effect": "Allow", "Action": [ "ec2:DescribeVolumes" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ec2:PauseVolumeIO" ], "Resource": "arn:aws:ec2:*:*:volume/*" }, { "Sid": "AllowInjectAPI", "Effect": "Allow", "Action": [ "ec2:InjectApiError" ], "Resource": [ "*" ], "Condition": { "ForAnyValue:StringEquals": { "ec2:FisActionId": [ "aws:ec2:api-insufficient-instance-capacity-error", "aws:ec2:asg-insufficient-instance-capacity-error" ] } } }, { "Sid": "DescribeAsg", "Effect": "Allow", "Action": [ "autoscaling:DescribeAutoScalingGroups" ], "Resource": [ "*" ] } ] } ``` ------ ## Scenario Content Scenario Content The following content defines the scenario. This JSON can be saved and used to create an [experiment template](https://docs.aws.amazon.com/fis/latest/userguide/experiment-templates.html) using the [create-experiment-template](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/create-experiment-template.html) command from the AWS Command Line Interface (AWS CLI). For the most recent version of the scenario, visit the scenario library in the FIS console. ``` { "targets": { "IAM-role": { "resourceType": "aws:iam:role", "resourceArns": [], "selectionMode": "ALL" }, "EBS-Volumes": { "resourceType": "aws:ec2:ebs-volume", "resourceTags": { "AzImpairmentPower": "ApiPauseVolume" }, "selectionMode": "COUNT(1)", "parameters": { "availabilityZoneIdentifier": "us-east-1a" }, "filters": [ { "path": "Attachments.DeleteOnTermination", "values": [ "false" ] } ] }, "EC2-Instances": { "resourceType": "aws:ec2:instance", "resourceTags": { "AzImpairmentPower": "StopInstances" }, "filters": [ { "path": "State.Name", "values": [ "running" ] }, { "path": "Placement.AvailabilityZone", "values": [ "us-east-1a" ] } ], "selectionMode": "ALL" }, "ASG": { "resourceType": "aws:ec2:autoscaling-group", "resourceTags": { "AzImpairmentPower": "IceAsg" }, "selectionMode": "ALL" }, "ASG-EC2-Instances": { "resourceType": "aws:ec2:instance", "resourceTags": { "AzImpairmentPower": "IceAsg" }, "filters": [ { "path": "State.Name", "values": [ "running" ] }, { "path": "Placement.AvailabilityZone", "values": [ "us-east-1a" ] } ], "selectionMode": "ALL" }, "Subnet": { "resourceType": "aws:ec2:subnet", "resourceTags": { "AzImpairmentPower": "DisruptSubnet" }, "filters": [ { "path": "AvailabilityZone", "values": [ "us-east-1a" ] } ], "selectionMode": "ALL", "parameters": {} }, "RDS-Cluster": { "resourceType": "aws:rds:cluster", "resourceTags": { "AzImpairmentPower": "DisruptRds" }, "selectionMode": "ALL", "parameters": { "writerAvailabilityZoneIdentifiers": "us-east-1a" } }, "ElastiCache-Cluster": { "resourceType": "aws:elasticache:replicationgroup", "resourceTags": { "AzImpairmentPower": "DisruptElasticache" }, "selectionMode": "ALL", "parameters": { "availabilityZoneIdentifier": "us-east-1a" } } }, "actions": { "Pause-Instance-Launches": { "actionId": "aws:ec2:api-insufficient-instance-capacity-error", "parameters": { "availabilityZoneIdentifiers": "us-east-1a", "duration": "PT30M", "percentage": "100" }, "targets": { "Roles": "IAM-role" } }, "Pause-EBS-IO": { "actionId": "aws:ebs:pause-volume-io", "parameters": { "duration": "PT30M" }, "targets": { "Volumes": "EBS-Volumes" }, "startAfter": [ "Stop-Instances", "Stop-ASG-Instances" ] }, "Stop-Instances": { "actionId": "aws:ec2:stop-instances", "parameters": { "completeIfInstancesTerminated": "true", "startInstancesAfterDuration": "PT30M" }, "targets": { "Instances": "EC2-Instances" } }, "Pause-ASG-Scaling": { "actionId": "aws:ec2:asg-insufficient-instance-capacity-error", "parameters": { "availabilityZoneIdentifiers": "us-east-1a", "duration": "PT30M", "percentage": "100" }, "targets": { "AutoScalingGroups": "ASG" } }, "Stop-ASG-Instances": { "actionId": "aws:ec2:stop-instances", "parameters": { "completeIfInstancesTerminated": "true", "startInstancesAfterDuration": "PT30M" }, "targets": { "Instances": "ASG-EC2-Instances" } }, "Pause-network-connectivity": { "actionId": "aws:network:disrupt-connectivity", "parameters": { "duration": "PT2M", "scope": "all" }, "targets": { "Subnets": "Subnet" } }, "Failover-RDS": { "actionId": "aws:rds:failover-db-cluster", "parameters": {}, "targets": { "Clusters": "RDS-Cluster" } }, "Pause-ElastiCache": { "actionId": "aws:elasticache:replicationgroup-interrupt-az-power", "parameters": { "duration": "PT30M" }, "targets": { "ReplicationGroups": "ElastiCache-Cluster" } } }, "stopConditions": [ { "source": "aws:cloudwatch:alarm", "value": "" } ], "roleArn": "", "tags": { "Name": "AZ Impairment: Power Interruption" }, "logConfiguration": { "logSchemaVersion": 2 }, "experimentOptions": { "accountTargeting": "single-account", "emptyTargetResolutionMode": "skip" }, "description": "Affect multiple resource types in a single AZ, targeting by tags and explicit ARNs, to approximate power interruption in one AZ." } ``` # AZ: Application Slowdown AZ: Application Slowdown You can use the AZ: Application Slowdown scenario to introduce additional latency between resources within a single Availability Zone (AZ). This latency creates many of the symptoms of an application slowdown, a partial disruption, sometimes known as a gray failure. It adds latency to network flows between target resources. Network flows represent the traffic between computing resources — the data packets carrying requests, responses, and other communications between your servers, containers, and services. The scenario can help to validate observability setups, tune alarm thresholds, discover application sensitivity to slowdowns, and practice critical operational decisions like AZ evacuation. By default, the scenario adds 200ms of latency to 100% of network flows between target resources within the selected AZ for a duration of 30 minutes. You can use the **Edit shared parameters** dialog in the AWS FIS console to adjust the following parameters at the scenario level, which then apply to the underlying actions: + Availability Zone - you can select the AZ to impair in the scenario. + Milliseconds (ms) of latency - adjust this based on your application’s sensitivity and needs. You can set latency lower for more sensitive applications or higher to test timeout handling, for example. Consider using multiples of your current application latency as a baseline. + Flows percentage - reduce to impair a subset of traffic. For example, you can add 200ms latency affecting 25% of the network flows for even more subtle testing. + Duration - set how long the experiment runs. You can shorten for quicker tests, or run longer sustained tests. For example, set the duration to 2 hours to test your recovery mechanisms under impaired conditions. + Resource targeting - you can define target resources for the overall scenario using tags (for EC2 instances or ECS tasks on EC2 or Fargate) or labels (for EKS pods on EC2). You can specify your own tags and labels, or use the defaults provided in the scenario. If you don’t wish to use tags or labels, you can edit the action to target resources by specifying other parameters. + Customization - If you don’t want to target EC2 or ECS resources, you can leave the actions with default tags. The experiment won’t find any resources to target and the action will be skipped. However, if you don’t want to target EKS resources, you should remove the EKS action and target from the scenario completely, as it requires an EKS cluster identifier to be provided. For even more granular customization, you can modify individual actions in the experiment template directly. ## Actions Actions Together, the following actions help create many of the symptoms of an application slowdown in a single AZ by introducing additional latency on the network flows, which then propagates through the application. These actions run in parallel, each adding 200ms latency for 30 minutes by default. After this period, latency returns to normal levels. The scenario needs at least one of the following resource types to run: EC2 instance, ECS task, or EKS pod. ### ECS Network Latency ECS Network Latency AZ: Application Slowdown includes [aws:ecs:task-network-latency](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#task-network-latency) to introduce latency for ECS tasks. The action targets tasks in the selected AZ. By default, it targets tasks with a [tag](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html) named `AZApplicationSlowdown` with a value of `LatencyForECS`. You can replace the default tag with your own, or add the scenario tag to your tasks. If no valid tasks are found this action will be skipped. Before running an experiment on ECS, you should follow the [setup steps for ECS task actions](https://docs.aws.amazon.com/fis/latest/userguide/ecs-task-actions.html). ### EKS Network Latency EKS Network Latency AZ: Application Slowdown includes [aws:eks:pod-network-latency](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#pod-network-latency) to introduce latency for EKS pods. The action targets pods in the selected AZ. By default, it targets pods within a cluster that have labels with the format key=value. The default label provided is `AZApplicationSlowdown=LatencyForEKS`. You can replace the default label with your own, or add this label to your pods. If no valid pods are found this action will be skipped. Before running an experiment on EKS, you should follow the [setup steps for EKS pod actions](https://docs.aws.amazon.com/fis/latest/userguide/eks-pod-actions.html). ### EC2 Network Latency EC2 Network Latency AZ: Application Slowdown uses the [aws:ssm:send-command](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#ssm-send-command) action to run the [AWSFIS-Run-Network-Latency-Sources](https://docs.aws.amazon.com/fis/latest/userguide/actions-ssm-agent.html#awsfis-run-network-latency-sources) document to introduce latency for EC2 instances. The action targets instances in the selected AZ. By default, it targets instances with a [tag](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) named `AZApplicationSlowdown` with a value of `LatencyForEC2`. You can replace the default tag with your own, or add this tag to your instances. If no valid instances are found this action will be skipped. Before running an experiment on EC2 using SSM, you should [configure the AWS Systems Manager agent](https://docs.aws.amazon.com/fis/latest/userguide/actions-ssm-agent.html). ## Limitations Limitations + This scenario does not include [stop conditions](https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html). The correct stop conditions for your application should be added to the experiment template. ## Requirements Requirements + Add the required permissions to the AWS FIS [experiment role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html). + You need to target one or more resources from any of the following 3 types within the selected AZ: EC2 instances, ECS tasks, or EKS pods. + All targets of the scenario must be in the same VPC. ## Permissions Permissions To run this scenario you need an IAM role with a trust policy that allows FIS to assume the role and the managed policies for the resource types you target in the experiment: EC2, ECS, and EKS. When you create an experiment template from the AZ: Application Slowdown scenario, FIS creates the role for you with the trust policy and the following AWS managed policies: + [AWSFaultInjectionSimulatorEC2Access](https://docs.aws.amazon.com/fis/latest/userguide/security-iam-awsmanpol.html#AWSFaultInjectionSimulatorEC2Access) + [AWSFaultInjectionSimulatorECSAccess](https://docs.aws.amazon.com/fis/latest/userguide/security-iam-awsmanpol.html#AWSFaultInjectionSimulatorECSAccess) + [AWSFaultInjectionSimulatorEKSAccess](https://docs.aws.amazon.com/fis/latest/userguide/security-iam-awsmanpol.html#AWSFaultInjectionSimulatorEKSAccess) If you’re using an existing [IAM role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html) to run the AZ: Application Slowdown scenario, you can attach the following policy to grant AWS FIS the necessary permissions: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "DescribeTasks", "Effect": "Allow", "Action": "ecs:DescribeTasks", "Resource": "*" }, { "Sid": "DescribeContainerInstances", "Effect": "Allow", "Action": "ecs:DescribeContainerInstances", "Resource": "arn:aws:ecs:*:*:container-instance/*/*" }, { "Sid": "DescribeInstances", "Effect": "Allow", "Action": "ec2:DescribeInstances", "Resource": "*" }, { "Sid": "DescribeSubnets", "Effect": "Allow", "Action": "ec2:DescribeSubnets", "Resource": "*" }, { "Sid": "DescribeCluster", "Effect": "Allow", "Action": "eks:DescribeCluster", "Resource": "arn:aws:eks:*:*:cluster/*" }, { "Sid": "TargetResolutionByTags", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" }, { "Sid": "SendCommand", "Effect": "Allow", "Action": [ "ssm:SendCommand" ], "Resource": [ "arn:aws:ec2:*:*:instance/*", "arn:aws:ssm:*:*:managed-instance/*", "arn:aws:ssm:*:*:document/*" ] }, { "Sid": "ListCommands", "Effect": "Allow", "Action": [ "ssm:ListCommands" ], "Resource": "*" }, { "Sid": "CancelCommand", "Effect": "Allow", "Action": [ "ssm:CancelCommand" ], "Resource": "*" } ] } ``` ## Scenario Content Scenario Content The following content defines the scenario. This JSON can be saved and used to create an [experiment template](https://docs.aws.amazon.com/fis/latest/userguide/experiment-templates.html) using the [create-experiment-template](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/create-experiment-template.html) command from the AWS Command Line Interface (AWS CLI). For the most recent version of the scenario, visit the scenario library in the FIS console and go to the **Content** tab. ``` { "tags": { "Name": "AZ: Application Slowdown" }, "description": "Add latency between resources within a single AZ.", "actions": { "LatencyForEKS": { "actionId": "aws:eks:pod-network-latency", "parameters": { "delayMilliseconds": "200", "duration": "PT30M", "flowsPercent": "100", "interface": "DEFAULT", "kubernetesServiceAccount": "fis-service-account", "sources": "us-east-1a" }, "targets": { "Pods": "TargetsForEKS" } }, "LatencyForEC2": { "actionId": "aws:ssm:send-command", "parameters": { "duration": "PT30M", "documentArn": "arn:aws:ssm:us-east-1::document/AWSFIS-Run-Network-Latency-Sources", "documentParameters": "{\"DelayMilliseconds\":\"200\",\"Sources\":\"us-east-1a\",\"Interface\":\"DEFAULT\",\"TrafficType\":\"egress\",\"DurationSeconds\":\"1800\",\"FlowsPercent\":\"100\",\"InstallDependencies\":\"True\"}" }, "targets": { "Instances": "TargetsForEC2" } }, "LatencyForECS": { "actionId": "aws:ecs:task-network-latency", "parameters": { "delayMilliseconds": "200", "duration": "PT30M", "flowsPercent": "100", "installDependencies": "true", "sources": "us-east-1a", "useEcsFaultInjectionEndpoints": "true" }, "targets": { "Tasks": "TargetsForECS" }, "startAfter": [] } }, "targets": { "TargetsForEKS": { "parameters": { "availabilityZoneIdentifier": "us-east-1a", "clusterIdentifier": "", "namespace": "default", "selectorType": "labelSelector", "selectorValue": "AZApplicationSlowdown=LatencyForEKS" }, "resourceType": "aws:eks:pod", "selectionMode": "ALL" }, "TargetsForEC2": { "filters": [ { "path": "Placement.AvailabilityZone", "values": [ "us-east-1a" ] } ], "resourceTags": { "AZApplicationSlowdown": "LatencyForEC2" }, "resourceType": "aws:ec2:instance", "selectionMode": "ALL" }, "TargetsForECS": { "filters": [ { "path": "AvailabilityZone", "values": [ "us-east-1a" ] } ], "resourceTags": { "AZApplicationSlowdown": "LatencyForECS" }, "resourceType": "aws:ecs:task", "selectionMode": "ALL" } }, "experimentOptions": { "accountTargeting": "single-account", "emptyTargetResolutionMode": "skip" }, "stopConditions": [ { "source": "none" } ] } ``` # Cross-AZ: Traffic Slowdown Cross-AZ: Traffic Slowdown You can use the Cross-AZ: Traffic Slowdown scenario to inject packet loss to disrupt and slow down traffic between Availability Zones (AZs). The packet loss impairs cross-AZ communication, a partial disruption, sometimes known as a gray failure. It injects packet loss on network flows between target resources. Network flows represent the traffic between computing resources — the data packets carrying requests, responses, and other communications between your servers, containers, and services. The scenario can help to validate observability setups, tune alarm thresholds, discover application sensitivity and dependencies in cross-AZ communication, and practice critical operational decisions like AZ evacuation. By default, the scenario injects 15% packet loss to 100% of outbound network flows for target resources from the selected AZ for a duration of 30 minutes. You can use the **Edit shared parameters** dialog in the AWS FIS console to adjust the following parameters at the scenario level, which then apply to the underlying actions: + Availability Zone - you can select the AZ to impair, and packet loss will be injected from that AZ to the other AZs within the Region. + Packet Loss - adjust the packet loss lower for subtle disruption testing, such as 5%, or higher to test severe communication degradation and recovery mechanisms, such as 50%, or even 100% for total connectivity impact. + Flows percentage - reduce to impair a subset of traffic. For example, you can inject 15% packet loss affecting 25% of the network flows for even more subtle testing. + Duration - set how long the experiment runs. You can shorten for quicker tests, or run longer sustained tests. For example, set the duration to 2 hours to help test recovery mechanisms under impaired conditions. + Resource targeting - you can define target resources for the overall scenario using tags (for EC2 instances or ECS tasks on EC2 or Fargate) or labels (for EKS pods on EC2). You can specify your own tags and labels, or use the defaults provided in the scenario. If you don’t wish to use tags or labels, you can edit the action to target resources by specifying other parameters. + Customization - If you don’t want to target EC2 or ECS resources, you can leave the actions with default tags. The experiment won’t find any resources to target and the action will be skipped. However, if you don’t want to target EKS resources, you should remove the EKS action and target from the scenario completely, as it requires an EKS cluster identifier to be provided. For even more granular customization, you can modify individual actions in the experiment template directly. ## Actions Actions Together, the following actions help create the symptoms of a traffic slowdown between AZs by introducing packet loss on outbound communication from the target AZ to other AZs in the region at the network layer. These actions run in parallel, each injecting 15% packet loss for 30 minutes by default. After this period, communication returns to normal. The scenario needs at least one of the following resource types in the selected AZ to run: EC2 instance, ECS task, or EKS pod. ### ECS Network Packet Loss ECS Network Packet Loss Cross-AZ: Traffic Slowdown includes [aws:ecs:task-network-packet-loss](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#task-network-packet-loss) to inject packet loss for ECS tasks. The action targets tasks in the selected AZ, and impairs their outbound communication to all other AZs in the Region. You can further customize the scope of the impact by editing the action and adding or removing AZs from the `Sources` field. By default, it targets tasks with a [tag](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-using-tags.html) named `CrossAZTrafficSlowdown` with a value of `PacketLossForECS`. You can replace the default tag with your own, or add the scenario tag to your tasks. If no valid tasks are found this action will be skipped. Before running an experiment on ECS, you should follow the [setup steps for ECS task actions](https://docs.aws.amazon.com/fis/latest/userguide/ecs-task-actions.html). ### EKS Network Packet Loss EKS Network Packet Loss Cross-AZ: Traffic Slowdown includes [aws:eks:pod-network-packet-loss](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#pod-network-packet-loss) to inject packet loss for EKS pods. The action targets pods in the selected AZ, and impairs their outbound communication to all other AZs in the region. You can further customize the scope of the impact by editing the action and adding or removing AZs from the `Sources` field. By default, it targets pods within a cluster that have labels with the format key=value. The default label provided is `CrossAZTraffic=PacketLossForEKS`. You can replace the default label with your own, or add this label to your pods. If no valid pods are found this action will be skipped. Before running an experiment on EKS, you should follow the [setup steps for EKS pod actions](https://docs.aws.amazon.com/fis/latest/userguide/eks-pod-actions.html). ### EC2 Network Packet Loss EC2 Network Packet Loss Cross-AZ: Traffic Slowdown uses the [aws:ssm:send-command](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#ssm-send-command) action to run the [AWSFIS-Run-Network-Packet-Loss-Sources](https://docs.aws.amazon.com/fis/latest/userguide/actions-ssm-agent.html#awsfis-run-network-packet-loss-sources) document to inject packet loss for EC2 instances, and impairs their outbound communication to all other AZs in the Region. You can further customize the scope of the impact by editing the action and adding or removing AZs from the `Sources` field. The action targets instances in the selected AZ. By default, it targets instances with a [tag](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) named `CrossAZTrafficSlowdown` with a value of `PacketLossForEC2`. You can replace the default tag with your own, or add this tag to your instances. If no valid instances are found this action will be skipped. Before running an experiment on EC2 using SSM, you should [configure the AWS Systems Manager agent](https://docs.aws.amazon.com/fis/latest/userguide/actions-ssm-agent.html). ## Limitations Limitations + This scenario does not include [stop conditions](https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html). The correct stop conditions for your application should be added to the experiment template. ## Requirements Requirements + Add the required permissions to the AWS FIS [experiment role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html). + You need to target one or more resources from any of the following 3 types within the selected AZ: EC2 instances, ECS tasks, or EKS pods. + All targets of the scenario must be in the same VPC. ## Permissions Permissions To run this scenario you need an IAM role with a trust policy that allows FIS to assume the role and the managed policies for the resource types you target in the experiment: EC2, ECS, and EKS. When you create an experiment template from the Cross-AZ: Traffic Slowdown scenario, FIS creates the role for you with the trust policy and the following AWS managed policies: + [AWSFaultInjectionSimulatorEC2Access](https://docs.aws.amazon.com/fis/latest/userguide/security-iam-awsmanpol.html#AWSFaultInjectionSimulatorEC2Access) + [AWSFaultInjectionSimulatorECSAccess](https://docs.aws.amazon.com/fis/latest/userguide/security-iam-awsmanpol.html#AWSFaultInjectionSimulatorECSAccess) + [AWSFaultInjectionSimulatorEKSAccess](https://docs.aws.amazon.com/fis/latest/userguide/security-iam-awsmanpol.html#AWSFaultInjectionSimulatorEKSAccess) If you’re using an existing [IAM role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html) to run the Cross-AZ: Traffic Slowdown scenario, you can attach the following policy to grant AWS FIS the necessary permissions: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "DescribeTasks", "Effect": "Allow", "Action": "ecs:DescribeTasks", "Resource": "*" }, { "Sid": "DescribeContainerInstances", "Effect": "Allow", "Action": "ecs:DescribeContainerInstances", "Resource": "arn:aws:ecs:*:*:container-instance/*/*" }, { "Sid": "DescribeInstances", "Effect": "Allow", "Action": "ec2:DescribeInstances", "Resource": "*" }, { "Sid": "DescribeSubnets", "Effect": "Allow", "Action": "ec2:DescribeSubnets", "Resource": "*" }, { "Sid": "DescribeCluster", "Effect": "Allow", "Action": "eks:DescribeCluster", "Resource": "arn:aws:eks:*:*:cluster/*" }, { "Sid": "TargetResolutionByTags", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" }, { "Sid": "SendCommand", "Effect": "Allow", "Action": [ "ssm:SendCommand" ], "Resource": [ "arn:aws:ec2:*:*:instance/*", "arn:aws:ssm:*:*:managed-instance/*", "arn:aws:ssm:*:*:document/*" ] }, { "Sid": "ListCommands", "Effect": "Allow", "Action": [ "ssm:ListCommands" ], "Resource": "*" }, { "Sid": "CancelCommand", "Effect": "Allow", "Action": [ "ssm:CancelCommand" ], "Resource": "*" } ] } ``` ## Scenario Content Scenario Content The following content defines the scenario. This JSON can be saved and used to create an [experiment template](https://docs.aws.amazon.com/fis/latest/userguide/experiment-templates.html) using the [create-experiment-template](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/create-experiment-template.html) command from the AWS Command Line Interface (AWS CLI). For the most recent version of the scenario, visit the scenario library in the FIS console and go to the **Content** tab. ``` { "tags": { "Name": "Cross-AZ: Traffic Slowdown" }, "description": "Inject packet loss to disrupt and slow down traffic between AZs.", "actions": { "PacketLossForEC2": { "actionId": "aws:ssm:send-command", "parameters": { "duration": "PT30M", "documentArn": "arn:aws:ssm:us-east-1::document/AWSFIS-Run-Network-Packet-Loss-Sources", "documentParameters": "{\"Sources\":\"us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f\",\"LossPercent\":\"15\",\"Interface\":\"DEFAULT\",\"TrafficType\":\"egress\",\"DurationSeconds\":\"1800\",\"FlowsPercent\":\"100\",\"InstallDependencies\":\"True\"}" }, "targets": { "Instances": "TargetsForEC2" } }, "PacketLossForECS": { "actionId": "aws:ecs:task-network-packet-loss", "parameters": { "sources": "us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f", "lossPercent": "15", "duration": "PT30M", "flowsPercent": "100", "installDependencies": "true", "useEcsFaultInjectionEndpoints": "true" }, "targets": { "Tasks": "TargetsForECS" } }, "PacketLossForEKS": { "actionId": "aws:eks:pod-network-packet-loss", "parameters": { "sources": "us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f", "lossPercent": "15", "duration": "PT30M", "flowsPercent": "100", "interface": "DEFAULT", "kubernetesServiceAccount": "fis-service-account" }, "targets": { "Pods": "TargetsForEKS" } } }, "targets": { "TargetsForEC2": { "filters": [ { "path": "Placement.AvailabilityZone", "values": [ "us-east-1a" ] } ], "resourceTags": { "CrossAZTrafficSlowdown": "PacketLossForEC2" }, "resourceType": "aws:ec2:instance", "selectionMode": "ALL" }, "TargetsForECS": { "filters": [ { "path": "AvailabilityZone", "values": [ "us-east-1a" ] } ], "resourceTags": { "CrossAZTrafficSlowdown": "PacketLossForECS" }, "resourceType": "aws:ecs:task", "selectionMode": "ALL" }, "TargetsForEKS": { "parameters": { "availabilityZoneIdentifier": "us-east-1a", "clusterIdentifier": "", "namespace": "default", "selectorType": "labelSelector", "selectorValue": "CrossAZTrafficSlowdown=PacketLossForEKS" }, "resourceType": "aws:eks:pod", "selectionMode": "ALL" } }, "experimentOptions": { "accountTargeting": "single-account", "emptyTargetResolutionMode": "skip" }, "stopConditions": [ { "source": "none" } ] } ``` # Cross-Region: Connectivity Cross-Region: Connectivity You can use the Cross-Region: Connectivity scenario to block application network traffic from the experiment Region to the destination Region and pause cross-Region replication for Amazon S3 and Amazon DynamoDB multi-Region global tables. Cross Region: Connectivity affects outbound application traffic from the Region in which you run the experiment (*experiment Region*). Stateless inbound traffic from the Region you wish to isolate from the *experiment region* (*destination Region*) may not be blocked. Traffic from AWS managed services may not be blocked. This scenario can be used to demonstrate that multi-Region applications operate as expected when resources in the destination Region are not accessible from the experiment Region. It includes blocking network traffic from the experiment Region to the destination Region by targeting transit gateways and route tables. It also pauses cross-Region replication for S3 and DynamoDB global tables. By default, actions for which no targets are found will be skipped. ## Actions Actions Together, the following actions block cross-Region connectivity for the included AWS services. The actions are run in parallel. By default, the scenario blocks traffic for 3 hours, which you can increase up to a maximum 12 Hour duration. ### Disrupt Transit Gateway Connectivity Disrupt Transit Gateway Connectivity Cross Region: Connectivity includes [aws:network:transit-gateway-disrupt-cross-region-connectivity](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#network-actions-reference) to block cross-Region network traffic from VPCs in the *experiment Region* to VPCs in the *destination Region* connected by a transit gateway. This does not affect access to VPC endpoints within the *experiment Region* but will block traffic from the *experiment Region* destined for a VPC endpoint in the *destination Region*. This action targets transit gateways connecting the *experiment Region* and the *destination Region*. By default, it targets transit gateways with a [tag](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html#tgw-tagging) named `DisruptTransitGateway` with a value of `Allowed`. You can add this tag to your transit gateways or replace the default tag with your own tag in the experiment template. By default, if no valid transit gateways are found this action will be skipped. ### Disrupt Subnet Connectivity Disrupt Subnet Connectivity Cross Region: Connectivity includes [aws:network:route-table-disrupt-cross-region-connectivity](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#network-actions-reference) to block cross-Region network traffic from VPCs in the *experiment Region* to public AWS IP blocks in the *destination Region*. These public IP blocks include AWS service endpoints in the *destination Region*, e.g. the S3 regional endpoint, and AWS IP blocks for managed services, e.g. the IP addresses used for load balancers and Amazon API Gateway. This action also blocks network connectivity over cross-Region VPC Peering connections from the *experiment Region* to the *destination Region*. It does not affect access to VPC endpoints in the *experiment Region* but will block traffic from the *experiment Region* destined for a VPC endpoint in the *destination Region*. This action targets subnets in the experiment Region. By default, it targets subnets with a [tag](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) named `DisruptSubnet` with a value of `Allowed`. You can add this tag to your subnets or replace the default tag with your own tag in the experiment template. By default, if no valid subnets are found this action will be skipped. ### Disrupt VPC Endpoint Connectivity Disrupt VPC Endpoint Connectivity Cross Region: Connectivity includes [aws:network:disrupt-vpc-endpoint](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#network-actions-reference) disrupt connectivity to a service associated with the target VPC endpoints. For example, if a VPC endpoint creates a Private Link to com.amazonaws.us-east-1.ec2, then the connectivity to that service will be disrupted. This action targets VPC endpoints in the experiment region. By default, it targets interface VPC endpoints with a [tag](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) named DisruptVpcEndpoint with a value `Allowed`. You can add this tag to your VPC endpoints or replace the default tag with your own tag in the experiment template. By default, if no valid VPC endpoints are found this action will be skipped. ### Pause S3 Replication Pause S3 Replication Cross Region: Connectivity includes [aws:s3:bucket-pause-replication](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#s3-actions-reference-fis) to pause S3 replication from the *experiment Region* to the *destination Region* for the targeted buckets. Replication from the *destination Region* to the *experiment Region* will be unaffected. After the scenario ends, bucket replication will resume from the point it was paused. Note that the time it takes for replication to keep all objects in sync will vary based on the duration of the experiment, and the rate of object upload to the bucket. This action targets S3 buckets in the experiment Region with [Cross-Region Replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) (CRR) enabled to an S3 bucket in the destination Region. By default, it targets buckets with a [tag](https://docs.aws.amazon.com/AmazonS3/latest/userguide/view-bucket-properties.html) named `DisruptS3` with a value of `Allowed`. You can add this tag to your buckets or replace the default tag with your own tag in the experiment template. By default, if no valid buckets are found this action will be skipped. ### Pause DynamoDB Replication Pause DynamoDB Replication Cross-Region: Connectivity includes [aws:dynamodb:global-table-pause-replication](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#dynamodb-actions-reference) to pause replication between the experiment Region and all other Regions, including the destination Region. This prevents replication into and out of the *experiment Region* but does not affect replication between other Regions. After the scenario ends, table replication will resume from the point it was paused. Note that the time it takes for replication to keep all data in sync will vary based on the duration of the experiment and the rate of changes to the table. This action targets both DynamoDB multi-Region strongly and eventually consistent global tables in the experiment Region. By default, it targets tables with a [tag](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tagging.html) named `DisruptDynamoDb` with a value of `Allowed`. You can add this tag to your tables or replace the default tag with your own tag in the experiment template. By default, if no valid global tables are found this action will be skipped. ### Pause MemoryDB multi-Region Replication Pause MemoryDB multi-Region Replication Cross-Region: Connectivity includes [aws:memorydb:multi-region-cluster-pause-replication](https://docs.aws.amazon.com/fis/latest/userguide/fis-actions-reference.html#memorydb-actions-reference) to pause replication from the regional member cluster in the experiment Region to the rest of the clusters in the targeted multi-Region cluster. Replication between other regional member clusters will be unaffected. After the scenario ends, replication will resume from the point it was paused. Note that the time for replication to sync data between the member clusters will vary based on the duration of the experiment, and the rate of data written to the clusters. This action targets MemoryDB Multi-Region clusters with a regional member in the experiment Region. By default, it targets multi-Region clusters with a [tag](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tagging.html) named `DisruptMemoryDB` with a value of `Allowed`. You can add this tag to your multi-Region clusters or replace the default tag with your own tag in the experiment template. By default, if no valid clusters are found this action will be skipped. ## Limitations Limitations + This scenario does not include [stop conditions](https://docs.aws.amazon.com/fis/latest/userguide/stop-conditions.html). The correct stop conditions for your application should be added to the experiment template. ## Requirements Requirements + Add the required permission to the AWS FIS [experiment role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html). + Resource tags must be applied to resources that are to be targeted by the experiment. These can use your own tagging convention or the default tags defined in the scenario. ## Permissions Permissions The following policy grants AWS FIS the necessary permissions to execute an experiment with the Cross-Region: Connectivity scenario. This policy must be attached to the [experiment role](https://docs.aws.amazon.com/fis/latest/userguide/getting-started-iam-service-role.html). ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "RouteTableDisruptConnectivity1", "Effect": "Allow", "Action": "ec2:CreateRouteTable", "Resource": "arn:aws:ec2:*:*:route-table/*", "Condition": { "StringEquals": { "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity2", "Effect": "Allow", "Action": "ec2:CreateRouteTable", "Resource": "arn:aws:ec2:*:*:vpc/*" }, { "Sid": "RouteTableDisruptConnectivity21", "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": "arn:aws:ec2:*:*:route-table/*", "Condition": { "StringEquals": { "ec2:CreateAction": "CreateRouteTable", "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity3", "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": "arn:aws:ec2:*:*:network-interface/*", "Condition": { "StringEquals": { "ec2:CreateAction": "CreateNetworkInterface", "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity4", "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": "arn:aws:ec2:*:*:prefix-list/*", "Condition": { "StringEquals": { "ec2:CreateAction": "CreateManagedPrefixList", "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity5", "Effect": "Allow", "Action": "ec2:DeleteRouteTable", "Resource": [ "arn:aws:ec2:*:*:route-table/*", "arn:aws:ec2:*:*:vpc/*" ], "Condition": { "StringEquals": { "ec2:ResourceTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity6", "Effect": "Allow", "Action": "ec2:CreateRoute", "Resource": "arn:aws:ec2:*:*:route-table/*", "Condition": { "StringEquals": { "ec2:ResourceTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity7", "Effect": "Allow", "Action": "ec2:CreateNetworkInterface", "Resource": "arn:aws:ec2:*:*:network-interface/*", "Condition": { "StringEquals": { "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity8", "Effect": "Allow", "Action": "ec2:CreateNetworkInterface", "Resource": [ "arn:aws:ec2:*:*:subnet/*", "arn:aws:ec2:*:*:security-group/*" ] }, { "Sid": "RouteTableDisruptConnectivity9", "Effect": "Allow", "Action": "ec2:DeleteNetworkInterface", "Resource": "arn:aws:ec2:*:*:network-interface/*", "Condition": { "StringEquals": { "ec2:ResourceTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity10", "Effect": "Allow", "Action": "ec2:CreateManagedPrefixList", "Resource": "arn:aws:ec2:*:*:prefix-list/*", "Condition": { "StringEquals": { "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity11", "Effect": "Allow", "Action": [ "ec2:DeleteManagedPrefixList", "ec2:ModifyManagedPrefixList" ], "Resource": "arn:aws:ec2:*:*:prefix-list/*", "Condition": { "StringEquals": { "aws:ResourceTag/managedByFIS": "true" } } }, { "Sid": "EC2DescribeResources", "Effect": "Allow", "Action": [ "ec2:DescribeNetworkInterfaces", "ec2:DescribeVpcs", "ec2:DescribeVpcPeeringConnections", "ec2:DescribeManagedPrefixLists", "ec2:DescribeSubnets", "ec2:DescribeRouteTables", "ec2:DescribeVpcEndpoints", "ec2:DescribeTransitGatewayPeeringAttachments", "ec2:DescribeTransitGatewayAttachments", "ec2:DescribeTransitGateways", "ec2:DescribeSecurityGroups" ], "Resource": "*" }, { "Sid": "RouteTableDisruptConnectivity14", "Effect": "Allow", "Action": "ec2:ReplaceRouteTableAssociation", "Resource": [ "arn:aws:ec2:*:*:subnet/*", "arn:aws:ec2:*:*:route-table/*" ] }, { "Sid": "RouteTableDisruptConnectivity15", "Effect": "Allow", "Action": "ec2:GetManagedPrefixListEntries", "Resource": "arn:aws:ec2:*:*:prefix-list/*" }, { "Sid": "RouteTableDisruptConnectivity16", "Effect": "Allow", "Action": "ec2:AssociateRouteTable", "Resource": [ "arn:aws:ec2:*:*:subnet/*", "arn:aws:ec2:*:*:route-table/*" ] }, { "Sid": "RouteTableDisruptConnectivity17", "Effect": "Allow", "Action": "ec2:DisassociateRouteTable", "Resource": "arn:aws:ec2:*:*:route-table/*", "Condition": { "StringEquals": { "ec2:ResourceTag/managedByFIS": "true" } } }, { "Sid": "RouteTableDisruptConnectivity18", "Effect": "Allow", "Action": "ec2:DisassociateRouteTable", "Resource": "arn:aws:ec2:*:*:subnet/*" }, { "Sid": "RouteTableDisruptConnectivity19", "Effect": "Allow", "Action": "ec2:ModifyVpcEndpoint", "Resource": "arn:aws:ec2:*:*:route-table/*", "Condition": { "StringEquals": { "ec2:ResourceTag/managedByFIS": "true" } } }, { "Sid": "TransitGatewayDisruptConnectivity1", "Effect": "Allow", "Action": [ "ec2:DisassociateTransitGatewayRouteTable", "ec2:AssociateTransitGatewayRouteTable" ], "Resource": [ "arn:aws:ec2:*:*:transit-gateway-route-table/*", "arn:aws:ec2:*:*:transit-gateway-attachment/*" ] }, { "Sid": "S3CrossRegion1", "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" }, { "Sid": "S3CrossRegion3", "Effect": "Allow", "Action": "s3:PauseReplication", "Resource": "arn:aws:s3:::*", "Condition": { "StringLike": { "s3:DestinationRegion": "*" } } }, { "Sid": "S3CrossRegion4", "Effect": "Allow", "Action": [ "s3:GetReplicationConfiguration", "s3:PutReplicationConfiguration" ], "Resource": "arn:aws:s3:::*", "Condition": { "BoolIfExists": { "s3:isReplicationPauseRequest": "true" } } }, { "Sid": "DynamoDbPauseReplication", "Effect": "Allow", "Action": [ "dynamodb:DescribeTable", "dynamodb:PutResourcePolicy", "dynamodb:GetResourcePolicy", "dynamodb:DeleteResourcePolicy" ], "Resource": [ "arn:aws:dynamodb:*:*:table/*" ] }, { "Sid": "DynamoDbMrscPauseReplication", "Effect": "Allow", "Action": [ "dynamodb:InjectError" ], "Resource": ["*"] }, { "Sid": "ResolveResourcesViaTags", "Effect": "Allow", "Action": "tag:GetResources", "Resource": "*" }, { "Sid": "MemDbCrossRegion", "Effect": "Allow", "Action": [ "memorydb:DescribeMultiRegionClusters", "memorydb:PauseMultiRegionClusterReplication" ], "Resource": [ "arn:aws:memorydb::*:multiregioncluster/*" ] }, { "Sid": "DisruptVPCE1", "Effect": "Allow", "Action": "ec2:CreateSecurityGroup", "Resource": [ "arn:aws:ec2:*:*:vpc/*", "arn:aws:ec2:*:*:security-group/*" ] }, { "Sid": "DisruptVPCE2", "Effect": "Allow", "Action": "ec2:CreateTags", "Resource": "arn:aws:ec2:*:*:security-group/*", "Condition": { "StringEquals": { "ec2:CreateAction": "CreateSecurityGroup", "aws:RequestTag/managedByFIS": "true" } } }, { "Sid": "DisruptVPCE3", "Effect": "Allow", "Action": [ "ec2:DeleteSecurityGroup", "ec2:RevokeSecurityGroupEgress" ], "Resource": "arn:aws:ec2:*:*:security-group/*", "Condition": { "StringEquals": { "aws:ResourceTag/managedByFIS": "true" } } }, { "Sid": "DisruptVPCE4", "Effect": "Allow", "Action": "vpce:AllowMultiRegion", "Resource": "arn:aws:ec2:*:*:vpc-endpoint/*" }, { "Sid": "ModifyVPCE", "Effect": "Allow", "Action": "ec2:ModifyVpcEndpoint", "Resource": [ "arn:aws:ec2:*:*:vpc-endpoint/*", "arn:aws:ec2:*:*:security-group/*" ] } ] } ``` ## Scenario Content Scenario Content The following content defines the scenario. This JSON can be saved and used to create an [experiment template](https://docs.aws.amazon.com/fis/latest/userguide/experiment-templates.html) using the [create-experiment-template](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/fis/create-experiment-template.html) command from the AWS Command Line Interface (AWS CLI). For the most recent version of the scenario, visit the scenario library in the FIS console. ``` { "targets": { "Transit-Gateway": { "resourceType": "aws:ec2:transit-gateway", "resourceTags": { "TgwTag": "TgwValue" }, "selectionMode": "ALL" }, "Subnet": { "resourceType": "aws:ec2:subnet", "resourceTags": { "SubnetKey": "SubnetValue" }, "selectionMode": "ALL", "parameters": {} }, "VPC-Endpoint": { "resourceType": "aws:ec2:vpc-endpoint", "resourceTags": { "DisruptPrivateLink": "Allowed" }, "selectionMode": "ALL" }, "S3-Bucket": { "resourceType": "aws:s3:bucket", "resourceTags": { "S3Impact": "Allowed" }, "selectionMode": "ALL" }, "DynamoDB-Global-Table": { "resourceType": "aws:dynamodb:global-table", "resourceTags": { "DisruptDynamoDb": "Allowed" }, "selectionMode": "ALL" }, "MemoryDB-Multi-Region-Cluster": { "resourceType": "aws:memorydb:multi-region-cluster", "resourceTags": { "DisruptMemoryDb": "Allowed" }, "selectionMode": "ALL" } }, "actions": { "Disrupt-Transit-Gateway-Connectivity": { "actionId": "aws:network:transit-gateway-disrupt-cross-region-connectivity", "parameters": { "duration": "PT3H", "region": "eu-west-1" }, "targets": { "TransitGateways": "Transit-Gateway" } }, "Disrupt-Subnet-Connectivity": { "actionId": "aws:network:route-table-disrupt-cross-region-connectivity", "parameters": { "duration": "PT3H", "region": "eu-west-1" }, "targets": { "Subnets": "Subnet" } }, "Disrupt-Vpc-Endpoint": { "actionId": "aws:network:disrupt-vpc-endpoint", "parameters": { "duration": "PT3H" }, "targets": { "VPCEndpoints": "VPC-Endpoint" } }, "Pause-S3-Replication": { "actionId": "aws:s3:bucket-pause-replication", "parameters": { "duration": "PT3H", "region": "eu-west-1" }, "targets": { "Buckets": "S3-Bucket" } }, "Pause-DynamoDB-Replication": { "actionId": "aws:dynamodb:global-table-pause-replication", "parameters": { "duration": "PT3H" }, "targets": { "Tables": "DynamoDB-Global-Table" } }, "Pause-MemoryDB-Multi-Region-Cluster-Replication": { "actionId": "aws:memorydb:multi-region-cluster-pause-replication", "parameters": { "duration": "PT3H", "region": "eu-west-1" }, "targets": { "MultiRegionClusters": "MemoryDB-Multi-Region-Cluster" } } }, "stopConditions": [ { "source": "none" } ], "roleArn": "", "logConfiguration": { "logSchemaVersion": 2 }, "tags": { "Name": "Cross-Region: Connectivity" }, "experimentOptions": { "accountTargeting": "single-account", "emptyTargetResolutionMode": "skip" }, "description": "Block application network traffic from experiment Region to target Region and pause cross-Region replication" } ```