# Automate starting and stopping AWS instances
The Instance Scheduler on AWS solution automates the starting and stopping of various AWS services including [Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) (Amazon EC2) and [Amazon Relational Database Service](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) (Amazon RDS) instances.
This solution helps reduce operational costs by stopping resources that are not in use and starting resources when their capacity is needed. For example, a company can use Instance Scheduler on AWS to automatically stop instances outside of business hours every day. If you leave all of your instances running at full utilization, this solution can result in up to 70% cost savings for those instances that are only necessary during regular business hours (weekly utilization reduced from 168 hours to 50 hours).
Instance Scheduler on AWS leverages Amazon Web Services (AWS) resource tags and [AWS Lambda](https://aws.amazon.com/lambda/) to automatically stop and restart instances across multiple AWS Regions and accounts on a customer-defined schedule. This solution also allows you to use hibernation for stopped EC2 instances.
This implementation guide provides an overview of the Instance Scheduler on AWS solution, its reference architecture and components, considerations for planning the deployment, and configuration steps for deploying the solution to the AWS Cloud.
This guide is intended for IT infrastructure architects, administrators, and DevOps professionals who want to implement Instance Scheduler on AWS in their environment.
Use this navigation table to quickly find answers to these questions:
| If you want to . . . | Read . . . |
| --- | --- |
| Know the cost for running this solution. The estimated cost for running this solution in the US East (N. Virginia) Region is USD \$113.15 per month. | [Cost](cost.md) |
| Understand the security considerations for this solution. | [AWS Well-Architected Security](aws-well-architected-design-considerations.md), [Security](security-1.md) |
| Configure schedules. | [Scheduler configuration table](scheduler-configuration-table.md) |
| Know which AWS Regions are supported for this solution. | [Supported AWS Regions](#supported-aws-regions) |
| View or download the AWS CloudFormation template included in this solution to automatically deploy the infrastructure resources (the "stack") for this solution. | [AWS CloudFormation templates](aws-cloudformation-templates.md) |
| Access the source code and optionally use the AWS Cloud Development Kit (AWS CDK) to deploy the solution. | [GitHub repository](https://github.com/aws-solutions/instance-scheduler-on-aws/tree/main) |
# Features and benefits
The Instance Schedule on AWS solution provides the following features:
**Cross-account instance scheduling**
This solution includes a template that creates the [AWS Identity and Access Management](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) (IAM) roles necessary to start and stop instances in secondary accounts. For more information, refer to the [Cross-account instance scheduling](cross-account-instance-scheduling-using-account-ids-or-aws-organization-id.md) section.
**Automated tagging**
Instance Scheduler on AWS can automatically add tags to all instances that it starts or stops. The solution also includes macros that allow you to add variable information to the tags.
**Configure schedules or periods using Scheduler CLI**
This solution includes a command line interface (CLI) that provides commands for configuring schedules and periods. The CLI allows customers to estimate cost savings for a given schedule. For more information, refer to the [Scheduler CLI](scheduler-cli.md).
**Manage schedules using Infrastructure as Code (IaC)**
This solution provides an AWS CloudFormation Custom Resource that you can use to manage schedules using Infrastructure as Code (IaC). For more information, refer to [Manage Schedules Using Infrastructure as Code](manage-schedules-using-infrastructure-as-code-iac.md).
**Integration with Systems Manager Maintenance Windows**
For Amazon EC2 instances, Instance Scheduler on AWS can integrate with [AWS Systems Manager](https://aws.amazon.com/systems-manager/) maintenance windows, defined in the same Region as those instances, to start and stop them in accordance with the maintenance window.
# Use cases
**Running instances only during working hours**
If you leave all of your instances running at full utilization, this solution can result in up to 76% cost savings for those instances that are only necessary during regular business hours (weekly utilization reduced from 168 hours to 40 hours). For more information, see the [Sample schedule](sample-schedules.md).
**Stopping instances after working hours**
If you want to make sure that development instances are off after hours and until needed again, you can use this solution to set an end period without a start period. For more information, see the [Sample schedule](sample-schedules.md).
# Concepts and definitions
This section describes key concepts and defines terminology specific to this solution:
**schedule**
Group of one or more periods that an instance is bound by.
**period**
Running period(s) defined by a start and stop time.
**instance**
A supported resource that is able to be scheduled. For example, an Amazon EC2 instance or an Amazon RDS cluster Amazon EC2 and Amazon RDS.
**regular business hours**
9:00 to 17:00 (9 AM - 5 PM) ET on weekdays
For a general reference of AWS terms, see the [AWS Glossary](https://docs.aws.amazon.com/general/latest/gr/glos-chap.html).
# Cost
You are responsible for the cost of the AWS services used while running Instance Scheduler. Understanding how costs scale with your deployment size helps you plan and optimize your implementation.
## Cost scaling factors
Instance Scheduler costs scale based on several factors:
**Number of scheduling targets**: The number of unique account-region-service combinations being managed. Each target requires a separate Lambda invocation per scheduling interval.
**Resources per target**: The number of resources (EC2 instances, RDS databases, etc.) within each target influences Lambda execution time and duration costs.
**Operational metrics complexity**: Optional CloudWatch metrics costs scale with the number of unique instance types and active schedules being tracked across your deployment.
**Scheduling frequency**: The solution runs based on your configured frequency (default: 5 minutes). More frequent checks increase Lambda invocations from 24 times daily (hourly) to 288 times daily (5-minute intervals).
## Calculating scheduling targets
A scheduling target is a unique combination of account-region-service that contains at least one actively managed instance. Multiple instances within the same account-region-service combination count as a single scheduling target.
**Example calculation:**
+ Account A, us-east-1, 5 EC2 instances = 1 scheduling target
+ Account A, us-east-1, 3 RDS databases = 1 scheduling target
+ Account A, us-east-1, 2 Auto Scaling groups = 1 scheduling target
+ Account A, us-west-2, 2 EC2 instances = 1 scheduling target
+ Account B, us-east-1, 10 EC2 instances = 1 scheduling target
Total: 5 scheduling targets
This means the solution will invoke 5 separate Lambda functions per scheduling interval to manage all resources across these account-region-service combinations.
**Note**
Targets can be in scope for scheduling but are not considered "active" for cost calculations until at least one resource is tagged for scheduling in that target.
For cost optimization, Instance Scheduler groups all Amazon RDS-related services into a single invocation. Therefore Amazon RDS, [Amazon Aurora](https://aws.amazon.com/rds/aurora/), [Amazon Neptune](https://aws.amazon.com/neptune/), and [Amazon DocDB scheduling](https://aws.amazon.com/documentdb/) all count as one "RDS" service for cost calculations.
## Cost optimization strategies
1. Deploy in a Region with lower Lambda pricing
1. Use the default 512 MB Lambda memory setting unless required to increase it by single-target scale limitations
1. Minimize the number of unique schedules and instance types in active use
1. Adjust scheduling frequency based on your requirements
1. Disable operational metrics dashboard if not planning to use it
See the pricing webpage for each [AWS service in this solution.](aws-services-used-in-this-solution.md)
We recommend creating a [budget](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-create.html) through AWS Cost Explorer to help manage costs. Prices are subject to change.
## Reference pricing examples (monthly)
The following examples demonstrate how costs scale across different deployment sizes. Use these as reference points to estimate costs for your specific deployment.
**Note**
All reference pricing are rough estimations of cost for the primary services used by the solution.
### Small deployment (\$1\$19 monthly)
This example represents a typical development or small production deployment:
+ 5 active targets
+ 20 managed resources
+ 3 active schedules
+ 2 instance types
+ 5-minute scheduling interval
+ 512 MB Lambda function, 5-second average runtime
| AWS service | Monthly Cost [USD] |
| --- | --- |
| AWS Lambda | \$1\$12.00 |
| AWS KMS | \$1\$11.50 |
| CloudWatch Logs | \$1\$10.30 |
| CloudWatch Metrics | \$1\$15.30 |
| Amazon DynamoDB | \$1\$10.05 |
| Total: | \$1\$19.15 |
### Medium deployment (\$1\$1161 monthly)
This example represents a mid-size enterprise deployment:
+ 250 active targets
+ 1000 managed resources
+ 15 active schedules
+ 15 instance types
+ 5-minute scheduling interval
+ 512 MB Lambda function, 5-second average runtime
+ 5 EC2 Maintenance Windows
| AWS service | Monthly Cost [USD] |
| --- | --- |
| AWS Lambda | \$1\$195.00 |
| Amazon DynamoDB | \$1\$11.00 |
| CloudWatch Logs | \$1\$110.00 |
| CloudWatch Metrics | \$1\$140.00 |
| AWS KMS | \$1\$115.00 |
| Total: | \$1\$1161.00 |
### Large deployment (\$1\$1630 monthly)
This example represents a large enterprise deployment:
+ 1000 active targets
+ 5000 managed resources
+ 500 active schedules
+ 50 instance types
+ 5-minute scheduling interval
+ 512 MB Lambda function, 5-second average runtime
+ 100 EC2 Maintenance Windows
| AWS service | Monthly Cost [USD] |
| --- | --- |
| AWS Lambda | \$1\$1380.00 |
| Amazon DynamoDB | \$1\$15.00 |
| CloudWatch Logs | \$1\$150.00 |
| CloudWatch Metrics | \$1\$1140.00 |
| AWS KMS | \$1\$155.00 |
| Total: | \$1\$1630.00 |
## Cost estimation for your deployment
To estimate costs for your specific deployment:
1. Count your total managed resources (EC2 instances, RDS databases, etc.)
1. Determine the number of accounts and Regions you’ll manage
1. Consider your required scheduling frequency
1. Decide whether you need operational metrics
1. Use the reference examples above to interpolate your expected costs
# Quotas
## Scaling limitations
Instance Scheduler scales on two primary axes to manage large enterprise deployments:
### Vertical scaling (resources per target)
Vertical scaling is bounded by the number of resources that a single Scheduling Request Lambda function can efficiently process within a single scheduling target (account/region/service combination).
Instance Scheduler is designed to be able to handle 1000s of EC2s, 100s of ASGs, and 100s of RDS dbs/clusters in a single [scheduling targets](cost.md#calculating-scheduling-targets) but may be limited by cross-region latency.
To ensure optimal performance, we recommend monitoring the execution time of the Scheduling Request Lambda (see [Operational insights dashboard](monitor-the-solution.md#operational-insights-dashboard)). We recommend keeping average runtime below 90 seconds with a maximum peak time no greater than 4 minutes.
### Horizontal scaling (number of targets)
Horizontal scaling is limited by the number of [active scheduling targets](cost.md#calculating-scheduling-targets) being managed. An active target is an account/region/service combination with at least one actively tagged resource. Instance Scheduler can be deployed to many more accounts and regions, but only targets with actively tagged resources affect performance.
With the default Lambda concurrency quota of 1000, you can run 1000 active targets simultaneously. Lambda queues additional executions automatically, allowing you to scale beyond this limit. We recommend keeping cumulative delay below 3 minutes for optimal performance.
For example, with a 15-second average runtime and lambda concurrency limit of 1000, you can manage up to 12000 active targets while keeping cumulative delay under 3 minutes (3 minutes ÷ 15 seconds × 1000 = 12000 targets).
For larger deployments, you can request a Lambda concurrency quota increase from AWS Support.
## Additional Considerations
**AWS resource tags**: AWS resources typically have a limit of 50 tags per resource. Instance Scheduler requires 6 informational and control tags for operating the solution. Ensure your resources have sufficient tag capacity to accommodate both Instance Scheduler tags and your existing tagging strategy.
**Lambda execution limits**: Each Scheduling Request Handler Lambda function has a 5-minute execution timeout.
**DynamoDB scaling**: The solution uses on-demand scaling for its [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) tables, automatically adjusting capacity based on your workload.
**API rate limits**: AWS service API throttling may occur with very large deployments. The solution includes retry logic to handle temporary throttling, but excessive throttling may reduce the upper scaling limits of the solution.
## AWS Service Quotas
### Service quotas for AWS services
Service quotas, also referred to as limits, are the maximum number of service resources or operations for your AWS account. Make sure you have sufficient quota for each of the services implemented in this solution. For more information, see [AWS service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html).
### AWS CloudFormation quotas
Your AWS account has AWS CloudFormation quotas that you should be aware of when launching the stack in this solution. By understanding these quotas, you can avoid limitation errors that would prevent you from deploying this solution successfully. For more information, see [AWS CloudFormation quotas](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html) in the *AWS CloudFormation User’s Guide*.
#### AWS Lambda quotas
Your account has a default AWS Lambda Concurrent Execution quota of 1000. For larger deployments, we recommend deploying Instance Scheduler to a dedicated account to avoid competing with other workloads for Lambda concurrency. This value is adjustable. For more information, refer to [AWS Lambda Getting started guide](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html).
## Supported AWS Regions
You can deploy Instance Scheduler in any AWS Region, including AWS GovCloud (US) Regions and some [opt-in Regions](https://docs.aws.amazon.com/glossary/latest/reference/glos-chap.html?icmpid=docs\_homepage\_addtlrcs#optinregion) (Regions that are disabled by default). After you deploy the solution, you can configure it to apply the appropriate start or stop actions to tagged EC2 and RDS DB instances in any Region(s) of your account. If you use cross-account instance scheduling, the solution applies actions to instances in all configured Regions in all accounts.
**Important**
Instance Scheduler on AWS actions affect appropriately tagged instances in all AWS Regions of your account, even though the Lambda function is running in a single Region.
You can use multiple deployments of the solution to schedule a large number of instances, or instances in many accounts and Regions. When you deploy multiple schedulers, use a different tag name for each stack, and configure a set of non-overlapping Regions for each deployment.
Each deployment checks every instance in every configured Region in an account for the tag key that identifies resources it should schedule. If the Regions for multiple deployments overlap, each instance will be checked by multiple deployments.
**Note**
Instance Scheduler on AWS can target instances within any opt-in region for scheduling, even when the solution stacks are deployed in standard AWS regions.
# Cross-account instance scheduling using account IDs or AWS Organization ID
This solution includes a template ([instance-scheduler-on-aws-remote.template](https://s3.amazonaws.com/solutions-reference/instance-scheduler-on-aws/latest/instance-scheduler-on-aws-remote.template)) that creates the [AWS Identity and Access Management (IAM)](https://aws.amazon.com/iam/) roles and other necessary resources to enable the solution to start scheduling in the secondary accounts. You can review and modify permissions in the remote template before you launch the stack.
## Enabling cross account scheduling using Account IDs
To apply automated start-stop schedules to resources in secondary accounts:
1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) and select the button to launch the [instance-scheduler-on-aws](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://s3.amazonaws.com/solutions-reference/instance-scheduler-on-aws/latest/instance-scheduler-on-aws.template) AWS CloudFormation template in the primary account.
1. Launch the remote template ([instance-scheduler-on-aws-remote](https://s3.amazonaws.com/solutions-reference/instance-scheduler-on-aws/latest/instance-scheduler-on-aws-remote.template)) in each applicable secondary account. When each remote stack is launched, it creates a cross-account role Amazon Resource Name (ARN).
1. Update the primary solution stack with the Account ID in the **Provide Organization** **ID** or **List of Remote Account IDs** parameters to allow the solution to perform start and stop actions on instances in the secondary accounts.
## Enabling cross account scheduling using AWS Organization ID
To apply automated start-stop schedules to resources in secondary accounts:
1. Sign in to the [AWS Management Console](https://console.aws.amazon.com/) and select the button to launch the [instance-scheduler-on-aws](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?templateURL=https://s3.amazonaws.com/solutions-reference/instance-scheduler-on-aws/latest/instance-scheduler-on-aws.template) AWS CloudFormation template in the primary account.
1. Set the CloudFormation parameter **Using AWS Organizations?** as Yes,and provide the organization ID in the **Provide** **Organization ID** OR **List of Remote Account IDs** CloudFormation parameters.
1. After deploying the stack in the primary account, launch the remote template `(instance-scheduler-on-aws-remote)` in each applicable secondary account in the same Region as the solution in the primary account. When each remote stack is launched successfully, the primary solution account will be updated with the account id without any further changes in the primary account.
## Managing Account IDs with AWS Systems Manager Parameter Store
Use AWS Systems Manager Parameter Store to store remote account IDs. You can store remote Account IDs as a list parameter where every item is an account ID, or as a string parameter that contains a comma-delimited list of remote account IDs. The parameter has the format \$1param:\$1name\$1\$1 where the name is the name of the parameter in Parameter Store.
To leverage this feature, you must launch the Instance Scheduler on AWS hub stack in the same account as your parameter store.
# Services supported for scheduling
Instance Scheduler on AWS currently supports scheduling of the following services:
+ Amazon EC2
+ Amazon EC2 Auto Scaling groups
+ Amazon RDS
+ Amazon Aurora clusters
+ Amazon DocumentDB
+ Amazon Neptune
# Instance shutdown behavior
## Amazon EC2
This solution is designed to automatically stop EC2 instances and assumes that instance *shutdown behavior* is set to Stop, not Terminate. Note that you cannot restart an Amazon EC2 instance after it is terminated.
By default, EC2 instances are configured to stop, not terminate, when shut down, but you can [modify this behavior](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingInstanceInitiatedShutdownBehavior). Therefore, make sure that the instances you control using the Instance Scheduler on AWS are configured with a Stop shutdown behavior; otherwise, they will be terminated.
## Amazon RDS, Amazon Neptune, and Amazon DocumentDB
This solution is designed to automatically stop, not delete, RDS, Neptune, and DocDB instances. You can use the **Create RDS Instance Snapshot** AWS CloudFormation template parameter to create snapshots of RDS DB instances before the solution stops the instances. Snapshots are kept until the next time the instance is stopped and a new snapshot is created.
**Note**
Snapshots are not available for Amazon Aurora clusters. You can use the **Schedule Aurora Clusters** template parameter to start and stop RDS DB instances that are part of an Aurora cluster or that manage Aurora databases. You must tag the cluster (not the individual instances) with the tag key you defined during initial configuration and the schedule name as the tag value to schedule that cluster.
For more information about limitations to starting and stopping an RDS DB instance, refer to [Stopping an Amazon RDS DB instance temporarily](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_StopInstance.html) in the *Amazon RDS User Guide*.
When an RDS DB instance is stopped, the cache is cleared, which might lead to slower performance when the instance is restarted.
## Amazon RDS maintenance window
Every RDS DB instance has a weekly [maintenance window](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html#Concepts.DBMaintenance) during which any system changes are applied. During the maintenance window, Amazon RDS will automatically start instances that have been stopped for more than seven days to apply maintenance. Amazon RDS will not stop the instance once the maintenance event is complete.
The solution allows you to specify whether to add the preferred maintenance window of an RDS DB instance as a running period to its schedule. The solution will start the instance at the beginning of the maintenance window and stop the instance at the end of the maintenance window if no other running period specifies that the instance should run, and if the maintenance event is completed.
If the maintenance event is not completed by the end of the maintenance window, the instance will run until the scheduling interval after the maintenance event is completed. For more information about the Amazon RDS maintenance window, refer to [Maintaining a DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.Maintenance.html) in the *Amazon RDS User Guide*.
## Amazon EC2 Auto Scaling groups
We designed this solution to automatically stop Amazon EC2 Auto Scaling groups by using scheduled scaling actions. You can use the solution to configure scheduled scaling actions on the Auto Scaling group (ASG). When an ASG is stopped by a scheduled scaling action, its minimum, desired, and maximum capacities will be set to `0` until the ASG is automatically started again. This will return the minimum, desired, and maximum capacities to their original values.