# Operator guide This guide is intended for users and operators of this solution and contains details on how to [configure schedules](#configure-schedules) and [monitor the solution](monitor-the-solution.md). ## Configure schedules **Important** Misconfigured schedules can result in instances running continuously and incurring unexpected costs. Before applying schedules to your resources, verify the following: The schedule name in the resource tag exactly matches a schedule defined in the configuration table. Misspelled or nonexistent schedule names will result in an `UnknownSchedule` error and the instance will not be stopped by the scheduler. Check for the `IS-Error` tag on your resources to identify this condition. If `stop_new_instances` is set to `false`, instances that are running outside of a scheduled period when first tagged will not be stopped until the next scheduled stop transition. This can result in instances running longer than expected. If `retain_running` is set to `true`, instances that are manually started before a running period begins will not be stopped at the end of that period. This is by design, but can lead to instances running indefinitely if not monitored. When using `enforced: false` (the default), the scheduler will not restart instances that are manually stopped during a running period, and will not stop instances that are manually started outside of a running period after the initial stop transition. We recommend enabling [informational tagging](monitor-the-solution.md#informational-tags) (enabled by default) and periodically reviewing the `IS-Error` and `IS-LastAction` tags on your resources to confirm that scheduling is operating as expected. Once the solution has been successfully deployed, you can begin to configure schedules. Instance Scheduler on AWS supports two methods of managing schedules as described below. **Note** The solution can support any number of schedules, each of which can contain one or more periods that define when instances controlled by that schedule should be running. For more information, refer to [Schedules](scheduler-cli.md) and [Periods](period-reference.md). ### Using Infrastructure as Code (recommended) Instance Scheduler on AWS provides an AWS CloudFormation CustomResource that you can use to manage your schedules and periods using Infrastructure as Code (IaC). For information on how to manage schedules using IaC, please refer to [Manage Schedules Using Infrastructure as Code (IaC)](manage-schedules-using-infrastructure-as-code-iac.md). ### Using the Amazon DynamoDB Console and Instance Scheduler on AWS CLI **Important** If you used the custom resource to manage any schedules using IaC, you must not use the DynamoDB console or scheduler CLI to delete or modify those schedules or their periods. If you do, you will create a conflict between the stored parameters in CloudFormation and the values in the table. Also, do not use periods managed by CloudFormation in schedules created using the DynamoDB console or the scheduler CLI. When deploying the Instance Scheduler on AWS hub stack, the solution created an Amazon DynamoDB table containing several sample periods and schedules that you can use as a reference to create your own custom periods and schedules. To create a schedule in DynamoDB, modify one of the schedules in the configuration table (ConfigTable) or create a new one. To create a schedule using the CLI, first [Install the Scheduler CLI](scheduler-cli-4.md#install-the-scheduler-cli) and then use the [Available commands](scheduler-cli-4.md#available-commands). **Note** For examples of how to create several sample schedules using IaC, DynamoDB, and the InstanceScheduler CLI, please refer to [Sample schedules](sample-schedules.md). This section provides instruction and reference on how to use, monitor and update the solution as well as troubleshooting and support information. ## Tag instances for scheduling When you deployed the AWS CloudFormation template, you defined the name (tag key) for the solution’s *custom tag*. For Instance Scheduler on AWS to recognize an Amazon EC2 or Amazon RDS instance, the tag key on that instance must match this custom tag key. Therefore, it is important that you apply tags consistently and correctly to all applicable instances. You can continue to use existing [tagging best practices](https://docs.aws.amazon.com/whitepapers/latest/tagging-best-practices/tagging-best-practices.html) for your instances while using this solution. For more information, refer to [Tag your Amazon EC2 resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) and [Tagging Amazon RDS resources](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Tagging.html). On the AWS Management Console, use the [Tag Editor](https://console.aws.amazon.com/resource-groups/tag-editor/) to apply or modify tags for multiple resources at a time. You can also apply and modify tags manually in the console. Shortly after tagging a resource, an IS-ManagedBy tag will be applied to the resource by Instance Scheduler to indicate that the resource is now being managed by the scheduler. You can look for this tag to confirm that the resource has been correctly registered for scheduling. ### Setting the tag value When you apply a tag to an instance, use the tag key you defined during initial configuration (by default the tag key is Schedule) and set the tag value to the name of the schedule that should apply to the instance. If you would like to change the tag key, you can do so by [updating the solution parameters](update-the-solution.md). **Note** For Amazon RDS instances, the tag value can be from 1 to 256 Unicode characters in length and cannot be prefixed with aws:. The string can contain only the set of Unicode letters, digits, white-space, '\$1', '.', '/', '=', '', '-' (Java regex: "^([\$1\$1p\$1L\$1\$1\$1p\$1Z\$1\$1\$1p\$1N\$1\$1.:/=\$1\$1-]\$1)\$1"). For more information, refer to [Tagging Amazon RDS resources](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Tagging.html). ### EC2 instances with encrypted EBS volumes If your EC2 DB instances have EBS volumes encrypted with customer-managed KMS keys, you must give the Instance Scheduler role the KMS:CreateGrant permission to be able to start those instances. For more information, refer to [Encrypted EC2 EBS Volumes](security-1.md#encrypted-ec2-ebs-volumes). ### EC2 instances managed in License Manager If your EC2 instances are managed in AWS License Manager, you must give the Instance Scheduler role the appropriate License Manager permissions to be able to start and stop those instances. For more information, refer to [EC2 License Manager](security-1.md#ec2-license-manager). # Schedule reference Schedules specify when instances tagged with that schedule should run. Each schedule must have a unique name, which is used as the tag *value* that identifies the schedule you want to apply to the tagged resource. ## Periods Each schedule must contain at least one period that defines the time(s) the instance should run. A schedule can contain more than one period. When more than one period is used in a schedule, Instance Scheduler on AWS will apply the appropriate start action when at least one of the periods is true. For more information, refer to [Period reference](period-reference.md). ## Time zone You can also specify a time zone for the schedule. If you do not specify a time zone, the schedule will use the default time zone you specify when you launch the solution. For a list of acceptable time zone values, refer to the **TZ** column of the [List of TZ database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). ## Stop new instances field The stop\$1new\$1instances field controls whether Instance Scheduler should stop an instance the first time it is tagged for scheduling if it is currently running outside of a running period. By default, this field is set to true. When set to true, if you tag a running instance that is outside its scheduled running period, Instance Scheduler will immediately stop the instance. When set to false, Instance Scheduler will leave the instance running until the next scheduled stop time. ## Hibernate field The hibernate field allows you to use hibernation for stopped Amazon EC2 instances. If this field is set to true, your EC2 instances must use an Amazon Machine Image (AMI) that supports hibernation. For more information, refer to [Supported Linux AMIs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/hibernating-prerequisites.html#hibernation-prereqs-supported-amis) in the *Amazon EC2 User Guide*. Hibernation saves the contents from the instance memory (RAM) to your Amazon Elastic Block Store (Amazon EBS) root volume. If this field is set to true, instances are hibernated instead of stopped when the solution stops them. If you set the solution to use hibernation, but your instances are not [configured for hibernation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enabling-hibernation.html) or they do not meet the [hibernation prerequisites](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/hibernating-prerequisites.html), the solution logs a warning and the instances are stopped without hibernation. For more information, refer to [Hibernate your On-Demand Instance or Spot Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Hibernate.html) in the *Amazon EC2 User Guide*. ## Enforced field Schedules contain an enforced field that allows you to prevent an instance from being manually started outside of a running period, or manually stopped during a running period. If this field is set to true and a user manually starts an instance outside of a running period, the solution will stop the instance. If this field is set to true, it also restarts an instance if it is manually stopped during a running period. ## Retain running field The retain\$1running field prevents the solution from stopping an instance at the end of a running period if the instance was manually started before the beginning of the period. For example, if an instance with a period that runs from 9 AM to 5 PM is manually started before 9 AM, the solution will not stop the instance at 5 PM. ## Systems Manager maintenance window field (only applies to EC2 instances) The ssm-maintenance-window field allows you to automatically add AWS Systems Manager maintenance windows as a running periods to a schedule. When you specify the name of a maintenance window that exists in the same account and AWS Region as your Amazon EC2 instances, the solution will start the instance at least 10 minutes before the start of the maintenance window and stop the instance at the end of the maintenance window if no other running period specifies that the instance should run. Once the SSM Maintenance window is created and the schedule is configured with the name of the SSM maintenance window, the changes are picked up at the next scheduled run of the Lambda. For example, if you selected a frequency of 5 minutes for the scheduler Lambda to run, the maintenance window changes will be picked up by the Lambda on the next 5-minute interval. Instance Scheduler on AWS will ensure that your instances are started at least 10 minutes before the maintenance window begins. Depending on the value you set for the **Scheduling Interval** AWS CloudFormation parameter, this may result in your instance being started 10\$1interval minutes before the beginning of the maintenance window in order to guarantee that the instance starts at least 10 minutes early. For example, if you set the Scheduling Interval to 30 minutes, the scheduler will start the instance between 10-40 minutes before the beginning of the maintenance window. **Note** In order to use this feature, the Enable EC2 SSM Maintenance Windows CloudFormation parameter in the solution hub stack must be set to `yes`. For more information, refer to [AWS Systems Manager Maintenance Windows](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-maintenance.html) in the *AWS Systems Manager user guide*. ## Instance type For Amazon EC2 instances only, a schedule allows you to specify an optional desired instance type for each period in a schedule. When you specify an instance type in the period, the solution will automatically resize EC2 instances to match the requested instance type. To specify an instance type, use the syntax *@*. For example, `weekends@t2.nano`. Note that if you specify an instance type for a period that schedules Amazon EC2 instances and Amazon RDS instances, the instance type will be ignored for Amazon RDS instances. If the instance type of a running instance is different than the instance type specified for the period, the solution will stop the running instance and restart the instance with the specified instance type. For more information, refer to [Change the instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html#resize-limitations) in the *Amazon EC2 User Guide for Linux Instances*. ## Schedule definitions The Instance Scheduler on AWS configuration table in Amazon DynamoDB contains schedule definitions. A schedule definition can contain the following fields: | Field | Description | | --- | --- | | `description` | An optional description of the schedule. | | `hibernate` | Choose whether to hibernate Amazon EC2 instances running Amazon Linux. When this field is set to true, the scheduler will hibernate instances when it stops them. Note that your instances must [turn on hibernation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Hibernate.html#enabling-hibernation) and must meet the [hibernation prerequisites](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Hibernate.html#hibernating-prerequisites). | | `enforced` | Choose whether to enforce the schedule. When this field is set to true, the scheduler will stop a running instance if it is manually started outside of the running period or it will start an instance if it is stopped manually during the running period. | | `name` | The name used to identify the schedule. This name must be unique and include only alpha-numerics, hyphens (-), and underscores (\$1). | | `periods` | The name of the periods that are used in this schedule. Enter the name(s) exactly as it appears in the period name field. You can also specify an instance type for the period using the syntax *@*. For example, `weekdays@t2.large`. | | `retain_running` | Choose whether to prevent the solution from stopping an instance at the end of a running period if the instance was manually started before the beginning of the period. | | `use_maintenance_window` | Choose whether to include Amazon RDS maintenance window as a running period to an Amazon RDS instance schedule, or an AWS Systems Manager maintenance window as a running period to an Amazon EC2 instance schedule. This field is enabled by default and can be disabled by setting its value to "false" | | `ssm_maintenance_window` | Choose whether to add AWS Systems Manager maintenance window(s) as additional running period for this schedule. Accepts a StringSet of maintenance window names that will be matched against the names of windows in the same account/region as scheduled EC2 instances. **Note:** This feature only applies to EC2 instances. | | `stop_new_instances` | Choose whether to stop an instance the first time it is tagged if it is running outside of the running period. By default, this field is set to true. | | `timezone` | The time zone the schedule will use. If no time zone is specified, the default time zone (UTC) is used. For a list of acceptable time zone values, refer to the **TZ** column of the [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). | | `use_metrics` | Choose whether to turn on CloudWatch metrics at the schedule level. This field overwrites the CloudWatch metrics setting you specified at deployment. **Note:** Enabling this feature will incur charges of \$10.90/month per schedule or scheduled service. | # Period reference Periods contain conditions that allow you to set the specific hours, days, and months an instance will run. A period can contain multiple conditions, but all conditions must be true for the Instance Scheduler on AWS to apply the appropriate start or stop action. ## Start and stop times The `begintime` and `endtime` fields define when the Instance Scheduler on AWS will start and stop instances. If you specify a start time only, the instance must be stopped manually. Note that if you specify a value in the [weekdays](#days-of-the-week) field, the solution uses that value to determine when to stop the instance. For example, if you specify a `begintime` of 9 AM with no `endtime` and a weekdays value of Monday through Friday, the instance will be stopped at 11:59 PM on Friday unless you have scheduled an adjacent period. Similarly, if you only specify a stop time, the instance must be started manually. If you don’t specify either time, this solution uses the days of the week, days of the month, or months rules to start and stop instances at the beginning/end of each day as appropriate. The `begintime` and `endtime` values for your period must be in the time zone specified in the schedule. If you do not specify a time zone in the schedule, the solution will use the time zone specified when you launch the solution. If your schedule contains multiple periods, we recommend that you always specify both a `begintime` and `endtime` in your periods. If you start an instance before the specified start time, the instance will run until the end of the running period. For example, a user might define a period that starts an instance daily at 9 AM and stops that instance at 5 PM. **9-5 scheduled start and stop** ![\[image6\]](http://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/images/image6.png) If you manually start that instance at 5 AM, the solution will stop the instance at 5 PM. If you use the [retain running field](schedule-reference.md#retain-running-field), the solution will not stop the instance at 5 PM. **5 AM scheduled stop** ![\[image7\]](http://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/images/image7.png) If you stop an instance before the specified stop time, the instance will not run until the beginning of the next running period. Continuing from the previous example, if the user stops the instance at 1 PM on Wednesday, the solution will not start the instance until 9 AM on Thursday. **Timeline showing scheduled start at 9 am, manual stop at 1 pm, and scheduled stop at 5 pm.** ![\[image8\]](http://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/images/image8.png) ### Adjacent periods The solution will not stop running instances if the schedule contains two adjacent running periods. For example, if you have a schedule with one period with an `endtime` of 11:59 PM and another period with a `begintime` of midnight the following day, the solution will not stop running instances, if there are no `weekdays`, `monthdays`, or `months` rules that stop the instances. To implement a schedule that runs instances from 9 AM Monday to 5 PM Friday, the solution requires three periods. The first period runs applicable instances from 9 AM to 11:59 PM Monday. The second period runs the instances from midnight Tuesday to 11:59 PM Thursday. The third period runs the instances from midnight Friday to 5 PM Friday. For more information, refer to [Sample schedules](sample-schedules.md). ## Days of the week The `weekdays` field defines which days during the week an instance will run. You can specify a list of days, a range of days, the n^th^ occurrence of that day in a month, or the last occurrence of that day in a month. The solution supports abbreviated day names (Mon) and numbers (0). ## Days of the month The `monthdays`, field defines which days during the month an instance will run. You can specify a list of days, a range of days, every n^th^ day of the month, the last day of the month, or the nearest weekday to a specific date. ## Months The `months` field defines which months an instance will run. You can specify a list of months, a range of months, or every n^th^ month. The solution supports abbreviated month names (Jan) and numbers (1). ## Period definitions The Instance Scheduler on AWS configuration table in Amazon DynamoDB contains period definitions. A period definition can contain the following fields. Note that some fields support [Cron non-standard characters](https://en.wikipedia.org/wiki/Cron#Non-Standard_Characters). **Important** You must specify at least one of the following items: begintime, endtime, weekdays, months, or monthdays. | Field | Description | | --- | --- | | `begintime` | The time, in **HH:MM** format, that the instance will start. | | `description` | An optional description of the period. | | `endtime` | The time, in **HH:MM** format, that the instance will stop. | | `months` | Enter a comma-delimited list of months, or a hyphenated range of months, during which the instance will run. For example, enter `jan, feb, mar` or `1, 2, 3` to run an instance during those months. Or, you can enter `jan-mar` or `1-3`. You can also schedule an instance to run every n^th^ month or every n^th^ month in a range. For example, enter `Jan/3` or `1/3` to run an instance every third month starting in January. Enter `Jan-Jul/2` to run every other month from January to July. | | `monthdays` | Enter a comma-delimited list of days of the month, or a hyphenated range of days, during which the instance will run. For example, enter `1, 2, 3` or `1-3` to run an instance during the first three days of the month. You can also enter multiple ranges. For example, enter `1-3`, `7-9` to run an instance from the 1^st^ to the 3^rd^ and the 7^th^ through the 9^th^. You can also schedule an instance to run every n^th^ day of the month or every n^th^ day of the month in a range. For example, enter `1/7` to run an instance every seventh day starting on the 1^st^. Enter `1-15/2` to run an instance every other day from the 1^st^ to the 15^th^. Enter `L` to run an instance on the last day of the month. Enter a date and W to run an instance on the nearest weekday to the specified date. For example, enter `15W` to run an instance on the nearest weekday to the 15^th^. | | `name` | The name used to identify the period. This name must be unique and include only alpha-numerics, hyphens (-), and underscores (\$1). | | `weekdays` | Enter a comma-delimited list of days of the week, or a range of days of the week, during which the instance will run. For example, enter `0, 1, 2` or `0-2` to run an instance Monday through Wednesday. You can also enter multiple ranges. For example, enter `0-2`, `4-6` to run an instance every day except Thursday. You can also schedule an instance to run every n^th^ occurrence of a weekday in the month. For example, enter `Mon#1` or `0#1` to run an instance the first Monday of the month. Enter a day and L to run an instance on the last occurrence of that weekday in the month. For example, enter `friL` or `4L` to run an instance on the last Friday of the month. | When a period contains multiple conditions, note that all conditions must be true for Instance Scheduler on AWS to apply the appropriate action. For example, a period that contains a `weekdays` field with a value of `Mon#1` and a months field with a value of `Jan/3` will apply the action on the first Monday of the quarter. # Sample schedules Instance Scheduler on AWS allows you to automatically start and stop Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Relational Database Service (Amazon RDS) instances. The following section provides some example schedules that can be adapted to many common use cases. ## Standard 9-5 working hours This schedule shows how to run instances on weekdays from 9 AM to 5 PM in London. ### Periods This period will start instances at 9 AM and stop instances at 5 PM on weekdays (Mon-Fri). | Field | Type | Value | | --- | --- | --- | | `begintime` | `String` | `09:00` | | `endtime` | `String` | `16:59` | | `name` | `String` | `weekdays-9-5` | | `weekdays` | `StringSet` | `mon-fri` | ### Schedule The schedule name provides the tag value that must be applied to instances and the timezone that will be used. | Field | Type | Value | | --- | --- | --- | | `name` | `String` | `london-working-hours` | | `periods` | `StringSet` | `weekdays-9-5` | | `timezone` | `String` | `Europe/London` | ### Instance tag To apply this schedule to instances, you must add the `Schedule=london-working-hours` tag to the instances. If you change the default tag name in the AWS CloudFormation **Instance Scheduler tag name** parameter, your tag will be different. For example, if you entered `Sked` as your tag name, your tag will be `Sked=london-working-hours`. For more information, refer to [Tag your resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-ec2-resources-table) in the *Amazon Elastic Compute Cloud User Guide*. ### Scheduler CLI To configure the above schedule using the [Instance Scheduler CLI](scheduler-cli-4.md) use the following commands: ``` scheduler-cli create-period --stack --name weekdays-9-5 --weekdays mon-fri --begintime 9:00 --endtime 16:59 scheduler-cli create-schedule --stack --name london-working-hours --periods weekdays-9-5 --timezone Europe/London Europe/London ``` ### Custom resource The following CloudFormation template will create the above schedule using the [schedule custom resource](manage-schedules-using-infrastructure-as-code-iac.md). To deploy this template, you will need to provide the **ServiceInstanceScheduleServiceToken** ARN that can be found in the AWS CloudFormation console by selecting the [previously deployed Instance Scheduler Hub Stack](step-1-launch-the-instance-scheduler-hub-stack.md) and then select **Outputs**. ``` AWSTemplateFormatVersion: 2010-09-09 Parameters: ServiceInstanceScheduleServiceTokenARN: Type: String Description: (Required) service token arn taken from InstanceScheduler outputs Metadata: 'AWS::CloudFormation::Designer': {} Resources: LondonWorkingWeek: Type: 'Custom::ServiceInstanceSchedule' Properties: NoStackPrefix: 'True' Name: london-working-hours Description: run instances from 9am to 5pm in London on weekdays ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN Timezone: Europe/London Periods: - Description: 9am to 5pm on weekdays BeginTime: '09:00' EndTime: '16:59' WeekDays: mon-fri ``` ## Stop instances after 5 PM Instances can be started freely at any time during the day and this schedule will ensure that a stop command is automatically sent to them at 5 PM ET every day. ### Periods This period will stop instances at 5 PM every day. | Field | Type | Value | | --- | --- | --- | | `endtime` | `String` | `16:59` | | `name` | `String` | `stop-at-5` | ### Schedule The schedule name provides the tag value that must be applied to instances and the timezone that will be used. | Field | | Value | | --- | --- | --- | | `name` | `String` | `stop-at-5-new-york` | | `periods` | `StringSet` | `stop-at-5` | | `timezone` | `String` | `America/New York` | ### Instance tag To apply this schedule to instances, you must add the `Schedule=stop-at-5-new-york` tag to the instances. If you changed the default tag name in the AWS CloudFormation **Instance Scheduler tag name** parameter, your tag will be different. For example, if you entered `Sked` as your tag name, your tag will be `Sked=stop-at-5-new-york` . For more information, refer to [Tag your resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-ec2-resources-table) in the *Amazon Elastic Compute Cloud User Guide*. ### Scheduler CLI To configure the above schedule using the [Instance Scheduler CLI](scheduler-cli-4.md), use the following commands: ``` scheduler-cli create-period --stack --name stop-at-5 --endtime 16:59 scheduler-cli create-schedule --stack --name stop-at-5-new-york --periods stop-at-5 --timezone America/New_York ``` ### Custom resource The following CloudFormation template will create the above schedule using the [schedule custom resource](manage-schedules-using-infrastructure-as-code-iac.md). To deploy this template, you will need to provide the **ServiceInstanceScheduleServiceToken** ARN that can be found in the AWS CloudFormation console by clicking on the [previously deployed Instance Scheduler Hub Stack](step-1-launch-the-instance-scheduler-hub-stack.md) and selecting **Outputs**. ``` AWSTemplateFormatVersion: 2010-09-09 Parameters: ServiceInstanceScheduleServiceTokenARN: Type: String Description: (Required) service token arn taken from InstanceScheduler outputs Metadata: 'AWS::CloudFormation::Designer': {} Resources: StopAfter5: Type: 'Custom::ServiceInstanceSchedule' Properties: NoStackPrefix: 'True' Name: stop-at-5-new-york Description: stop instances at 5pm ET every day ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN Timezone: America/New_York Periods: - Description: stop at 5pm EndTime: '16:59' ``` ## Stop instances over the weekend This schedule shows how to run instances from Monday 9 AM ET to Friday 5 PM ET. Since Monday and Friday are not full days, this schedule includes three periods to accommodate: Monday, Tuesday-Thursday, and Friday. ### Periods The first period starts tagged instances at 9 AM Monday and stops at midnight. This period includes the following fields and values. | Field | Type | Value | | --- | --- | --- | | `begintime` | `String` | `09:00` | | `endtime` | `String` | `23:59` | | `name` | `String` | `mon-start-9am` | | `weekdays` | `StringSet` | `mon` | The second period runs tagged instances all day Tuesday through Thursday. This period includes the following fields and values. | Field | | Value | | --- | --- | --- | | `name` | `String` | `tue-thu-full-day` | | `weekdays` | `StringSet` | `tue-thu` | The third period stops tagged instances at 5 PM on Friday. This period includes the following fields and values. | Field | | Value | | --- | --- | --- | | `begintime` | `String` | `00:00` | | `endtime` | `String` | `16:59` | | `name` | `String` | `fri-stop-5pm` | | `weekdays` | `StringSet` | `fri` | ### Schedule The schedule combines the three periods into the schedule for tagged instances. The schedule includes the following fields and values. | Field | | Value | | --- | --- | --- | | `name` | `String` | mon-9am-fri-5pm | | `periods` | `StringSet` | mon-start-9am,tue-thu-full-day,fri-stop-5pm | | `timezone` | `String` | America/New\$1York | ### Instance tag To apply this schedule to instances, you must add the `Schedule=mon-9am-fri-5pm` tag to the instances. Note that if you changed the default tag name in the AWS CloudFormation **Instance Scheduler tag name** parameter, your tag will be different. For example, if you entered Sked as your tag name, your tag will be `Sked=mon-9am-fri-5pm`. For more information, refer to [Tag your resources](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-ec2-resources-table) in the *Amazon Elastic Compute Cloud User Guide*. ### Scheduler CLI To configure the above schedule using the [Instance Scheduler CLI](scheduler-cli-4.md), use the following commands: ``` scheduler-cli create-period --stack --name mon-start-9am --weekdays mon --begintime 9:00 --endtime 23:59 scheduler-cli create-period --stack --name tue-thu-full-day --weekdays tue-thu scheduler-cli create-period --stack --namefri-stop-5pm --weekdays fri --begintime 0:00 --endtime 17:00 scheduler-cli create-schedule --stack --name mon-9am-fri-5pm --periods mon-start-9am,tue-thu-full-day,fri-stop-5pm -timezone America/New_York ``` ### Custom resource The following CloudFormation template will create the above schedule using the [schedule custom resource](manage-schedules-using-infrastructure-as-code-iac.md). To deploy this template, you will need to provide the **ServiceInstanceScheduleServiceToken** ARN that can be found in the AWS CloudFormation console by selecting the [previously deployed Instance Scheduler Hub Stack](step-1-launch-the-instance-scheduler-hub-stack.md) and then select **Outputs**. ``` AWSTemplateFormatVersion: 2010-09-09 Parameters: ServiceInstanceScheduleServiceTokenARN: Type: String Description: (Required) service token arn taken from InstanceScheduler outputs Metadata: 'AWS::CloudFormation::Designer': {} Resources: StopOnWeekends: Type: 'Custom::ServiceInstanceSchedule' Properties: NoStackPrefix: 'True' Name: mon-9am-fri-5pm Description: start instances at 9am on monday and stop them at 5pm on friday ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN Timezone: America/New_York Periods: - Description: 9am monday start BeginTime: '09:00' EndTime: '23:59' WeekDays: mon - Description: all day tuesday-thursday WeekDays: tue-thu - Description: 5pm friday stop BeginTime: '00:00' EndTime: '16:59' WeekDays: fri ``` ## Solution resources The following resources are created as part of the Instance Scheduler on AWS stack. | Resource name | Type | Description | | --- | --- | --- | | **Main** | `AWS::Lambda::Function` | Instance Scheduler on AWS Lambda function. | | **Scheduler Config Helper** | `Custom::ServiceSetup` | Stores global configuration settings in Amazon DynamoDB. | | **Scheduler Invoke Permission** | `AWS::Lambda::Permission` | Allows the Amazon CloudWatch event to invoke the Instance Scheduler’s AWS Lambda function. | | **Scheduler Logs** | `AWS::Logs::LogGroup` | CloudWatch Log Group for Instance Scheduler. | | **Scheduler Policy** | `AWS::IAM::Policy` | Policy that allows scheduler to perform start and stop actions, change Amazon EC2 instance attributes, set tags, and access scheduler resources. | | **Scheduler Rule** | `AWS::Events::Rule` | Amazon EventBridge event rule that invokes the scheduler’s Lambda function. | | **Configuration Metrics Event Rule** | `AWS::Events::Rule` | Amazon EventBridge event rule that periodically invokes the configuration description anonymized metrics function. Disabled when anonymized metrics are disabled. | | **State Table** | `AWS::DynamamoDB::Table` | DynamoDB table that stores last desired state of instances. | | **Config Table** | `AWS::DynamamoDB::Table` | DynamoDB table that stores global configuration, schedule, and period data. | | **Instance Scheduler SNS Topic** | `AWS::SNS::Topic` | Sends warning and error messages to subscribed email addresses. | # Scheduler CLI The Instance Scheduler on AWS command line interface (CLI) allows you to configure schedules and periods, and estimate cost savings for a given schedule. ## Prerequisites The CLI in this solution requires Python 3.8\$1 and the latest version of boto3. ## Credentials To use the scheduler CLI, you must have credentials for the AWS CLI. For more information, refer to [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html) in the *AWS CLI User Guide*. Your credentials must have the following permissions: + `lambda:InvokeFunction` - To invoke the InstanceSchedulerMain function in the scheduler stack, and to update the schedule and period information in the scheduler configuration database from the command line + `cloudformation:DescribeStackResource` - To retrieve the physical resource ID of the AWS Lambda function from the stack to handle the CLI request Requests made by the scheduler CLI and responses are logged in the `AdminCliRequestHandler-yyyymmdd` log stream. **Note** If you specify a profile using the profile-name argument, the profile you specify must have these permissions. For more information about the profile-name argument, refer to [Common Arguments](#common-arguments). ## Install the Scheduler CLI 1. [Download](https://s3.amazonaws.com/solutions-reference/instance-scheduler-on-aws/latest/instance_scheduler_cli.zip) the scheduler CLI package ** *(instance\$1scheduler\$1cli.zip)* ** and place it in a directory on your computer. **Important** The installation will fail if you don’t place the files into their own directory, and then install them from that directory. 1. Unzip the zip archive into its own directory (instance\$1scheduler\$1cli). 1. From the same directory in which you placed the unzipped CLI package, install the scheduler-cli to your environment: **Note** Scheduler-CLI requires Python 3.8 or above and the latest versions of pip and boto3. If you do not have all of these installed on your local machine, please refer to [pip’s official documentation](https://pip.pypa.io/en/stable/getting-started/) for installation instructions before attempting to install the Scheduler-CLI. ``` pip install --no-index --find-links=instance_scheduler_cli instance_scheduler_cli ``` 1. Verify the installation succeeded with: ``` scheduler-cli --help ``` **Note** If preferred, an [sdist of the CLI](https://s3.amazonaws.com/solutions-reference/instance-scheduler-on-aws/latest/instance_scheduler_cli-3.0.0.tar.gz) and can be installed using the same process as above. ## Command structure The scheduler CLI uses a multipart structure on the command line. The next part specifies the scheduler CLI python script. The scheduler CLI has commands that specify the operations to perform on periods and schedules. The specific arguments for an operation can be specified on the command line in any order. ``` scheduler-cli ``` ## Common arguments The scheduler CLI supports the following arguments that all commands can use: | Argument | Description | | --- | --- | | `--stack [replaceable]` | The name of the scheduler stack. **Important:** This argument is required for all commands. | | `--region [replaceable]` | The name of the region where the scheduler stack is deployed. **Note:** You must use this argument when the default configuration and credential files are not installed in the same region as the solution stack. | | `--profile-name [replaceable] ` | The name of the profile to use to run commands. If no profile name is specified, the default profile is used. | | `--query` | A JMESPath expression that controls the command output. For more information on controlling output, refer to [Controlling Command Output from the AWS Command Line Interface](https://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html) in the *AWS CLI User Guide*. | | `--help` | Shows valid commands and arguments for the scheduler CLI. When used with a specific command, it shows valid subcommands and arguments for that command. | | `--version` | Shows the version number of the scheduler CLI. | ## Available commands + [create-period](#create-period) + [create-schedule](#create-schedule) + [delete-period](#delete-period) + [delete-schedule](#delete-schedule) + [describe-periods](#describe-periods) + [describe-schedules](#describe-schedules) + [describe-schedule-usage](#describe-schedule-usage) + [update-period](#update-period) + [update-schedule](#update-schedule) + [help](#help) ## create-period ### Description Creates a period. A period must contain at least one of the following items: `begintime`, `endtime`, `weekdays`, `months`, or `monthdays`. ### Arguments `--name` + The name of the period Type: String Required: Yes `--description` + A description of the period Type: String Required: No `--begintime` + The time when the running period starts. If `begintime` and `endtime` are not specified, the running period is 00:00 - 23:59. Type: String Constraints: `H:MM` or `HH:MM` format Required: No `--endtime` + The time when the running period stops. If `begintime` and `endtime` are not specified, the running period is 00:00 - 23:59. Type: String Constraints: `H:MM` or `HH:MM` format Required: No `--weekdays` + The days of the week for the period Type: String Constraints: Comma-delimited list of abbreviated day names (mon) or numbers (0). Use - to specify a range. Use / to specify every n^th^ day of the week. Required: No `--months` + The months of the period Type: String Constraints: Comma-delimited list of abbreviated months names (jan) or numbers (1). Use - to specify a range. Use / to specify every n^th^ month. Required: No `--monthdays` + The days of the month for the period Type: String Constraints: Comma-delimited list of abbreviated months names (jan) or numbers (1). Use - to specify a range. Use / to specify every n^th^ day of the month. Required: No ### Example ``` $ scheduler-cli create-period --name "weekdays" --begintime 09:00 --endtime 18:00 --weekdays mon-fri --stack Scheduler { "Period": { "Name": "weekdays", "Endtime": "18:00", "Type": "period", "Begintime": "09:00", "Weekdays": [ "mon-fri" ] } } ``` ## create-schedule ### Description Creates a schedule. ### Arguments `--name` + The name of the schedule Type: String Required: Yes `--description` + A description of the schedule Type: String Required: No `--enforced` + Enforces the scheduled state for the instance Required: No `--use-metrics` + Collect Amazon CloudWatch metrics Required: No `--periods` + A list of running periods for the schedule. If multiple periods are specified, the solution will start an instance if one of the periods evaluates to `true`. Type: String Constraints: Comma-delimited list of periods. Use `@[replaceable]` to specify an instance type for a period. For example, `weekdays@t2.large`. Required: Yes `--retain-running` + Prevents an instance from being stopped by the solution at the end of a running period, if the instance was manually started before the beginning of the period. Required: No `--ssm-maintenance-window` + Adds an AWS Systems Manager maintenance window as a running period to an Amazon EC2 instance schedule. Type: String Required: No `--do-not-stop-new-instances` + Do not stop an instance the first time it is tagged if it is running outside of a running period Required: No `--timezone` + The time zone the schedule will use Type: Array of strings Required: No (If this argument is not used, the default time zone from main solution stack is used.) `--use-maintenance-window` + Adds an Amazon RDS maintenance window as a running period to an Amazon RDS instance schedule, or an AWS Systems Manager maintenance window as a running period to an Amazon EC2 instance schedule Type: true/false Required: No (default true) ### Example ``` $ scheduler-cli create-schedule --name LondonOfficeHours --periods weekdays,weekends --timezone Europe/London --stack Scheduler { "Schedule": { "Enforced": false, "Name": "LondonOfficeHours", "StopNewInstances": true, "Periods": [ "weekends", "weekdays" ], "Timezone": "Europe/London", "Type": "schedule" } } ``` ## delete-period `--name` + The name of the applicable period Type: String Required: Yes **Important** If the period is used in existing schedules, you must remove it from those schedules *before* you delete it. **Example** ``` $ scheduler-cli delete-period --name weekdays --stack Scheduler { "Period": "weekdays" } ``` ## delete-schedule ### Description Deletes an existing schedule ### Arguments `--name` + The name of the applicable schedule Type: String Required: Yes ### Example ``` $ scheduler-cli delete-schedule --name LondonOfficeHours --stack Scheduler { "Schedule": "LondonOfficeHours" } ``` ## describe-periods ### Description Lists the configured periods for the Instance Scheduler stack ### Arguments `--name` + The name of a specific period you want described Type: String Required: No ### Example ``` $ scheduler-cli describe-periods --stack Scheduler { "Periods": [ { "Name": "first-monday-in-quarter", "Months": [ "jan/3" ], "Type": "period", "Weekdays": [ "mon#1" ], "Description": "Every first Monday of each quarter" }, { "Description": "Office hours", "Weekdays": [ "mon-fri" ], "Begintime": "09:00", "Endtime": "17:00", "Type": "period", "Name": "office-hours" }, { "Name": "weekdays", "Endtime": "18:00", "Type": "period", "Weekdays": [ "mon-fri" ], "Begintime": "09:00" }, { "Name": "weekends", "Type": "period", "Weekdays": [ "sat-sun" ], "Description": "Days in weekend" } ] } ``` ## describe-schedules ### Description Lists the configured schedules for the Instance Scheduler stack. ### Arguments `--name` + The name of a specific schedule you want described Type: String Required: No ### Example ``` $ scheduler-cli describe-schedules --stack Scheduler { "Schedules": [ { "OverrideStatus": "running", "Type": "schedule", "Name": "Running", "UseMetrics": false }, { "Timezone": "UTC", "Type": "schedule", "Periods": [ "working-days@t2.micro", "weekends@t2.nano" ], "Name": "scale-up-down" }, { "Timezone": "US/Pacific", "Type": "schedule", "Periods": [ "office-hours" ], "Name": "seattle-office-hours" }, { "OverrideStatus": "stopped", "Type": "schedule", "Name": "stopped", "UseMetrics": true } ] } ``` ## describe-schedule-usage ### Description Lists all the periods running within a schedule and calculates the billing hours for instances. Use this command to simulate a schedule to calculate potential savings, and running periods after creating or updating a schedule. ### Arguments `--name` + The name of the applicable schedule Type: String Required: Yes `--startdate` + The start date of the period used for calculation. The default date is the current date. Type: String Required: No `--enddate` + The end date of the period used for calculation. The default date is the current date. Type: String Required: No ### Example ``` $ scheduler-cli describe-schedule-usage --stack InstanceScheduler --name seattle-office-hours { "Usage": { "2017-12-04": { "BillingHours": 8, "RunningPeriods": { "Office-hours": { "Begin": "12/04/17 09:00:00", "End": "12/04/17 17:00:00", "BillingHours": 8, "BillingSeconds": 28800 } }, "BillingSeconds": 28800 } }, "Schedule": "seattle-office-hours" ``` ## update-period ### Description Updates an existing period ### Arguments The `update-period` command supports the same arguments as the `create-period` command. For more information on the arguments, refer to the [create period command](#create-period). **Important** If you do not specify an argument, that argument will be removed from the period. ## update-schedule ### Description Updates an existing schedule ### Arguments The `update-schedule` command supports the same arguments as the `create-schedule` command. For more information on the arguments, refer to the [create schedule command](#create-schedule). **Important** If you do not specify an argument, that argument will be removed from the schedule. ## help ### Description Displays a list of valid commands and arguments for the scheduler CLI. ### Example ``` $ scheduler-cli --help usage: scheduler-cli [-h] [--version] {create-period,create-schedule,delete-period,delete-schedule,describe-periods,describe-schedule-usage,describe-schedules,update-period,update-schedule} ... optional arguments: -h, --help show this help message and exit --version show program's version number and exit subcommands: Valid subcommands {create-period,create-schedule,delete-period,delete-schedule,describe-periods,describe-schedule-usage,describe-schedules,update-period,update-schedule} Commands help create-period Creates a period create-schedule Creates a schedule delete-period Deletes a period delete-schedule Deletes a schedule describe-periods Describes configured periods describe-schedule-usage Calculates periods and billing hours in which instances are running describe-schedules Described configured schedules update-period Updates a period update-schedule Updates a schedule ``` When used with a specific command, the `--help` argument shows valid subcommands and arguments for that command. ### Specific command example ``` $ scheduler-cli describe-schedules --help usage: scheduler-cli describe-schedules [-h] [--name NAME] [--query QUERY] [--region REGION] --stack STACK optional arguments: -h, --help show this help message and exit --name NAME Name of the schedule --query QUERY JMESPath query to transform or filter the result --region REGION Region in which the Instance Scheduler stack is deployed --stack STACK, -s STACK Name of the Instance Scheduler stack ``` # Update global configuration settings When you first deployed Instance Scheduler’s Hub template in AWS CloudFormation, a number of global configuration settings were selected as parameter inputs. These global configuration parameters can be updated at any time within the CloudFormation console. To update the global configuration of Instance Scheduler, log into the account/region containing your hub deployment and go to the AWS CloudFormation console. Find the Instance Scheduler Hub Stack and select ** *Update → Use Existing Template* **. Update any global configuration parameters you would like to change, and then select ** *next → next → submit* ** to perform a CloudFormation update of the relevant solution resources. # Manage schedules using Infrastructure as Code (IaC) **Important** Deploy schedules using a separate template after the hub stack deployment is complete. Instance Scheduler on AWS provides a custom resource (`ServiceInstanceSchedule`) that you can use to configure and manage schedules through AWS CloudFormation. The custom resource uses PascalCase keys for the same data as the Instance Scheduler config table in Amazon DynamoDB (see template below for examples). For more information on the fields for schedules, refer to [Schedule Definitions](schedule-reference.md#schedule-definitions). For more information on the fields for periods, refer to [Period Definitions](period-reference.md#period-definitions). When you use the custom resource to create a schedule, the name of that schedule is the logical resource name of the custom resource by default. To specify a different name, use the Name property of the custom resource. The solution also adds the stack name to the schedule name as a prefix by default. If you do not want to add the stack name as a prefix, use the `NoStackPrefix` property. When you use the Name and `NoStackPrefix` properties, make sure you choose unique schedule names. If a schedule with the same name already exists, the resource will not be created or updated. To get started managing schedules using IaC, copy and paste the following sample template and customize as many or as few schedules as you like. Save the file as a .template file (for example: `my-schedules.template`), and then deploy your new template using AWS CloudFormation. For examples of completed schedule templates, refer to [Sample Schedules](period-reference.md#period-definitions). ``` AWSTemplateFormatVersion: 2010-09-09 Parameters: ServiceInstanceScheduleServiceTokenARN: Type: String Description: (Required) service token arn taken from InstanceScheduler outputs Metadata: 'AWS::CloudFormation::Designer': {} Resources: SampleSchedule1: Type: 'Custom::ServiceInstanceSchedule' Properties: ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN #do not edit this line NoStackPrefix: 'False' Name: my-renamed-sample-schedule Description: a full sample template for creating cfn schedules showing all possible values Timezone: America/New_York Enforced: 'True' Hibernate: 'True' RetainRunning: 'True' StopNewInstances: 'True' UseMaintenanceWindow: 'True' SsmMaintenanceWindow: 'my_window_name' Periods: - Description: run from 9-5 on the first 3 days of March BeginTime: '9:00' EndTime: '17:00' InstanceType: 't2.micro' MonthDays: '1-3' Months: '3' - Description: run from 2pm-5pm on the weekends BeginTime: '14:00' EndTime: '17:00' InstanceType: 't2.micro' WeekDays: 'Sat-Sun' SampleSchedule2: Type: 'Custom::ServiceInstanceSchedule' Properties: ServiceToken: !Ref ServiceInstanceScheduleServiceTokenARN #do not edit this line NoStackPrefix: 'True' Description: a sample template for creating simple cfn schedules Timezone: Europe/Amsterdam Periods: - Description: stop at 5pm every day EndTime: '17:00' ``` When deploying the template, you must provide ServiceTokenARN for your deployment of Instance Scheduler on AWS. This ARN can be found within CloudFormation by navigating to your deployed Instance Scheduler stack, selecting **Outputs**, and looking for `ServiceInstanceScheduleServiceToken.` **Important** Do not use the DynamoDB console or scheduler CLI to delete or modify schedules and periods that were configured using the custom resource. If you do, you will create a conflict between the stored parameters in the stack and the values in the table. Also, do not use periods configured using the custom resource in schedules created using the DynamoDB console or the scheduler CLI. Before you delete the main Instance Scheduler stack, you must delete all additional stacks that contain schedules and periods created using the custom resource because the custom resource stacks contain dependencies on the main stack’s DynamoDB table. In the configuration DynamoDB table, schedules and periods that were configured with the custom resource can be identified by the **configured\$1in\$1stack** attribute. The attribute contains the Amazon Resource Name of the stack that was used to create the item. # Handling EC2 Insufficient Capacity Errors When Instance Scheduler fails to start an instance due to insufficient capacity, its default behavior is to issue a start-failed event (see [EventBridge Events](monitor-the-solution.md#monitoring-eventbridge-events)) and try again on the next scheduling interval. Alternatively, Instance Scheduler can be configured to resize your instance to alternative instance types before retrying the start operation. This feature helps improve instance availability in capacity constrained environments. ## Configuration To enable alternate instance types for an EC2 instance, add the `IS-PreferredInstanceTypes` tag to the instance with a comma-separated list of instance types in order of preference (most preferred first): ``` IS-PreferredInstanceTypes: t3.medium,t3.large,m5.large ``` ## How it works The alternate instance types list is provided in order of preference, with the first type being the most preferred. When Instance Scheduler attempts to start an EC2 instance: 1. If the instance is not currently the most preferred size, attempts to resize it to the most preferred size before starting 1. If the start operation succeeds, no further alternates are attempted 1. If the start operation fails due to insufficient capacity: 1. Attempts to resize to the next alternate instance type in the list 1. Retries the start operation 1. If still unsuccessful, tries the next alternate type 1. Continues until successful or all alternates are exhausted ## Requirements and limitations **Instance compatibility**: Alternate instance types must be compatible with the instance’s current configuration (AMI, subnet, security groups, etc.). For more information, refer to [Change the instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-resize.html#resize-limitations) in the *Amazon EC2 User Guide*. **Tag format**: The `IS-PreferredInstanceTypes` tag value must be a comma-separated list of valid EC2 instance types. ## Example For an instance originally configured as `t3.small`, you might configure: ``` Schedule: office-hours IS-PreferredInstanceTypes: t3.small,t3.medium,t3.large,m5.large ``` If the `t3.small` instance fails to start due to capacity issues, Instance Scheduler will attempt to resize and start the instance as `t3.medium`, then `t3.large`, then `m5.large` until successful or all options are exhausted. # EC2 Auto Scaling Group Scheduling Instance Scheduler on AWS supports scheduling of EC2 Auto Scaling groups (ASGs) using Scheduled Scaling Actions. This differs from the implementation of EC2/RDS scheduling and will be further explained in this section Refer to [Scheduled scaling for Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-scheduled-scaling.html) for more information on Scheduled Scaling Actions. ## ASG scheduling overview ASGs can be scheduled by applying a schedule tag as described in [Tagging instances for scheduling](operator-guide.md#tag-instances-for-scheduling) ## Definition of Running/Stopped for ASGs When an Auto Scaling Group is configured, a user specifies a minimum, desired, and maximum capacity for that ASG. Instance Scheduler refers to these values as the min-desired-max or MDM of an ASG. The ** *running state* ** of an ASG is defined using the `IS-MinDesiredMax` control tag. This tag should contain the desired MDM values in the format `min,desired,max` (for example: `1,3,5`). If the `IS-MinDesiredMax` tag is not provided when the ASG is first tagged for scheduling, it will be automatically generated from the current size of the ASG at the time of tagging. For all ASGs the ** *stopped state* ** is defined as an MDM of 0-0-0. ## ASG Start/Stop Behavior When Instance Scheduler starts or stops an ASG, it modifies the ASG’s capacity settings: **Starting an ASG**: Sets the minimum, desired, and maximum capacity to the values defined in the `IS-MinDesiredMax` tag (or the automatically generated values from when the ASG was first tagged). **Stopping an ASG**: Sets the minimum, desired, and maximum capacity to 0-0-0, which causes all instances in the ASG to be terminated. ### Limitations ASG scheduling is performed by converting Instance Scheduler on AWS schedules into scheduled scaling rules compatible with the ASG service. This translation works best for simple single-period schedules that do not use complex cron expressions. The following schedule features are not supported for ASG scheduling: + Advanced schedule flags such as *enforced* and *retain running.* + N-th weekday, nearest weekday, and last weekday expressions in periods. + Multi-period schedules with immediately adjacent or overlapping periods. + When configuring scheduled scaling actions for multi-period schedules, Instance Scheduler on AWS directly translate the beginning/end of periods to start/stop actions for the ASG even when another overlapping or adjacent period would normally cause that action to be skipped. # Monitor the solution ## Logging and notifications Instance Scheduler uses structured logging optimized for CloudWatch Logs Insights queries. This solution logs processing information for each tagged instance, the results of the period evaluation for the instance, the desired state of the instance during that period, the applied action, and debugging messages. Logs are written to Amazon CloudWatch Logs in two log groups: `{stackName}-{namespace}-administrative-logs` Logs for resource registration and deregistration, custom resource operations, CLI requests, and other administrative activities. `{stackName}-{namespace}-scheduling-logs` Logs for scheduling operations including orchestration and request handler execution. Warning and error logs are also forwarded to a solution-created Amazon SNS topic, which can be configured to send messages to a subscribed email address. You can find the name of the Amazon SNS topic in the **Outputs** tab of the solution stack. ## Informational tags When informational tagging is enabled (the default), Instance Scheduler writes tags directly to managed resources to provide at-a-glance visibility into the solution’s scheduling activity. These tags are applied using the AWS Resource Groups Tagging API and are updated each time the scheduler processes a resource. You can enable or disable this feature using the **Enable informational tagging** parameter on the hub stack. For more information, refer to [Update global configuration settings](update-global-configuration-settings.md). ### Informational tag keys The following tags are written to managed resources: | Tag key | Description | | --- | --- | | `IS-ManagedBy` | The ARN of the Instance Scheduler hub stack managing this resource. Applied when a resource is first registered for scheduling and on each subsequent scheduling action. | | `IS-LastAction` | The last scheduling action taken on the resource, along with a UTC timestamp. For example: `Started 2025-06-15 09:00:00 UTC` or `Stopped 2025-06-15 17:00:00 UTC`. This tag is only updated when the scheduler actively starts or stops a resource (not when it evaluates a resource and determines no action is needed). | | `IS-Error` | If the scheduler encounters an error while processing a resource, this tag contains the error code and a UTC timestamp. For example: `StartFailed 2025-06-15 09:00:05 UTC`. This tag is automatically cleared on the next successful scheduling action. | | `IS-ErrorMessage` | A human-readable description of the error. This tag is only present when `IS-Error` is also present and is cleared alongside it. | ### Error codes The following error codes may appear in the `IS-Error` tag: | Error code | Description | | --- | --- | | `UnknownSchedule` | The schedule name specified in the resource’s schedule tag does not match any schedule defined in the configuration table. | | `UnsupportedResource` | The resource type is not supported for scheduling (for example, a read replica of another RDS instance). | | `IncompatibleSchedule` | The schedule assigned to the resource is not compatible with the resource type (for example, an ASG schedule that uses unsupported cron expressions). | | `StartFailed` | The scheduler attempted to start the resource but the operation failed. | | `StopFailed` | The scheduler attempted to stop the resource but the operation failed. | | `ConfigurationFailed` | The scheduler attempted to configure scheduled scaling rules on an Auto Scaling group but the operation failed. | | `UnknownError` | An unexpected error occurred while processing the resource. | ### Tag behavior + When a resource is first registered for scheduling, the `IS-ManagedBy` tag is applied immediately. + When a resource is deregistered (the schedule tag is removed), all informational tags (`IS-ManagedBy`, `IS-LastAction`, `IS-Error`, `IS-ErrorMessage`) are removed from the resource. + Error tags are not re-written on each scheduling interval if the same error persists and the existing tag is still present on the resource. They are only updated when the error code changes. + All tag values are truncated to 256 characters to comply with AWS tagging limits. ### Tag governance considerations **Important** Instance Scheduler creates and updates the tags listed above on managed resources as part of normal operation. If your organization enforces tag governance through AWS Config rules, tag policies, service control policies, or automated remediation, ensure that your change management controls are configured to allow the following tag keys: `IS-ManagedBy` `IS-LastAction` `IS-Error` `IS-ErrorMessage` `IS-PreferredInstanceTypes` (if using alternate instance types) `IS-MinDesiredMax` (if scheduling Auto Scaling groups) If you cannot accommodate these tags in your governance policies, disable informational tagging by setting the **Enable informational tagging** parameter to `No` on the hub stack. Note that this will also disable the `IS-ManagedBy` tag that is used to confirm resource registration. ### Control tags In addition to informational tags, Instance Scheduler uses the following control tags for specific features: | Tag key | Description | | --- | --- | | `IS-PreferredInstanceTypes` | A comma-separated list of alternate EC2 instance types to try when starting an instance fails due to insufficient capacity. For more information, refer to [Handling EC2 Insufficient Capacity Errors](specifying-alternate-instance-types-for-ec2.md). | | `IS-MinDesiredMax` | The minimum, desired, and maximum capacity values for an Auto Scaling group in the format `min,desired,max`. For more information, refer to [EC2 Auto Scaling Group Scheduling](ec2-auto-scaling-group-scheduling.md). | ### Tag capacity **Important** AWS resources typically have a limit of 50 tags per resource. Instance Scheduler may use up to 6 tags on a resource (4 informational tags plus up to 2 control tags). Ensure your resources have sufficient tag capacity to accommodate Instance Scheduler tags alongside your existing tagging strategy. If a resource is at or near the 50-tag limit, informational tag writes may fail. The scheduler logs these failures but continues scheduling operations. Check CloudWatch Logs if you suspect tagging issues. ## CloudWatch Logs Insights queries Instance Scheduler’s structured logging format enables efficient querying using CloudWatch Logs Insights. You can use Logs Insights to search, analyze, and visualize log data to troubleshoot operational issues and monitor scheduling activity. Instance Scheduler provides pre-formatted log queries that you can access from the Saved Queries section in the CloudWatch Logs console: `SchedulingHistory` Query scheduling actions performed on resources, including start and stop operations. `RegistrationEvents` Query resource registration and deregistration events. `Errors` Query error logs to troubleshoot issues with the solution. For more information about CloudWatch Logs Insights, refer to [Analyzing log data with CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) in the *Amazon CloudWatch Logs User Guide*. ## Operational insights dashboard The Operational Insights dashboard provides visibility into solution performance and cost savings from scheduled instance management. To access the dashboard, ensure **Operational Monitoring** is set to "enabled" in the hub stack parameters. Navigate to CloudWatch and select "Dashboards" from the navigation menu. The dashboard name is *\$1\$1stack-name\$1-Operational-Insights-Dashboard\$1*. The dashboard displays managed instance counts, running hours saved, and Lambda function performance metrics. **Operational insights dashboard overview** ![\[OpsDashboardOverview\]](http://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/images/OpsDashboardOverview.png) **Note** The information in these graphs is dependent upon the scheduling interval configured on the solution hub stack. When updating the solution’s scheduling interval, the dashboard will only display scheduling metrics from after the most recent update to the scheduling interval. Monitor Lambda execution times to ensure optimal performance (see [Quotas](solution-quotas.md)). If execution times consistently approach the timeout threshold, consider increasing the Lambda size property or deploying Instance Scheduler to a region with lower latency to your managed regions. **Lambda metrics showing duration and error counts** ![\[OpsDashboardLambdaMetrics\]](http://docs.aws.amazon.com/solutions/latest/instance-scheduler-on-aws/images/OpsDashboardLambdaMetrics.png) ### Additional costs associated with this feature This operational dashboard is powered by custom CloudWatch metrics collected by the solution that will incur additional cost. This feature can be turned off by disabling "Operational Monitoring" on the solution hub stack. This feature costs an additional \$13.00/month plus additional scaling costs based on the size of your deployment. The costs are as follows: | Custom CloudWatch Dashboard | \$13 | | --- | --- | | Per-instance-type metrics | \$10.90 per instance type\$1 | | API usage | \$1\$10.10 per [active target](cost.md#calculating-scheduling-targets)\$1\$1 | ** *\$1These costs are tracked per service category (EC2/RDS) and only for instance types actually used for scheduling.* ** ** **\$1** ## Monitoring EventBridge Events Instance Scheduler publishes scheduling and registration events to EventBridge event buses to provide visibility into solution operations and enable integration with other AWS services. ### Event types The solution publishes two main categories of events: **Scheduling events**: Published when Instance Scheduler takes action to start, stop, or configure managed resources. These events include details about the instance, schedule, and action taken. start, stop, or configure managed resources. These events include details about the instance, schedule, and action taken. **Registration events**: Published when resources are registered or unregistered for scheduling based on tagging operations. ### Event destinations **IS-LocalEvents event buses**: An `IS-LocalEvents` event bus is deployed in each managed region of each member account (including the hub account). Each bus receives events for scheduling actions and resource registrations within that region. **IS-GlobalEvents event bus**: The `IS-GlobalEvents` event bus in the hub account receives a copy of each event sent to any `IS-LocalEvents` event bus, providing centralized monitoring across all accounts and regions. ### Using EventBridge events You can create EventBridge rules to: + Monitor scheduling operations across your infrastructure + Trigger notifications when instances are started or stopped + Integrate with other AWS services for automated workflows + Implement compliance monitoring and alerting ### Event structure All events use the standard EventBridge format. The following examples show the structure for each event type: **Scheduling event:** ``` { "Source": "instance-scheduler", "DetailType": "Scheduling Action", "Resources": ["arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0"], "Detail": { "account": "123456789012", "region": "us-east-1", "service": "ec2", "resource_id": "i-1234567890abcdef0", "requested_action": "Start", "action_taken": "Started", "schedule": "office-hours" } } ``` **Registration event:** ``` { "Source": "instance-scheduler", "DetailType": "Resource Registered", "Resources": ["arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0"], "Detail": { "account": "123456789012", "region": "us-east-1", "service": "ec2", "resource_id": "i-1234567890abcdef0", "schedule": "office-hours" } } ``` Each event contains these key fields: + `Source` - Identifies the event source as "instance-scheduler" + `DetailType` - Specifies the event category: "Scheduling Action" for instance operations or "Resource Registered" for tagging events + `Resources` - Array containing the ARNs of affected AWS resources + `Detail` - Contains the event payload with account ID, region, service type (ec2/rds), resource ID, schedule name, and for scheduling events, both the requested action and actual result Possible `requested_action` values for scheduling events: + `Start`: Scheduler intended to start the instance + `Stop`: Scheduler intended to stop the instance + `Configure`: Scheduler intended to configure the instance Possible `action_taken` values for scheduling events: + `Started`: Instance was started + `Stopped`: Instance was stopped + `Hibernated`: Instance was hibernated + `Configured`: Instance configuration was modified + `Error`: An error occurred during the scheduling operation ### Creating EventBridge rules To monitor Instance Scheduler events: 1. Navigate to the EventBridge console in your AWS account 1. Create a new rule targeting either the `IS-GlobalEvents` event bus (for centralized monitoring) or `IS-LocalEvents` event bus (for local monitoring) 1. Define event patterns to match Instance Scheduler events 1. Configure targets such as SNS topics, Lambda functions, or CloudWatch Logs For more information about EventBridge, refer to [What is Amazon EventBridge?](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) in the *Amazon EventBridge User Guide*.