# Rotate AWS Secrets Manager secrets Rotate secrets *Rotation* is the process of periodically updating a secret. When you rotate a secret, you update the credentials in both the secret and the database or service. In Secrets Manager, you can set up automatic rotation for your secrets. There are two forms of rotation: + [Managed rotation](rotate-secrets_managed.md) – For most [managed secrets](service-linked-secrets.md), you use managed rotation, where the service configures and manages rotation for you. Managed rotation doesn't use a Lambda function. + [Rotate Secrets Manager managed external secrets](rotate-secrets_external.md) – For secrets held by Secrets Manager partners, you use managed external secrets rotation to update the secret on the partner's system. This doesn't require a Lambda function. + [Rotation by Lambda function](rotate-secrets_lambda.md) – For other types of secrets, Secrets Manager rotation uses a Lambda function to update the secret and the database or service. # Managed rotation for AWS Secrets Manager secrets Managed rotation Some services offer *managed rotation*, where the service configures and manages rotation for you. With managed rotation, you don't use an AWS Lambda function to update the secret and the credentials in the database. The following services offer managed rotation: + **Amazon Aurora** offers managed rotation for master user credentials. For more information, see [Password management with Amazon Aurora and AWS Secrets Manager](https://docs.aws.amazon.com//AmazonRDS/latest/AuroraUserGuide/rds-secrets-manager.html) in the *Amazon Aurora User Guide*. + **Amazon ECS** Service Connect offers managed rotation for AWS Private Certificate Authority TLS certificates. For more information, see [TLS with Service Connect](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-tls.html) in the *Amazon Elastic Container Service Developer Guide*. + **Amazon RDS** offers managed rotation for master user credentials. For more information, see [Password management with Amazon RDS and AWS Secrets Manager](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-secrets-manager.html) in the *Amazon RDS User Guide*. + **Amazon DocumentDB** offers managed rotation for master user credentials. For more information, see [Password management with Amazon DocumentDB and AWS Secrets Manager](https://docs.aws.amazon.com/documentdb/latest/developerguide/docdb-secrets-manager.html) in the *Amazon DocumentDB User Guide*. + **Amazon Redshift** offers managed rotation for admin passwords. For more information, see [Managing Amazon Redshift admin passwords using AWS Secrets Manager](https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-secrets-manager-integration.html) in the *Amazon Redshift Management Guide*. + **managed external secrets** offers managed rotation for secrets held by Secrets Manager partners. For more information, see [Using AWS Secrets Manager managed external secrets to manage Third Party secrets](managed-external-secrets.md). **Tip** For all other types of secrets, see [Rotation by Lambda function](rotate-secrets_lambda.md). Rotation for managed secrets typically completes within one minute. During rotation, new connections that retrieve the secret may get the previous version of the credentials. In applications, we strongly recommend that you follow the best practice of using a database user created with the minimal privileges required for your application, rather than using the master user. For application users, for highest availability, you can use the [Alternating users rotation strategy](rotation-strategy.md). For secrets held by Secrets Manager partners, **To change the schedule for managed rotation** 1. Open the managed secret in the Secrets Manager console. You can follow a link from the managing service, or [search for the secret](service-linked-secrets.md) in the Secrets Manager console. 1. Under **Rotation schedule**, enter your schedule in UTC time zone in either the **Schedule expression builder** or as a **Schedule expression**. Secrets Manager stores your schedule as a `rate()` or `cron()` expression. The rotation window automatically starts at midnight unless you specify a **Start time**. You can rotate a secret as often as every four hours. For more information, see [Rotation schedules](rotate-secrets_schedule.md). 1. (Optional) For **Window duration**, choose the length of the window during which you want Secrets Manager to rotate your secret, for example **3h** for a three hour window. The window must not extend into the next rotation window. If you don't specify **Window duration**, for a rotation schedule in hours, the window automatically closes after one hour. For a rotation schedule in days, the window automatically closes at the end of the day. 1. Choose **Save**. **To change the schedule for managed rotation (AWS CLI)** + Call [https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/rotate-secret.html](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/rotate-secret.html). The following example rotates the secret between 16:00 and 18:00 UTC on the 1st and 15th day of the month. For more information, see [Rotation schedules](rotate-secrets_schedule.md). ``` aws secretsmanager rotate-secret \ --secret-id MySecret \ --rotation-rules \ "{\"ScheduleExpression\": \"cron(0 16 1,15 * ? *)\", \"Duration\": \"2h\"}" ``` # Rotate Secrets Manager managed external secrets Rotate managed external secrets Secrets Manager has partnered with select software vendors to offer managed external secrets. This feature helps customers manage the secret lifecycle by handling rotations automatically. With managed external secrets, customers no longer need to maintain specific rotation logic for each secret stored with different partners. This will be handled by Secrets Manager. To view the list of partners onboarded with Secrets Manager, see [Managed external secrets Partners](https://docs.aws.amazon.com/secretsmanager/latest/userguide/mes-partners.html). ## Set Up Rotation in the Console To configure rotation for an existing managed external secret, created by specifying the secret type and value as specified by the respective [integration partners](https://docs.aws.amazon.com/secretsmanager/latest/userguide/mes-partners.html), use the following steps: 1. Open the Secrets Manager console. 1. Select your managed external secret from the list. 1. Choose the **Configuration** tab. 1. In the **Rotation configuration** section, choose **Edit rotation**. 1. Turn on **Automatic rotation**. 1. Under **Rotation metadata**, add any partner-specific metadata required for rotation: Follow the guidelines provided by your integration partner for other required metadata 1. In **Service permissions for secret rotation**, select or create an IAM role for rotation: + Choose **Create a new role** to automatically create a role with necessary permissions + Or select an existing role with appropriate permissions for your partner By default, permissions are scoped to the individual partner in the region where the secret is created 1. Set your **Rotation schedule** (for example, rotate automatically every 30 days). 1. Choose **Save** to apply the rotation configuration. The two key metadata fields configured during this process are: | Field | Description | | --- | --- | | ExternalSecretRotationMetadata | Partner-specific metadata required for rotation, such as API version for Salesforce | | ExternalSecretRotationRoleArn | The ARN of the IAM role used for rotation, with permissions scoped to the integration partner | For more information on these fields, see Using Secrets Manager [managed external secrets to manage Third Party secrets](https://docs.aws.amazon.com/secretsmanager/latest/userguide/managed-external-secrets.html). ## Set Up Rotation Using the CLI Run the following command to set up rotation for a Salesforce secret. This command specifies the secret ID, the IAM role ARN for rotation, the rotation schedule, and any partner-specific metadata required for the rotation process. ``` aws secretsmanager rotate-secret \ --secret-id SampleSecret \ --external-secret-rotation-role-arn arn:aws:iam::123412341234:role/xyz \ --rotation-rules AutomaticallyAfterDays=1 \ --external-secret-rotation-metadata '[{"Key":"apiVersion","Value":"v65.0"}]' ``` # Rotation by Lambda function For many types of secrets, Secrets Manager uses an AWS Lambda function to update the secret and the database or service. For information about the costs of using a Lambda function, see [Pricing](intro.md#asm_pricing). For some [Secrets managed by other services](service-linked-secrets.md), you use *managed rotation*. To use [Managed rotation](rotate-secrets_managed.md), you first create the secret through the managing service. During rotation, Secrets Manager logs events that indicate the state of rotation. For more information, see [Log AWS Secrets Manager events with AWS CloudTrail](monitoring-cloudtrail.md). To rotate a secret, Secrets Manager calls a [ Lambda function](rotate-secrets_lambda-functions.md) according to the rotation schedule you set up. If you also manually update your secret value while automatic rotation is set up, then Secrets Manager considers that a valid rotation when it calculates the next rotation date. During rotation, Secrets Manager calls the same function several times, each time with different parameters. Secrets Manager invokes the function with the following JSON request structure of parameters: ``` { "Step" : "request.type", "SecretId" : "string", "ClientRequestToken" : "string", "RotationToken" : "string" } ``` **Parameters:** + **Step** – The rotation step: `create_secret`, `set_secret`, `test_secret`, or `finish_secret`. For more information, see [Four steps in a rotation function](rotate-secrets_lambda-functions.md#rotate-secrets_lambda-functions-code). + **SecretId** – The ARN of the secret to rotate. + **ClientRequestToken** – A unique identifier for the new version of the secret. This value helps ensure idempotency. For more information, see [PutSecretValue: ClientRequestToken](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_PutSecretValue.html#SecretsManager-PutSecretValue-request-ClientRequestToken) in the *AWS Secrets Manager API Reference*. + **RotationToken** – A unique identifier that indicates the source of the request. Required for secret rotation using an assumed role or cross-account rotation, in which you rotate a secret in one account by using a Lambda rotation function in another account. In both cases, the rotation function assumes an IAM role to call Secrets Manager and then Secrets Manager uses the rotation token to validate the IAM role identity. If any rotation step fails, Secrets Manager retries the entire rotation process multiple times. **Topics** + [Automatic rotation for database secrets (console)](rotate-secrets_turn-on-for-db.md) + [Automatic rotation for non-database secrets (console)](rotate-secrets_turn-on-for-other.md) + [Automatic rotation (AWS CLI)](rotate-secrets_turn-on-cli.md) + [ # Lambda function rotation strategies ](rotation-strategy.md) + [ # Lambda rotation functions ](rotate-secrets_lambda-functions.md) + [Rotation function templates](reference_available-rotation-templates.md) + [Permissions for rotation](rotating-secrets-required-permissions-function.md) + [ # Network access for AWS Lambda rotation function ](rotation-function-network-access.md) + [Troubleshoot rotation](troubleshoot_rotation.md) # Set up automatic rotation for Amazon RDS, Amazon Aurora, Amazon Redshift, or Amazon DocumentDB secrets Automatic rotation for database secrets (console) This tutorial describes how to set up [Rotation by Lambda function](rotate-secrets_lambda.md) for database secrets. Rotation is the process of periodically updating a secret. When you rotate a secret, you update the credentials in both the secret and the database. In Secrets Manager, you can set up automatic rotation for your database secrets. To set up rotation using the console, you need to first choose a rotation strategy. Then you configure the secret for rotation, which creates a Lambda rotation function if you don't already have one. The console also sets permissions for the Lambda function execution role. The last step is to make sure that the Lambda rotation function can access both Secrets Manager and your database through the network. **Warning** To turn on automatic rotation, you must have permission to create an IAM execution role for the Lambda rotation function and attach a permission policy to it. You need both `iam:CreateRole` and `iam:AttachRolePolicy` permissions. Granting these permissions allows an identity to grant themselves any permissions. **Topics** + [ ## Step 1: Choose a rotation strategy and (optionally) create a superuser secret ](#rotate-secrets_turn-on-for-db_step1) + [ ## Step 2: Configure rotation and create a rotation function ](#rotate-secrets_turn-on-for-db_step2) + [ ## Step 3: (Optional) Set additional permissions conditions on the rotation function ](#rotate-secrets_turn-on-for-db_step3) + [ ## Step 4: Set up network access for the rotation function ](#rotate-secrets_turn-on-for-db_step4) + [ ## Next steps ](#rotate-secrets_turn-on-for-db_stepnext) ## Step 1: Choose a rotation strategy and (optionally) create a superuser secret For information about the strategies offered by Secrets Manager, see [Lambda function rotation strategies](rotation-strategy.md). If you choose the *alternating users strategy*, you must [Create secrets](create_secret.md) and store database superuser credentials in it. You need a secret with superuser credentials because rotation clones the first user, and most users do not have that permission. Note that Amazon RDS Proxy does not support the alternating users strategy. ## Step 2: Configure rotation and create a rotation function **To turn on rotation for an Amazon RDS, Amazon DocumentDB, or Amazon Redshift secret** 1. Open the Secrets Manager console at [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/). 1. On the **Secrets** page, choose your secret. 1. On the **Secret details** page, in the **Rotation configuration** section, choose **Edit rotation**. 1. In the **Edit rotation configuration** dialog box, do the following: 1. Turn on **Automatic rotation**. 1. Under **Rotation schedule**, enter your schedule in UTC time zone in either the **Schedule expression builder** or as a **Schedule expression**. Secrets Manager stores your schedule as a `rate()` or `cron()` expression. The rotation window automatically starts at midnight unless you specify a **Start time**. You can rotate a secret as often as every four hours. For more information, see [Rotation schedules](rotate-secrets_schedule.md). 1. (Optional) For **Window duration**, choose the length of the window during which you want Secrets Manager to rotate your secret, for example **3h** for a three hour window. The window must not extend into the next rotation window. If you don't specify **Window duration**, for a rotation schedule in hours, the window automatically closes after one hour. For a rotation schedule in days, the window automatically closes at the end of the day. 1. (Optional) Choose **Rotate immediately when the secret is stored** to rotate your secret when you save your changes. If you clear the checkbox, then the first rotation will begin on the schedule you set. If rotation fails, for example because Steps 3 and 4 are not yet completed, Secrets Manager retries the rotation process multiple times. 1. Under **Rotation function**, do one of the following: + Choose **Create a new Lambda function** and enter a name for your new function. Secrets Manager adds `SecretsManager` to the beginning of the function name. Secrets Manager creates the function based on the appropriate [template](reference_available-rotation-templates.md) and sets the necessary [permissions](rotating-secrets-required-permissions-function.md) for the Lambda execution role. + Choose **Use an existing Lambda function** to reuse a rotation function you used for another secret. The rotation functions listed under **Recommended VPC configurations** have the same VPC and security group as the database, which helps the function access the database. 1. For **Rotation strategy**, choose the **Single user** or **Alternating users** strategy. For more information, see [Step 1: Choose a rotation strategy and (optionally) create a superuser secret](#rotate-secrets_turn-on-for-db_step1). 1. Choose **Save**. ## Step 3: (Optional) Set additional permissions conditions on the rotation function In the resource policy for your rotation function, we recommend that you include the context key [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) to help prevent Lambda from being used as a [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). For some AWS services, to avoid the confused deputy scenario, AWS recommends that you use both the [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn) and [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) global condition keys. However, if you include the `aws:SourceArn` condition in your rotation function policy, the rotation function can only be used to rotate the secret specified by that ARN. We recommend that you include only the context key `aws:SourceAccount` so that you can use the rotation function for multiple secrets. **To update your rotation function resource policy** 1. In the Secrets Manager console, choose your secret, and then on the details page, under **Rotation configuration**, choose the Lambda rotation function. The Lambda console opens. 1. Follow the instructions at [Using resource-based policies for Lambda](https://docs.aws.amazon.com/lambda/latest/dg/access-control-resource-based.html) to add a `aws:sourceAccount` condition. ``` "Condition": { "StringEquals": { "AWS:SourceAccount": "123456789012" } }, ``` If the secret is encrypted with a KMS key other than the AWS managed key `aws/secretsmanager`, Secrets Manager grants the Lambda execution role permission to use the key. You can use the [SecretARN encryption context](security-encryption.md#security-encryption-encryption-context) to limit the use of the decrypt function, so the rotation function role only has access to decrypt the secret it is responsible for rotating. **To update your rotation function execution role** 1. From the Lambda rotation function, choose **Configuration**, and then under **Execution role**, choose the **Role name**. 1. Follow the instructions at [Modifying a role permissions policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_permissions-policy) to add a `kms:EncryptionContext:SecretARN` condition. ``` "Condition": { "StringEquals": { "kms:EncryptionContext:SecretARN": "SecretARN" } }, ``` ## Step 4: Set up network access for the rotation function For more information, see [Network access for AWS Lambda rotation function](rotation-function-network-access.md). ## Next steps See [Troubleshoot AWS Secrets Manager rotation](troubleshoot_rotation.md). # Set up automatic rotation for non-database AWS Secrets Manager secrets Automatic rotation for non-database secrets (console) This tutorial describes how to set up [Rotation by Lambda function](rotate-secrets_lambda.md) for non-database secrets. Rotation is the process of periodically updating a secret. When you rotate a secret, you update the credentials in both the secret and the database or service that the secret is for. For database secrets, see [Automatic rotation for database secrets (console)](rotate-secrets_turn-on-for-db.md). **Warning** To turn on automatic rotation, you must have permission to create an IAM execution role for the Lambda rotation function and attach a permission policy to it. You need both `iam:CreateRole` and `iam:AttachRolePolicy` permissions. Granting these permissions allows an identity to grant themselves any permissions. **Topics** + [ ## Step 1: Create a generic rotation function ](#rotate-secrets_turn-on-for-other_create) + [ ## Step 2: Write the rotation function code ](#rotate-secrets_turn-on-for-other_write) + [ ## Step 3: Configure the secret for rotation ](#rotate-secrets_turn-on-for-other_configure) + [ ## Step 4: Allow the rotation function to access Secrets Manager and your database or service ](#rotate-secrets_turn-on-for-other_perms) + [ ## Step 5: Allow Secrets Manager to invoke the rotation function ](#rotate-secrets_turn-on-for-other_perms2) + [ ## Step 6: Set up network access for the rotation function ](#rotate-secrets_turn-on-for-other_network) + [ ## Next steps ](#rotate-secrets_turn-on-for-other_stepnext) ## Step 1: Create a generic rotation function To begin, create a Lambda rotation function. It will not have the code in it to rotate your secret, so you'll write that in a later step. For information about how a rotation function works, see [Lambda rotation functions](rotate-secrets_lambda-functions.md). In supported Regions, you can use AWS Serverless Application Repository to create the function from a template. For a list of supported Regions, see [AWS Serverless Application Repository FAQs](https://aws.amazon.com/serverless/serverlessrepo/faqs/). In other Regions, you create the function from scratch and copy the template code into the function. **To create a generic rotation function** 1. To determine whether AWS Serverless Application Repository is supported in your Region, see [AWS Serverless Application Repository endpoints and quotas](https://docs.aws.amazon.com/general/latest/gr/serverlessrepo.html) in the *AWS General Reference*. 1. Do one of the following: + If AWS Serverless Application Repository is supported in your Region: 1. In the Lambda console, choose **Applications** and then choose **Create application**. 1. On the **Create application** page, choose the **Serverless application** tab. 1. In the search box under **Public applications**, enter **SecretsManagerRotationTemplate**. 1. Select **Show apps that create custom IAM roles or resource policies**. 1. Choose the **SecretsManagerRotationTemplate** tile. 1. On the **Review, configure and deploy** page, in the **Application settings** tile, fill in the required fields. + For **endpoint**, enter the endpoint for your Region, including **https://**. For a list of endpoints, see [AWS Secrets Manager endpoints](asm_access.md#endpoints). + To put the Lambda function in a VPC, include **vpcSecurityGroupIds** and **vpcSubnetIds**. 1. Choose **Deploy**. + If AWS Serverless Application Repository isn't supported in your Region: 1. In the Lambda console, choose **Functions** and then choose **Create function**. 1. On the **Create function** page, do the following: 1. Choose **Author from scratch**. 1. For **Function name**, enter a name for your rotation function. 1. For **Runtime**, choose **Python 3.10**. 1. Choose **Create function**. ## Step 2: Write the rotation function code In this step, you write the code that updates the secret and the service or database that the secret is for. For information about what a rotation function does, including tips on writing your own rotation function, see [Lambda rotation functions](rotate-secrets_lambda-functions.md). You can also use the [Rotation function templates](reference_available-rotation-templates.md) as reference. ## Step 3: Configure the secret for rotation In this step, you set a rotation schedule for your secret and connect the rotation function to the secret. **To configure rotation and create an empty rotation function** 1. Open the Secrets Manager console at [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/). 1. On the **Secrets** page, choose your secret. 1. On the **Secret details** page, in the **Rotation configuration** section, choose **Edit rotation**. In the **Edit rotation configuration** dialog box, do the following: 1. Turn on **Automatic rotation**. 1. Under **Rotation schedule**, enter your schedule in UTC time zone in either the **Schedule expression builder** or as a **Schedule expression**. Secrets Manager stores your schedule as a `rate()` or `cron()` expression. The rotation window automatically starts at midnight unless you specify a **Start time**. You can rotate a secret as often as every four hours. For more information, see [Rotation schedules](rotate-secrets_schedule.md). 1. (Optional) For **Window duration**, choose the length of the window during which you want Secrets Manager to rotate your secret, for example **3h** for a three hour window. The window must not extend into the next rotation window. If you don't specify **Window duration**, for a rotation schedule in hours, the window automatically closes after one hour. For a rotation schedule in days, the window automatically closes at the end of the day. 1. (Optional) Choose **Rotate immediately when the secret is stored** to rotate your secret when you save your changes. If you clear the checkbox, then the first rotation will begin on the schedule you set. 1. Under **Rotation function**, choose the Lambda function you created in Step 1. 1. Choose **Save**. ## Step 4: Allow the rotation function to access Secrets Manager and your database or service The Lambda rotation function needs permission to access the secret in Secrets Manager, and it needs permission to access your database or service. In this step, you grant these permissions to the Lambda execution role. If the secret is encrypted with a KMS key other than the AWS managed key `aws/secretsmanager`, then you need to grant the Lambda execution role permission to use the key. You can use the [SecretARN encryption context](security-encryption.md#security-encryption-encryption-context) to limit the use of the decrypt function, so the rotation function role only has access to decrypt the secret it is responsible for rotating. For policy examples, see [Permissions for rotation](rotating-secrets-required-permissions-function.md). For instructions, see [Lambda execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html) in the *AWS Lambda Developer Guide*. ## Step 5: Allow Secrets Manager to invoke the rotation function To allow Secrets Manager to invoke the rotation function on the rotation schedule you set up, you need to grant `lambda:InvokeFunction` permission to the Secrets Manager service principal in the resource policy of the Lambda function. In the resource policy for your rotation function, we recommend that you include the context key [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) to help prevent Lambda from being used as a [confused deputy](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html). For some AWS services, to avoid the confused deputy scenario, AWS recommends that you use both the [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourcearn) and [https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html#condition-keys-sourceaccount) global condition keys. However, if you include the `aws:SourceArn` condition in your rotation function policy, the rotation function can only be used to rotate the secret specified by that ARN. We recommend that you include only the context key `aws:SourceAccount` so that you can use the rotation function for multiple secrets. To attach a resource policy to a Lambda function, see [Using resource-based policies for Lambda](https://docs.aws.amazon.com/lambda/latest/dg/access-control-resource-based.html). The following policy allows Secrets Manager to invoke a Lambda function. ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Id": "default", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "secretsmanager.amazonaws.com" }, "Action": "lambda:InvokeFunction", "Condition": { "StringEquals": { "AWS:SourceAccount": "123456789012" } }, "Resource": "arn:aws:lambda:us-east-1:123456789012:function:function-name" } ] } ``` ------ ## Step 6: Set up network access for the rotation function In this step, you allow the rotation function to connect to both Secrets Manager and the service or database the secret is for. The rotation function must have access to both to be able to rotate the secret. See [Network access for AWS Lambda rotation function](rotation-function-network-access.md). ## Next steps When you configured rotation in Step 3, you set a schedule for rotating the secret. If rotation fails when it is scheduled, Secrets Manager will attempt the rotation multiple times. You can also start a rotation immediately by following the instructions in [Rotate a secret immediately](rotate-secrets_now.md). If rotation fails, see [Troubleshoot rotation](troubleshoot_rotation.md). # Set up automatic rotation using the AWS CLI Automatic rotation (AWS CLI) This tutorial describes how to set up [Rotation by Lambda function](rotate-secrets_lambda.md) by using the AWS CLI. When you rotate a secret, you update the credentials in both the secret and the database or service that the secret is for. You can also set up rotation using the console. For database secrets, see [Automatic rotation for database secrets (console)](rotate-secrets_turn-on-for-db.md). For all other types of secrets, see [Automatic rotation for non-database secrets (console)](rotate-secrets_turn-on-for-other.md). To set up rotation using the AWS CLI, if you are rotating a database secret, you first need to choose a rotation strategy. If you choose the alternating users strategy, you must store a separate secret with credentials for a database superuser. Next, you write the rotation function code. Secrets Manager provides templates you can base your function on. Then you create a Lambda function with your code and set permissions for both the Lambda function and the Lambda execution role. The next step is to make sure that the Lambda function can access both Secrets Manager and your database or service through the network. Finally, you configure the secret for rotation. **Topics** + [ ## Prerequisite for database secrets: Choose a rotation strategy ](#rotate-secrets_turn-on-cli_step1) + [ ## Step 1: Write the rotation function code ](#rotate-secrets_turn-on-cli_write) + [ ## Step 2: Create the Lambda function ](#w2aac21c11c25c15) + [ ## Step 3: Set up network access ](#w2aac21c11c25c17) + [ ## Step 4: Configure the secret for rotation ](#w2aac21c11c25c19) + [ ## Next steps ](#w2aac21c11c25c21) ## Prerequisite for database secrets: Choose a rotation strategy For information about the strategies offered by Secrets Manager, see [Lambda function rotation strategies](rotation-strategy.md). ### Option 1: Single user strategy If you choose the *single user strategy*, you can continue with Step 1. ### Option 2: Alternating users strategy If you choose the *alternating users strategy*, you must: + [Create a secret](create_secret.md#create_secret_cli) and store database superuser credentials in it. You need a secret with superuser credentials because alternating users rotation clones the first user, and most users do not have that permission. + Add the ARN of the superuser secret to the original secret. For more information, see [JSON structure of AWS Secrets Manager secrets](reference_secret_json_structure.md). Note that Amazon RDS Proxy does not support the alternating users strategy. ## Step 1: Write the rotation function code To rotate a secret, you need a rotation function. A rotation function is a Lambda function that Secrets Manager calls to rotate your secret. For more information, see [Rotation by Lambda function](rotate-secrets_lambda.md). In this step, you write the code that updates the secret and the service or database that the secret is for. Secrets Manager provides templates for Amazon RDS, Amazon Aurora, Amazon Redshift, and Amazon DocumentDB database secrets in [Rotation function templates](reference_available-rotation-templates.md). **To write the rotation function code** 1. Do one of the following: + Check the list of [rotation function templates](reference_available-rotation-templates.md). If there is one that matches your service and rotation strategy, copy the code. + For other types of secrets, you write your own rotation function. For instructions, see [Lambda rotation functions](rotate-secrets_lambda-functions.md). 1. Save the file in a ZIP file *my-function.zip* along with any required dependencies. ## Step 2: Create the Lambda function In this step, you create the Lambda function using the ZIP file you created in Step 1. You also set the [Lambda execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html), which is the role that Lambda assumes when the function is invoked. **To create a Lambda rotation function and execution role** 1. Create a trust policy for the Lambda execution role and save it as a JSON file. For examples and more information, see [Lambda rotation function execution role permissions for AWS Secrets Manager](rotating-secrets-required-permissions-function.md). The policy must: + Allow the role to call Secrets Manager operations on the secret. + Allow the role to call the service that the secret is for, for example, to create a new password. 1. Create the Lambda execution role and apply the trust policy you created in the previous step by calling [https://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html](https://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html). ``` aws iam create-role \ --role-name rotation-lambda-role \ --assume-role-policy-document file://trust-policy.json ``` 1. Create the Lambda function from the ZIP file by calling [https://docs.aws.amazon.com/cli/latest/reference/lambda/create-function.html](https://docs.aws.amazon.com/cli/latest/reference/lambda/create-function.html). ``` aws lambda create-function \ --function-name my-rotation-function \ --runtime python3.7 \ --zip-file fileb://my-function.zip \ --handler .handler \ --role arn:aws:iam::123456789012:role/service-role/rotation-lambda-role ``` 1. Set a resource policy on the Lambda function to allow Secrets Manager to invoke it by calling [https://docs.aws.amazon.com/cli/latest/reference/lambda/add-permission.html](https://docs.aws.amazon.com/cli/latest/reference/lambda/add-permission.html). ``` aws lambda add-permission \ --function-name my-rotation-function \ --action lambda:InvokeFunction \ --statement-id SecretsManager \ --principal secretsmanager.amazonaws.com \ --source-account 123456789012 ``` ## Step 3: Set up network access For more information, see [Network access for AWS Lambda rotation function](rotation-function-network-access.md). ## Step 4: Configure the secret for rotation To turn on automatic rotation for your secret, call [https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/rotate-secret.html](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/rotate-secret.html). You can set a rotation schedule with a `cron()` or `rate()` schedule expression, and you can set a rotation window duration. For more information, see [Rotation schedules](rotate-secrets_schedule.md). ``` aws secretsmanager rotate-secret \ --secret-id MySecret \ --rotation-lambda-arn arn:aws:lambda:Region:123456789012:function:my-rotation-function \ --rotation-rules "{\"ScheduleExpression\": \"cron(0 16 1,15 * ? *)\", \"Duration\": \"2h\"}" ``` ## Next steps See [Troubleshoot AWS Secrets Manager rotation](troubleshoot_rotation.md). # Lambda function rotation strategies For [Rotation by Lambda function](rotate-secrets_lambda.md), for database secrets, Secrets Manager offers two rotation strategies. ## Rotation strategy: single user Single user This strategy updates credentials for one user in one secret. For Amazon RDS Db2 instances, because users can't change their own passwords, you must provide admin credentials in a separate secret. **This is the simplest rotation strategy, and it is appropriate for most use cases.** In particular, we recommend you use this strategy for credentials for one-time (ad hoc) or interactive users. When the secret rotates, open database connections are not dropped. While rotation is happening, there is a short period of time between when the password in the database changes and when the secret is updated. During this time, there is a low risk of the database denying calls that use the rotated credentials. You can mitigate this risk with an [appropriate retry strategy](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/). After rotation, new connections use the new credentials. ## Rotation strategy: alternating users Alternating users This strategy updates credentials for two users in one secret. You create the first user, and during the first rotation, the rotation function clones it to create the second user. Every time the secret rotates, the rotation function alternates which user's password it updates. Because most users don't have permission to clone themselves, you must provide the credentials for a `superuser` in another secret. We recommend using the single-user rotation strategy when cloned users in your database don't have the same permissions as the original user, and for credentials for one-time (ad hoc) or interactive users. This strategy is appropriate for databases with permission models where one role owns the database tables and a second role has permission to access the database tables. It is also appropriate for applications that require high availability. If an application retrieves the secret during rotation, the application still gets a valid set of credentials. After rotation, both `user` and `user_clone` credentials are valid. There is even less chance of applications getting a deny during this type of rotation than single user rotation. If the database is hosted on a server farm where the password change takes time to propagate to all servers, there is a risk of the database denying calls that use the new credentials. You can mitigate this risk with an [appropriate retry strategy](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/). Secrets Manager creates the cloned user with the same permissions as the original user. If you change the original user's permissions after the clone is created, you must also change the cloned user's permissions. For example, if you create a secret with a database user's credentials, the secret contains one version with those credentials. ![\[The secret contains one secret version labeled AWSCURRENT. The username for the AWSCURRENT version is MyUser.\]](http://docs.aws.amazon.com/secretsmanager/latest/userguide/images/AlternatingUsers1.png) **First rotation** – The rotation function creates a clone of your user with a generated password, and those credentials become the current secret version. ![\[The secret contains two secret versions, one labeled AWSCURRENT and one labeled AWSPREVIOUS. The username for the AWSCURRENT version is MyUser_clone.\]](http://docs.aws.amazon.com/secretsmanager/latest/userguide/images/AlternatingUsers2.png) **Second rotation** – The rotation function updates the password for the original user. ![\[The secret contains two secret versions, one labeled AWSCURRENT and one labeled AWSPREVIOUS. The username for the AWSCURRENT version is MyUser.\]](http://docs.aws.amazon.com/secretsmanager/latest/userguide/images/AlternatingUsers3.png) **Third rotation** – The rotation function updates the password for the cloned user. ![\[The secret contains two secret versions, one labeled AWSCURRENT and one labeled AWSPREVIOUS. The username for the AWSCURRENT version is MyUser_clone.\]](http://docs.aws.amazon.com/secretsmanager/latest/userguide/images/AlternatingUsers4.png) # Lambda rotation functions In [Rotation by Lambda function](rotate-secrets_lambda.md), an AWS Lambda function rotates the secret. AWS Secrets Manager uses [staging labels](whats-in-a-secret.md) to identify secret versions during rotation. If AWS Secrets Manager doesn't provide a [rotation function template](reference_available-rotation-templates.md) for your secret type, you can create a custom rotation function. Follow these guidelines when writing your rotation function: **Best practices for custom rotation functions** + Use the [generic rotation template](reference_available-rotation-templates.md#OTHER_rotation_templates) as a starting point. + Be cautious with debugging or logging statements. They can write information to Amazon CloudWatch Logs. Ensure logs don't contain sensitive information. For log statement examples, see the [AWS Secrets Manager rotation function templates](reference_available-rotation-templates.md) source code. + For security, AWS Secrets Manager only allows a Lambda rotation function to rotate the secret directly. The rotation function can't call another Lambda function to rotate the secret. + For debugging guidance, see [Testing and debugging serverless applications](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-test-and-debug.html). + If you use external binaries and libraries, for example to connect to a resource, you're responsible for patching and updating them. + Package your rotation function and any dependencies in a ZIP file, such as *my-function.zip*. **Warning** Setting the provisioned concurrency parameter to a value lower than 10 can cause throttling due to insufficient execution threads for the Lambda function. For more information, see [ Understanding reserved concurrency and provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html#reserved-and-provisioned) in the AWS Lambda AWS Lambda Developer Guide. ## Four steps in a rotation function **Topics** + [ ### `createSecret`: Create a new version of the secret ](#w2aac21c11c29c11b5) + [ ### **setSecret**: Change the credentials in the database or service ](#w2aac21c11c29c11b7) + [ ### **testSecret**: Test the new secret version ](#w2aac21c11c29c11b9) + [ ### **finishSecret**: Finish the rotation ](#w2aac21c11c29c11c11) ### `createSecret`: Create a new version of the secret The method `createSecret` first checks if a secret exists by calling [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_secret_value) with the passed-in `ClientRequestToken`. If there's no secret, it creates a new secret with [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.create_secret](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.create_secret) and the token as the `VersionId`. Then it generates a new secret value with [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_random_password](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.get_random_password). Next it calls [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.put_secret_value](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.put_secret_value) to store it with the staging label `AWSPENDING`. Storing the new secret value in `AWSPENDING` helps ensure idempotency. If rotation fails for any reason, you can refer to that secret value in subsequent calls. See [How do I make my Lambda function idempotent](https://aws.amazon.com/premiumsupport/knowledge-center/lambda-function-idempotent/). **Tips for writing your own rotation function** + Ensure the new secret value only includes characters that are valid for the database or service. Exclude characters by using the `ExcludeCharacters` parameter. + As you test your function, use the AWS CLI to see version stages: call [https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/describe-secret.html](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/describe-secret.html) and look at `VersionIdsToStages`. + For Amazon RDS MySQL, in alternating users rotation, Secrets Manager creates a cloned user with a name no longer than 16 characters. You can modify the rotation function to allow longer usernames. MySQL version 5.7 and higher supports usernames up to 32 characters, however Secrets Manager appends "\$1clone" (six characters) to the end of the username, so you must keep the username to a maximum of 26 characters. ### **setSecret**: Change the credentials in the database or service The method `setSecret` changes the credential in the database or service to match the new secret value in the `AWSPENDING` version of the secret. **Tips for writing your own rotation function** + If you pass statements to a service that interprets statements, like a database, use query parameterization. For more information, see [Query Parameterization Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html) on the *OWASP web site*. + The rotation function is a privileged deputy that has the authorization to access and modify customer credentials in both the Secrets Manager secret and the target resource. To prevent a potential [confused deputy attack](https://docs.aws.amazon.com/IAM/latest/UserGuide/confused-deputy.html), you need to make sure that an attacker cannot use the function to access other resources. Before you update the credential: + Check that the credential in the `AWSCURRENT` version of the secret is valid. If the `AWSCURRENT` credential isn't valid, abandon the rotation attempt. + Check that the `AWSCURRENT` and `AWSPENDING` secret values are for the same resource. For a username and password, check that the `AWSCURRENT` and `AWSPENDING` usernames are the same. + Check that the destination service resource is the same. For a database, check that the `AWSCURRENT` and `AWSPENDING` host names are the same. + In rare cases, you might want to customize an existing rotation function for a database. For example, with alternating users rotation, Secrets Manager creates the cloned user by copying the [runtime configuration parameters](https://www.postgresql.org/docs/8.0/runtime-config.html) of the first user. If you want to include more attributes, or change which ones are granted to the cloned user, you need to update the code in the `set_secret` function. ### **testSecret**: Test the new secret version Next, the Lambda rotation function tests the `AWSPENDING` version of the secret by using it to access the database or service. Rotation functions based on [Rotation function templates](reference_available-rotation-templates.md) test the new secret by using read access. ### **finishSecret**: Finish the rotation Finally, the Lambda rotation function moves the label `AWSCURRENT` from the previous secret version to this version, which also removes the `AWSPENDING` label in the same API call. Secrets Manager adds the `AWSPREVIOUS` staging label to the previous version, so that you retain the last known good version of the secret. The method **finish\$1secret** uses [https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.update_secret_version_stage](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.update_secret_version_stage) to move the staging label `AWSCURRENT` from the previous secret version to the new secret version. Secrets Manager automatically adds the `AWSPREVIOUS` staging label to the previous version, so that you retain the last known good version of the secret. **Tips for writing your own rotation function** + Don't remove `AWSPENDING` before this point, and don't remove it by using a separate API call, because that can indicate to Secrets Manager that the rotation did not complete successfully. Secrets Manager adds the `AWSPREVIOUS` staging label to the previous version, so that you retain the last known good version of the secret. When rotation is successful, the `AWSPENDING` staging label might be attached to the same version as the `AWSCURRENT` version, or it might not be attached to any version. If the `AWSPENDING` staging label is present but not attached to the same version as `AWSCURRENT`, then any later invocation of rotation assumes that a previous rotation request is still in progress and returns an error. When rotation is unsuccessful, the `AWSPENDING` staging label might be attached to an empty secret version. For more information, see [Troubleshoot rotation](troubleshoot_rotation.md). # AWS Secrets Manager rotation function templates Rotation function templates AWS Secrets Manager provides a set of rotation function templates that help automate the secure management of credentials for various database systems and services. The templates are ready-to-use Lambda functions that implement best practices for credential rotation, helping you maintain your security posture without manual intervention. The templates support two primary rotation strategies: + *Single-user rotation* which updates the credentials for a single user. + *Alternating-users rotation* which maintains two separate users to help eliminate downtime during credential changes. Secrets Manager also provides a generic template that serves as a starting point for any type of secret. To use the templates, see: + [Automatic rotation for database secrets (console)](rotate-secrets_turn-on-for-db.md) + [Automatic rotation for non-database secrets (console)](rotate-secrets_turn-on-for-other.md) To write your own rotation function, see [Write a rotation function](rotate-secrets_lambda-functions.md). **Contents** + [ ## Amazon RDS and Amazon Aurora ](#RDS_rotation_templates) + [ ### Amazon RDS Db2 single user ](#sar-template-db2-singleuser) + [ ### Amazon RDS Db2 alternating users ](#sar-template-db2-multiuser) + [ ### Amazon RDS MariaDB single user ](#sar-template-mariadb-singleuser) + [ ### Amazon RDS MariaDB alternating users ](#sar-template-mariadb-multiuser) + [ ### Amazon RDS and Amazon Aurora MySQL single user ](#sar-template-mysql-singleuser) + [ ### Amazon RDS and Amazon Aurora MySQL alternating users ](#sar-template-mysql-multiuser) + [ ### Amazon RDS Oracle single user ](#sar-template-oracle-singleuser) + [ ### Amazon RDS Oracle alternating users ](#sar-template-oracle-multiuser) + [ ### Amazon RDS and Amazon Aurora PostgreSQL single user ](#sar-template-postgre-singleuser) + [ ### Amazon RDS and Amazon Aurora PostgreSQL alternating users ](#sar-template-postgre-multiuser) + [ ### Amazon RDS Microsoft SQLServer single user ](#sar-template-sqlserver-singleuser) + [ ### Amazon RDS Microsoft SQLServer alternating users ](#sar-template-sqlserver-multiuser) + [ ## Amazon DocumentDB (with MongoDB compatibility) ](#NON-RDS_rotation_templates) + [ ### Amazon DocumentDB single user ](#sar-template-mongodb-singleuser) + [ ### Amazon DocumentDB alternating users ](#sar-template-mongodb-multiuser) + [ ## Amazon Redshift ](#template-redshift) + [ ### Amazon Redshift single user ](#sar-template-redshift-singleuser) + [ ### Amazon Redshift alternating users ](#sar-template-redshift-multiuser) + [ ## Amazon Timestream for InfluxDB ](#template-TimeStream) + [ ### Amazon Timestream for InfluxDB single user ](#template-TimeStream-singleuser) + [ ### Amazon Timestream for InfluxDB alternating users ](#template-TimeStream-multiuser) + [ ## Amazon ElastiCache ](#template-ELC) + [ ## Active Directory ](#template-AD) + [ ### Active Directory credentials ](#template-AD-password) + [ ### Active Directory keytab ](#template-AD-keytab) + [ ## Other types of secrets ](#OTHER_rotation_templates) ## Amazon RDS and Amazon Aurora ### Amazon RDS Db2 single user + **Template name:** SecretsManagerRDSDb2RotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **`SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSDb2RotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSDb2RotationSingleUser/lambda_function.py) + **Dependency: **[python-ibmdb](https://github.com/ibmdb/python-ibmdb) ### Amazon RDS Db2 alternating users + **Template name:** SecretsManagerRDSDb2RotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **`SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code:** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSDb2RotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSDb2RotationMultiUser/lambda_function.py) + **Dependency: **[python-ibmdb](https://github.com/ibmdb/python-ibmdb) ### Amazon RDS MariaDB single user + **Template name:** SecretsManagerRDSMariaDBRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **`SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMariaDBRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMariaDBRotationSingleUser/lambda_function.py) + **Dependency: **PyMySQL 1.0.2. If you use sha256 password for authentication, PyMySQL[rsa]. For information about using packages with compiled code in a Lambda runtime, see [How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?](https://repost.aws/knowledge-center/lambda-python-package-compatible) in *AWS Knowledge Center*. ### Amazon RDS MariaDB alternating users + **Template name:** SecretsManagerRDSMariaDBRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **`SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code:** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMariaDBRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMariaDBRotationMultiUser/lambda_function.py) + **Dependency: **PyMySQL 1.0.2. If you use sha256 password for authentication, PyMySQL[rsa]. For information about using packages with compiled code in a Lambda runtime, see [How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?](https://repost.aws/knowledge-center/lambda-python-package-compatible) in *AWS Knowledge Center*. ### Amazon RDS and Amazon Aurora MySQL single user + **Template name:** SecretsManagerRDSMySQLRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code:** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMySQLRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMySQLRotationSingleUser/lambda_function.py) + **Dependency: **PyMySQL 1.0.2. If you use sha256 password for authentication, PyMySQL[rsa]. For information about using packages with compiled code in a Lambda runtime, see [How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?](https://repost.aws/knowledge-center/lambda-python-package-compatible) in *AWS Knowledge Center*. ### Amazon RDS and Amazon Aurora MySQL alternating users + **Template name:** SecretsManagerRDSMySQLRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMySQLRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSMySQLRotationMultiUser/lambda_function.py) + **Dependency: **PyMySQL 1.0.2. If you use sha256 password for authentication, PyMySQL[rsa]. For information about using packages with compiled code in a Lambda runtime, see [How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?](https://repost.aws/knowledge-center/lambda-python-package-compatible) in *AWS Knowledge Center*. ### Amazon RDS Oracle single user + **Template name:** SecretsManagerRDSOracleRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSOracleRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSOracleRotationSingleUser/lambda_function.py) + **Dependency: **[python-oracledb 2.4.1](https://github.com/oracle/python-oracledb) ### Amazon RDS Oracle alternating users + **Template name:** SecretsManagerRDSOracleRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSOracleRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSOracleRotationMultiUser/lambda_function.py) + **Dependency: **[python-oracledb 2.4.1](https://github.com/oracle/python-oracledb) ### Amazon RDS and Amazon Aurora PostgreSQL single user + **Template name:** SecretsManagerRDSPostgreSQLRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSPostgreSQLRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSPostgreSQLRotationSingleUser/lambda_function.py) + **Dependency: **PyGreSQL 5.2.5 ### Amazon RDS and Amazon Aurora PostgreSQL alternating users + **Template name:** SecretsManagerRDSPostgreSQLRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSPostgreSQLRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSPostgreSQLRotationMultiUser/lambda_function.py) + **Dependency: **PyGreSQL 5.2.5 ### Amazon RDS Microsoft SQLServer single user + **Template name:** SecretsManagerRDSSQLServerRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSSQLServerRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSSQLServerRotationSingleUser/lambda_function.py) + **Dependency: **Pymssql 2.2.2 ### Amazon RDS Microsoft SQLServer alternating users + **Template name:** SecretsManagerRDSSQLServerRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **Expected `SecretString` structure:** [Amazon RDS and Aurora credentials](reference_secret_json_structure.md#reference_secret_json_structure_rds). + **Source code: **[https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSSQLServerRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRDSSQLServerRotationMultiUser/lambda_function.py) + **Dependency: **Pymssql 2.2.2 ## Amazon DocumentDB (with MongoDB compatibility) Amazon DocumentDB ### Amazon DocumentDB single user + **Template name:** SecretsManagerMongoDBRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **Expected `SecretString` structure:** [Amazon DocumentDB credentials](reference_secret_json_structure.md#reference_secret_json_structure_docdb). + **Source code:** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerMongoDBRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerMongoDBRotationSingleUser/lambda_function.py) + **Dependency: **PyMongo 4.2.0 ### Amazon DocumentDB alternating users + **Template name:** SecretsManagerMongoDBRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **Expected `SecretString` structure:** [Amazon DocumentDB credentials](reference_secret_json_structure.md#reference_secret_json_structure_docdb). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerMongoDBRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerMongoDBRotationMultiUser/lambda_function.py) + **Dependency: **PyMongo 4.2.0 ## Amazon Redshift ### Amazon Redshift single user + **Template name:** SecretsManagerRedshiftRotationSingleUser + **Rotation strategy:** [Rotation strategy: single user](rotation-strategy.md#rotating-secrets-one-user-one-password). + **Expected `SecretString` structure:** [Amazon Redshift credentials](reference_secret_json_structure.md#reference_secret_json_structure_RS). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRedshiftRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRedshiftRotationSingleUser/lambda_function.py) + **Dependency: **PyGreSQL 5.2.5 ### Amazon Redshift alternating users + **Template name:** SecretsManagerRedshiftRotationMultiUser + **Rotation strategy:** [Rotation strategy: alternating users](rotation-strategy.md#rotating-secrets-two-users). + **Expected `SecretString` structure:** [Amazon Redshift credentials](reference_secret_json_structure.md#reference_secret_json_structure_RS). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRedshiftRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRedshiftRotationMultiUser/lambda_function.py) + **Dependency: **PyGreSQL 5.2.5 ## Amazon Timestream for InfluxDB To use these templates, see [How Amazon Timestream for InfluxDB uses secrets](https://docs.aws.amazon.com/timestream/latest/developerguide/timestream-for-influx-security-db-secrets.html) in the *Amazon Timestream Developer Guide*. ### Amazon Timestream for InfluxDB single user + **Template name:** SecretsManagerInfluxDBRotationSingleUser + **Expected `SecretString` structure:** [Amazon Timestream for InfluxDB secret structure](reference_secret_json_structure.md#reference_secret_json_structure_TIME). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerInfluxDBRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerInfluxDBRotationSingleUser/lambda_function.py) + **Dependency: **InfluxDB 2.0 python client ### Amazon Timestream for InfluxDB alternating users + **Template name:** SecretsManagerInfluxDBRotationMultiUser + **Expected `SecretString` structure:** [Amazon Timestream for InfluxDB secret structure](reference_secret_json_structure.md#reference_secret_json_structure_TIME). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerInfluxDBRotationMultiUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerInfluxDBRotationMultiUser/lambda_function.py) + **Dependency: **InfluxDB 2.0 python client ## Amazon ElastiCache To use this template, see [Automatically rotating passwords for users](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/User-Secrets-Manager.html) in the *Amazon ElastiCache User Guide*. + **Template name:** SecretsManagerElasticacheUserRotation + **Expected `SecretString` structure:** [Amazon ElastiCache credentials](reference_secret_json_structure.md#reference_secret_json_structure_ELC). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerElasticacheUserRotation/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerElasticacheUserRotation/lambda_function.py) ## Active Directory ### Active Directory credentials + **Template name:** SecretsManagerActiveDirectoryRotationSingleUser + **Expected `SecretString` structure:** [Active Directory credentials](reference_secret_json_structure.md#reference_secret_json_structure_AD). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerActiveDirectoryRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerActiveDirectoryRotationSingleUser/lambda_function.py) ### Active Directory keytab + **Template name:** SecretsManagerActiveDirectoryAndKeytabRotationSingleUser + **Expected `SecretString` structure:** [Active Directory credentials](reference_secret_json_structure.md#reference_secret_json_structure_AD). + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerActiveDirectoryAndKeytabRotationSingleUser/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerActiveDirectoryAndKeytabRotationSingleUser/lambda_function.py) + **Dependencies:** msktutil ## Other types of secrets Secrets Manager provides this template as a starting point for you to create a rotation function for any type of secret. + **Template name:** SecretsManagerRotationTemplate + **Source code: ** [https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRotationTemplate/lambda\$1function.py](https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/tree/master/SecretsManagerRotationTemplate/lambda_function.py) # Lambda rotation function execution role permissions for AWS Secrets Manager Permissions for rotation For [Rotation by Lambda function](rotate-secrets_lambda.md), when Secrets Manager uses a Lambda function to rotate a secret, Lambda assumes an [IAM execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html) and provides those credentials to the Lambda function code. For instructions about how to set up automatic rotation, see: + [Automatic rotation for database secrets (console)](rotate-secrets_turn-on-for-db.md) + [Automatic rotation for non-database secrets (console)](rotate-secrets_turn-on-for-other.md) + [Automatic rotation (AWS CLI)](rotate-secrets_turn-on-cli.md) The following examples show inline policies for Lambda rotation function execution roles. To create an execution role and attach a permissions policy, see [AWS Lambda execution role](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html). **Topics** + [ ## Policy for a Lambda rotation function execution role ](#rotating-secrets-required-permissions-function-example) + [ ## Policy statement for customer managed key ](#rotating-secrets-required-permissions-function-cust-key-example) + [ ## Policy statement for alternating users strategy ](#rotating-secrets-required-permissions-function-alternating-example) ## Policy for a Lambda rotation function execution role The following example policy allows the rotation function to: + Run Secrets Manager operations for *SecretARN*. + Create a new password. + Set up the required configuration if your database or service runs in a VPC. See [Configuring a Lambda function to access resources in a VPC](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "secretsmanager:UpdateSecretVersionStage" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:secretName-AbCdEf" }, { "Effect": "Allow", "Action": [ "secretsmanager:GetRandomPassword" ], "Resource": "*" }, { "Action": [ "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DetachNetworkInterface" ], "Resource": "*", "Effect": "Allow" } ] } ``` ------ ## Policy statement for customer managed key If the secret is encrypted with a KMS key other than the AWS managed key `aws/secretsmanager`, then you need to grant the Lambda execution role permission to use the key. You can use the [SecretARN encryption context](security-encryption.md#security-encryption-encryption-context) to limit the use of the decrypt function, so the rotation function role only has access to decrypt the secret it is responsible for rotating. The following example shows a statement to add to the execution role policy to decrypt the secret using the KMS key. ``` { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:DescribeKey", "kms:GenerateDataKey" ], "Resource": "KMSKeyARN", "Condition": { "StringEquals": { "kms:EncryptionContext:SecretARN": "SecretARN" } } } ``` To use the rotation function for multiple secrets that are encrypted with a customer managed key, add a statement like the following example to allow the execution role to decrypt the secret. ``` { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:DescribeKey", "kms:GenerateDataKey" ], "Resource": "KMSKeyARN", "Condition": { "StringEquals": { "kms:EncryptionContext:SecretARN": [ "arn1", "arn2" ] } } } ``` ## Policy statement for alternating users strategy For information about the *alternating users rotation strategy*, see [Lambda function rotation strategies](rotation-strategy.md). For a secret that contains Amazon RDS credentials, if you are using the alternating users strategy and the superuser secret is [managed by Amazon RDS](rotate-secrets_managed.md), then you must also allow the rotation function to call read-only APIs on Amazon RDS so that it can get the connection information for the database. We recommend you attach the AWS managed policy [AmazonRDSReadOnlyAccess](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonRDSReadOnlyAccess.html). The following example policy allows the function to: + Run Secrets Manager operations for *SecretARN*. + Retrieve the credentials in the superuser secret. Secrets Manager uses the credentials in the superuser secret to update the credentials in the rotated secret. + Create a new password. + Set up the required configuration if your database or service runs in a VPC. For more information, see [Configuring a Lambda function to access resources in a VPC](https://docs.aws.amazon.com/lambda/latest/dg/vpc.html). ------ #### [ JSON ] **** ``` { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:DescribeSecret", "secretsmanager:GetSecretValue", "secretsmanager:PutSecretValue", "secretsmanager:UpdateSecretVersionStage" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:secretName-AbCdEf" }, { "Effect": "Allow", "Action": [ "secretsmanager:GetSecretValue" ], "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:secretName-AbCdEf" }, { "Effect": "Allow", "Action": [ "secretsmanager:GetRandomPassword" ], "Resource": "*" }, { "Action": [ "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeNetworkInterfaces", "ec2:DetachNetworkInterface" ], "Resource": "*", "Effect": "Allow" } ] } ``` ------ # Network access for AWS Lambda rotation function For [Rotation by Lambda function](rotate-secrets_lambda.md), when Secrets Manager uses a Lambda function to rotate a secret, the Lambda rotation function must be able to access the secret. If your secret contains credentials, then the Lambda function must also be able to access the source of those credentials, such as a database or service. **To access a secret** Your Lambda rotation function must be able to access a Secrets Manager endpoint. If your Lambda function can access the internet, then you can use a public endpoint. To find an endpoint, see [AWS Secrets Manager endpoints](asm_access.md#endpoints). If your Lambda function runs in a VPC that doesn't have internet access, we recommend you configure Secrets Manager service private endpoints within your VPC. Your VPC can then intercept requests addressed to the public regional endpoint and redirect them to the private endpoint. For more information, see [VPC endpoints (AWS PrivateLink)](vpc-endpoint-overview.md). Alternatively, you can enable your Lambda function to access a Secrets Manager public endpoint by adding a [NAT gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) or an [internet gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) to your VPC, which allows traffic from your VPC to reach the public endpoint. This exposes your VPC to more risk because an IP address for the gateway can be attacked from the public Internet. **(Optional) To access the database or service** For secrets such as API keys, there is no source database or service that you need to update along with the secret. If your database or service is running on an Amazon EC2 instance in a VPC, we recommend that you configure your Lambda function to run in the same VPC. Then the rotation function can communicate directly with your service. For more information, see [Configuring VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#vpc-configuring). To allow the Lambda function to access the database or service, you must make sure that the security groups attached to your Lambda rotation function allow outbound connections to the database or service. You must also make sure that the security groups attached to your database or service allow inbound connections from the Lambda rotation function. # Troubleshoot AWS Secrets Manager rotation Troubleshoot rotation For many services, Secrets Manager uses a Lambda function to rotate secrets. For more information, see [Rotation by Lambda function](rotate-secrets_lambda.md). The Lambda rotation function interacts with the database or service the secret is for as well as Secrets Manager. When rotation doesn't work the way you expect, you should first check the CloudWatch logs. **Note** Some services can manage secrets for you, including managing automatic rotation. For more information, see [Managed rotation for AWS Secrets Manager secrets](rotate-secrets_managed.md). **Topics** + [ ## How to troubleshoot secret rotation failures in AWS Lambda functions ](#troubleshooting-secret-rotation-failures) + [ ## No activity after "Found credentials in environment variables" ](#troubleshoot_rotation_timing-out) + [ ## No activity after "createSecret" ](#troubleshoot_rotation_createSecret) + [ ## Error: "Access to KMS is not allowed" ](#troubleshoot_rotation_kms-key) + [ ## Error: "Key is missing from secret JSON" ](#tshoot-lambda-mismatched-secretvalue) + [ ## Error: "setSecret: Unable to log into database" ](#troubleshoot_rotation_setSecret) + [ ## Error: "Unable to import module 'lambda\$1function'" ](#tshoot-python-version) + [ ## Upgrade an existing rotation function from Python 3.7 to 3.9 ](#troubleshoot_rotation_python39) + [ ## Upgrade an existing rotation function from Python 3.9 to 3.10 ](#troubleshoot_rotation_python_310) + [ ## AWS Lambda secret rotation with `PutSecretValue` failed ](#troubleshoot_rotation_putsecretvalue) + [ ## Error: "Error when executing lambda ** during ** step" ](#concurrency-related-failures) ## How to troubleshoot secret rotation failures in AWS Lambda functions Troubleshooting steps If you're experiencing secret rotation failures with your Lambda functions, use the following steps to troubleshoot and resolve the issue. ### Possible causes + Insufficient concurrent executions for the Lambda function + Race conditions due to multiple API calls during rotation + Incorrect Lambda function logic + Networking issues between the Lambda function and the database ### General troubleshooting steps 1. Analyze CloudWatch logs: + Look for specific error messages or unexpected behavior in the Lambda function logs + Verify that all rotation steps (**CreateSecret**, **SetSecret**, **TestSecret**, **FinishSecret**) are being attempted 1. Review API calls during rotation: + Avoid making mutating API calls on the secret during Lambda rotation + Ensure there's no race condition between **RotateSecret** and **PutSecretValue** calls 1. Verify Lambda function logic: + Confirm you're using the latest AWS sample code for secret rotation + If using custom code, review it for proper handling of all rotation steps 1. Check network configuration: + Verify security group rules allow the Lambda function to access the database + Ensure proper VPC endpoint or public endpoint access for Secrets Manager 1. Test secret versions: + Verify that the AWSCURRENT version of the secret allows database access + Check if AWSPREVIOUS or AWSPENDING versions are valid 1. Clear pending rotations: + If rotation consistently fails, clear the AWSPENDING staging label and retry rotation 1. Check Lambda concurrency settings: + Verify that concurrency settings are appropriate for your workload + If you suspect concurrency issues, see the "Troubleshooting concurrency-related rotation failures" section ## No activity after "Found credentials in environment variables" If there is no activity after "Found credentials in environment variables", and the task duration is long, for example the default Lambda timeout of 30000ms, then the Lambda function may be timing out while trying to reach the Secrets Manager endpoint. Your Lambda rotation function must be able to access a Secrets Manager endpoint. If your Lambda function can access the internet, then you can use a public endpoint. To find an endpoint, see [AWS Secrets Manager endpoints](asm_access.md#endpoints). If your Lambda function runs in a VPC that doesn't have internet access, we recommend you configure Secrets Manager service private endpoints within your VPC. Your VPC can then intercept requests addressed to the public regional endpoint and redirect them to the private endpoint. For more information, see [VPC endpoints (AWS PrivateLink)](vpc-endpoint-overview.md). Alternatively, you can enable your Lambda function to access a Secrets Manager public endpoint by adding a [NAT gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) or an [internet gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) to your VPC, which allows traffic from your VPC to reach the public endpoint. This exposes your VPC to more risk because an IP address for the gateway can be attacked from the public Internet. ## No activity after "createSecret" The following are issues that can cause rotation to stop after createSecret: **The VPC Network ACLs do not allow HTTPS traffic in and out.** For more information, see [Control traffic to subnets using Network ACLs](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html) in the *Amazon VPC User Guide*. **Lambda function timeout configuration is too short to perform the task. ** For more information, see [Configuring Lambda function options](https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html) in the *AWS Lambda Developer Guide*. **The Secrets Manager VPC endpoint does not allow the VPC CIDRs on ingress in the assigned security groups. ** For more information, see [Control traffic to resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) in the *Amazon VPC User Guide*. **The Secrets Manager VPC endpoint policy does not allow Lambda to use the VPC endpoint. ** For more information, see [Using an AWS Secrets Manager VPC endpoint](vpc-endpoint-overview.md). **The secret uses alternating users rotation, the superuser secret is managed by Amazon RDS, and the Lambda function can't access the RDS API.** For [alternating users rotation](rotation-strategy.md#rotating-secrets-two-users) where the superuser secret is [managed by another AWS service](service-linked-secrets.md), the Lambda rotation function must be able to call the service endpoint to get the database connection information. We recommend that you configure a VPC endpoint for the database service. For more information, see: + [Amazon RDS API and interface VPC endpoints](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/vpc-interface-endpoints.html) in the *Amazon RDS User Guide*. + [Working with VPC endpoints](https://docs.aws.amazon.com/redshift/latest/mgmt/enhanced-vpc-working-with-endpoints.html) in the *Amazon Redshift Management Guide*. ## Error: "Access to KMS is not allowed" If you see `ClientError: An error occurred (AccessDeniedException) when calling the GetSecretValue operation: Access to KMS is not allowed`, the rotation function does not have permission to decrypt the secret using the KMS key that was used to encrypt the secret. There might be a condition in the permissions policy that limits the encryption context to a specific secret. For information about the required permission, see [Policy statement for customer managed key](rotating-secrets-required-permissions-function.md#rotating-secrets-required-permissions-function-cust-key-example). ## Error: "Key is missing from secret JSON" A Lambda rotation function requires the secret value to be in a specific JSON structure. If you see this error, then the JSON might be missing a key that the rotation function tried to access. For information about the JSON structure for each type of secret, see [JSON structure of AWS Secrets Manager secrets](reference_secret_json_structure.md). ## Error: "setSecret: Unable to log into database" The following are issues that can cause this error: **The rotation function can't access the database.** If the task duration is long, for example over 5000ms, then the Lambda rotation function might not be able to access the database over the network. If your database or service is running on an Amazon EC2 instance in a VPC, we recommend that you configure your Lambda function to run in the same VPC. Then the rotation function can communicate directly with your service. For more information, see [Configuring VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#vpc-configuring). To allow the Lambda function to access the database or service, you must make sure that the security groups attached to your Lambda rotation function allow outbound connections to the database or service. You must also make sure that the security groups attached to your database or service allow inbound connections from the Lambda rotation function. **The credentials in the secret are incorrect.** If the task duration is short, then the Lambda rotation function might not be able to authenticate with the credentials in the secret. Check the credentials by logging in manually with the information in the `AWSCURRENT` and `AWSPREVIOUS` versions of the secret using the AWS CLI command [https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/get-secret-value.html](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/get-secret-value.html). **The database uses `scram-sha-256` to encrypt passwords.** If your database is Aurora PostgreSQL version 13 or later and uses `scram-sha-256` to encrypt passwords, but the rotation function uses `libpq` version 9 or older which does not support `scram-sha-256`, then the rotation function can't connect to the database. **To determine which database users use `scram-sha-256` encryption** + See *Checking for users with non-SCRAM passwords* in the blog [SCRAM Authentication in RDS for PostgreSQL 13](https://aws.amazon.com/blogs/database/scram-authentication-in-rds-for-postgresql-13/). **To determine which version of `libpq` your rotation function uses** 1. On a Linux-based computer, on the Lambda console, navigate to your rotation function and download the deployment bundle. Uncompress the zip file into a work directory. 1. At a command line, in the work directory, run: `readelf -a libpq.so.5 | grep RUNPATH` 1. If you see the string *`PostgreSQL-9.4.x`*, or any major version less than 10, then the rotation function doesn't support `scram-sha-256`. + Output for a rotation function that doesn't support `scram-sha-256`: `0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild-a1b2c/workspace/build/PostgreSQL/PostgreSQL-9.4.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild-a1b2c/workspace/src/PostgreSQL/build/private/install/lib]` + Output for a rotation function that supports `scram-sha-256`: `0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild-a1b2c/workspace/build/PostgreSQL/PostgreSQL-10.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild-a1b2c/workspace/src/PostgreSQL/build/private/install/lib]` + Output for a rotation function that supports `scram-sha-256`: `0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild- a1b2c /workspace/build/PostgreSQL/PostgreSQL-14.x_client_only. 123456 .0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil-path/build.libfarm/lib:/local/p4clients/pkgbuild- a1b2c /workspace/src/PostgreSQL/build/private/install/lib]` + Output for a rotation function that supports `scram-sha-256`: `0x000000000000001d (RUNPATH) Library runpath: [/local/p4clients/pkgbuild- a1b2c/workspace/build/PostgreSQL/PostgreSQL- 14.x_client_only.123456.0/AL2_x86_64/DEV.STD.PTHREAD/build/private/tmp/brazil- path/build.libfarm/lib:/local/p4clients/pkgbuild- a1b2c/workspace/src/PostgreSQL/build/private/install/lib]` If you set up automatic secret rotation before December 30, 2021, your rotation function bundled an earlier version of `libpq` that doesn't support `scram-sha-256`. To support `scram-sha-256`, you need to [recreate your rotation function](rotate-secrets_turn-on-for-db.md). **The database requires SSL/TLS access.** If your database requires an SSL/TLS connection, but the rotation function uses an unencrypted connection, then the rotation function can't connect to the database. Rotation functions for Amazon RDS (except Oracle and Db2) and Amazon DocumentDB automatically use Secure Socket Layer (SSL) or Transport Layer Security (TLS) to connect to your database, if it is available. Otherwise they use an unencrypted connection. If you set up automatic secret rotation before December 20, 2021, your rotation function might be based on an earlier template that did not support SSL/TLS. To support connections that use SSL/TLS, you need to [recreate your rotation function](rotate-secrets_turn-on-for-db.md). **To determine when your rotation function was created** 1. In the Secrets Manager console [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/), open your secret. In the **Rotation configuration** section, under **Lambda rotation function**, you see the **Lambda function ARN**, for example, `arn:aws:lambda:aws-region:123456789012:function:SecretsManagerMyRotationFunction `. Copy the function name from the end of the ARN, in this example ` SecretsManagerMyRotationFunction `. 1. In the AWS Lambda console [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/), under **Functions**, paste your Lambda function name in the search box, choose Enter, and then choose the Lambda function. 1. In the function details page, on the **Configuration** tab, under **Tags**, copy the value next to the key **aws:cloudformation:stack-name**. 1. In the AWS CloudFormation console [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/), under **Stacks**, paste the key value in the search box, and then choose Enter. 1. The list of stacks filters so that only the stack that created the Lambda rotation function appears. In the **Created date** column, view the date the stack was created. This is the date the Lambda rotation function was created. ## Error: "Unable to import module 'lambda\$1function'" You might receive this error if you're running an earlier Lambda function that was automatically upgraded from Python 3.7 to a newer version of Python. To resolve the error, you can change the Lambda function version back to Python 3.7, and then [Upgrade an existing rotation function from Python 3.7 to 3.9](#troubleshoot_rotation_python39). For more information, see [Why did my Secrets Manager Lambda function rotation fail with a “pg module not found“ error?](https://repost.aws/knowledge-center/secrets-manager-lambda-rotation) in *AWS re:Post*. ## Upgrade an existing rotation function from Python 3.7 to 3.9 Some rotation functions created before November 2022 used Python 3.7. The AWS SDK for Python stopped supporting Python 3.7 in December 2023. For more information, see [Python support policy updates for AWS SDKs and Tools](https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/). To switch to a new rotation function that uses Python 3.9, you can add a runtime property to an existing rotation function or recreate the rotation function. **To find which Lambda rotation functions use Python 3.7** 1. Sign in to the AWS Management Console and open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the list of **Functions**, filter for **SecretsManager**. 1. In the filtered list of functions, under **Runtime**, look for Python 3.7. **Topics** + [ ### Option 1: Recreate the rotation function using CloudFormation ](#update-python-opt-1) + [ ### Option 2: Update the runtime for the existing rotation function using CloudFormation ](#update-python-opt-2) + [ ### Option 3: For AWS CDK users, upgrade the CDK library ](#update-python-opt-3) ### Option 1: Recreate the rotation function using CloudFormation When you use the Secrets Manager console to turn on rotation, Secrets Manager uses CloudFormation to create the necessary resources, including the Lambda rotation function. If you used the console to turn on rotation, or you created the rotation function using a CloudFormation stack, you can use the same CloudFormation stack to recreate the rotation function with a new name. The new function uses the more recent version of Python. **To find the CloudFormation stack that created the rotation function** + On the Lambda function details page, on the **Configuration** tab, choose **Tags**. View the ARN next to **aws:cloudformation:stack-id**. The stack name is embedded in the ARN, as shown in the following example. + ARN: `arn:aws:cloudformation:us-west-2:408736277230:stack/SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda-3CUDHZMDMBO8/79fc9050-2eef-11ed-80f0-021fb13c0537` + Stack name: **SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda** **To recreate a rotation function (CloudFormation)** 1. In CloudFormation, search for the stack by name, and then choose **Update**. If a dialog box appears recommending you update the root stack, choose **Go to root stack**, and then choose **Update**. 1. On the **Update stack** page, under **Prepare template**, choose **Edit in Application Composer**, and then under **Edit template in Application Composer**, choose the button **Edit in Application Composer**. 1. In Application Composer, do the following: 1. In the template code, in `SecretRotationScheduleHostedRotationLambda`, replace the value for `"functionName": "SecretsManagerTestRotationRDS"` with a new function name, for example in JSON, `"functionName": "SecretsManagerTestRotationRDSupdated"` 1. Choose **Update template**. 1. In the **Continue to CloudFormation** dialog box, choose **Confirm and continue to CloudFormation**. 1. Continue through the CloudFormation stack workflow and then choose **Submit**. ### Option 2: Update the runtime for the existing rotation function using CloudFormation When you use the Secrets Manager console to turn on rotation, Secrets Manager uses CloudFormation to create the necessary resources, including the Lambda rotation function. If you used the console to turn on rotation, or you created the rotation function using a CloudFormation stack, you can use the same CloudFormation stack to update the runtime for the rotation function. **To find the CloudFormation stack that created the rotation function** + On the Lambda function details page, on the **Configuration** tab, choose **Tags**. View the ARN next to **aws:cloudformation:stack-id**. The stack name is embedded in the ARN, as shown in the following example. + ARN: `arn:aws:cloudformation:us-west-2:408736277230:stack/SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda-3CUDHZMDMBO8/79fc9050-2eef-11ed-80f0-021fb13c0537` + Stack name: **SecretsManagerRDSMySQLRotationSingleUser5c2-SecretRotationScheduleHostedRotationLambda** **To update the runtime for a rotation function (CloudFormation)** 1. In CloudFormation, search for the stack by name, and then choose **Update**. If a dialog box appears recommending you update the root stack, choose **Go to root stack**, and then choose **Update**. 1. On the **Update stack** page, under **Prepare template**, choose **Edit in Application Composer**, and then under **Edit template in Application Composer**, choose the button **Edit in Application Composer**. 1. In Application Composer, do the following: 1. In the template JSON, for the `SecretRotationScheduleHostedRotationLambda`, under `Properties`, under `Parameters`, add **"runtime": "python3.9"**. 1. Choose **Update template**. 1. In the **Continue to CloudFormation** dialog box, choose **Confirm and continue to CloudFormation**. 1. Continue through the CloudFormation stack workflow and then choose **Submit**. ### Option 3: For AWS CDK users, upgrade the CDK library If you used the AWS CDK prior to version v2.94.0 to set up rotation for your secret, you can update the Lambda function by upgrading to v2.94.0 or later. For more information, see the [AWS Cloud Development Kit (AWS CDK) v2 Developer Guide](https://docs.aws.amazon.com/cdk/v2/guide/home.html). ## Upgrade an existing rotation function from Python 3.9 to 3.10 Secrets Manager is transitioning from Python 3.9 to 3.10 for Lambda rotation functions. To switch to a new rotation function that uses Python 3.10, you'll need to follow the upgrade path based on your deployment method. Use the following procedures to upgrade both the Python version and the underlying dependencies. **To find which Lambda rotation functions use Python 3.9** 1. Sign in to the AWS Management Console and open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. In the list of **Functions**, filter for **SecretsManager**. 1. In the filtered list of functions, under **Runtime**, look for **Python 3.9**. ### Update paths by deployment method The Lambda rotation functions identified in this list can be deployed through Secrets Manager console, AWS Serverless Application Repository apps, or CloudFormation transforms. Each of these deployment strategies have a distinct update path. Use one of the following procedures to update your Lambda rotation functions, depending on how your function was deployed. ------ #### [ AWS Secrets Manager console-deployed functions ] A new Lambda function must be deployed through AWS Secrets Manager console as you cannot manually update dependencies for existing Lambda functions. Use the following procedure to upgrade AWS Secrets Manager console-deployed functions. 1. Open the Secrets Manager console at [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/). 1. Under **AWS Secrets Manager**, select **Secrets**. Select the secret that uses the Lambda function you want to update. 1. Navigate to the **Rotations** tab and select the **Update rotation configurations** option. 1. Under **Rotation functions**, choose **Create a new function**, and enter a new name for the Lambda rotation function. 1. (Optional) Once the update is complete, you can test the updated Lambda function to confirm it works as expected. Under the **Rotation** tab, select **Rotate Secret Immediately** to initiate an immediate rotation. 1. (Optional) You can view your function logs and the Python version used at runtime in Amazon CloudWatch. For more information, see [Viewing CloudWatch Logs for Lambda functions](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs-view.html#monitoring-cloudwatchlogs-console) in the *AWS Lambda Developer Guide*. 1. Once the new rotation function is set up, you can delete the old rotation function. ------ #### [ AWS Serverless Application Repository deployments ] The following procedure shows how to upgrade AWS Serverless Application Repository deployments. The Lambda functions deployed through AWS Serverless Application Repository have a banner stating `This function belongs to an application. Click here to manage it.` which includes a link to the Lambda application to which the function belongs. **Important** AWS Serverless Application Repository availability is AWS Region dependent. Use the following procedure to update AWS Serverless Application Repository deployed functions. 1. Open the AWS Lambda console at [https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/). 1. Navigate to the **Configurations** tab of the Lambda function that needs to be updated. 1. You'll need the following information about your function when updating the deployed AWS Serverless Application Repository application. You can find this information in the Lambda console. + **Lambda application's name** + The Lambda application name can be found by using the link in the banner. For example, the banner states the following `serverlessrepo-SecretsManagerRedshiftRotationSingleUser`. The name in this example is `SecretsManagerRedshiftRotationSingleUser`. + **Lambda rotation function name** + **Secrets Manager endpoint** + The endpoint can be found under the **Configurations** and the **Environment variables** tabs assigned to the **SECRETS\$1MANAGER\$1ENDPOINT** variable. 1. To upgrade Python, you must update the semantic version of the serverless application. See [Updating Applications](https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverlessrepo-how-to-consume-new-version.html#update-applications) in the *AWS Serverless Application Repository Developer Guide*. ------ #### [ Custom Lambda rotation functions ] If you created custom Lambda rotation functions, you’ll need to upgrade each package dependencies and runtimes for these functions. For more information, see [Upgrade Lambda function runtime to latest version](https://repost.aws/knowledge-center/lambda-upgrade-function-runtime). ------ #### [ AWS::SecretsManager-2024-09-16 transform macro ] If the Lambda function is deployed through this transform, [updating the stacks using existing template](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-direct.html) will allow you to use the updated Lambda runtime. Use the following procedure to update CloudFormation stack using existing template. 1. Open the CloudFormation console at [https://console.aws.amazon.com/cloudformation](https://console.aws.amazon.com/cloudformation/). 1. On the **Stacks** page, select the stack that you want to update. 1. Choose **Update** on the stack details pane. 1. For **Choose a template update method**, select **Direct update**. 1. On the **Specify template** page, select **Use existing template**. 1. Keep all other options at their default values, and then choose **Update stack**. If you experience issues updating the stack, see [Determine the cause of a stack failure](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/determine-root-cause-for-stack-failures.html) in the *CloudFormation User Guide*. ------ #### [ AWS::SecretsManager-2020-07-23 transform macro ] We recommend you migrate to the newer transform version if you're using `AWS::SecretsManager-2020-07-23`. See [Introducing an enhanced version of the AWS Secrets Manager transform: AWS::SecretsManager-2024-09-16](https://aws.amazon.com/blogs/security/introducing-an-enhanced-version-of-the-aws-secrets-manager-transform-awssecretsmanager-2024-09-16/) in the *AWS Security Blog* for more information. If you continue to use `AWS::SecretsManager-2020-07-23`, you can experience a mismatch error between your runtime version and the Lambda function code artifacts. For more information, see [AWS::SecretsManager::RotationSchedule HostedRotationLambda](https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/aws-properties-secretsmanager-rotationschedule-hostedrotationlambda.html#cfn-secretsmanager-rotationschedule-hostedrotationlambda-runtime) in the *CloudFormation Template Reference*. If you experience issues updating the stack, [Determine the cause of a stack failure](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/determine-root-cause-for-stack-failures.html) in the *CloudFormation User Guide*. ------ **Verify Python upgrade** To verify the Python upgrade, open the Lambda console ([https://console.aws.amazon.com/lambda/](https://console.aws.amazon.com/lambda/)) and access the **Function** page. Select the function you updated. Under **Code source** section, review the files included in the directory and ensure the Python .so file is version `3.10`. ## AWS Lambda secret rotation with `PutSecretValue` failed If you use an assumed role or a cross-account rotation with Secrets Manager and you find a **RotationFailed** event in CloudTrail with the message: Pending secret version *VERSION\$1ID *for Secret *SECRET\$1ARN * was not created by Lambda *LAMBDA\$1ARN.* Remove the `AWSPENDING` staging label and restart rotation, then you need to update your Lambda function to use the `RotationToken` parameter. ## Update Lambda rotation function to include `RotationToken` 1. Download the Lambda function code + Open the Lambda console + In the navigation pane, choose **Functions** + Select your Lambda secret rotation function for **Function name** + For **Download**, choose one of **Function code .zip**, **AWS SAM file**, **Both** + Choose **OK** to save the function on your local machine. 1. Edit `Lambda_handler` Include the rotation\$1token parameter in the create\$1secret step for cross-account rotation: ``` def lambda_handler(event, context): """Secrets Manager Rotation Template This is a template for creating an AWS Secrets Manager rotation lambda Args: event (dict): Lambda dictionary of event parameters. These keys must include the following: - SecretId: The secret ARN or identifier - ClientRequestToken: The ClientRequestToken of the secret version - Step: The rotation step (one of createSecret, setSecret, testSecret, or finishSecret) - RotationToken: the rotation token to put as parameter for PutSecretValue call context (LambdaContext): The Lambda runtime information Raises: ResourceNotFoundException: If the secret with the specified arn and stage does not exist ValueError: If the secret is not properly configured for rotation KeyError: If the event parameters do not contain the expected keys """ arn = event['SecretId'] token = event['ClientRequestToken'] step = event['Step'] # Add the rotation token rotation_token = event['RotationToken'] # Setup the client service_client = boto3.client('secretsmanager', endpoint_url=os.environ['SECRETS_MANAGER_ENDPOINT']) # Make sure the version is staged correctly metadata = service_client.describe_secret(SecretId=arn) if not metadata['RotationEnabled']: logger.error("Secret %s is not enabled for rotation" % arn) raise ValueError("Secret %s is not enabled for rotation" % arn) versions = metadata['VersionIdsToStages'] if token not in versions: logger.error("Secret version %s has no stage for rotation of secret %s." % (token, arn)) raise ValueError("Secret version %s has no stage for rotation of secret %s." % (token, arn)) if "AWSCURRENT" in versions[token]: logger.info("Secret version %s already set as AWSCURRENT for secret %s." % (token, arn)) return elif "AWSPENDING" not in versions[token]: logger.error("Secret version %s not set as AWSPENDING for rotation of secret %s." % (token, arn)) raise ValueError("Secret version %s not set as AWSPENDING for rotation of secret %s." % (token, arn)) # Use rotation_token if step == "createSecret": create_secret(service_client, arn, token, rotation_token) elif step == "setSecret": set_secret(service_client, arn, token) elif step == "testSecret": test_secret(service_client, arn, token) elif step == "finishSecret": finish_secret(service_client, arn, token) else: raise ValueError("Invalid step parameter") ``` 1. Edit `create_secret` code Revise the `create_secret` function to accept and use the `rotation_token` parameter: ``` # Add rotation_token to the function def create_secret(service_client, arn, token, rotation_token): """Create the secret This method first checks for the existence of a secret for the passed in token. If one does not exist, it will generate a new secret and put it with the passed in token. Args: service_client (client): The secrets manager service client arn (string): The secret ARN or other identifier token (string): The ClientRequestToken associated with the secret version rotation_token (string): the rotation token to put as parameter for PutSecretValue call Raises: ResourceNotFoundException: If the secret with the specified arn and stage does not exist """ # Make sure the current secret exists service_client.get_secret_value(SecretId=arn, VersionStage="AWSCURRENT") # Now try to get the secret version, if that fails, put a new secret try: service_client.get_secret_value(SecretId=arn, VersionId=token, VersionStage="AWSPENDING") logger.info("createSecret: Successfully retrieved secret for %s." % arn) except service_client.exceptions.ResourceNotFoundException: # Get exclude characters from environment variable exclude_characters = os.environ['EXCLUDE_CHARACTERS'] if 'EXCLUDE_CHARACTERS' in os.environ else '/@"\'\\' # Generate a random password passwd = service_client.get_random_password(ExcludeCharacters=exclude_characters) # Put the secret, using rotation_token service_client.put_secret_value(SecretId=arn, ClientRequestToken=token, SecretString=passwd['RandomPassword'], VersionStages=['AWSPENDING'], RotationToken=rotation_token) logger.info("createSecret: Successfully put secret for ARN %s and version %s." % (arn, token)) ``` 1. Upload the updated Lambda function code After updating your Lambda function code, [upload it to rotate your secret](https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-zip.html#configuration-function-update). ## Error: "Error when executing lambda ** during ** step" Intermittent rotation failures If you're experiencing intermittent secret rotation failures with your Lambda function getting stuck in a loop of sets, for example between **CreateSecret** and **SetSecret**, the issue may be related to concurrency settings. ### Concurrency troubleshooting steps **Warning** Setting the provisioned concurrency parameter to a value lower than 10 can cause throttling due to insufficient execution threads for the Lambda function. For more information, see [ Understanding reserved concurrency and provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html#reserved-and-provisioned) in the AWS Lambda AWS Lambda Developer Guide. 1. Check and adjust Lambda concurrency settings: + Verify that `reserved_concurrent_executions` is not set too low (for example, 1) + If using reserved concurrency, set it to at least 10 + Consider using unreserved concurrency for more flexibility 1. For provisioned concurrency: + Don't set the provisioned concurrency parameter explicitly (for example, in Terraform). + If you must set it, use a value of at least 10. + Test thoroughly to make sure the chosen value works for your use case. 1. Monitor and adjust concurrency: + Calculate concurrency using this formula: Concurrency = (average requests per second) \$1 (average request duration in seconds). For more information, see [Estimating reserved concurrency](https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html#estimating-reserved-concurrency). + Observe and record values during rotations to determine the appropriate concurrency settings. + Be careful when setting low concurrency values. They can cause throttling if there aren't enough available execution threads. For more information on configuring Lambda concurrency, see [Configuring reserved concurrency](https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html) and [Configuring provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html) in the AWS Lambda Developer Guide. # Rotation schedules Secrets Manager rotates your secret on a schedule during a rotation window that you set. To set the schedule and window, you use a **cron()** or **rate()** expression along with a window duration. Secrets Manager rotates your secret at any time during the rotation window. You can rotate a secret as often as every four hours within a rotation window as small as one hour. To turn on rotation, see: + [Managed rotation for AWS Secrets Manager secrets](rotate-secrets_managed.md) + [Set up automatic rotation for Amazon RDS, Amazon Aurora, Amazon Redshift, or Amazon DocumentDB secrets](rotate-secrets_turn-on-for-db.md) + [Set up automatic rotation for non-database AWS Secrets Manager secrets](rotate-secrets_turn-on-for-other.md) Secrets Manager rotation schedules use UTC time zone. ## Rotation windows A Secrets Manager rotation window is similar to a maintenance window. You set the rotation window when you want your secret rotated, and Secrets Manager rotates your secret at some time during the rotation window. Secrets Manager rotation windows always start on the hour. For a rotation schedule that uses a `rate()` expression in days, the rotation window starts at midnight. You can set the start time for the rotation window by using a `cron()` expression. For examples, see [Cron expressions](#rotate-secrets_schedule-cron). By default, the rotation window closes after one hour for a rotation schedule in *hours*, and at the end of the day for a rotation schedule in *days*. To change the length of the rotation window, set the **Window duration**. You can set the rotation window as small as one hour. The rotation window must not extend into the next rotation window. In other words, for a rotation schedule in *hours*, confirm that the rotation window is less than or equal to the number of hours between rotations. For a rotation schedule in *days*, confirm that the start hour plus the window duration is less than or equal to 24 hours. ## Rate expressions Secrets Manager rate expressions have the following format, where *Value* is a positive integer and *Unit* can be `hour`, `hours`, `day`, or `days`: ``` rate(Value Unit) ``` You can rotate a secret as often as every four hours. The maximum rotation period is 999 days. Examples: + `rate(4 hours)` means the secret is rotated every four hours. + `rate(1 day)` means the secret is rotated every day. + `rate(10 days)` means the secret is rotated every 10 days. ## Cron expressions Secrets Manager cron expressions have the following format: ``` cron(Minutes Hours Day-of-month Month Day-of-week Year) ``` A cron expression that includes increments of hours resets each day. For example, `cron(0 4/12 * * ? *)` means 4:00 AM, 4:00 PM, and then the next day 4:00 AM, 4:00 PM. Secrets Manager rotation schedules use UTC time zone. | Example schedule | Expression | | --- | --- | | Every eight hours starting at midnight. | `cron(0 /8 * * ? *)` | | Every eight hours starting at 8:00 AM. | `cron(0 8/8 * * ? *)` | | Every ten hours, starting at 2:00 AM. The rotation windows will start at 2:00, 12:00, and 22:00, and then the next day at 2:00, 12:00, and 22:00. | `cron(0 2/10 * * ? *)` | | Every day at 10:00 AM. | `cron(0 10 * * ? *)` | | Every Saturday at 6:00 PM. | `cron(0 18 ? * SAT *)` | | The first day of every month at 8:00 AM. | `cron(0 8 1 * ? *)` | | Every three months on the first Sunday at 1:00 AM. | `cron(0 1 ? 1/3 SUN#1 *)` | | The last day of every month at 5:00 PM. | `cron(0 17 L * ? *)` | | Monday through Friday at 8:00 AM. | `cron(0 8 ? * MON-FRI *)` | | First and 15th day of every month at 4:00 PM. | `cron(0 16 1,15 * ? *)` | | First Sunday of every month at midnight. | `cron(0 0 ? * SUN#1 *)` | | Starting in January, every 11 months on the first Monday at midnight. | `cron(0 0 ? 1/11 2#1 *)` | ### Cron expression requirements in Secrets Manager Secrets Manager has some restrictions on what you can use for cron expressions. A cron expression for Secrets Manager must have **0** in the minutes field because Secrets Manager rotation windows start on the hour. It must have **\$1** in the year field, because Secrets Manager does not support rotation schedules that are more than a year apart. The following table shows the options you can use. | **Fields** | **Values** | **Wildcards** | | --- | --- | --- | | Minutes | Must be 0 | None | | Hours | 0–23 | Use **/** (forward slash) to specify increments. For example `2/10` means every 10 hours beginning at 2:00 AM. You can rotate a secret as often as every four hours. | | Day-of-month | 1–31 | Use **,** (comma) to include additional values. For example `1,15` means the first and 15th day of the month. Use **-** (dash) to specify a range. For example `1–15` means days 1 through 15 of the month. Use **\$1** (asterisk) to includes all values in the field. For example `*` means every day of the month. The **?** (question mark) wildcard specifies one or another. You can't specify the `Day-of-month` and `Day-of-week` fields in the same cron expression. If you specify a value in one of the fields, you must use a **?** (question mark) in the other. Use **/** (forward slash) to specify increments. For example, `1/2` means every two days starting on day 1, in other words, day 1, 3, 5, and so on. Use **L** to specify the last day of the month. Use ***DAY*L** to specify the last named day of the month. For example `SUNL` means the last Sunday of the month. | | Month | 1–12 or JAN–DEC | Use **,** (comma) to include additional values. For example, `JAN,APR,JUL,OCT` means January, April, July, and October. Use **-** (dash) to specify a range. For example `1–3` means months 1 through 3 of the year. Use **\$1** (asterisk) to includes all values in the field. For example `*` means every month. Use **/** (forward slash) to specify increments. For example, `1/3` means every third month, starting on month 1, in other words month 1, 4, 7, and 10. | | Day-of-week | 1–7 or SUN–SAT | Use **\$1** to specify the day of the week within a month. For example, `TUE#3` means the third Tuesday of the month. Use **,** (comma) to include additional values. For example `1,4` means the first and fourth day of the week. Use **-** (dash) to specify a range. For example `1–4` means days 1 through 4 of the week. Use **\$1** (asterisk) to includes all values in the field. For example `*` means every day of the week. The **?** (question mark) wildcard specifies one or another. You can't specify the `Day-of-month` and `Day-of-week` fields in the same cron expression. If you specify a value in one of the fields, you must use a **?** (question mark) in the other. Use **/** (forward slash) to specify increments. For example, `1/2` means every second day of the week, starting on the first day, so day 1, 3, 5, and 7. Use **L** to specify the last day of the week. | | Year | Must be \$1 | None | # Rotate an AWS Secrets Manager secret immediately Rotate a secret immediately You can only rotate a secret that has rotation configured. To determine whether a secret has been configured for rotation, in the console, view the secret and scroll down to the **Rotation configuration** section. If **Rotation status** is **Enabled**, then the secret is configured for rotation. If not, see [Rotate AWS Secrets Manager secrets](rotating-secrets.md). **To rotate a secret immediately (console)** 1. Open the Secrets Manager console at [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/). 1. Choose your secret. 1. On the secret details page, under **Rotation configuration**, choose **Rotate secret immediately**. 1. In the **Rotate secret** dialog box, choose **Rotate**. ## AWS CLI **Example Rotate a secret immediately** The following [https://docs.aws.amazon.com//cli/latest/reference/secretsmanager/rotate-secret.html](https://docs.aws.amazon.com//cli/latest/reference/secretsmanager/rotate-secret.html) example starts an immediate rotation. The secret must already have rotation configured. ``` $ aws secretsmanager rotate-secret \ --secret-id MyTestSecret ``` # Find secrets that aren't rotated You can use AWS Config to evaluate your secrets to see if they are rotating in compliance with your standards. You define your internal security and compliance requirements for secrets using AWS Config rules. Then AWS Config can identify secrets that don't conform to your rules. You can also track changes to secret metadata, rotation configuration, the KMS key used for secret encryption, the Lambda rotation function, and tags associated with a secret. If you have secrets in multiple AWS accounts and AWS Regions in your organization, you can aggregate that configuration and compliance data. For more information, see [Multi-account Multi-Region data aggregation](https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html). **To assess whether secrets are rotating** 1. Follow the instructions on [Evaluating your resources with AWS Config rules](https://docs.aws.amazon.com/config/latest/developerguide/evaluating-your-resources.html), and choose from of the following rules: + `[secretsmanager-rotation-enabled-check](https://docs.aws.amazon.com/config/latest/developerguide/secretsmanager-rotation-enabled-check.html)` — Checks whether rotation is configured for secrets stored in Secrets Manager. + `[secretsmanager-scheduled-rotation-success-check](https://docs.aws.amazon.com/config/latest/developerguide/secretsmanager-scheduled-rotation-success-check.html)`— Checks whether the last successful rotation is within the configured rotation frequency. The minimum frequency for the check is daily. + `[secretsmanager-secret-periodic-rotation](https://docs.aws.amazon.com/config/latest/developerguide/secretsmanager-secret-periodic-rotation.html)`— Checks whether secrets were rotated within the specified number of days. 1. Optionally, configure AWS Config to notify you when secrets aren't compliant. For more information, see [Notifications that AWS Config sends to an Amazon SNS topic](https://docs.aws.amazon.com/config/latest/developerguide/notifications-for-AWS-Config.html). # Cancel automatic rotation in Secrets Manager Cancel automatic rotation If you configured [automatic rotation](rotating-secrets.md) for a secret and you want to stop rotating it, you can cancel rotation. **To cancel automatic rotation** 1. Open the Secrets Manager console at [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/). 1. Choose your secret. 1. On the secret details page, under **Rotation configuration**, choose **Edit rotation**. 1. In the **Edit rotation configuration** dialog box, turn off **Automatic rotation**, and then choose **Save**. Secrets Manager retains the rotation configuration information so that you can use it in the future if you decide to turn rotation back on.