This module shows how to use AWS Secrets Manager for rotating the password for a publicly-facing Amazon RDS database.
Here's a diagram of what we are going to build.
This environment consists of a VPC in the us-east-1 region with two public subnets. One subnet containts an Amazon EC2 bastion host running Amazon Linux 2 used for running AWS CLI commands. The other subnet contains an Amazon RDS MySQL database instance. Both subnets are exposed to the internet. When we initiate a rotation of the RDS database, the AWS Lambda function that Secrets Manager uses will reach out to the Amazon RDS database to perform the rotation.
| Important Security Notes |
|---|
| For the sake of simplicity, the CloudFormation template used in the module creates shell scripts that use jq to parse the secret value into shell variables to allow for easy command line manipulation. This is NOT a security best practice for a production environment. In a production environment, we recommend that you don't store passwords in environment variables, and work with them in plaintext at the command line. |
| Also, the MySQL database that's configured in this module is OPEN TO THE PUBLIC INTERNET on port 3306, again for simplicity in setup for the module. The Lambda function must be able to access both the public AWS Secrets Manager service endpoint and your database. Making the database publicly accessible is the easiest way to do this. We strongly recommend that you follow the clean up instructions at the end of this module to remove the CloudFormation stack and the database. |
In order to complete this workshop you'll need the following:
- an AWS Account with administrator access to services used in the module including AWS Secrets Manager, AWS CloudFormation, Amazon EC2, Amazon RDS, and Amazon VPC
- an Amazon EC2 key pair in the region you have chosen to create a bastion instance from which you will execute various commands
- a program (e.g. putty to connect to the Amazon EC2 bastion host using the secure shell (ssh) protocol
You will incur charges for the services used in these modules. The pricing for each service is available on that service's pricing page.
We will be adding presentation materials. Stay tuned!
You need to choose a region that offers AWS Secrets Manager and Amazon RDS. For information about service availability, check the region table.
Tip: The AWS region name is always listed in the upper-right corner of the AWS Management Console, in the navigation bar.
Make a note of the AWS region name, for example, US East (N. Virginia). For more information about regions, see: AWS Regions and Endpoints.
(expand for details)
In this section, you will use AWS CloudFormation to build the environment shown in the diagram above.
Complete all the steps below unless they are marked "optional."
-
2.1. Download the AWS Cloudformation template and save it to your workstation.
-
2.2. In the AWS Management Console, Under Management Tools, Select CloudFormation.
-
2.3. Click the Create stack button.
-
2.4. Select the Upload a template to Amazon S3 radio button.
-
2.5. Click the Choose file button and use the file navigation pop-up to select the CloudFormation template you just downloaded.
-
2.6. Click the Next button.
Enter the values below into Stack name and parameter fields:
| Field name | Value |
|---|---|
| Stack name | You can choose or just use smdemo |
| Choose an Availability Zone | Select any Availability Zone |
| Choose a second Availability Zone | Select a different Availability Zone |
| Choose a key pair | Select the Amazon EC2 key pair |
| Enter the name of the database | Accept the default value of smdemo |
| Enter the TCP port for the database endpoint | Accept the default value of 3306 |
| Enter a prefix for the Name tag | Accept the default value of smdemo |
| Enter the value for the Project tag | Accept the default value of smproj |
-
2.7. Click the Next button.
-
2.8. On the Options page, click the Next button.
-
2.9. On the Review page, check the acknowledgement box and click the Create button. AWS CloudFormation will the build the resource stack. This may take 15-20 minutes to complete. Wait until the status of the stack changes to "CREATE_COMPLETE." You can periodically refresh the page to update the status. If the stack creation fails, look at the bottom of the page and select the Events tab. You should be able to see the reason for the failure in the Status Reason column. A common reason for a failure is not having specified some of the parameters when creating the stack.
-
2.10. Let's take a look at the Output section of the CloudFormation stack.
The outputs values include the RDS master username and password, both of which are set to randon alphanumeric strings for added security. You will store the DBUser and DBPassword values as a secret in Secrets Manager. You will se the DBEndpoint value to help you select the correct database in Secrets Manager. Another output value is the IP address of the bastion host. You will need this IP address to set up a connection to the bastion host later in this module. Keep this window open.
(expand for details)
In this section, you will store the RDS database credentials in AWS Secrets Manager. You will use the DBUser and DBPassword values that were output from the AWS CloudFormation.
-
3.1. Open the AWS Secrets Manager Console in a new tab or window.
-
3.2. Click Store a new secret.
-
3.3. Select the Credentials for RDS database radio button.
-
3.4. Copy the values for the DBUser and DBPassword CloudFormation output values that you got from item 2.10 above into the User name and Password fields respectively. Scroll down to the bottom of the page and you will see a list of your RDS instances. Using the DBendpoint output value that you got from item 2.10 above, select the RDS instance.
-
3.5. Click Next.
-
3.6. Enter a name for the secret. You can pick a name or just use smdemo as shown below.
-
3.7. Click Next.
-
3.8 Select Disable automatic rotation and then click Next. We will enable rotation later in this module.
- 3.9. Click Store.
You have now stored your secret value as shown below.
(expand for details)
In this section, you will connect to the bastion host so you can run scripts that the CloudFormation template has created on the instance.
Complete all the steps below unless they are marked "optional."
-
4.1. Locate the Amazon EC2 key pair thst you provided to AWS CloudFormation.
-
4.2. Locate the IP address of the bastion host that appears as an output to AWS CloudFormation
-
4.3. Connect to the bastion host using the EC2 key pair and the IP address. Your user ID will be [email protected] where X.X.X.X is the bastion IP address. Remain logged in for the remainder of this module.
-
4.4. Use the ls command to list the contents of the home directory. You will see two shell scripts.
-
mysql.oldway.sh - This shell script connects to the database the "old" way, using a hard-coded password.
-
mysql.newway.sh - This shell script connects to the database the "new" way, using AWS Secrets Manager.
-
-
4.5. Let's take a look at the file mysql.oldway.sh. You can use the cat command to do this. In the example below, the values PASSWORD, USER, and ENDPOINT represent the hard-coded database password, username, and host endpoint.
#/bin/bash
# mysql.oldway.sh
# This is the old way of accessing a database, with a hard-coded password.
# This script will only work right after the CloudFormation template runs.
# After you store and rotate the secret, you will need to use the
# mysql.newway.sh script.
mysql \
-pPASSWORD \
-u USER \
-P 3306 \
-h ENDPOINT
$
- 4.6. Now let's try this script by running the following commands. The first command invokes the script. The subsequent commands select the database, display the table names in the database, query the rows in the table, and exit MySQL.
./mysql.oldway.sh
use smdemo;
show tables;
select * from bookinfo;
quit;
You can see an example of the output below. This shows that you can access the database, the "old" way, with a hard-coded user name and password. You may be wondering why MariaDB appears in the image below. Amazon Linux 2 includes the MariaDB port of the mysql command as an "extras" module. The mysql program is compatible with both MySQL and MariaDB.
- 4.7. Let's take a look at the file mysql.newway.sh. You can use the cat command to do this. As mentioned above, for the sake of simplicity, the scripts used in the tutorial use jq to parse the secret value into shell variables to allow for easy command line manipulation. This is NOT a security best practice for a production environment. In a production environment, we recommend that you don't store passwords in environment variables, and work with them in plaintext at the command line.
$ cat mysql.newway.sh
#/bin/bash
# This is the new way of accessing a database, with AWS Secrets Manager.
if [ $# -ne 1 ]
then
echo usage: $0 SecretName
exit 1
fi
secret=$(aws secretsmanager get-secret-value --secret-id $1 --region us-east-1 | jq .SecretString | jq fromjson)
user=$(echo $secret | jq -r .username)
password=$(echo $secret | jq -r .password)
endpoint=$(echo $secret | jq -r .host)
port=$(echo $secret | jq -r .port)
mysql \
-p$password \
-u $user \
-P $port \
-h $endpoint
$
- 4.8. Now let's try this script by running the following commands. The first command invokes the script. Note that you must specify the name of the secret! The subsequent commands select the database, display the table names in the database, query the rows in the table, and exit MySQL.
./mysql.newway.sh smdemo
use smdemo;
show tables;
select * from bookinfo;
quit;
You can see an example of the output below. This shows that you can access the database, the "new" way, using AWS Secrets Manager.
(expand for details)
In this section, you will enable the rotation of the secret you created in AWS Secrets Manager.
-
5.1. Go to the main screen of the AWS Secrets Manager console.
-
5.2. Click on the secret that you previously created.
-
5.3. Click Edit rotation.
-
5.4. Select Enable automatic rotation. Choose 30 days for the rotation interval. Click Use this secret because we will be using the credentials of this secret to access the database and then rotate the same credentials. Click Save to begin the process.
- 5.5. You will see a message telling you that the rotation is beginning and that you should remain on the page until it is complete.
A message will appear when the rotation is complete.
- 5.6. Click Retrieve secret value to see the new password value.
(expand for details)
Let's try to connect to the database again, both the "old" way with a hard-coded password, the "new" way with AWS Secrets Manager.
- 6.1. On the bastion host, repeat step 4.6 with the mysql.oldway.sh script. You should receive an error message (access denied) because the mysql.oldway.sh script has the same hard-coded password.
- 6.2. Repeat step 4.8 with the mysql.newway.sh script. You should be able to connect to the database just as you did before since this script uses AWS Secrets Manager to fetch the updated credentials.
| Important Security Note |
|---|
| Now that you have seen how AWS Secrets Manager can rotate the credentials for a publicly-facing database, please follow these steps to remove the resources you created, including the publicly-facing database. |
-
7.1. Delete the CloudFormation stack that you created in steps 2.1 through 2.9. The stack deletion process may take several minutes to complete.
-
7.2. Delete the secret you created in Secrets Manager. Note that when you delete a secret, the deletion is scheduled for a minimum of seven days in the future.
-
7.3. When you enabled rotation on your secret, AWS Secrets Manager used AWS CloudFormation to create an AWS Lambda function to do the rotation using the AWS Serverless Application Repository. Go to the AWS CloudFormation console and delete this stack as well. The name of the stack begins with the following string:
aws-serverless-repository-SecretsManagerRDSMySQLRotationSingleUser
Look for a stack with this naming convention that was created at about the same time as you enabled rotation.
You have completed this module. Visit the main Secrets Managager workshop site for more workshops as they become available.