Boto3 Secrets Manager – Complete Tutorial

AWS Secrets Manager helps you protect the secrets needed to access your applications, services, and IT resources. This service lets you rotate, manage, and retrieve database credentials, API keys, passwords, and other secrets throughout their lifecycle. AWS Boto3 is the Python Software Development Kit (SDK) for the AWS cloud platform that helps to interact with AWS resources from Python code. This Boto3 Secrets Manager tutorial covers how to use the Boto3 library to manage your secrets in AWS Secrets Manager.

What is AWS Secrets Manager?

There are many scenarios where you might need to use credentials, tokens, API keys, etc., to access certain services. For example, you might need to use SQL server credentials to access a DB from the application. Storing those credentials in the codebase as a plain text file is a security vulnerability. Anyone with access to your codebase could read those secrets and get unauthorized access to your services to perform malicious activities. The AWS Secrets Manager allows you to store sensitive information and get access to it by keys that you can safely save in your application config file or code.

AWS Parameter Store vs. Secrets Manager

The AWS Secrets Manager is designed specifically for confidential information (like database credentials and API Keys) that needs to be encrypted, so the creation of a secret entry has encrypted by default.

The AWS Systems Manager Parameter Store is designed to cater to a wider use case, not just secrets or passwords but also application configuration variables like URLs, custom settings, etc.

Both Secrets Manager and Parameter Store can use AWS KMS to encrypt values. The AWS Parameter Store provides the option to store data unencrypted. On the other hand, with Secrets Manager, there’s no option to store unencrypted data. AWS KMS provides a highly available key storage, management, and auditing solution for you to encrypt data within your applications and control the encryption of stored data across AWS services. With KMS and with the help of IAM, you can use policies to control permissions on which IAM users and roles have permission to decrypt the value.

Parameter Store comes with no additional charges, but there is a limit on the number of parameters you can store, currently 10,000.

Parameter Store stores individual values using a hierarchical key. You can create keys like /my-app/prod/db/password, /my-app/dev/db/password and you can retrieve them individually or all keys that start with /my-app.

aws ssm get-parameters-by-path /my-app/prod

You can write your function that updates credentials managed by Parameter Store and invoke it via a CloudWatch scheduled event or EventBridge.

AWS Secrets Manager does come with additional costs. Learn more about AWS Secrets Manager pricing from here.

AWS Secret Manager has built-in integration for rotating MySQL, Postgres SQL, Amazon Aurora, and RDS database credentials. For services with which it doesn’t integrate, it allows Lambda functions to rotate these other forms of stored secrets. Similar to other AWS services, the built-in integration will only grow to include more AWS services in the future. Eventually, you can manage all secrets of your entire AWS platform from one place.

AWS Secrets Manager also can generate random secrets. You can randomly generate passwords in CloudFormation and store the password in Secrets Manager.

You can share AWS Secrets Manager secrets across multiple accounts.

How do I access AWS Secrets Manager in Python?

To access AWS Secrets Manager, you must install Boto3, an AWS SDK for Python. Also, you need to have AWS CLI configured to use the Boto3 library. Boto3 uses your AWS Access Key Id and Secret Access Key to programmatically manage AWS resources.

First, you must install AWS CLI from here, depending on the Operating System.

After installing AWS CLI, run aws configure in your terminal to configure your AWS account with AWS CLI. It will prompt Access Key Id and Secret Access Key which you can find from IAM in the AWS Console.

Second, install the boto3 library using the pip install boto3 command.

For more in-depth information, we recommend you check out the Introduction to Boto3 library and How to use aws-vault to securely access multiple AWS accounts articles.

Now you’re good to go.

How do I get my secret from AWS Secrets Manager?

In the AWS Console, search for the Secret Manager; you will see all your Secrets there.

Click on one of the secrets and then click on the Retrieve secret value button to see the secret value.

Here’s what you should get:

Boto3 Secrets Manager – create a secret

Secrets Manager stores the encrypted secret data in a collection of “versions” associated with the secret. Each version contains a copy of the encrypted secret data. Each version is associated with one or more “staging labels” that identify the version in the rotation cycle.

The Boto3 Secrets Manager client is a low-level class that provides methods to connect to AWS Secrets Manager, similar to the AWS API service. All service APIs in the Boto3 client map 1:1 to the AWS service API.

You can provide the secret data to be encrypted by putting text in either the SecretString parameter or binary data in the SecretBinary parameter, but not both. If you include SecretString or SecretBinary then Secrets Manager also creates an initial secret version and automatically attaches the staging label AWSCURRENT to the new version.

#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.create_secret(
    Name='DatabaseProdSecrets',
    SecretString='{"username": "prod", "password": "hello-world-prod"}'
)

Output:

If you need to provide a custom KMS key, you can use the KmsKeyId parameter in create_secret() method that Specifies the ARN, Key ID, or alias of the Amazon Web Services KMS customer master key (CMK) to be used to encrypt the SecretString or SecretBinary values in the versions stored in this secret.

If you don’t provide theKmsKeyId, then Secrets Manager uses the account’s default CMK (the one named aws/secretsmanager ). If an Amazon Web Services KMS CMK with that name doesn’t exist, then Secrets Manager will create it for you automatically the first time it needs to encrypt a version’s SecretString or SecretBinary fields.

How to list secrets in AWS Secrets Manager using Boto3?

You can use the list_secrets() method to list all secrets stored in AWS Secrets Manager. When listing secrets, you can also filter and limit the number of results to a specific number.

Note: The encrypted fields SecretString and SecretBinary are not included in the output. To get that information, you need to call the GetSecretValue operation.

#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.list_secrets()
print(response['SecretList'])

Here’s an execution output:

How to retrieve a secret value from AWS Secrets Manager using Boto3?

To retrieve a secret value from AWS Secrets Manager using Boto3, you need to use theget_secret_value() method.

The following code example will get the secret with SecretId (or Name when creating) of DatabaseProdSecrets. For more information, please read the Boto3 Secrets Manager documentation.

You also need to parse the SecretString value using the json.loads which converts JSON string into the Python dictionary so that you can access the items of a dictionary.

#!/usr/bin/env python3
import boto3 
import json 
client = boto3.client('secretsmanager')
response = client.get_secret_value(
    SecretId='DatabaseProdSecrets'
)
database_secrets = json.loads(response['SecretString'])
print(database_secrets['password'])

Here’s an execution output:

Permissions required to retrieve a secret

To retrieve the secret, you need to allow the secretsmanager:GetSecretValue API call in your IAM policy.

If you’re using a customer-managed Amazon Web Services KMS key to encrypt the secret, you also need to have kms:Decrypt permission.

Alternatively, you can attach the SecretsManagerReadWrite policy to the user who needs permission to manage AWS Secrets Manager.

Retrieve secret values from the Python code

To retrieve a secret value from AWS Secrets Manager using Boto3, you need to use theget_secret_value() method.

#!/usr/bin/env python3
import boto3
import json 
client = boto3.client('secretsmanager')
response = client.get_secret_value(
    SecretId='DatabaseProdSecrets'
)
database_secrets = json.loads(response['SecretString'])
print(database_secrets['password'])

Retrieve secret values in Bash

Using the AWS CLI, you can retrieve secret values in the Bash shell

aws secretsmanager get-secret-value --secret-id <SecretId>

Retrieve secret values in Powershell

The cmdlets in the AWS Tools for PowerShell for each service are based on the methods provided by the AWS SDK for the service.

You can use Get-SECSecretValue cmdlets to retrieve secrets. Read more about this cmdlet from here.

Get-SECSecretValue -SecretId <SecretId>

How to update an existing secret in AWS Secrets Manager using Boto3?

There are two methods for updating secrets in Boto3.

The first one is put_secret_value(). This method creates a new version and attaches it to the secret

#!/usr/bin/env python3
import boto3 
import json 
client = boto3.client('secretsmanager')
response = client.put_secret_value(
    SecretId='DatabaseProdSecrets',
    SecretString='{"username": "prod", "password": "hello-world-updated2"}'
)
print(response)

Output:

The second one is the update_secret() method. This method modifies many of the details of the specified secret. If you include a ClientRequestToken and either SecretString or SecretBinary then it also creates a new version attached to the secret.

#!/usr/bin/env python3
import boto3 
import json
client = boto3.client('secretsmanager')
response = client.update_secret(
    SecretId='DatabaseProdSecrets',
    Description='Description updated'
)
print(response)

Here’s an execution output:

How to create a new version of the secret in AWS Secrets Manager using Boto3?

You can use either put_secret_value or update_secret to create a new version of the secret.

The put_secret_value creates a new version and attaches it to the secret.

The update_secret method creates a new version attached to the secret when a ClientRequestToken and either SecretString or SecretBinary parameters are used.

How to delete a secret in AWS Secrets Manager using Boto3?

You can use the delete_secret() function from Boto3 to delete secret and all of its versions. You can optionally include a recovery window during which you can restore the secret. If you don’t specify a recovery window value, the secret will be deleted within 30 days.

At any time before the recovery window ends, you can use RestoreSecret to remove the DeletionDate and cancel the deletion of the secret.

#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.delete_secret(
    SecretId='DatabaseProdSecrets',
    RecoveryWindowInDays=10,
    ForceDeleteWithoutRecovery=False
)
print(response)

Here’s an execution output:

To restore deleted secrets before the recovery window end, you can use the restore_secret() method.

#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.restore_secret(
    SecretId='DatabaseProdSecrets'
)
print(response)

Here’s an execution output:

Summary

This article covered Python to interact with AWS Secret Manager to create, update, and delete secrets using the Boto3 Python SDK.

Additional Learning Resources

Free hands-on AWS workshops

We recommend the following free AWS Lambda Workshops to get more hands-on experience on the subject:

• AWS Lambda Workshop
• Migrate Spring Boot applications to AWS Lambda
• Building CI/CD pipelines for Lambda canary deployments using AWS CDK
• Content Generation Lambda@Edge

Also, we recommend the following free AWS Step Functions Workshops:

• The AWS Step Functions Workshop

Paid courses

From our experience, these are the best hands-on paid learning materials today related to Serverless, AWS Lambda, and Step Functions: