Working with Secrets Manager in Python using Boto3

AWS Secrets Manager helps you protect secrets needed to access your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, passwords, and other secrets throughout their lifecycle. AWS Boto3 is the Python Software Development Kit (SDK) for AWS cloud platform that helps to interact with AWS resources from Python code. This article will cover how you can use the Boto3 library to manage your secrets in AWS Secrets Manager.

What is AWS Secrets Manager?

There are many scenarios where you might need to use credentials, tokens, API keys, etc., to access certain services. For example, you might need to use SQL server credentials to access a DB from the application. Storing those credentials in the codebase as a plain text file is a security vulnerability. Anyone with access to your codebase would be able to read those secrets, get unauthorized access to your services to perform malicious activities. The AWS Secrets Manager allows you to store sensitive information and get access to it by keys that you can safely save in your application config file or code.

AWS Parameter Store vs. Secrets Manager

The AWS Secrets Manager is designed specifically for confidential information (like database credentials, API Keys) that needs to be encrypted, so the creation of a secret entry has encrypted by default.

The AWS Systems Manager Parameter Store is designed to cater to a wider use case, not just secrets or passwords, but also application configuration variables like URLs, custom settings, etc.

Both Secrets Manager and Parameter Store can use AWS KMS to encrypt values. The AWS Parameter Store provides the option to store data unencrypted. On the other hand, with Secrets Manager, there’s no option to store unencrypted data. AWS KMS provides a highly available key storage, management, and auditing solution for you to encrypt data within your own applications and control the encryption of stored data across AWS services. With KMS, and with the help of IAM, you can use policies to control permissions on which IAM users and roles have permission to decrypt the value.

Parameter Store comes with no additional charges, but there is a limit on the number of parameters you can store, currently 10,000.

Parameter Store stores individual values using a hierarchical key. You can create keys like /my-app/prod/db/password, /my-app/dev/db/password and you can retrieve them individually or all keys that start with /my-app.

aws ssm get-parameters-by-path /my-app/prod

You can write your own function that updates credentials managed by Parameter Store and invoking it via a CloudWatch scheduled event or EventBridge.

AWS Secrets Manager does come with additional costs. Learn more about AWS Secrets Manager pricing from here.

AWS Secret Manager has built-in integration for rotating MySQL, Postgres SQL, Amazon Aurora, and RDS database credentials. For services that it doesn’t have integration with, it allows Lambda functions to rotate these other forms of stored secrets. Similar to other AWS services, the built-in integration will only grow to include more AWS services in the future. This means eventually, you will be able to manage all secrets of your entire AWS platform from one place.

AWS Secrets Manager also has the ability to generate random secrets. You can randomly generate passwords in CloudFormation and store the password in Secrets Manager.

You can share AWS Secrets Manager secrets across multiple accounts.

How do I access AWS Secrets Manager in Python?

To access AWS Secrets Manager, you need to install Boto3, an AWS SDK for Python. Also, you need to have AWS CLI configured to use the Boto3 library. Boto3 uses your AWS Access Key Id and Secret Access Key to programmatically manage AWS resources.

First, you need to install AWS CLI from here, depending on the Operating System.

After installing AWS CLI, run aws configure in your terminal to configure your AWS account with AWS CLI. It will prompt Access Key Id and Secret Access Key which you can find from IAM in the AWS Console.

Second install boto3 library using pip install boto3

For more in-depth information, we recommend you check out the Introduction to Boto3 library article.

Now you’re good to go.

How do I get my secret from AWS Secrets Manager?

In the AWS Console, search for Secret Manager and there you will see all of your Secrets.

Click on one of the secrets and then click on Retrieve secret value button to see the secret value.

Here’s what you should get:

How to create a secret in AWS Secrets Manager using Boto3?

Secrets Manager stores the encrypted secret data in a collection of “versions” associated with the secret. Each version contains a copy of the encrypted secret data. Each version is associated with one or more “staging labels” that identify the version in the rotation cycle.

The Boto3 Secrets Manager client is a low-level class that provides methods to connect to AWS Secrets Manager similar to the AWS API service. All service APIs that are available in the Boto3 client map 1:1 to the AWS service API.

You can provide the secret data to be encrypted by putting text in either the SecretString parameter or binary data in the SecretBinary parameter, but not both. If you include SecretString or SecretBinary then Secrets Manager also creates an initial secret version and automatically attaches the staging label AWSCURRENT to the new version.

#!/usr/bin/env python3

import boto3

client = boto3.client('secretsmanager')

response = client.create_secret(
    Name='DatabaseProdSecrets',
    SecretString='{"username": "prod", "password": "hello-world-prod"}'
)

Output:

If you need to provide a custom KMS key, you can use the KmsKeyId parameter in create_secret() method that Specifies the ARN, Key ID, or alias of the Amazon Web Services KMS customer master key (CMK) to be used to encrypt the SecretString or SecretBinary values in the versions stored in this secret.

If you don’t provide the KmsKeyId, then Secrets Manager use the account’s default CMK (the one named aws/secretsmanager ). If an Amazon Web Services KMS CMK with that name doesn’t exist, then Secrets Manager will create it for you automatically the first time it needs to encrypt a version’s SecretString or SecretBinary fields.

How to list secrets in AWS Secrets Manager using Boto3?

You can use the list_secrets() method to list all secrets stored in AWS Secrets Manager. When listing secrets, you can also filter and limit the number of results to specific number.

Note: The encrypted fields SecretString and SecretBinary are not included in the output. To get that information, you need to call the GetSecretValue operation.

#!/usr/bin/env python3

import boto3

client = boto3.client('secretsmanager')

response = client.list_secrets()

print(response['SecretList'])

Here’s an execution output:

How to retrieve a secret value from AWS Secrets Manager using Boto3?

To retrieve a secret value from AWS Secrets Manager using Boto3, you need to use the get_secret_value() method.

The following code example will get the secret with SecretId (or Name when creating) of DatabaseProdSecrets. For more information, please read the Boto3 Secrets Manager documentation.

You also need to parse the SecretString value using the json.loads which converts JSON string into the Python dictionary so that you can access the items of a dictionary.

#!/usr/bin/env python3

import boto3 
import json 

client = boto3.client('secretsmanager')

response = client.get_secret_value(
    SecretId='DatabaseProdSecrets'
)

database_secrets = json.loads(response['SecretString'])

print(database_secrets['password'])

Here’s an execution output:

Permissions required to retrieve a secret

To retrieve secret, you need to allow the secretsmanager:GetSecretValue API cal in your IAM policy.

If you’re using customer-managed Amazon Web Services KMS key to encrypt the secret, you also need to have kms:Decrypt permission.

Alternatively, you can attach the SecretsManagerReadWrite policy to the user who needs permissions to manage AWS Secrets Manager.

Retrieve a secret values from the Python code

To retrieve a secret value from AWS Secrets Manager using Boto3, you need to use the get_secret_value() method.

#!/usr/bin/env python3

import boto3
import json 

client = boto3.client('secretsmanager')

response = client.get_secret_value(
    SecretId='DatabaseProdSecrets'
)

database_secrets = json.loads(response['SecretString'])

print(database_secrets['password'])

Retrieve a secret values in Bash

Using the AWS CLI, you can retrieve secret values in the Bash shell

aws secretsmanager get-secret-value --secret-id <SecretId>

Retrieve a secret values in Powershell

The cmdlets in the AWS Tools for PowerShell for each service are based on the methods provided by the AWS SDK for the service.

You can use Get-SECSecretValue cmdlets to retrieve secrets. Read more about this cmdlet from here.

Get-SECSecretValue -SecretId <SecretId>

How to update an existing secret in AWS Secrets Manager using Boto3?

There are two methods for updating secrets in Boto3.

The first one is put_secret_value(). This method creates a new version and attaches it to the secret

#!/usr/bin/env python3

import boto3 
import json 

client = boto3.client('secretsmanager')

response = client.put_secret_value(
    SecretId='DatabaseProdSecrets',
    SecretString='{"username": "prod", "password": "hello-world-updated2"}'
)

print(response)

Output:

The second one is the update_secret() method. This method modifies many of the details of the specified secret. If you include a ClientRequestToken and either SecretString or SecretBinary then it also creates a new version attached to the secret.

#!/usr/bin/env python3

import boto3 
import json

client = boto3.client('secretsmanager')

response = client.update_secret(
    SecretId='DatabaseProdSecrets',
    Description='Description updated'
)

print(response)

Here’s an execution output:

How to create a new version of the secret in AWS Secrets Manager using Boto3?

You can use either put_secret_value or update_secret to create a new version of the secret.

The put_secret_value creates a new version and attaches it to the secret.

The update_secret method creates a new version attached to the secret when a ClientRequestToken and either SecretString or SecretBinary parameters are used.

How to delete a secret in AWS Secrets Manager using Boto3?

You can use the delete_secret() function from Boto3 to delete secret and all of its versions. You can optionally include a recovery window during which you can restore the secret. If you don’t specify a recovery window value, the secret will be deleted within 30 days.

At any time before the recovery window ends, you can use RestoreSecret to remove the DeletionDate and cancel the deletion of the secret.

#!/usr/bin/env python3

import boto3

client = boto3.client('secretsmanager')

response = client.delete_secret(
    SecretId='DatabaseProdSecrets',
    RecoveryWindowInDays=10,
    ForceDeleteWithoutRecovery=False
)

print(response)

Here’s an execution output:

To restore deleted secrets before the recovery window end, you can use the restore_secret() method.

#!/usr/bin/env python3

import boto3

client = boto3.client('secretsmanager')

response = client.restore_secret(
    SecretId='DatabaseProdSecrets'
)

print(response)

Here’s an execution output:

Summary

This article covered how to use Python to interact with AWS Secret Manager to create, update, and delete secrets using the Boto3 Python SDK.

If you’d like to learn more about using the Boto3 library, especially in combination with AWS Lambda, we encourage you to check out one of the top-rated Udemy courses on the topic – AWS Automation with Boto3 of Python and Lambda Functions.

Related articles

• Quick Intro to Python for AWS Automation Engineers
• Introduction to AWS Systems Manager
• Introduction to Boto3 library
• Working with S3 in Python using Boto3
• Working with EC2 Instances using Boto3 in Python
• Working with Snapshots and AMIs using Boto3 in Python