AWS Secrets Manager helps you protect secrets needed to access your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, passwords, and other secrets throughout their lifecycle. AWS Boto3 is the Python Software Development Kit (SDK) for AWS cloud platform that helps to interact with AWS resources from Python code. This article will cover how you can use the Boto3 library to manage your secrets in AWS Secrets Manager.
Table of contents
- What is AWS Secrets Manager?
- How do I access AWS Secrets Manager in Python?
- How do I get my secret from AWS Secrets Manager?
- How to create a secret in AWS Secrets Manager using Boto3?
- How to list secrets in AWS Secrets Manager using Boto3?
- How to retrieve a secret value from AWS Secrets Manager using Boto3?
- How to update an existing secret in AWS Secrets Manager using Boto3?
- How to create a new version of the secret in AWS Secrets Manager using Boto3?
- How to delete a secret in AWS Secrets Manager using Boto3?
- Summary
- Related articles
What is AWS Secrets Manager?
There are many scenarios where you might need to use credentials, tokens, API keys, etc., to access certain services. For example, you might need to use SQL server credentials to access a DB from the application. Storing those credentials in the codebase as a plain text file is a security vulnerability. Anyone with access to your codebase would be able to read those secrets, get unauthorized access to your services to perform malicious activities. The AWS Secrets Manager allows you to store sensitive information and get access to it by keys that you can safely save in your application config file or code.
AWS Parameter Store vs. Secrets Manager
The AWS Secrets Manager is designed specifically for confidential information (like database credentials, API Keys) that needs to be encrypted, so the creation of a secret entry has encrypted by default.
The AWS Systems Manager Parameter Store is designed to cater to a wider use case, not just secrets or passwords, but also application configuration variables like URLs, custom settings, etc.
Both Secrets Manager and Parameter Store can use AWS KMS to encrypt values. The AWS Parameter Store provides the option to store data unencrypted. On the other hand, with Secrets Manager, there’s no option to store unencrypted data. AWS KMS provides a highly available key storage, management, and auditing solution for you to encrypt data within your own applications and control the encryption of stored data across AWS services. With KMS, and with the help of IAM, you can use policies to control permissions on which IAM users and roles have permission to decrypt the value.
Parameter Store comes with no additional charges, but there is a limit on the number of parameters you can store, currently 10,000.
Parameter Store stores individual values using a hierarchical key. You can create keys like /my-app/prod/db/password
, /my-app/dev/db/password
and you can retrieve them individually or all keys that start with /my-app
.
aws ssm get-parameters-by-path /my-app/prod
You can write your own function that updates credentials managed by Parameter Store and invoking it via a CloudWatch scheduled event or EventBridge.
AWS Secrets Manager does come with additional costs. Learn more about AWS Secrets Manager pricing from here.
AWS Secret Manager has built-in integration for rotating MySQL, Postgres SQL, Amazon Aurora, and RDS database credentials. For services that it doesn’t have integration with, it allows Lambda functions to rotate these other forms of stored secrets. Similar to other AWS services, the built-in integration will only grow to include more AWS services in the future. This means eventually, you will be able to manage all secrets of your entire AWS platform from one place.
AWS Secrets Manager also has the ability to generate random secrets. You can randomly generate passwords in CloudFormation and store the password in Secrets Manager.
You can share AWS Secrets Manager secrets across multiple accounts.
How do I access AWS Secrets Manager in Python?
To access AWS Secrets Manager, you need to install Boto3, an AWS SDK for Python. Also, you need to have AWS CLI configured to use the Boto3 library. Boto3 uses your AWS Access Key Id
and Secret Access Key
to programmatically manage AWS resources.
First, you need to install AWS CLI from here, depending on the Operating System.
After installing AWS CLI, run aws configure
in your terminal to configure your AWS account with AWS CLI. It will prompt Access Key Id
and Secret Access Key
which you can find from IAM in the AWS Console.
Second install boto3
library using pip install boto3
For more in-depth information, we recommend you check out the Introduction to Boto3 library article.
Now you’re good to go.
How do I get my secret from AWS Secrets Manager?
In the AWS Console, search for Secret Manager
and there you will see all of your Secrets.
Click on one of the secrets and then click on Retrieve secret value
button to see the secret value.
Here’s what you should get:
How to create a secret in AWS Secrets Manager using Boto3?
Secrets Manager stores the encrypted secret data in a collection of “versions” associated with the secret. Each version contains a copy of the encrypted secret data. Each version is associated with one or more “staging labels” that identify the version in the rotation cycle.
The Boto3 Secrets Manager client
is a low-level class that provides methods to connect to AWS Secrets Manager similar to the AWS API service. All service APIs that are available in the Boto3 client map 1:1 to the AWS service API.
You can provide the secret data to be encrypted by putting text in either the SecretString
parameter or binary data in the SecretBinary
parameter, but not both. If you include SecretString
or SecretBinary
then Secrets Manager also creates an initial secret version and automatically attaches the staging label AWSCURRENT
to the new version.
#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.create_secret(
Name='DatabaseProdSecrets',
SecretString='{"username": "prod", "password": "hello-world-prod"}'
)
Output:
If you need to provide a custom KMS key, you can use the KmsKeyId
parameter in create_secret()
method that Specifies the ARN, Key ID, or alias of the Amazon Web Services KMS customer master key (CMK) to be used to encrypt the SecretString
or SecretBinary
values in the versions stored in this secret.
If you don’t provide the KmsKeyId
, then Secrets Manager use the account’s default CMK (the one named aws/secretsmanager
). If an Amazon Web Services KMS CMK with that name doesn’t exist, then Secrets Manager will create it for you automatically the first time it needs to encrypt a version’s SecretString
or SecretBinary
fields.
How to list secrets in AWS Secrets Manager using Boto3?
You can use the list_secrets() method to list all secrets stored in AWS Secrets Manager. When listing secrets, you can also filter and limit the number of results to specific number.
Note: The encrypted fields SecretString
and SecretBinary
are not included in the output. To get that information, you need to call the GetSecretValue
operation.
#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.list_secrets()
print(response['SecretList'])
Here’s an execution output:
How to retrieve a secret value from AWS Secrets Manager using Boto3?
To retrieve a secret value from AWS Secrets Manager using Boto3, you need to use the get_secret_value() method.
The following code example will get the secret with SecretId
(or Name
when creating) of DatabaseProdSecrets
. For more information, please read the Boto3 Secrets Manager documentation.
You also need to parse the SecretString
value using the json.loads
which converts JSON string into the Python dictionary so that you can access the items of a dictionary.
#!/usr/bin/env python3
import boto3
import json
client = boto3.client('secretsmanager')
response = client.get_secret_value(
SecretId='DatabaseProdSecrets'
)
database_secrets = json.loads(response['SecretString'])
print(database_secrets['password'])
Here’s an execution output:
Permissions required to retrieve a secret
To retrieve secret, you need to allow the secretsmanager:GetSecretValue
API cal in your IAM policy.
If you’re using customer-managed Amazon Web Services KMS key to encrypt the secret, you also need to have kms:Decrypt
permission.
Alternatively, you can attach the SecretsManagerReadWrite
policy to the user who needs permissions to manage AWS Secrets Manager.
Retrieve a secret values from the Python code
To retrieve a secret value from AWS Secrets Manager using Boto3, you need to use the get_secret_value() method.
#!/usr/bin/env python3
import boto3
import json
client = boto3.client('secretsmanager')
response = client.get_secret_value(
SecretId='DatabaseProdSecrets'
)
database_secrets = json.loads(response['SecretString'])
print(database_secrets['password'])
Retrieve a secret values in Bash
Using the AWS CLI, you can retrieve secret values in the Bash shell
aws secretsmanager get-secret-value --secret-id <SecretId>
Retrieve a secret values in Powershell
The cmdlets in the AWS Tools for PowerShell for each service are based on the methods provided by the AWS SDK for the service.
You can use Get-SECSecretValue
cmdlets to retrieve secrets. Read more about this cmdlet from here.
Get-SECSecretValue -SecretId <SecretId>
How to update an existing secret in AWS Secrets Manager using Boto3?
There are two methods for updating secrets in Boto3.
The first one is put_secret_value(). This method creates a new version and attaches it to the secret
#!/usr/bin/env python3
import boto3
import json
client = boto3.client('secretsmanager')
response = client.put_secret_value(
SecretId='DatabaseProdSecrets',
SecretString='{"username": "prod", "password": "hello-world-updated2"}'
)
print(response)
Output:
The second one is the update_secret() method. This method modifies many of the details of the specified secret. If you include a ClientRequestToken
and either SecretString
or SecretBinary
then it also creates a new version attached to the secret.
#!/usr/bin/env python3
import boto3
import json
client = boto3.client('secretsmanager')
response = client.update_secret(
SecretId='DatabaseProdSecrets',
Description='Description updated'
)
print(response)
Here’s an execution output:
How to create a new version of the secret in AWS Secrets Manager using Boto3?
You can use either put_secret_value
or update_secret
to create a new version of the secret.
The put_secret_value
creates a new version and attaches it to the secret.
The update_secret
method creates a new version attached to the secret when a ClientRequestToken
and either SecretString
or SecretBinary
parameters are used.
How to delete a secret in AWS Secrets Manager using Boto3?
You can use the delete_secret() function from Boto3 to delete secret and all of its versions. You can optionally include a recovery window during which you can restore the secret. If you don’t specify a recovery window value, the secret will be deleted within 30 days.
At any time before the recovery window ends, you can use RestoreSecret
to remove the DeletionDate
and cancel the deletion of the secret.
#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.delete_secret(
SecretId='DatabaseProdSecrets',
RecoveryWindowInDays=10,
ForceDeleteWithoutRecovery=False
)
print(response)
Here’s an execution output:
To restore deleted secrets before the recovery window end, you can use the restore_secret() method.
#!/usr/bin/env python3
import boto3
client = boto3.client('secretsmanager')
response = client.restore_secret(
SecretId='DatabaseProdSecrets'
)
print(response)
Here’s an execution output:
Summary
This article covered how to use Python to interact with AWS Secret Manager to create, update, and delete secrets using the Boto3 Python SDK.
If you’d like to learn more about using the Boto3 library, especially in combination with AWS Lambda, we encourage you to check out one of the top-rated Udemy courses on the topic – AWS Automation with Boto3 of Python and Lambda Functions.
Related articles
- Quick Intro to Python for AWS Automation Engineers
- Introduction to AWS Systems Manager
- Introduction to Boto3 library
- Working with S3 in Python using Boto3
- Working with EC2 Instances using Boto3 in Python
- Working with Snapshots and AMIs using Boto3 in Python
How useful was this post?
Click on a star to rate it!
We are sorry that this post was not useful for you!
Let us improve this post!
Please, tell us what's wrong with this post, and we'll fix it ASAP!
Hi, Tuvshinsanaa Tuul from Mongolia. I have a Bachelor of Information System. I’m a Software Engineer experienced with JavaScript, AWS, Python, and PHP/Laravel.