Introduction to Boto3 library

Every Cloud Automation Engineer must have a minimal understanding of the AWS SDK for his favorite programming language. This article covers not only the installation and configuration of the Boto3 library but also advanced topics such boto3.client(), boto3.resource() and boto3.Session() objects. The content of this article will allow you to start managing the AWS cloud using Python SDK in just 5 minutes.

What is the Boto3 library?

The Boto3 library is the AWS Software Development Kit (SDK) for Python that allows you to create, configure, and manage AWS services using APIs. Boto3 SDK provides an object-oriented API and low-level access to AWS services. The Boto3 library heavily relies on another botocore librarythat handles  Python code to take care of low-level operations required to send secure HTTPS API requests to AWS services and transform XML responses to Python dictionary data type.

What AWS accesses does the Boto3 library give you?

The Boto3 library in an AWS SDK for Python allows you easily integrate your Python applications, libraries, or script with AWS services and gives you complete control over all AWS resources like EC2, VPC, S3, DynamoDB, and others. Although, you need to keep in mind that the user or service making API calls must have the actual permissions to call AWS APIs. Those permissions are controlled by the AWS IAM service, defined in IAM Policy, and granted to the user or service through the IAM Role.

How do I use Python SDK for AWS?

To start using the AWS SDK for Python to interact with AWS APIs, you have to install the Python boto3 module (covered in the next section), import it from your Python program code, and use boto3.client() or boto3.resource() methods to connect to a specific AWS service API endpoint. Finally, you must use the Boto3 client or resource object methods to interact with service-specific API.

Installing AWS SDK for Python and AWS CLI tools

If your computer doesn’t have Boto3 or AWS tools, you must install them first.

There are two Python modules you need to be able to manage AWS resources from the command line (CLI) and your Python scripts:

• boto3 – theAWS SDK for Python
• awscli – command-line tools allowing you to manage AWS resources from your terminal

We need to install the AWS CLI tools to configure the AWS environment on your machine.

We’ll do it after the installation process.

Installing Boto3 library

To install the Boto3 library, you have to run the following command in your terminal:

pip install boto3

The command above will install the Boto3 library globally in your system. Alternatively, you can configure a Python development environment to isolate your dependencies and manage them separately.

Installing AWS CLI tools

To install theAWS CLI tools, you have to run another command in your terminal:

pip install awscli

Configuring AWS environment

AWS CLI is a set of command-line tools for accessing AWS from the terminal shell. Those tools are available for you through the aws command. In this section, we’ll use a subcommand named configure to set up an AWS environment on your laptop, workstation, or server.

To configure the AWS environment, type the following command in your terminal:

aws configure

This command will walk you through an environment configuration process and ask you for 4 things:

• AWS Access Key
• AWS Secret Access Key
• Default region name – enter us-east-1
• Default output format – enter json

The command above will configure the default profile used by your Python scripts and applications relying on the Boto3. The default profile allows you to use the same credentials to interact with AWS cloud APIs in the same AWS region.

For more information about AWS regions, check out the “AWS Global Infrastructure: Regions and Availability Zones” article.

Creating AWS Access and Secret Keys

AWS Access Key and AWS Secret Access key are used to authenticate your API calls. To create them for your user, you need to log into your AWS account with administrative permissions and go to the IAM service:

Go to the Users interface using the link at the left menu or an IAM Dashboard page:

Select a user from the list:

Go to the Security credentials tab:

Hit Create access key button:

What the “aws configure” command does?

The aws configure command updates a text file that the awscli, boto3 or any other software which is interacting with AWS. Python developers using the macOS or Linux can find the AWS credentials file at this path:

~/.aws/credentials

You can view the AWS credentials file using the cat utility:

cat ~/.aws/credentials

And the credentials file content should look like this:

[default]
aws_access_key_id = AKIAYQV4J5VM5FWGJMGB
aws_secret_access_key = YSgRFYeVopQaiKWBF6QdO2e2mK0ZSypKro7xsO1o

The aws configure command allows you not to store your AWS credentials (the AWS Access and Secret Keys) in your Python scripts.

Keeping your AWS credentials in your scripts is not secure, and you should never do this!

Note: even storing AWS Access and Secret Keys in a plain text file (~/.aws/credentials) is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.

Testing AWS credentials

Once you’ve configured your AWS credentials, you can test that everything’s ready to move forward.

Testing AWS credentials via AWS CLI

My favorite way of testing AWS credentials is by running the following command:

aws sts get-caller-identity | tee

This command will produce information about your AWS Account and User IDs:

{
    "UserId": "AKIAYQV4J5VM5FWGJMGB",
    "Account": "012345678901",
    "Arn": "arn:aws:iam::012345678901:user/admin"
}

Testing AWS credentials via Boto3

We can request the same information by using the Boto3 library. Here’s an example of a Python script (get_caller_identity.py) to do that:

#!/usr/bin/env python3
import json
import boto3
client = boto3.client('sts')
response = client.get_caller_identity()
user_id = response['UserId']
account = response['Account']
arn = response['Arn']
output = {
    'UserId': user_id,
    'Account': account,
    'Arn': arn
}
print(json.dumps(output, indent=4))

In the Python code above, we did the following things:

• Import of a built-in json module to process JSON responses returned from theget_caller_identity() method (line 3)
• Import of the boto3 module to have the ability to make calls to AWS APIs (line 4)
• Instantiate the AWS Security Token Service (STS) client (line 6)
• Make a call to the AWS STS service to get information about AWS Account and User IDs and save the service response to the variable (line 8)

Note: All AWS API calls at the Boto3 library return a Python dictionary as a response.

Now, run theget_caller_identity.py script by executing the following command to get the same output as you got from the AWS CLI call:

python3 get_caller_identity.py

The output will be absolutely the same.

Now you’re ready to start using the Boto3 library and continue this course.

Advanced Boto3 library topics

This section will cover advanced topics of using the Boto3 library.

How does Boto3 library work under the hood?

The AWS CLI and Boto3 library are built on top of a common botocore module.  This module is  a low-level Python library that takes care of low-level operations required to send secure HTTPS API requests to AWS.

So, the botocore module is responsible for the following things:

• Configure and handle AWS user sessions and credentials during API calls
• Provides a fine-granular interface to manage low-level AWS API calls to a specific service (for example, AWS STS GetCallerIdentity API call)
• Serializes and deserializes input parameters and XML responses received from API calls into Python dictionaries
• Provides low-level clients and high-level resources abstractions that we’re using to interact with AWS resources from Python

In short, botocore is a utility module that allows you to forget about dealing with really low-level API calls to AWS resources and use more high-level methods to interact with them.

AWS CLI vs. botocore vs. Boto3

In this article, we’ve mentioned AWS CLI tools, botocore, and boto3 modules. Moreover, we started using the Boto3 client to make an API call to the AWS STS service. At the same time, you may find many examples on the internet where people use the client() instead of the resource() method. So, what are the differences?

The botocore module is a common lower-level utility library used by the AWS CLI and the boto3 module:

At the same time, the boto3 module allows you to use a lower-level client to AWS API or higher-level object-oriented resource abstraction.

The major differences between boto3.client() and boto3.resource():

In summary, using the resource allows us to write more readable code and avoid processing response dictionaries provided by service clients. In addition, it will enable you to lazily iterate over a large number of returned objects without thinking about pagination or memory utilization.

For example, let’s get all IAM Roles from the AWS account using the client() and the following code:

import boto3
client = boto3.client('iam')
paginator = client.get_paginator('list_roles')
aws_roles = []
for page in paginator.paginate():
    for role in page['Roles']:
        aws_roles.append(role['RoleName'])
print('\n'.join(aws_roles))

Now, let’s do the same thing but with the help of resource() instead:

import boto3
iam_resource = boto3.resource('iam')
aws_roles = []
for role in iam_resource.roles.all():
    aws_roles.append(role.name)
print('\n'.join(aws_roles))

As you can see, the resource() simplifies our code by removing the need for a paginator and additional nested for-loop.

Finally, you can get access to the client() methods from the resource():

iam_resource = boto3.resource('iam')
# getting access to the client from resource object
iam_client = iam_resource.meta.client
paginator = iam_client.get_paginator('list_roles')
aws_roles = []
for page in paginator.paginate():
    for role in page['Roles']:
        aws_roles.append(role['RoleName'])
print('\n'.join(aws_roles))

How to pass AWS credentials to the Boto3 client?

To pass AWS credentials to the Boto3 client, you have to provide them in the aws_access_key_id and aws_secret_access_key variables, for example:

import boto3
client = boto3.client(
    'iam',
    aws_access_key_id="XXXXXXX",
    aws_secret_access_key="YYYYYYY"
)

How to specify AWS Region in the Boto3 client?

To pass AWS Region to the Boto3 client, you have to provide it in the region_name variable, for example:

import boto3
client = boto3.client(
    'ec2',
    region_name="us-east-2"
)

How to use Sessions in Boto3?

The boto3.Session() is a class that stores the configuration state and allows you to create the Boto3 clients() and resources(). Dealing with multi-region and multi-account AWS API calls is out of this article’s scope. Still, I have to mention that it is preferable to use session when you’re working with multi-region or multi-account code:

import boto3
session = boto3.Session()
s3 = session.client('s3')
ddb = session.resource('dynamodb')

Using boto3.Session() becomes even more important when you need to assume roles across different AWS accounts.

For more information about using boto3.Sessions() I highly recommend you to check out the “boto3 Sessions, and Why You Should Use Them” article.

How to assume the IAM role using Boto3?

To assume an IAM role using Boto3, you need to use the assume_role() method of the STS client.

Note: you need to ensure that your current user has been granted permission to assume that role.

import boto3
CURRENT_ACCOUNT_SESSION = boto3.Session()
STS_CLIENT = CURRENT_ACCOUNT_SESSION.client('sts')
AWS_ACCOUNT_ID = '585584209241'
ROLE_NAME = 'demo-role'
assumed_role_object = STS_CLIENT.assume_role(
    RoleArn=f'arn:aws:iam::{AWS_ACCOUNT_ID}:role/{ROLE_NAME}',
    RoleSessionName=f'{ROLE_NAME}-Session'
)
assumed_role_credentials = assumed_role_object['Credentials']
ASSUMED_ROLE_SESSION = boto3.Session(
    aws_access_key_id=assumed_role_credentials['AccessKeyId'],
    aws_secret_access_key=assumed_role_credentials['SecretAccessKey'],
    aws_session_token=assumed_role_credentials['SessionToken'],
)
instances = ASSUMED_ROLE_SESSION.resource('ec2', region_name="us-east-2").instances.all()
for instance in instances:
    print(f'Instance ID: {instance.id}')

Here’s an execution output:

Additional Learning Resources

Free hands-on AWS workshops

We recommend the following free AWS Lambda Workshops to get more hands-on experience on the subject:

• AWS Lambda Workshop
• Migrate Spring Boot applications to AWS Lambda
• Building CI/CD pipelines for Lambda canary deployments using AWS CDK
• Content Generation Lambda@Edge

Also, we recommend the following free AWS Step Functions Workshops:

• The AWS Step Functions Workshop

Paid courses

From our experience, these are the best hands-on paid learning materials today related to Serverless, AWS Lambda, and Step Functions:

Summary

In this article, we’ve covered how to install and configure Boto3 to start managing AWS services, described the botocore module, and the differences between boto3.client() and boto3.resource(), and some advanced Boto3 library usage topics.