Introduction to Boto3 library

Andrei Maksimov

Andrei Maksimov

5
(1)

Every Cloud Automation Engineer must have minimal understanding of the AWS SDK for his favorite programming language. This article covers not only the installation and configuration of the Boto3 library but also advanced topics such boto3.client(), boto3.resource() and boto3.Session() objects. The content of this article will allow you to start programmatically managing AWS services in just 5 minutes.

What is the Boto3 library?

The Boto3 library is the AWS Software Development Kit (SDK) for Python that allows you to create, configure, and manage AWS services using AWS APIs. Boto3 SDK provides not only an object-oriented API but also low-level access to AWS services. The Boto3 library heavily relies on another botocore library, that handles  Python code to take care of low-level operations required to send secure HTTPS API requests to AWS services and transform XML responses to Python dictionary data type.

What AWS accesses does the Boto3 library give you?

The Boto3 library allows you easily integrate your Python applications, libraries, or script with AWS services and gives you full control over all AWS services like Amazon EC2, Amazon VPC, Amazon S3, Amazon DynamoDB, and others. Although, you need to keep in mind that the user or service who is making API calls must have the actual permissions to call AWS APIs. Those permissions are controlled by the AWS IAM service, defined in IAM Policy, and granted to the user or service through the IAM Role.

How do I use Boto3 on AWS?

To start using the Boto3 library to interact with AWS APIs, you have to install Python boto3 module (covered in the next section), import it from your Python program code, and use boto3.client() or boto3.resource() methods to connect to a specific AWS service API endpoint. Finally, you have to use the Boto3 client or resource object methods to interact with service-specific API.

Installing Boto3 library and AWS tools

If your computer doesn’t have Boto3 or AWS tools yet, you have to install them first.

There are two Python modules you need to be able to manage AWS services from the command-line (CLI) and your Python scripts:

  • boto3 – the AWS SDK for Python
  • awscli – command-line tools allowing you to manage AWS services from your terminal

We need to install the AWS CLI tools to configure the AWS environment on your machine.

We’ll do it after the installation process.

Installing Boto3 library

To install the Boto3 library, you have to run the following command in your terminal:

pip install boto3

Installing AWS CLI tools

To install the AWS CLI tools, you have to run another command in your terminal:

pip install awscli

Configuring AWS environment

AWS CLI is a set of command-line tools for accessing AWS from the terminal shell. Those tools are available for you through the aws command. In this section, we’ll use a subcommand named configure to set up an AWS environment on your laptop, workstation, or server.

To configure the AWS environment, type the following command in your terminal:

aws configure

This command will walk you through an environment configuration process and ask you for 4 things:

  • AWS Access Key
  • AWS Secret Access Key
  • Default region name – enter us-east-1
  • Default output format – enter json

Creating AWS Access and Secret Keys

AWS Access Key and AWS Secret Access key are used to authenticate your API calls. To create them for your user, you need to log into your AWS account with administrative permissions and go to the IAM service:

AWS Web Console - Open IAM service

Go to the Users interface using the link at the left menu or an IAM Dashboard page:

AWS Web Console - Open IAM users

Select user from the list:

AWS Web Console - Open IAM user

Go to the Security credentials tab:

AWS Web Console - Open IAM user Security Credentials

Hit Create access key button:

AWS Web Console - Create IAM user AWS Access Keys

What aws configure does?

The aws configure command updates a text file that the awscli, boto3 or any other software which is interacting with AWS is using. If you are on macOS or Linux user, the aws configure command updates the text file at this path:

~/.aws/credentials

You can view it using the cat utility:

cat ~/.aws/credentials

And the file should look like this:

[default]
aws_access_key_id = AKIAYQV4J5VM5FWGJMGB
aws_secret_access_key = YSgRFYeVopQaiKWBF6QdO2e2mK0ZSypKro7xsO1o

The aws configure tool allows you not to store your AWS credentials (the AWS Access and Secret Keys) in your Python scripts.

Storing your AWS credentials in your scripts is not secure and, you should never do this!

Note: even storing AWS Access and Secret Keys in a plain text file (~/.aws/credentials) is not very secure. The better and more secure way is to store AWS Access and Secret Keys in the encrypted store, for example, aws-vault.

Testing AWS credentials

As soon as you’ve configured your AWS credentials, you can test that everything’s ready to move forward.

Testing AWS credentials via AWS CLI

My favorite way of testing AWS credentials is by running the following command:

aws sts get-caller-identity | tee

This command will produce information about your AWS Account and User IDs:

{
    "UserId": "AKIAYQV4J5VM5FWGJMGB",
    "Account": "012345678901",
    "Arn": "arn:aws:iam::012345678901:user/admin"
}

Testing AWS credentials via Boto3

We can request the same information by using the Boto3 library. Here’s an example of a Python script (get_caller_identity.py) to do that:

#!/usr/bin/env python3

import json
import boto3

client = boto3.client('sts')

response = client.get_caller_identity()

user_id = response['UserId']
account = response['Account']
arn = response['Arn']

output = {
    'UserId': user_id,
    'Account': account,
    'Arn': arn
}

print(json.dumps(output, indent=4))

In the Python code above, we did the following things:

Import of a built-in json module to process JSON response from the get_caller_identity() method (line 3)

Import of the boto3 module to have an ability to make calls to AWS APIs (line 4)

Instantiate the AWS Security Token Service (STS) client (line 6)

Make a call to the AWS STS service to get information about AWS Account and User IDs and save the service response to the variable (line 8)

All AWS API calls at the Boto3 library returning response information in the form of a Python dictionary, so I highly encourage you to check out our article “Working with Dictionaries in Python,” as we’ll work with such data structure a lot.

Now, run the get_caller_identity.py script by executing the following command to get the same output as you got from the AWS CLI call:

python3 get_caller_identity.py

The output will be absolutely the same.

Now you’re ready to start using the Boto3 library and move on with this course.

Advanced Boto3 library topics

In this section, we’ll cover advanced topics of using the Boto3 library.

How Boto3 library works under the hood?

The AWS CLI and Boto3 library are built on top of a common botocore module.  This module is  a low-level Python library that takes care of low-level operations required to send secure HTTPS API requests to AWS and respond back.

So, the botocore module is responsible for the following things:

  • Configure and handle AWS user session and credentials during API calls
  • Provides a fine-granular interface to manage low-level AWS API calls to a specific service (for example, AWS STS GetCallerIdentity API call)
  • Serializes an deserializes input parameters, and XML responses received from API calls into Python dictionaries
  • Provides low-level clients and high-level resources abstractions that we’re using to interact with AWS services from Python

In short, botocore is a utility module that allows you to forget about dealing with really low-level API calls to AWS services and use more high-level methods to interact with them.

AWS CLI vs. botocore vs. Boto3

In this article, we’ve mentioned AWS CLI tools, botocore, and boto3 modules. Moreover, we started using the Boto3 client to make an API call to the AWS STS service. At the same time, you may find lots of examples on the internet, where instead of the client() the resource() method is used. So, what are the differences?

The botocore module is a common lower-level utility library used by the AWS CLI and the boto3 module:

Introduction to Boto3 library - aws-cli vs boto3 vs botocore

At the same time, the boto3 module allows you to use a lower-level client to AWS API or higher-level object-oriented resource abstraction.

The major differences between boto3.client() and boto3.resource():

boto3.client()boto3.resource()
Returns a response from AWS service API in the form of a Python dictionaryReturns a Python class, which you’re using instead of dealing with Python dictionaries
Thread-safe and can be used in multithreading or multiprocessing codeNot thread-safe and can’t be used in multithreading or multiprocessing code

As a summary, using the resource() allows us to write a more readable code and avoid processing response dictionaries. In addition to that, it allows you to lazily iterate over a large number of returned objects without thinking about pagination or memory utilization.

For example, let’s get all IAM Roles from the AWS account using client():

import boto3

client = boto3.client('iam')
paginator = client.get_paginator('list_roles')

aws_roles = []

for page in paginator.paginate():
    for role in page['Roles']:
        aws_roles.append(role['RoleName'])

print('\n'.join(aws_roles))

Now, let’s do the same thing but with the help of resource() instead:

import boto3

iam_resource = boto3.resource('iam')

aws_roles = []

for role in iam_resource.roles.all():
    aws_roles.append(role.name)

print('\n'.join(aws_roles))

As you can see, the resource() simplifies our code by removing the need to use a paginator and additional nested for-loop.

Finally, you can get access to the client() methods from the resource():

iam_resource = boto3.resource('iam')

# getting access to the client from resource object
iam_client = iam_resource.meta.client

paginator = iam_client.get_paginator('list_roles')

aws_roles = []

for page in paginator.paginate():
    for role in page['Roles']:
        aws_roles.append(role['RoleName'])

print('\n'.join(aws_roles))

How to pass AWS credentials to Boto3 client?

To pass AWS credentials to the Boto3 client, you have to provide them in the aws_access_key_id and aws_secret_access_key variables, for example:

import boto3

client = boto3.client(
    'iam',
    aws_access_key_id="XXXXXXX",
    aws_secret_access_key="YYYYYYY"
)

How to specify AWS Region in Boto3 client?

To pass AWS Region to the Boto3 client, you have to provide it in the region_name variable, for example:

import boto3

client = boto3.client(
    'ec2',
    region_name="us-east-2"
)

How to use Sessions in Boto3?

The boto3.Session() is a class that stores configuration state and allows you to create the Boto3 clients() and resources(). Dealing with multi-region and multi-account AWS API calls topic is out of the scope of this article. Still, I have to mention that it is more preferable to use session when you’re working with multi-region or multi-account code:

import boto3

session = boto3.Session()

s3 = session.client('s3')
ddb = session.resource('dynamodb')

Using boto3.Session() becomes even more important when you need to assume roles across different AWS accounts.

For more information about using boto3.Sessions() I highly recommend you to check out the “boto3 Sessions, and Why You Should Use Them” article.

How to assume IAM role using Boto3?

To assume an IAM role using Boto3 you need to use the assume_role() method of the STS client.

Note: you need to make sure, that your current user has been granted permission to assume that role.

import boto3

CURRENT_ACCOUNT_SESSION = boto3.Session()
STS_CLIENT = CURRENT_ACCOUNT_SESSION.client('sts')
AWS_ACCOUNT_ID = '585584209241'
ROLE_NAME = 'demo-role'

assumed_role_object = STS_CLIENT.assume_role(
    RoleArn=f'arn:aws:iam::{AWS_ACCOUNT_ID}:role/{ROLE_NAME}',
    RoleSessionName=f'{ROLE_NAME}-Session'
)

assumed_role_credentials = assumed_role_object['Credentials']

ASSUMED_ROLE_SESSION = boto3.Session(
    aws_access_key_id=assumed_role_credentials['AccessKeyId'],
    aws_secret_access_key=assumed_role_credentials['SecretAccessKey'],
    aws_session_token=assumed_role_credentials['SessionToken'],
)

instances = ASSUMED_ROLE_SESSION.resource('ec2', region_name="us-east-2").instances.all()

for instance in instances:
    print(f'Instance ID: {instance.id}')

Here’s an execution output:

1. Introduction to Boto3 library - Assuming IAM role

Summary

In this article, we’ve covered how to install and configure Boto3 to start managing AWS services, described the botocore module, and the differences between boto3.client() and boto3.resource(), and some advanced Boto3 library usage topics.

How useful was this post?

Click on a star to rate it!

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Top rated Udemy Courses to improve you career

Subscribe to our updates

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use https://codeshare.io/ or a similar tool as Facebook comments are breaking code formatting.