Working with EBS volumes in Python using Boto3

Abhinav Dumbre

Abhinav Dumbre

5
(2)

One of the core services of AWS cloud is Amazon Elastic Compute Cloud (Amazon EC2). Amazon EC2 is a web service that provides secure and resizable compute capacity in the cloud in the form of virtual servers. Each virtual server is started from the Amazon Machine Images (AMIs). The AMI containing the information about EBS volume snapshots and their mapping to the EC2 instance. This article will cover using Python to interact with the Amazon EC2 service to manage EBS volumes using the Boto3 library programmatically.

Prerequisites

To start automating Amazon EC2 and making API calls to manage EBS volumes, snapshots, and AMIs, you must first configure your Python environment.

In general, here’s what you need to have installed:

  • Python 3
  • Boto3
  • AWS CLI tools

What is an EBS volume?

EC2 instances use block-level storage called Amazon Elastic Block Store (EBS). Amazon EBS offers durable, high-performance, and highly scalable volumes ranging from one GiB up to 16 TiB of storage size for EC2 instances. The data stored on EBS volume is automatically replicated across the Availability Zone (AZ) within AWS Region.

Working with EBS volumes, snapshots, and AMIs using Boto3 in Python - Workflow diagram

EBS volumes are easy to use, whether you need to add them to or remove them from the EC2 instance, modify their size, or change their type. When choosing your EBS volume types, you’ll find multiple options based on your business application requirements.

How to connect to Amazon EC2 using Boto3?

The Boto3 library provides you two ways to access APIs for managing AWS services:

  • The client that allows you to access the low-level API data. For example, you can get access to API response data in JSON format.
  • The resource that allows you to use AWS services in a higher-level object-oriented way. For more information on the topic, take a look at AWS CLI vs. botocore vs. Boto3.

Here’s how you can instantiate the Boto3 EC2 client to start working with Amazon EC2 APIs:

import boto3

AWS_REGION = "us-east-1"

ec2_client = boto3.client("ec2", region_name=AWS_REGION)

You can similarly instantiate the Boto3 EC2 resource:

import boto3

AWS_REGION = "us-east-1"

ec2_resource = boto3.resource("ec2", region_name=AWS_REGION)

How to create EBS volume using Boto3?

One of the daily operations for AWS EBS volumes is creating new volumes and attaching them to the running EC2 instances. To create a new EBS volume, you can use the create_volume() method of the Boto3 EC2 client or resource.

Creating an EBS volume using Boto3 client

To create a new EBS volume, you can use the create_volume() method of the Boto3 EC2 client:

import boto3

AWS_REGION = "us-east-2"

ec2_client = boto3.client('ec2', region_name=AWS_REGION)

new_volume = ec2_client.create_volume(
    AvailabilityZone=f'{AWS_REGION}c',
    Size=10,
    VolumeType='gp2',
    TagSpecifications=[
        {
            'ResourceType': 'volume',
            'Tags': [
                {
                    'Key': 'Name',
                    'Value': 'hands-on-cloud-ebs-boto3'
                }
            ]
        }
    ]
)

print(f'Created volume ID: {new_volume["VolumeId"]}')

The required arguments are:

  • AvailabilityZone – specifies the Availability Zone where to create the Volume
  • Size – specifies the size of the Volume in GiB

The rest of the arguments are optional, but we set them up for demo purposes:

  • VolumeType – defines an EBS volume type. This parameter can be one of the following values:
    • General Purpose SSDgp2 | gp3
    • Provisioned IOPS SSDio1 | io2
    • Throughput Optimized HDDst1
    • Cold HDDsc1
    • Magneticstandard
  • TagSpecifications allow you to define the Tags for the volume.

The new_volume variable contains an AWS EC2 service response serialized in the form of the Python dictionary, so, to get access to the volume ID, we have to use the VolumeId key.

Here’s an execution output:

2. Working with EBS volumes using Boto3 - Creating an EBS volume using Boto3 client

Creating an EBS volume using Boto3 resource

To create a new EBS volume, you can use the create_volume() method of the Boto3 EC2 resource:

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

new_volume = ec2_resource.create_volume(
    AvailabilityZone=f'{AWS_REGION}c',
    Size=10,
    VolumeType='gp2',
    TagSpecifications=[
        {
            'ResourceType': 'volume',
            'Tags': [
                {
                    'Key': 'Name',
                    'Value': 'hands-on-cloud-ebs-boto3'
                }
            ]
        }
    ]
)

print(f'Created volume ID: {new_volume.id}')

Here’s an execution output:

1. Working with EBS volumes using Boto3 - Creating an EBS volume using Boto3 resource

How to list all EC2 volumes using Boto3?

The easiest way to list all EC2 volumes is to use the all() method of the volumes collection of the EC2 resource:

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

for volume in ec2.volumes.all():
    print(volume)

Here’s an execution output:

3. Working with EBS volumes using Boto3 - Listing all EC2 volumes using Boto3

How to search EC2 volumes Boto3?

The easiest way to search for EC2 volumes by the specified condition is to use the filter() method of the volumes resource.

Searching EC2 volumes by Tag

In the following example, we’ll find all volumes, which has the Tag Name and Tag value hands-on-cloud-ebs-boto3:

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

for volume in ec2_resource.volumes.filter(
    Filters=[
        {
            'Name': 'tag:Name',
            'Values': [
                'hands-on-cloud-ebs-boto3',
            ]
        }
    ]
):
    print(f'Volume {volume.id} ({volume.size} GiB) -> {volume.state}')

Here’s an execution output:

4. Working with EBS volumes using Boto3 - Searching EC2 volumes by Tag using Boto3

Searching EC2 volumes by ID

In the following example, we’ll find all volumes that belong to a list of IDs:

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

for volume in ec2_resource.volumes.filter(
    VolumeIds=[
        'vol-07f77d0a13239ef7e',
        'vol-01e5646dd54edd848',
    ],
):
    print(f'Volume {volume.id} ({volume.size} GiB) -> {volume.state}')

Here’s an execution output:

5. Working with EBS volumes using Boto3 - Searching EC2 volumes by ID using Boto3

How to delete EC2 volumes using Boto3?

The easiest way to delete for EC2 volume is to use the delete() method of the Volume Boto3 class.

Deleting single EC2 volume

To delete a single EC2 volume, you can use the following code snippet:

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

volume = ec2_resource.Volume('vol-01e5646dd54edd848')

if volume.state == "available":
    volume.delete()
    print(f'Volume successfully deleted')
else:
    print(f"Can't delete volume attached to EC2 instance")

Note: you can delete only volumes in the state available.

Here’s an execution output:

6. Working with EBS volumes using Boto3 - Deleting single EC2 volume

Deleting multiple EC2 volumes

To delete multiple EC2 volumes, you can filter for required modules and then use for-loop to delete them one by one. Let’s delete all volumes in the state available. This is the common code that allows cleaning up unused EC2 volume resources:

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

for volume in ec2_resource.volumes.filter(
    Filters=[
        {
            'Name': 'status',
            'Values': [
                'available',
            ]
        }
    ]
):
    if volume.state == "available":
        volume_id = volume.id
        volume.delete()
        print(f'Volume {volume_id} successfully deleted')
    else:
        print(f"Can't delete volume attached to EC2 instance")

Here’s an execution output:

7. Working with EBS volumes using Boto3 - Deleting unused EC2 volumes

How to describe EBS volumes using Boto3?

The most straightforward way to describe the volume is to use the describe_volumes() method of the EC2 client because it provides complete information about the volume in JSON format.

The describe_volumes() method accepts the same arguments as the filter() method, so you can specify various conditions to find required volumes. We’ll use VolumeID to simplify the example:

import json
from datetime import date, datetime
import boto3

AWS_REGION = "us-east-2"
EC2_CLIENT = boto3.client('ec2', region_name=AWS_REGION)


# Helper method to serialize datetime fields
def json_datetime_serializer(obj):
    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    raise TypeError ("Type %s not serializable" % type(obj))


describe_response = EC2_CLIENT.describe_volumes(
    VolumeIds=[
        'vol-0c0ce77e0b27ed800'
    ]
)

print('Volumes information:')
print(json.dumps(
        describe_response,
        indent=4,
        default=json_datetime_serializer
    )
)

In the example above, we’re using additional json_datetime_serializer() method to serialize (convert to string) datetime.datetime fields returned by the describe_volumes() method.

Here’s an example output:

7. Working with EBS volumes using Boto3 - Describing EBS volumes using Boto3

How to attach EBS volume to EC2 instance using Boto3?

To attach the EBS volume to the EC2 instance, you need to use attach_to_instance() method of the EC2 resource.

Note: you can only attach EBS volumes to the EC2 instances within the same Availability Zone. If you try to attach EBS volume to the instance from another Availability zone, you’ll receive the botocore.exceptions.ClientError: An error occurred (InvalidVolume.ZoneMismatch) when calling the AttachVolume operation exception.

import boto3

AWS_REGION = "us-east-2"

ec2_resource = boto3.resource('ec2', region_name=AWS_REGION)

volume = ec2_resource.Volume('vol-0d4abbb9e7da7ed9f')

print(f'Volume {volume.id} status -> {volume.state}')

volume.attach_to_instance(
    Device='/dev/sdh',
    InstanceId='i-07125d93aed65ea84'
)

print(f'Volume {volume.id} status -> {volume.state}')

Here’s an execution output:

8. Working with EBS volumes using Boto3 - Attaching EBS volume to EC2 Instance

How to detach EBS volume from EC2 instance using Boto3?

To detach the EBS volume from the EC2 instance, you need to use the detach_from_instance() method of the Volume class. In addition to that, you have to use the waiter.VolumeAvailable class checks every 15 seconds that the EBS volume has been successfully disconnected from the EC2 instance by running the describe_volumes() method in the background. That’s a correct way to wait for the successful completion of detaching operation.

import boto3

AWS_REGION = "us-east-2"
EC2_RESOURCE = boto3.resource('ec2', region_name=AWS_REGION)
EC2_CLIENT = boto3.client('ec2', region_name=AWS_REGION)

volume = EC2_RESOURCE.Volume('vol-0d4abbb9e7da7ed9f')

print(f'Volume {volume.id} status -> {volume.state}')

volume.detach_from_instance(
    Device='/dev/sdh',
    InstanceId='i-07125d93aed65ea84'
)

# Vaiting for volume to become available
waiter = EC2_CLIENT.get_waiter('volume_available')
waiter.wait(
    VolumeIds=[
        volume.id,
    ]
)

print(f'Volume {volume.id} status -> {volume.state}')

Here’s an execution output:

9. Working with EBS volumes using Boto3 - Detaching EBS volume from EC2 instance

How to increase size of the EBS volume using Boto3?

If you need to increase this size based on application requirements, you can use the modify_volume() method of the EC2 client.

You can only increase the size of EBS volume. There is no way to decrease the size of volume from AWS console, CLI, OR API.

Request modifications to your EBS volumes
import time
import boto3

AWS_REGION = "us-east-2"
EC2_CLIENT = boto3.client('ec2', region_name=AWS_REGION)
VOLUME_ID = 'vol-029786bbbf29faa8d'

def get_modification_state(volume_id):
    response = EC2_CLIENT.describe_volumes_modifications(
        VolumeIds=[
            VOLUME_ID
        ]
    )
    return response['VolumesModifications'][0]['ModificationState']

modify_volume_response = EC2_CLIENT.modify_volume(
    VolumeId=VOLUME_ID,
    Size=30
)

while True:
    state = get_modification_state(VOLUME_ID)
    if state == 'completed' or state == None:
        break
    elif state == 'failed':
        raise Exception('Failed to modify volume size')
    else:
        time.sleep(60)

print(f'Volume {VOLUME_ID} successfully resized')

In the example above, we have to use the get_modification_state() helper method, which returns the EBS volume modification status.

The while loop helps to wait till the volume modification operation finishes.

Note: pay attention that for large volumes, the volume modification operation might take many hours.

Here’s an execution output:

10. Working with EBS volumes using Boto3 - Increasing the size of the EBS volume

Summary

This article covered how to use the Python Boto3 library to implement the most common EBS volume operations such as creating, listing, filtering (searching), deleting, and modifying.

Related articles

How useful was this post?

Click on a star to rate it!

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Top rated Udemy Courses to improve you career

Subscribe to our updates

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use https://codeshare.io/ or a similar tool as Facebook comments are breaking code formatting.