Working with DynamoDB in Python using Boto3

Related Content

Amazon DynamoDB (DDB) is a key-value and document database that is provided as a fully managed service in AWS which provides the possibility to build applications handling millions of requests and scale to meet the market’s demands. Amazon DynamoDB has been proven efficient for use in various industries, from consumer to enterprise apps. In this article, we will look at how to create tables, load data, perform CRUD operations, and query tables in AWS DynamoDB using Python and Boto3 library.

Prerequisites

To start working with DynamoDB using Python, you must set up your Python environment.

In summary, this is what you will need:

  • Python 3
  • Boto3
  • AWS CLI tools

Alternatively, you can quickly set up and use Cloud9 IDE.

Introduction to Amazon DynamoDB

Amazon DynamoDB enables the building of flexible microservices, serverless web applications, and mobile backends, and it acts as a highly scalable and fully managed database for applications.

The most interesting topic of DynamoDB is how to design your tables to get maximum performance and efficiency from the DynamoDB service. Regardless of your current experience, I highly recommend you a couple of videos on that topic before getting started:

Now, as soon as the concepts of designing DynamoDB tables are covered, you may look at how to start working with this service in Python using the Boto3 library.

Pro tip: Whenever you’re searching for something related to Amazon DynamoDB in Google, you can use “ddb” keyword instead of “dynamodb” in a search query, for example: “boto3 ddb”. Google is smart enough to understand you. Check it out!

Connecting to DynamoDB APIs using Boto3

The first thing you need to define in your Python script or Lambda function is to tell Boto3 how to connect to DynamoDB. To connect to Amazon DynamoDB API, you need to use the client and resource boto3 library methods.

Here’s an example of using the Boto DynamoDB client() method:

import boto3

dynamodb_client = boto3.client('dynamodb')

Here’s an example of using the Boto DynamoDB resource() method:

import boto3

dynamodb_resource = boto3.resource('dynamodb')

Right after that, you can use the dynamodb_client variable to get access to the Boto3 DynamoDB client methods, and dynamodb_resource variable to get access to the Boto3 DynamoDB resource methods and collections.

The boto3.resource('dynamodb') resource allows developers to create, update, and delete DynamoDB tables and items. This resource supports table-level security via ACLs and item-level security via Condition Expressions. The boto3.resource('dynamodb') resource can be used in conjunction with AWS Lambda functions to provide serverless access to DynamoDB data.

Create Tables in DynamoDB using Boto3

First, import the boto3 module and then create a Boto3 DynamoDB resource.

Next, create a table named Employees with a primary key that has the following attributes;

Name a partition key with AttributeType set to S for string.

Email a sort key with AttributeType set to S for string.

ProvisionedThroughput means the maximum number of consistent reads and writes per second on your table.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.create_table (
    TableName = 'Employees',
       KeySchema = [
           {
               'AttributeName': 'Name',
               'KeyType': 'HASH'
           },
           {
               'AttributeName': 'Email',
               'KeyType': 'RANGE'
           }
           ],
           AttributeDefinitions = [
               {
                   'AttributeName': 'Name',
                   'AttributeType': 'S'
               },
               {
                   'AttributeName':'Email',
                   'AttributeType': 'S'
               }
            ],
            ProvisionedThroughput={
                'ReadCapacityUnits':1,
                'WriteCapacityUnits':1
            }
          
    )
print(table)

Here is the execution output:

Alternatively, you can create the DynamoDB table using AWS CLI: How to use AWS CLI to manage Amazon DynamoDB.

List Tables in DynamoDB using Boto3

To list all tables in DynamoDB, use the dynamodb.tables.all() method.

import boto3

dynamodb = boto3.resource('dynamodb')

print(list(dynamodb.tables.all()))

Here is the execution output.

CRUD Operations in DynamoDB using Boto3

Tables, items, and attributes are the core components that make up DynamoDB. A table is a collection of items, and each item is a collection of attributes.

The following section will look at the four basic ways to perform CRUD operations on Items in DynamoDB Tables.

Create Items

To add a new item to a table, use the put_item() method and add an item with a primary of Name and Email as shown in the example.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Employees')

response = table.put_item(
Item = { 
     'Name': 'Kelvin Galabuzi',
     'Email': 'kelvingalabuzi@handson.cloud'
       }
)
print(response)

Here is the Execution output:

Batch Write Items

The batch_writer() method in Boto3 implements the BatchWriteItem AWS API call, which allows you to write multiple items to an Amazon DynamoDB table in a single request. This can be useful when you want to perform many write operations in a single request or to write items spread across multiple partitions. The BatchWriteItem API call supports both put and delete operations. Each request can write up to 100 items, and each item can be up to 400 KB in size. If you need to write more than 100 items, or if your items are larger than 400 KB, you can use the boto3 BatchWriteItem API call instead. BatchWriteItem API call has the following limitations:

  • You can only write items to a single table in each request.
  • You cannot use filters when writing items (e.g., you cannot specify a condition expression).
  • All items in the request must belong to the same partition key value.
  • You cannot update an existing item; you can only insert new items or delete existing ones. If you need to update an item, you must delete it and then insert a new item with the updated data.

For more information about the BatchWriteItem API call, see the Amazon DynamoDB Developer Guide.

NB: You can also add multiple attributes to items as required.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Employees')

with table.batch_writer() as batch:
    batch.put_item(Item={"Name": "Luzze John", "Email": "john@handson.cloud",
        "Department": "IT", "Section": { "QA": "QA-1", "Reporting Line": "L1" } })
    batch.put_item(Item={"Name": "Lugugo Joshua", "Email": "joshua@handson.cloud",
        "Department": "IT", "Section": { "Development": "SD-1", "Reporting Line": "L1" } })
    batch.put_item(Item={"Name": "Robert Nsamba", "Email": "robert@handson.cloud",
        "Department": "IT", "Section": { "PM": "PM-1", "Reporting Line": "L1" } })
print(batch)

Here is the execution output:

Read Items

To read a single item from the DynamoDB table, use the get_item() method with the key-value pair of items in your table.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Employees')

# boto3 dynamodb getitem
response = table.get_item(
    Key={
        'Name': 'Kelvin Galabuzi',
        'Email': 'kelvingalabuzi@handson.cloud'
    }
)
print(response['Item'])

Here is the Execution output:

To all read Items in the table, you can use the scan() method to return all items in the table.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Employees')

response = table.scan()
response['Items']

print(response)

Here is the Execution output:

Update Items

To update items, you can use the update_item() method to edit an existing item’s attributes. You can also use the same method to add or remove new attributes.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Employees')

response = table.update_item(
    Key={'Name': 'Luzze John', 'Email': 'john@handson.cloud'},
    ExpressionAttributeNames={
        "#section": "Section",
        "#qa": "QA",
        },
        ExpressionAttributeValues={
            ':id': 'QA-2'
        },
        UpdateExpression="SET #section.#qa = :id",
    )
print(response)

Delete Items

To delete an item, use the delete_item() method and specify the item’s primary key.

import boto3

dynamodb = boto3.resource('dynamodb')

table = dynamodb.Table('Employees')

response = table.delete_item(Key = {'Name': 'Peter Matovu', 'Email': 'petermatovu@handson.cloud'})

print(response)

Here is the Execution output.

Query Tables in DynamoDB using Boto3

To query items in DynamoDB, you can use the query() method to fetch items based on primary key values. In addition, you can use the KeyConditionExpression to specify the value of the partition key and return all items from the table with that partition key.

Ensure to use the DynamoDB conditions where for example, the eq() method will fetch all the attributes that match the specified key.

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Employees')

# boto3 ddb query
response = table.query(KeyConditionExpression=Key('Name').eq('Luzze John'))

print("The query returned the following items:")
for item in response['Items']:
    print(item)

Here is the Execution output.

Scan Tables in DynamoDB using Boto3

Scanning can be used to access all the data in tables. The scan() method will read every item in the table and return the data. When you provide additional options such as the FilterExpression, you can return fewer and desired results from the table scan operation.

You can also add additional DynamoDB conditions and retrieve items with specific attributes from the table.

import boto3
from boto3.dynamodb.conditions import Key, Attr

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Employees')

response = table.scan(FilterExpression=Attr('Department').eq('IT'))

print("The query returned the following items:")
for item in response['Items']:
    print(item)

Here is the Execution output.

Global Secondary Index

A Global Secondary Index allows you to query attributes not part of the main table’s primary key. This will help you avoid the slowness and inefficiencies associated with a full table scan operation.

In addition, the global secondary index will contain attributes from the main table but will be organized by a primary key of its own, enabling faster queries.

To learn more about Global Secondary Indexes, consider looking at the information shared in the AWS documentation on this link.

Creating a Global Secondary Index using Boto3

In the following example, I will create a global secondary index to the CapacityBuildingLevel attribute with all employees who have achieved specific capacity-building levels.

import boto3

client = dynamodb = boto3.client('dynamodb')

try:
    resp = client.update_table(
        TableName="Employees",
        AttributeDefinitions=[
            {
                "AttributeName": "CapacityBuildingLevel",
                "AttributeType": "S"
            },
        ],
        GlobalSecondaryIndexUpdates=[
            {
                "Create": {
                    "IndexName": "CapacityBuildingIndex",
                    "KeySchema": [
                        {
                            "AttributeName": "CapacityBuildingLevel",
                            "KeyType": "HASH"
                        }
                    ],
                    "Projection": {
                        "ProjectionType": "ALL"
                    },
                    "ProvisionedThroughput": {
                        "ReadCapacityUnits": 1,
                        "WriteCapacityUnits": 1,
                    }
                }
            }
        ],
    )
    print("Secondary index added!")
except Exception as e:
    print("Error updating table:")
    print(e)

Here is the Execution output.

Querying a Global Secondary Index

Next, you can use the created secondary index to retrieve all Employees with the specific CapacityBuildingLevel set to 4.

import time

import boto3
from boto3.dynamodb.conditions import Key

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Employees')

while True:
    if not table.global_secondary_indexes or table.global_secondary_indexes[0]['IndexStatus'] != 'ACTIVE':
        print('Waiting for index to backfill...')
        time.sleep(5)
        table.reload()
    else:
        break

# boto3 dynamo query
resp = table.query(
    IndexName="CapacityBuildingIndex",
    KeyConditionExpression=Key('CapacityBuildingLevel').eq('Level 4'),
)

print("The query returned the following items:")
for item in resp['Items']:
    print(item)

Here is the Execution output.

Backup a DynamoDB Table using Boto3

To create on-demand backups for tables using Boto3, use the create_backup() method and pass the Table name and the destination backup Table name.

import boto3

client = boto3.client('dynamodb')

response = client.create_backup(
    TableName='Employees',
    BackupName='Employees-Backup-01'
)
print(response)

Here is the execution output.

Delete a DynamoDB Table using Boto3

To delete a DynamoDB table using Boto3, use the delete() table method and insert the name of the Dynamodb table you want to delete.

import boto3

client = boto3.client('dynamodb')

try:
    resp = client.delete_table(
        TableName="Test_Delete",
    )
    print("Table deleted successfully!")
except Exception as e:
    print("Error deleting table:")
    print(e)

Here is the execution output.

Summary

This article covered how to use Python to interact with Amazon DynamoDB to create, list, and perform CRUD operations, query tables, and additional management operations in Amazon DynamoDB.

Suppose you’d like to learn more about using the Boto3 library, especially in combination with AWS Lambda. In that case, we encourage you to check out one of the top-rated Udemy courses on the topic – AWS Automation with Boto3 of Python and Lambda Functions.

LIKE THIS ARTICLE?
Facebook
Twitter
LinkedIn
Pinterest
WANT TO BE AN AUTHOR OF ANOTHER POST?

We’re looking for skilled technical authors for our blog!

Table of Contents