How to migrate Google DataStore to AWS DynamoDB

In this small post I’ll show you how you can migrate Google DataStore kind to AWS DynamoDB table.

Virtual environment

virtualenv venv
source venv/bin/activate
pip install boto3
pip install google-cloud-datastore

Migration process

First of all we’ll need to get DataStore data:

from import datastore

datastore_client = datastore.Client()
query = datastore.Query(datastore_client, kind='MyKind')

ds_results = []
for result in query.fetch():

After that we need to connect to our DynamoDB table:

import boto3
import json
import decimal
from boto3.dynamodb.conditions import Key, Attr

# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
                return int(o)
        return super(DecimalEncoder, self).default(o)

dynamodb = boto3.resource('dynamodb', region_name='us-east-1')

table = dynamodb.Table('MyTable')

And finally we need to make a migration.

To make everything’s happen we need:

  • convert GCP DataStore Entity to JSON representation
  • remove keys with possible empty values
  • send and item to DynamoDB
for item in ds_results:
    i = json.loads(json.dumps(item))
    i = dict((k, v) for k, v in i.iteritems() if v)

Hope, this will save somebody’s time!


Author avatar
Andrei Maksimov

I’m a passionate Cloud Infrastructure Architect with more than 15 years of experience in IT.

Let’s discuss your AWS questions if you still have them.

This post represents my personal experience and opinion about the topic.