Boto3 ECS – Complete Tutorial

AWS ECS (Elastic Container Service) is a managed Docker container service to provision and orchestrate Docker containers and container-based applications. Automating and managing ECS services might be tricky. This Boto3 ECS tutorial covers managing AWS ECS clusters, tasks, task definitions, and services using Python and the Boto3 library.

Prerequisites

To get started with the AWS Elastic Container Service automation using Boto3, you need to set up your Python environment on your laptop.

In summary, you need to install:

• Python 3
• Boto3
• AWS CLI Tools

Alternatively, you can set up and launch a Cloud9 IDE Instance.

Boto3 ECS – Managing ECS Cluster

The components of AWS ECS form the following hierarchy:

• Cluster – A cluster is a logical grouping of tasks or services
• Task Definition – The task definition is a text file in JSON format that describes one or more containers, up to a maximum of ten, that form your application. It can be thought of as a blueprint for your application.
• Service allows you to run and maintain a specified number of instances of a task definition simultaneously in an AWS ECS cluster
• Task – is the instantiation of a task definition within a cluster

An ECS cluster launches the groups of infrastructure resources (services and tasks). The infrastructure capacity is provided by AWS ECS EC2 based and Fargate, where Fargate is a much-preferred option for lower management overhead.

For more information about ECS components, we strongly recommend you to check out The role of AWS Fargate in the container world article.

Create Cluster

To create an ECS cluster using Boto3, you need to use the create_cluster() method of the ECS client. This method requires the clusterName as a parameter.

The following example is the most straightforward way of launching the ECS Fargate cluster:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
response = client.create_cluster(clusterName="WebServices")
print(json.dumps(response, indent=4))

In the console, the script returns the cluster name, ARN, status, and other helpful metadata information:

In the AWS console, search for ECS in the search tab. The cluster resides under the Clusters tab of the ECS console.

List Clusters

To list all created ECS clusters in the AWS account, you need to use the list_clusters() method of the Boto3 ECS client.

The following code lists the clusters’ ARNs in the N. Virginia (us-east-1) region:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_clusters')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
})
for each_page in response_iterator:
    for each_arn in each_page['clusterArns']:
        print(each_arn)

Here’s an execution output:

Describe Clusters

To describe the ECS cluster and get all cluster’s metadata information, including running tasks, pending tasks, active services, failed tasks, etc., you need to use the describe_clusters() method of the Boto3 ECS client:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_clusters')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
})
for each_page in response_iterator:
    for each_arn in each_page['clusterArns']:
        response = client.describe_clusters(clusters=[each_arn])
        print(json.dumps(response, indent=4))

We’re using a paginator object to iterate through the entire list of clusters.

Each iterator object returns the list of the clusterArns, which we can use in combination with the describe_clusters() function to return information about all clusters.

Delete Cluster

To delete the ECS cluster using the Boto3 library, you need to use thedelete_cluster() method of the ECS client.

Note: you need to delete all running tasks and services before deleting the ECS cluster.

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
response = client.delete_cluster(cluster="WebServices")
print(json.dumps(response, indent=4))

Here’s an execution output:

Managing ECS Task Definitions

An ECS Task Definition specifies the configuration parameters for the ECS task to assume during its launch. Task Definitions are similar to Dockerfiles that act as blueprints for spinning up the tasks (i.e., containers).

Create Task Definition

To create a Task Definition for ECS using Boto3, you need to use the register_task_definition() method of the ECS client. This method takes the containerDefinitions list as an argument, containing a list of configurations of containers’ properties such as CPU, memory, image, mount points, etc.

Here’s a complete set of supported parameters for the task definition.

Upon execution, the following code registers a task definition that is used to create a simple container with the PHP-based webpage (amazon/amazon-ecs-sample Docker image):

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
response = client.register_task_definition(
        containerDefinitions=[
            {
                "name": "AmazonSampleImage",
                "image": "amazon/amazon-ecs-sample",
                "cpu": 0,
                "portMappings": [],
                "essential": True,
                "environment": [],
                "mountPoints": [],
                "volumesFrom": [],
                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {
                        "awslogs-group": "/ecs/AWSSampleApp",
                        "awslogs-region": "us-east-1",
                        "awslogs-stream-prefix": "ecs"
                    }
                }
            }
        ],
        executionRoleArn="arn:aws:iam::585584209241:role/ecsTaskExecutionRole",
        family= "AWSSampleApp2",
        networkMode="awsvpc",
        requiresCompatibilities= [
            "FARGATE"
        ],
        cpu= "256",
        memory= "512")
print(json.dumps(response, indent=4, default=str))

Here’s an execution output:

The presence of a taskDefinitionArn shows that the task definition has been successfully registered. This task definition ARN is used by ECS client methods to deploy tasks and services.

The image parameter in the task definition above references a public Docker image responsible for some automation. For example, you can execute AWS CLI to manage Amazon S3 buckets and objects within your Docker image.

List Task Definition Families

When you register a task definition, you set up its Family (AWSSampleApp2 in our previous example), similar to a name for multiple versions of the task definition, specified with a revision number. The first task definition registered into a particular family has a revision 1. Any task definitions registered after that are given a sequential revision number.

To list all ECS Task Definition Families, you need to use the list_task_definition_families() method of the Boto3 ECS client.

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_task_definition_families')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    print(each_page['families'])

In the example above, we’re using the paginator to process a complete list of Task Definition Families in the AWS account.

Here’s an execution output:

List Task Definitions

To list Task Definitions in the ECS service, you need to use the list_task_definitions() method of the Boto3 ECS client, which returns versioned and newly created task definitions:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_task_definitions')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_task in each_page['taskDefinitionArns']:
        print(each_task)

The output from the example above returns ARNs of the task definitions that are present in your AWS account:

Describe Task Definition

To describe the ECS task definition, you need to use the describe_task_definition() method of the Boto3 ECS client.

Let’s iterate through the list of the task definitions and describe each of them:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_task_definitions')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_task_definition in each_page['taskDefinitionArns']:
        response = client.describe_task_definition(taskDefinition=each_task_definition)
        print(json.dumps(response, indent=4, default=str))

The output contains image name, CPU, memory, port mappings, and other Task Definition’s attributes:

Deregister Task Definition

To deregister ECS Task Definition, you need to use the deregister_task_definition() method of the Boto3 ECS client. This method requires Task Definition ARN as a parameter:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_task_definitions')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_task_definition in each_page['taskDefinitionArns']:
        response = client.deregister_task_definition(
                        taskDefinition=each_task_definition)
        print(json.dumps(response, indent=4, default=str))

Here’s an execution output:

Managing ECS Tasks

ECS cluster launches tasks based on their Task Definitions, and Boto3 provides all required methods to run and manage these tasks effectively. In this part of the article, we will run a simple web server application from our previously defined Task Definition, which runs theamazon/amazon-ecs-sample Docker image.

Run Task

To run a Task on the ECS Fargate cluster, you need to use the run_task() method of the Boto3 ECS library and provide it required arguments such as Task Definition, launch configuration, network configuration, public IP preferences, and other required arguments.

An important point to note here is that the public exposure of the Task depends on the network configuration and the assignment of public IPs to the tasks.

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
ec2 = boto3.resource('ec2')
response = client.run_task(
    taskDefinition='console-sample-app',
    launchType='FARGATE',
    cluster='default',
    platformVersion='LATEST',
    count=1,
    networkConfiguration={
        'awsvpcConfiguration': {
            'subnets': [
                'subnet-9e7c0fc1',
            ],
            'assignPublicIp': 'ENABLED',
            'securityGroups': ["sg-0bfa5becbde957cd3"]
        }
    }
)
print(json.dumps(response, indent=4, default=str))

The returned response data contains detailed information about the launched Task.

The details include the success metrics and reasons for failures in case of any issues:

The following AWS console screenshot shows the successfully launched Task:

The task Details tab shows details about the public and private IPs:

You can use the public IP to access the demo web page served by the launched Docker container:

List Tasks

To list ECS Tasks, you need to use the list_tasks() method of the Boto3 ECS client. This method show tasks launched in the default cluster if the cluster ID is not present.

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_tasks')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
counter = 1
for each_page in response_iterator:
    for each_task in each_page['taskArns']:
        print(each_task)

Here’s an execution output:

Describe Tasks

To describe ECS Tasks, you need to use the describe_tasks() method of the Boto3 ECS client.

This method returns a detailed description of the ECS tasks present in the corresponding AWS region:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_tasks')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_task in each_page['taskArns']:
        response = client.describe_tasks(tasks=[each_task])
        print(json.dumps(response, indent=4, default=str))

Here’s an execution output:

Stop Task

To stop the ECS Task, you need to use the stop_task() method of the Boto3 ECS client. This method requires the taskArn as an argument.

Combined with thelist_tasks() method, you can stop all tasks in the specific cluster.

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
paginator = client.get_paginator('list_tasks')
response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_task in each_page['taskArns']:
        response = client.stop_task(task=each_task)
        print(json.dumps(response, indent=4, default=str))

Optionally, you can use additional if statements to filter the required Task by the name of Family.

This generates the following output containing the details of stopped tasks:

Managing ECS Service

An ECS Services allows you to launch multiple tasks using their task definition and keep them in the desired running state. The typical use-case for the ECS Service is to launch several ECS Tasks acting as a web server.

Create Service

To create and launch ECS Service, you need to use the create_service() method of the Boto3 ECS client. This method is similar to the create_task() method, but it requires an additional argument to specify the desired amount of Tasks launched by the Service (desiredCount):

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
clusters = client.list_clusters()
cluster_name = clusters['clusterArns'][0]
response = client.create_service(cluster=cluster_name, 
                serviceName="SimpleWebServer",
                taskDefinition='console-sample-app',
                desiredCount=1,
                networkConfiguration={
                    'awsvpcConfiguration': {
                        'subnets': [
                            'subnet-9e7c0fc1',
                        ],
                        'assignPublicIp': 'ENABLED',
                        'securityGroups': ["sg-0bfa5becbde957cd3"]
                    }
                },
                launchType='FARGATE',
            )
print(json.dumps(response, indent=4, default=str))

The response object contains the details about the newly launched Service:

List Services

To list ECS Services, you need to use the list_services() method of the Boto3 ECS client.

Let’s retrieve a list of the Service ARNs that are launched in the specific cluster:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
clusters = client.list_clusters()
cluster_name = clusters['clusterArns'][0]
paginator = client.get_paginator('list_services')
response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        print(each_arn)

Here’s an execution output:

Describe Services

To get the information about all ECS cluster Services, you need to use the describe_services() method of the Boto3 ECS client:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
clusters = client.list_clusters()
cluster_name = clusters['clusterArns'][0]
paginator = client.get_paginator('list_services')
response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        response = client.describe_services(
            services=[
                each_arn
            ]
        )
        print(json.dumps(response, indent=4, default=str))

The above code returns the description of each Service, including the count of running tasks, launch type, load balancers, and other Service-related information:

Update Service

To update the existing ECS Service, you need to use theupdate_service() method of the Boto3 ECS client.

In the following example, we will update the desiredCount of the Service Task:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
clusters = client.list_clusters()
cluster_name = clusters['clusterArns'][0]
paginator = client.get_paginator('list_services')
response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        response = client.update_service(
            service = each_arn,
            desiredCount = 2
        )
        print(json.dumps(response, indent=4, default=str))

Here’s what the Service looks like in the AWS console before the update operation.

Notice that the desired count is 1.

Upon execution of the code, the output shows that the desired count has been changed from 1 to 2:

The same information is available in the AWS console under the Services tab:

Delete Service

To stop and delete the ECS Service, you need to use the delete_service() method of the Boto3 ECS client. This method takes the serviceArn as an argument to delete the corresponding Service:

import boto3
import json
client = boto3.client("ecs", region_name="us-east-1")
clusters = client.list_clusters()
cluster_name = clusters['clusterArns'][0]
paginator = client.get_paginator('list_services')
response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)
for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        response = client.delete_service(
                service = each_arn,
                force = True)
        print(json.dumps(response, indent=4, default=str))

The output shows the full description of the Service that has been deleted.

Notice that the status has now turned to DRAINING, which means that ECS is stopping all Tasks launched by the Service before deleting the Service itself:

Additional Learning Resources

Free hands-on AWS workshops

We recommend the following free AWS Lambda Workshops to get more hands-on experience on the subject:

• AWS Lambda Workshop
• Migrate Spring Boot applications to AWS Lambda
• Building CI/CD pipelines for Lambda canary deployments using AWS CDK
• Content Generation Lambda@Edge

Also, we recommend the following free AWS Step Functions Workshops:

• The AWS Step Functions Workshop

Paid courses

From our experience, these are the best hands-on paid learning materials today related to Serverless, AWS Lambda, and Step Functions:

Summary

This article covered how to manage AWS ECS clusters, tasks, task definitions, and services using Python and the Boto3 library.