Working with ECS in Python using Boto3

Sathiya Sarathi

Sathiya Sarathi

0
(0)

AWS ECS (Elastic Container Service) is a managed Docker container service to provision and orchestrate Docker containers and container-based applications. Automating and managing ECS services might be tricky. This article will cover how to manage AWS ECS clusters, tasks, task definitions, and services using Python and the Boto3 library.

Prerequisites

To get started with the AWS Elastic Container Service automation using Boto3, you need to set up your Python environment on your laptop.

As a summary, you need to install:

  • Python 3
  • Boto3
  • AWS CLI Tools

Alternatively, you can set up and launch a Cloud9 IDE Instance.

Managing ECS Cluster

The components of AWS ECS form the following hierarchy:

  • Cluster – A cluster is a logical grouping of tasks or services
  • Task Definition – The task definition is a text file, in JSON format, that describes one or more containers, up to a maximum of ten, that form your application. It can be thought of as a blueprint for your application.
  • Service allows you to run and maintain a specified number of instances of a task definition simultaneously in an AWS ECS cluster
  • Task – is the instantiation of a task definition within a cluster

An ECS cluster launches the groups of infrastructure resources (services and tasks). The infrastructure capacity is provided by both AWS ECS EC2 based and Fargate, where Fargate is a much-preferred option for lower management overhead.

For more information about ECS components, we strongly recommend you to check out The role of AWS Fargate in the container world article.

Create Cluster

To create an ECS cluster using Boto3, you need to use the create_cluster() method of the ECS client. This method requires the clusterName as a parameter.

The following example is the most straightforward way of launching the ECS Fargate cluster:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

response = client.create_cluster(clusterName="WebServices")

print(json.dumps(response, indent=4))

In the console, the script returns the cluster name, ARN, status, and other helpful metadata information:

Working with ECS - Create Cluster
Working with ECS – Create Cluster

In the AWS console, search for ECS in the search tab. The cluster resides under the Clusters tab of the ECS console.

Working with ECS - Create Cluster Console View
Working with ECS – Create Cluster Console View

List Clusters

To list all created ECS clusters in the AWS account, you need to use the list_clusters() method of the Boto3 ECS client.

The following code lists the clusters’ ARNs in the N. Virginia (us-east-1) region:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_clusters')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
})

for each_page in response_iterator:
    for each_arn in each_page['clusterArns']:
        print(each_arn)

Here’s an execution output:

Working with ECS - List Clusters
Working with ECS – List Clusters

Describe Clusters

To describe the ECS cluster and get all cluster’s metadata information, including running tasks, pending tasks, active services, failed tasks, etc., you need to use the describe_clusters() method of the Boto3 ECS client:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_clusters')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
})

for each_page in response_iterator:
    for each_arn in each_page['clusterArns']:
        response = client.describe_clusters(clusters=[each_arn])
        print(json.dumps(response, indent=4))

We’re using a paginator object to iterate through the entire list of clusters.

Each iterator object returns the list of the clusterArns, which we can use in combination with the describe_clusters() function to return information about all clusters.

Working with ECS - Describe Clusters
Working with ECS – Describe Clusters

Delete Cluster

To delete the ECS cluster using the Boto3 library, you need to use the delete_cluster() method of the ECS client.

Note: you need to delete all running tasks and services before deleting the ECS cluster.

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

response = client.delete_cluster(cluster="WebServices")

print(json.dumps(response, indent=4))

Here’s an execution output:

Working with ECS -  Delete Cluster
Working with ECS – Delete Cluster

Managing ECS Task Definitions

An ECS Task Definition specifies the configuration parameters for the ECS task to assume during its launch. Task Definitions are similar to Dockerfiles that act as blueprints for spinning up the tasks (i.e., containers).

Create Task Definition

To create a Task Definition for ECS using Boto3, you need to use the register_task_definition() method of the ECS client. This method takes the containerDefinitions list as an argument, containing a list of configurations of containers’ properties such as CPU, memory, image, mount points, etc.

Here’s a complete set of supported parameters for the task definition.

Upon execution, the following code registers a task definition that is used to create a simple container with the PHP-based webpage (amazon/amazon-ecs-sample Docker image):

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

response = client.register_task_definition(
        containerDefinitions=[
            {
                "name": "AmazonSampleImage",
                "image": "amazon/amazon-ecs-sample",
                "cpu": 0,
                "portMappings": [],
                "essential": True,
                "environment": [],
                "mountPoints": [],
                "volumesFrom": [],
                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {
                        "awslogs-group": "/ecs/AWSSampleApp",
                        "awslogs-region": "us-east-1",
                        "awslogs-stream-prefix": "ecs"
                    }
                }
            }
        ],
        executionRoleArn="arn:aws:iam::585584209241:role/ecsTaskExecutionRole",
        family= "AWSSampleApp2",
        networkMode="awsvpc",
        requiresCompatibilities= [
            "FARGATE"
        ],
        cpu= "256",
        memory= "512")

print(json.dumps(response, indent=4, default=str))

Here’s an execution output:

Working with ECS - Register Task Definitions
Working with ECS – Register Task Definitions

The presence of a taskDefinitionArn shows that the task definition has been successfully registered. This task definition ARN is used by ECS client methods to deploy tasks and services.

List Task Definition Families

When you register a task definition, you set up its Family (AWSSampleApp2 in our previous example), similar to a name for multiple versions of the task definition, specified with a revision number. The first task definition registered into a particular family has a revision 1, and any task definitions registered after that are given a sequential revision number.

To list all ECS Task Definition Families, you need to use the list_task_definition_families() method of the Boto3 ECS client.

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_task_definition_families')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    print(each_page['families'])

In the example above, we’re using the paginator to process a complete list of Task Definition Families in the AWS account.

Here’s an execution output:

Working with ECS - List Task Definition Families
Working with ECS – List Task Definition Families

List Task Definitions

To list Task Definitions in the ECS service, you need to use the list_task_definitions() method of the Boto3 ECS client, which returns versioned and newly created task definitions:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_task_definitions')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_task in each_page['taskDefinitionArns']:
        print(each_task)

The output from the example above returns ARNs of the task definitions that are present in your AWS account:

Working with ECS - List Task Definitions
Working with ECS – List Task Definitions

Describe Task Definition

To describe the ECS task definition, you need to use the describe_task_definition() method of the Boto3 ECS client.

Let’s iterate through the list of the task definitions and describe each of them:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_task_definitions')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_task_definition in each_page['taskDefinitionArns']:
        response = client.describe_task_definition(taskDefinition=each_task_definition)
        print(json.dumps(response, indent=4, default=str))

The output contains image name, CPU, memory, port mappings, and other Task Definition’s attributes:

Working with ECS – Describe Task Definition

Deregister Task Definition

To deregister ECS Task Definition, you need to use the deregister_task_definition() method of the Boto3 ECS client. This method requires Task Definition ARN as a parameter:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_task_definitions')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_task_definition in each_page['taskDefinitionArns']:
        response = client.deregister_task_definition(
                        taskDefinition=each_task_definition)
        print(json.dumps(response, indent=4, default=str))

Here’s an execution output:

Working with ECS - Deregister Task Definitions
Working with ECS – Deregister Task Definitions

Managing ECS Tasks

ECS cluster launches tasks based on their Task Definitions, and Boto3 provides all required methods to run and manage these tasks effectively. In this part of the article, we will run a simple web server application from our previously defined Task Definition, which runs the amazon/amazon-ecs-sample Docker image.

Run Task

To run a Task on the ECS Fargate cluster, you need to use the run_task() method of the Boto3 ECS library and provide it required arguments such as Task Definition, launch configuration, network configuration, public IP preferences, and other required arguments.

An important point to note here is that the public exposure of the Task depends on the network configuration and the assignment of public IPs to the tasks.

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

ec2 = boto3.resource('ec2')

response = client.run_task(
    taskDefinition='console-sample-app',
    launchType='FARGATE',
    cluster='default',
    platformVersion='LATEST',
    count=1,
    networkConfiguration={
        'awsvpcConfiguration': {
            'subnets': [
                'subnet-9e7c0fc1',
            ],
            'assignPublicIp': 'ENABLED',
            'securityGroups': ["sg-0bfa5becbde957cd3"]
        }
    }
)

print(json.dumps(response, indent=4, default=str))

The returned response data contains detailed information about the launched Task.

The details include the success metrics and reasons for failures in case of any issues:

Working with ECS - Run Task
Working with ECS – Run Task

The following AWS console screenshot shows the successfully launched Task:

Working with ECS – Run Task Console View

The task Details tab shows details about the public and private IPs:

 Working with ECS - Run Task Console View Detailed with PublicIP
Working with ECS – Run Task Console View Detailed with PublicIP

You can use the public IP to access the demo web page served by the launched Docker container:

Working with ECS - App run from the created Task
Working with ECS – App run from the created Task

List Tasks

To list ECS Tasks, you need to use the list_tasks() method of the Boto3 ECS client. This method show tasks launched in the default cluster if the cluster ID is not present.

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_tasks')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

counter = 1
for each_page in response_iterator:
    for each_task in each_page['taskArns']:
        print(each_task)

Here’s an execution output:

Working with ECS - List Tasks
Working with ECS – List Tasks

Describe Tasks

To describe ECS Tasks, you need to use the describe_tasks() method of the Boto3 ECS client.

This method returns a detailed description of the ECS tasks present in the corresponding AWS region:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_tasks')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_task in each_page['taskArns']:
        response = client.describe_tasks(tasks=[each_task])
        print(json.dumps(response, indent=4, default=str))

Here’s an execution output:

Working with ECS - Describe Tasks
Working with ECS – Describe Tasks

Stop Task

To stop the ECS Task, you need to use the stop_task() method of the Boto3 ECS client. This method requires the taskArn as an argument.

In combination with the list_tasks() method, you can stop all tasks in the specific cluster.

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

paginator = client.get_paginator('list_tasks')

response_iterator = paginator.paginate(
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_task in each_page['taskArns']:
        response = client.stop_task(task=each_task)
        print(json.dumps(response, indent=4, default=str))

Optionally, you can use additional if statements to filter the required Task by the name of Family.

This generates the following output containing the details of stopped tasks:

Working with ECS - Stop Task
Working with ECS – Stop Task

Managing ECS Service

An ECS Services allows you to launch multiple tasks using their task definition and keep them in the desired running state. The typical use-case for the ECS Service is to launch several ECS Tasks acting as a web server, for example.

Create Service

To create and launch ECS Service, you need to use the create_service() method of the Boto3 ECS client. This method is similar to the create_task() method, but it requires an additional argument to specify the desired amount of Tasks launched by the Service (desiredCount):

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

clusters = client.list_clusters()

cluster_name = clusters['clusterArns'][0]

response = client.create_service(cluster=cluster_name, 
                serviceName="SimpleWebServer",
                taskDefinition='console-sample-app',
                desiredCount=1,
                networkConfiguration={
                    'awsvpcConfiguration': {
                        'subnets': [
                            'subnet-9e7c0fc1',
                        ],
                        'assignPublicIp': 'ENABLED',
                        'securityGroups': ["sg-0bfa5becbde957cd3"]
                    }
                },
                launchType='FARGATE',
            )

print(json.dumps(response, indent=4, default=str))

The response object contains the details about the newly launched Service:

Working with ECS - Create Service
Working with ECS – Create Service

List Services

To list ECS Services, you need to use the list_services() method of the Boto3 ECS client.

Let’s retrieve a list of the Service ARNs that are launched in the specific cluster:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

clusters = client.list_clusters()

cluster_name = clusters['clusterArns'][0]

paginator = client.get_paginator('list_services')

response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        print(each_arn)

Here’s an execution output:

Working with ECS - List Services
Working with ECS – List Services

Describe Services

To get the information about all ECS cluster Services, you need to use the describe_services() method of the Boto3 ECS client:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

clusters = client.list_clusters()

cluster_name = clusters['clusterArns'][0]

paginator = client.get_paginator('list_services')

response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        response = client.describe_services(
            services=[
                each_arn
            ]
        )
        print(json.dumps(response, indent=4, default=str))

The above code returns the description of each Service, including the count of running tasks, launch type, load balancers, and other Service-related information:

Working with ECS - Describe Services
Working with ECS – Describe Services

Update Service

To update the existing ECS Service, you need to use the update_service() method of the Boto3 ECS client.

In the following example, we will update the desiredCount of the Service Task:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

clusters = client.list_clusters()

cluster_name = clusters['clusterArns'][0]

paginator = client.get_paginator('list_services')

response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        response = client.update_service(
            service = each_arn,
            desiredCount = 2
        )
        print(json.dumps(response, indent=4, default=str))

Here’s how the Service looks like in the AWS console before the update operation.

Notice that the desired count is 1.

Before Update - Service: Console View
Before Update – Service: Console View

Upon execution of the code, the output shows that the desired count has been changed from 1 to 2:

Update Service - Running count change for the running tasks
Update Service – Running count change for the running Tasks

The same information is available in the AWS console under the Services tab:

 After Update - Service: Console View
After Update – Service: Console View

Delete Service

To stop and delete the ECS Service, you need to use the delete_service() method of the Boto3 ECS client. This method takes the serviceArn as an argument to delete the corresponding Service:

import boto3
import json

client = boto3.client("ecs", region_name="us-east-1")

clusters = client.list_clusters()

cluster_name = clusters['clusterArns'][0]

paginator = client.get_paginator('list_services')

response_iterator = paginator.paginate(
    cluster=cluster_name,
    PaginationConfig={
        'PageSize':100
    }
)

for each_page in response_iterator:
    for each_arn in each_page['serviceArns']:
        response = client.delete_service(
                service = each_arn,
                force = True)
        print(json.dumps(response, indent=4, default=str))

The output shows the full description of the Service that has been deleted.

Notice that the status has now turned to DRAINING, which means that ECS is stopping all Tasks launched by the Service before deleting the Service itself:

Working with ECS - Delete Service
Working with ECS – Delete Service

Summary

In this article, we’ve covered how to manage AWS ECS clusters, tasks, task definitions, and services by using Python and the Boto3 library.

How useful was this post?

Click on a star to rate it!

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Top rated Udemy Courses to improve you career

Subscribe to our updates

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use https://codeshare.io/ or a similar tool as Facebook comments are breaking code formatting.