AWS SQS – The Most Important Information

Amazon Simple Queue Service (SQS) is a web service that provides a secure, durable, and highly available distributed fully managed message queuing system and enables companies to integrate and decouple distributed software systems and components in their architecture. SQS allows applications to send messages to queues and then deliver them to other parts of the application for processing in a microservice architecture. As a result, different services can communicate and scale independently.

In this AWS SQS tutorial, we’ll cover everything you need to know about the oldest AWS service – Amazon SQS queue. Let’s get started.

Terminology

There are several terms we need to know when speaking about Amazon SQS:

• Queue – the actual message queue that is responsible for messages delivery
• Producer – an application, process, or service that sends messages to the queue
• Consumer – an application, process, or service that receives (polls) messages from the queue, processes them, and deletes them from the queue

You can have as many producers and consumers sending and receiving messages through the queue.

Every SQS queue plays the role of the buffer to decouple producers and consumers.

Queue types

You can create one of the two SQS queue types:

Standard queues

The primary purpose of the Standard SQS queue is to decouple applications, software components, or microservices, allocate messages for consumers, and batch messages for future consumer processing. In summary, we use message queues to handle task processing asynchronously.

Standard queue features:

• Unlimited throughput and number of messages in the queue.
• The default retention period for messages: 4 days (14 days max).
• Low latency (< 10ms on publish and receive operations)
• Maximum message size: 256 KB

When using the Standard queue, you can have duplicate or out-of-order messages.

An example of using the Standard queue is processing files uploaded to the S3 bucket.

API call for sending messages to Amazon SQS – SendMessage.

For more information on using Amazon SQS API, check the Boto3 SQS – Complete Tutorial article or the official Boto3 SQS documentation.

FIFO queues

First In First Out (FIFO) SQS queue guarantees that messages will be delivered between software components in distributed systems in order and exactly once.

AWS SQS Limits

Let’s review the most important AWS SQS limitations.

Standard SQS queue

Amazon Standard SQS has the following limitations:

1. Maximum of 120,000 inflight messages per queue (might be extended through a support service request)
2. Maximum visibility timeout of 12 hours for a single message
3. Limited support for large messages (up to 256KB)
4. The message retention period is limited to 14 days

FIFO SQS queue

1. Maximum of 20,000 inflight messages per queue (hard limit)
2. Maximum visibility timeout of 12 hours for a single message
3. Limited support for large messages (up to 256KB)
4. The message retention period is limited to 14 days
5. Maximum throughput is limited to 3000 msgs/s with batching or 300 msgs/s without batching

I recommend checking the “Dissecting SQS FIFO Queues — Does Ordered and Exactly Once Messaging Really Exist?” article for more information about FIFO SQS queue limitations.

Create an AWS SQS queue

You can create an Amazon SQS queue using Amazon SQS console (Web Management Console), AWS CLI, API calls to Amazon SQS service, or Infrastructure as Code (IaC) solutions, such as Terraform, CloudFormation, AWS CDK, or AWS SAM.

Let’s look at how to create an Amazon SQS queue using Web Management Console.

Navigate to the SQS Service part of the web interface:

Click the “Create queue” button:

Select the required queue type and enter the queue name:

Optionally, change visibility timeout, message retention period, delivery delay, maximum message size, and receive message wait time configuration parameters:

Choose required queue encryption settings:

Adjust queue Access policy if required:

Finally, set up Dead Letter Queue settings and Tags, and hit the “Create queue” button.

Amazon SQS message

The Amazon Simple Queue service message can contain 256 KB of text data, which is persisted in the queue until the consumer deletes it. If you need to send larger messages, it is recommended to store the message payload in S3 and send only a reference to the S3 object through the queue.

As soon as the Amazon SQS receives the message, it is called an “in flight” message.

You can send as many messages to the queue as you need, but there’s a quota for inflight messages – 120,000 for a single Standard SQS queue (might be increased) and 20,000 for a single FIFO SQS queue (hard limit).

Consuming SQS messages

You have multiple different options to receive and process messages from the Amazon SQS:

• AWS Lambda
• EC2 instances
• Containers (Fargate and Kubernetes)
• Applications at on-premises

Every Simple Queue Service consumer can poll SQS for up to 10 messages at a time using the ReceiveMessage API call, process them, and delete processed messages from the queue using the DeleteMessage API call.

In the case of multiple consumers, each consumer will get a different set of messages from the queue.

If the message is not processed fast enough by the consumer, it might become visible and picked up by other consumers polling the queue (the same message might be processed several times). There are two ways of solving this “issue”:

• Add additional consumers to improve processing throughput
• Increase queue visibility timeout (starts when the consumer receives a message)
• Use AWS SQS delay queues to delay the visibility of all incoming messages in the queue
• Use SQS message timers to delay the delivery of individual messages

Using SQS with Auto Scaling Group (ASG)

EC2 instances, Fargate tasks, or Kubernetes pods can be auto-scaled. When using auto-scaling to manage the number of consumers processing the queue, you need to rely on the ApproximateNumberOfMessages metric in CloudWatch Metric service, which monitors the queue length.

SQS Pricing

AWS SQS pricing is straightforward. The first million requests are free, then $0.40 / 1M requests for a Standard queue and $0.50 / 1M requests for a FIFO queue.

SQS Security

Let’s cover the most important AWS Simple Queue Service security features.

Encryption

Amazon SQS supports several encryption mechanisms for transmitting sensitive data:

• AWS SQS has encryption in transit – All API calls to the service are encrypted (HTTPS) by default
• AWS SQS supports encryption at rest using KMS keys and client-side encryption if you’d like to encrypt messages yourself.

Access controls

Access to SQS APIs is controlled by:

• AWS IAM roles’ policies used by users and services
• SQS Access Policy
• KMS keys IAM policies

You can use SQS Access Policy to:

• Grant cross-account access to message queues
• Grant S3, SNS, or other AWS services permissions to send messages to the queue

FAQ