AWS SQS (Simple Queue Service)

SQS provides a reliable and durable method to send and receive messages.
Once a message is sent / queued, it guaranteed to be delivered at least once to those polling the queue.
Once your process has read a message it should be deleted to prevent processing it a 2nd time. To delete a message from the queue a receipt handle is required – in addition to authority to delete it – which is usually the case if you can read it.

Use Case

While there are lots of specific use cases for SQS, the most important general case in my mind, is the fact that by using SQS in multi-step applications, you decouple the application as a whole. Removing points of failure, allowing restarts and distributed processing. And in general making the application more robust – less prone to failure.

Message Retention Max and Defaults

Message retention defaults to 4 days and a maximum of 14 days – although it is defined in seconds.
There is also a delay queue by setting a delay value greater than zero / 0. The default is zero, there is no delay.

AWS SQS (Simple Queue Service) Message Visibility Timeout

To help prevent messages from being processed multiple times by distributed applications SQS implements Visibility Timeout.
Per Amazon:
“Immediately after a message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message. The default visibility timeout for a message is 30 seconds. The maximum is 12 hours.”

Message Queue Polling

On message queue polling – long polling is strongly preferred as short polling (spinning on the queue) can consume large amounts of CPU / resource. The max and default long poll is 20 seconds.
To change to long polling set ReceiveMessageWaitTimeSeconds see (you can do this two way 1.) when reading, 2.) change the attribute of the queue as below):

https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-sqs-long-polling.html

import boto3

# Create SQS client
sqs = boto3.client('sqs')

queue_url = 'SQS_QUEUE_URL'

# Enable long polling on an existing SQS queue
sqs.set_queue_attributes(
    QueueUrl=queue_url,
    Attributes={'ReceiveMessageWaitTimeSeconds': '20'}

SQS Queue Types

There are multiple types of queues:

Standard Queues
FIFO Queues
Dead Letter Queues

SQS Standard Queues

Two of the best features of SQS Service is it’s extremely high availability and scalability. AWS achieves this especially for standard queues by distributing the service across many many nodes. While taking the distributed approach to the system certainly massively improves high availability and scalability, for pure scalabilities sake standard queues do not guarantee first in first out reading of messages from Standard Queues.
With a standard queue you could add a timestamp to the message data itself and order on that if you choose. Or you can choose the less scalable FIFO queue type to guarantee ordered reads.

SQS fifo QUEUES

FIFO queues gaurantee messages are read in the order they were place in the queue. You cannot modify a standard queue to FIFO – drop and recreate instead. FIFO queues do not scale nearly as well as standard, so there is a trade off.

Other SQS Facts

Each Queue Can be individually configured
Messages can contain up to 256 KB of text data, including XML, JSON, and unformatted text.
Standard and FIFO queues support server-side encryption (new feature 2017).

References:
https://aws.amazon.com/documentation/sqs/
More from LonzoDB on AWS

Post Views: 9