SQS and SWS
Amazon Simple Queue Service
Amazon SQS is a fast, reliable, scalable, and fully managed message queuing service. You can use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be continuously available.
An Amazon SQS queue is basically a buffer between the application components that receive data and those components that process the data in your system. Amazon SQS ensures delivery of each message at least once and supports multiple readers and writers interacting with the same queue.
Amazon SQS is engineered to be highly available and to deliver messages reliably and efficiently; however, the service does not guarantee First In, First Out (FIFO) delivery of messages. For many distributed applications, each message can stand on its own and, if all messages are delivered, the order is not important. If your system requires that order be preserved, you can place sequencing information in each message so that you can reorder the messages when they are retrieved from the queue.
Below figure describes the lifecycle of an Amazon SQS message, called Message A, from creation to deletion. Assume that a queue already exists.
- Component 1 sends Message A to a queue, and the message is redundantly distributed across the Amazon SQS servers.
- When Component 2 is ready to process a message, it retrieves messages from the queue, and Message A is returned. While Message A is being processed, it remains in the queue and is not returned to subsequently receive requests for the duration of the visibility timeout.
- Component 2 deletes Message A from the queue to prevent the message from being received and processed again after the visibility timeout expires.
Delay Queues and Visibility Timeouts:
Delay queues allow you to postpone the delivery of new messages in a queue for a specific number of seconds. If you create a delay queue, any message that you send to that queue will be invisible to consumers for the duration of the delay period.
To create a delay queue, use CreateQueue and set the DelaySeconds attribute to any value between 0 and 900 (15 minutes). You can also turn an existing queue into a delay queue by using SetQueueAttributesto set the queue’s DelaySeconds attribute. The default value for DelaySeconds is 0.
Delay queues are similar to visibility timeouts in that both features make messages unavailable to consumers for a specific period of time. The difference is that a delay queue hides a message when it is first added to the queue, whereas a visibility timeout hides a message only after that message is retrieved from the queue. Figure illustrates the functioning of a visibility timeout.
When a message is in the queue but is neither delayed nor in a visibility timeout, it is considered to be “in flight.” You can have up to 120,000 messages in flight at any given time. Amazon SQS supports up to 12 hours’ maximum visibility timeout.
Queue Operations, Unique IDs, and Metadata
The defined operations for Amazon SQS queues are
Only the AWS account owner or an AWS identity that has been granted the proper permissions can perform operations.
Your messages are identified via a globally unique ID that Amazon SQS returns when the message is delivered to the queue. The ID isn’t required in order to perform any further actions on the message, but it’s useful for tracking whether a particular message in the queue has been received. When you receive a message from the queue, the response includes a receipt handle, which you must provide when deleting the message.
Queue and Message Identifiers
Amazon SQS uses three identifiers that you need to be familiar with: queue URLs, message IDs, and receipt handles.
When creating a new queue, you must provide a queue name that is unique within the scope of all of your queues. Amazon SQS assigns each queue an identifier called a queue URL, which includes the queue name and other components that Amazon SQS determines. Whenever you want to perform an action on a queue, you must provide its queue URL.
Amazon SQS assigns each message a unique ID that it returns to you in the SendMessage response. This identifier is useful for identifying messages, but note that to delete a message, you need the message’s receipt handle instead of the message ID. The maximum length of a message ID is 100 characters.
Each time you receive a message from a queue, you receive a receipt handle for that message. The handle is associated with the act of receiving the message, not with the message itself. As stated previously, to delete the message or to change the message visibility, you must provide the receipt handle and not the message ID. This means you must always receive a message before you can delete it (that is, you can’t put a message into the queue and then recall it). The maximum length of a receipt handle is 1,024 characters.
Amazon SQS provides support for message attributes. Message attributes allow you to provide structured metadata items (such as timestamps, geospatial data, signatures, and identifiers) about the message. Message attributes are optional and separate from, but sent along with, the message body. The receiver of the message can use this information to help decide how to handle the message without having to process the message body first. Each message can have up to 10 attributes. To specify message attributes, you can use the AWS Management Console, AWS Software Development Kits (SDKs), or a query API.
When your application queries the Amazon SQS queue for messages, it calls the function ReceiveMessage. ReceiveMessage will check for the existence of a message in the queue and return immediately, either with or without a message. If your code makes periodic calls to the queue, this pattern is sufficient.
If your SQS client is just a loop that repeatedly checks for new messages, however, then this pattern becomes problematic, as the constant calls to ReceiveMessage burn CPU cycles and tie up a thread.
In this situation, you will want to use long polling. With long polling, you send a WaitTimeSeconds argument to ReceiveMessage of up to 20 seconds. If there is no message in the queue, then the call will wait up to WaitTimeSeconds for a message to appear before returning. If a message appears before the time expires, the call will return the message right away. Long polling drastically reduces the amount of load on your client.
Dead Letter Queues
Amazon SQS provides support for dead letter queues. A dead letter queue is a queue that other (source) queues can target to send messages that for some reason could not be successfully processed. A primary benefit of using a dead letter queue is the ability to sideline and isolate the unsuccessfully processed messages.
You can then analyze any messages sent to the dead letter queue to try to determine the cause of failure.
Messages can be sent to and received from a dead letter queue, just like any other Amazon SQS queue. You can create a dead letter queue from the Amazon SQS API and the Amazon SQS console.
While IAM can be used to control the interactions of different AWS identities with queues, there are often times when you will want to expose queues to other accounts. These situations may include:
- You want to grant another AWS account a particular type of access to your queue (for example, SendMessage).
- You want to grant another AWS account access to your queue for a specific period of time.
- You want to grant another AWS account access to your queue only if the requests come from your Amazon EC2 instances.
- You want to deny another AWS account access to your queue.
While close coordination between accounts may allow these types of actions through the use of IAM roles,that level of coordination is frequently unfeasible.
Amazon SQS Access Control allows you to assign policies to queues that grant specific interactions to other accounts without that account having to assume IAM roles from your account. These policies are written in the same JSON language as IAM. For example, the following sample policy gives the developer with AWS account number 111122223333 the SendMessage permission for the queue named 444455556666/queue1 in the US East (N. Virginia) region.
Amazon Simple Workflow Service (Amazon SWF)
Amazon SWF makes it easy to build applications that coordinate work across distributed components. In Amazon SWF, a task represents a logical unit of work that is performed by a component of your application. Coordinating tasks across the application involves managing inter-task dependencies, scheduling, and concurrency in accordance with the logical flow of the application.
When using Amazon SWF, you implement workers to perform tasks. These workers can run either on cloud infrastructure, such as Amazon EC2, or on your own premises.
Amazon SWF stores tasks, assigns them to workers when they are ready, monitors their progress, and maintains their state, including details on their completion. To coordinate tasks, you write a program that gets the latest state of each task from Amazon SWF and uses it to initiate subsequent tasks.
Using Amazon SWF, you can implement distributed, asynchronous applications as workflows. Workflows coordinate and manage the execution of activities that can be run asynchronously across multiple computing devices and that can feature both sequential and parallel processing.
When designing a workflow, analyze your application to identify its component tasks, which are represented in Amazon SWF as activities. The workflow’s coordination logic determines the order in which activities are executed.
Domains provide a way of scoping Amazon SWF resources within your AWS account. You must specify a domain for all the components of a workflow, such as the workflow type and activity types. It is possible to have more than one workflow in a domain; however, workflows in different domains cannot interact with one another.
The workflow history is a detailed, complete, and consistent record of every event that occurred since the workflow execution started. An event represents a discrete change in your workflow execution’s state, such as scheduled and completed activities, task timeouts, and signals.
Amazon SWF consists of a number of different types of programmatic features known as actors. Actors can be workflow starters, deciders, or activity workers. These actors communicate with Amazon SWF through its API. You can develop actors in any programming language.
A workflow starter is any application that can initiate workflow executions. For example, one workflow starter could be an e-commerce website where a customer places an order. Another workflow starter could be a mobile application where a customer orders takeout food or requests a taxi.
Activities within a workflow can run sequentially, in parallel, synchronously, or asynchronously. The logic that coordinates the tasks in a workflow is called the decider. The decider schedules the activity tasks and provides input data to the activity workers. The decider also processes events that arrive while the workflow is in progress and closes the workflow when the objective has been completed.
An activity worker is a single computer process (or thread) that performs the activity tasks in your workflow. Different types of activity workers process tasks of different activity types, and multiple activity workers can process the same type of task. When an activity worker is ready to process a new activity task, it polls Amazon SWF for tasks that are appropriate for that activity worker. After receiving a task, the activity worker processes the task to completion and then returns the status and result to Amazon SWF. The activity worker then polls for a new task.
Amazon SWF provides activity workers and deciders with work assignments, given as one of three types of tasks: activity tasks, AWS Lambda tasks, and decision tasks.
An activity task tells an activity worker to perform its function, such as to check inventory or charge a credit card. The activity task contains all the information that the activity worker needs to perform its function.
An AWS Lambda task is similar to an activity task, but executes an AWS Lambda function instead of a traditional Amazon SWF activity. For more information about how to define an AWS Lambda task, see the AWS documentation on AWS Lambda tasks.
A decision task tells a decider that the state of the workflow execution has changed so that the decider can determine the next activity that needs to be performed. The decision task contains the current workflow history.
Task lists provide a way of organizing the various tasks associated with a workflow. You could think of task lists as similar to dynamic queues. When a task is scheduled in Amazon SWF, you can specify a queue (task list) to put it in. Similarly, when you poll Amazon SWF for a task, you determine which queue (task list) to get the task from.
Task lists provide a flexible mechanism to route tasks to workers as your use case necessitates. Task lists are dynamic in that you don’t need to register a task list or explicitly create it through an action—simply scheduling a task creates the task list if it doesn’t already exist.
Deciders and activity workers communicate with Amazon SWF using long polling. The decider or activity worker periodically initiates communication with Amazon SWF, notifying Amazon SWF of its availability to accept a task, and then specifies a task list to get tasks from. Long polling works well for high-volume task processing. Deciders and activity workers can manage their own capacity.
Amazon SWF objects are uniquely identified by workflow type, activity type, decision and activity tasks, and workflow execution:
A registered workflow type is identified by its domain, name, and version. Workflow types are specified in the call to RegisterWorkflowType.
A registered activity type is identified by its domain, name, and version. Activity types are specified in the call to RegisterActivityType.
Each decision task and activity task is identified by a unique task token. The task token is generated by Amazon SWF and is returned with other information about the task in the response from PollForDecisionTask or PollForActivityTask. Although the token is most commonly used by the process that received the task, that process could pass the token to another process, which could then report the completion or failure of the task.
A single execution of a workflow is identified by the domain, workflow ID, and run ID. The first two are parameters that are passed to StartWorkflowExecution. The run ID is returned by StartWorkflowExecution.
Workflow Execution Closure
After you start a workflow execution, it is open. An open workflow execution can be closed as completed, canceled, failed, or timed out. It can also be continued as a new execution, or it can be terminated. The decider, the person administering the workflow, or Amazon SWF can close a workflow execution.
Lifecycle of a Workflow Execution
From the start of a workflow execution to its completion, Amazon SWF interacts with actors by assigning them appropriate tasks: either activity tasks or decision tasks.
Amazon SWF workflow illustration
- A workflow starter calls an Amazon SWF action to start the workflow execution for an order, providing order information.
- Amazon SWF receives the start workflow execution request and then schedules the first decision task.
- The decider receives the task from Amazon SWF, reviews the history, and applies the coordination logic to determine that no previous activities occurred. It then makes a decision to schedule the Verify Order activity with the information the activity worker needs to process the task and returns the decision to Amazon SWF.
- Amazon SWF receives the decision, schedules the Verify Order activity task, and waits for the activity task to complete or time out.
- An activity worker that can perform the Verify Order activity receives the task, performs it, and returns the results to Amazon SWF.
- Amazon SWF receives the results of the Verify Order activity, adds them to the workflow history, and schedules a decision task.
- The decider receives the task from Amazon SWF, reviews the history, applies the coordination logic, makes a decision to schedule a Charge Credit Card activity task with information the activity worker needs to process the task, and returns the decision to Amazon SWF.
- Amazon SWF receives the decision, schedules the Charge Credit Card activity task, and waits for it to complete or time out.
- An activity worker activity receives the Charge Credit Card task, performs it, and returns the results to Amazon SWF.
- Amazon SWF receives the results of the Charge Credit Card activity task, adds them to the workflow history, and schedules a decision task.
- The decider receives the task from Amazon SWF, reviews the history, applies the coordination logic, makes a decision to schedule a Ship Order activity task with the information the activity worker needs to perform the task, and returns the decision to Amazon SWF.
- Amazon SWF receives the decision, schedules a Ship Order activity task, and waits for it to complete or time out.
- An activity worker that can perform the Ship Order activity receives the task, performs it, and returns the results to Amazon SWF.
- Amazon SWF receives the results of the Ship Order activity task, adds them to the workflow history, and schedules a decision task.
- The decider receives the task from Amazon SWF, reviews the history, applies the coordination logic, makes a decision to schedule a Record Completion activity task with the information the activity worker needs, performs the task, and returns the decision to Amazon SWF.
- Amazon SWF receives the decision, schedules a Record Completion activity task, and waits for it to complete or time out.
- An activity worker Record Completion receives the task, performs it, and returns the results to Amazon SWF.
- Amazon SWF receives the results of the Record Completion activity task, adds them to the workflow history, and schedules a decision task.
- The decider receives the task from Amazon SWF, reviews the history, applies the coordination logic, makes a decision to close the workflow execution, and returns the decision along with any results to Amazon SWF.
- Amazon SWF closes the workflow execution and archives the history for future reference.