Amazon Simple Queue Service

Last updated
Amazon Simple Queue Service
Developer(s) Amazon.com
License Proprietary software
Website aws.amazon.com/sqs/

Amazon Simple Queue Service (Amazon SQS) is a distributed message queuing service introduced by Amazon.com as a beta in late 2004, and generally available in mid 2006. [1] [2] It supports programmatic sending of messages via web service applications as a way to communicate over the Internet. SQS is intended to provide a highly scalable hosted message queue that resolves issues arising from the common producer–consumer problem or connectivity between producer and consumer.

Contents

Amazon SQS can be described as commoditization of the messaging service. Well-known examples of messaging service technologies include IBM WebSphere MQ and Microsoft Message Queuing. Unlike these technologies, users do not need to maintain their own server. Amazon does it for them and sells the SQS service at a per-use rate.

API

Amazon provides SDKs in several programming languages including Java, Ruby, Python, .NET, PHP, Go and JavaScript. A Java Message Service (JMS) 1.1 client for Amazon SQS was released in December 2014.

Authentication

Amazon SQS provides authentication procedures to allow for secure handling of data. Amazon uses its Amazon Web Services (AWS) identification to do this, requiring users to have an AWS enabled account with Amazon.com. AWS assigns a pair of related identifiers, your AWS access keys, to an AWS enabled account to perform identification. The first identifier is a public 20-character Access Key. This key is included in an AWS service request to identify the user. If the user is not using SOAP with WS-Security, a digital signature is calculated using the Secret Access Key. The Secret Access Key is a 40-character private identifier. AWS uses the Access Key ID provided in a service request to look up an account's Secret Access Key. Amazon.com then calculates a digital signature with the key. If they match then the user is considered authentic, if not then the authentication fails and the request is not processed.

Message delivery

Amazon SQS guarantees at-least-once delivery. Messages are stored on multiple servers for redundancy and to ensure availability. If a message is delivered while a server is not available, it may not be removed from that server's queue and may be resent. As of 2007, Amazon SQS does not guarantee that the recipient will receive the messages in the order they were sent by the sender. If message ordering is important, it is required that the application place sequencing information within the messages to allow for reordering after delivery.

Messages can be of any type, and the data contained within is not restricted. Message bodies were initially limited to 8KB in size but was later raised to 64KB on 2010-07-01 [3] and then 256KB on 2013-06-18. [4] For larger messages, the user has a few options to get around this limitation. A large message can be split into multiple segments that are sent separately, or the message data can be stored using Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB with just a pointer to the data transmitted in the SQS message. Amazon has made an Extended Client Library available for this purpose. [5]

The service supports both unlimited queues and message traffic.

Message deletion

SQS does not automatically delete messages once they are sent. When a message is delivered, a receipt handle is generated for that delivery and sent to the recipient. These receipts are not sent with the message but in addition to it. SQS requires the recipient to provide the receipt in order to delete a message. This feature is new as of 2008 where only the message ID was required for message deletion. Because the system is distributed, a message may be sent more than once. In this case, the most recent receipt handle is needed to delete the message. Furthermore, the receipt handle may have other validity constraints; for instance, the receipt handle may only be valid during the visibility timeout (see below).

Once a message is delivered, it has a visibility timeout to prevent other components from consuming it. The "clock" for the visibility timeout starts once a message is sent, the default time being 30 seconds. If the queue is not told to delete the message during this time, the message becomes visible again and will be present.

Each queue also consists of a retention parameter defaulting to 4 days. Any message residing in the queue for longer will be purged automatically. The retention can be modified from 1 minute up to 14 days by the user. If the retention is changed while messages are already in the queue, any message that has been in the queue for longer than the new retention will be purged.

Notable usage

Dropbox, Netflix [6] and Nextdoor [7] are examples of companies that use SQS extensively. SQS is also widely used within Amazon.com.

See also

Related Research Articles

The Simple Mail Transfer Protocol (SMTP) is an Internet standard communication protocol for electronic mail transmission. Mail servers and other message transfer agents use SMTP to send and receive mail messages. User-level email clients typically use SMTP only for sending messages to a mail server for relaying, and typically submit outgoing email to the mail server on port 587 or 465 per RFC 8314. For retrieving messages, IMAP is standard, but proprietary servers also often implement proprietary protocols, e.g., Exchange ActiveSync.

Jakarta Enterprise Beans is one of several Java APIs for modular construction of enterprise software. EJB is a server-side software component that encapsulates business logic of an application. An EJB web container provides a runtime environment for web related software components, including computer security, Java servlet lifecycle management, transaction processing, and other web services. The EJB specification is a subset of the Java EE specification.

The Jakarta Messaging API is a Java application programming interface (API) for message-oriented middleware. It provides generic messaging models, able to handle the producer–consumer problem, that can be used to facilitate the sending and receiving of messages between software systems. Jakarta Messaging is a part of Jakarta EE and was originally defined by a specification developed at Sun Microsystems before being guided by the Java Community Process.

<span class="mw-page-title-main">Email client</span> Computer program used to access and manage a users email

An email client, email reader or, more formally, message user agent (MUA) or mail user agent is a computer program used to access and manage a user's email.

<span class="mw-page-title-main">News server</span> Type of server software

A news server is a collection of software used to handle Usenet articles. It may also refer to a computer itself which is primarily or solely used for handling Usenet. Access to Usenet is only available through news server providers.

In computer science, message queues and mailboxes are software-engineering components typically used for inter-process communication (IPC), or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content. Group communication systems provide similar kinds of functionality.

Memcached is a general-purpose distributed memory-caching system. It is often used to speed up dynamic database-driven websites by caching data and objects in RAM to reduce the number of times an external data source must be read. Memcached is free and open-source software, licensed under the Revised BSD license. Memcached runs on Unix-like operating systems and on Microsoft Windows. It depends on the libevent library.

Microsoft Message Queuing (MSMQ) is a message queue implementation developed by Microsoft and deployed in its Windows Server operating systems since Windows NT 4 and Windows 95. Windows Server 2016 and Windows 10 also includes this component. In addition to its mainstream server platform support, MSMQ has been incorporated into Microsoft Embedded platforms since 1999 and the release of Windows CE 3.0.

Representational state transfer (REST) is a software architectural style that describes a uniform interface between physically separate components, often across the Internet in a client-server architecture. REST defines four interface constraints:

IBM MQ is a family of message-oriented middleware products that IBM launched in December 1993. It was originally called MQSeries, and was renamed WebSphere MQ in 2002 to join the suite of WebSphere products. In April 2014, it was renamed IBM MQ. The products that are included in the MQ family are IBM MQ, IBM MQ Advanced, IBM MQ Appliance, IBM MQ for z/OS, and IBM MQ on IBM Cloud. IBM MQ also has containerised deployment options.

Push technology or server push is a style of Internet-based communication where the request for a given transaction is initiated by the publisher or central server. It is contrasted with pull/get, where the request for the transmission of information is initiated by the receiver or client.

Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services (AWS) that provides object storage through a web service interface. Amazon S3 uses the same scalable storage infrastructure that Amazon.com uses to run its e-commerce network. Amazon S3 can store any type of object, which allows uses like storage for Internet applications, backups, disaster recovery, data archives, data lakes for analytics, and hybrid cloud storage.

The Sun Java System Calendar Server was Sun's calendar (scheduling) server. The currently available version is 6.3 which is now part of the Sun Java System Communications Suite.

A message queueing service is a message-oriented middleware or MOM deployed in a compute cloud using software as a service model. Service subscribers access queues and or topics to exchange data using point-to-point or publish and subscribe patterns.

<span class="mw-page-title-main">Elasticsearch</span> Search engine

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is dual-licensed under the source-available Server Side Public License and the Elastic license, while other parts fall under the proprietary (source-available) Elastic License. Official clients are available in Java, .NET (C#), PHP, Python, Ruby and many other languages. According to the DB-Engines ranking, Elasticsearch is the most popular enterprise search engine.

Amazon Route 53 is a scalable and highly available Domain Name System (DNS) service. Released on December 5, 2010, it is part of Amazon.com's cloud computing platform, Amazon Web Services (AWS). The name is a possible reference to U.S. Routes, and "53" is a reference to the TCP/UDP port 53, where DNS server requests are addressed. In addition to being able to route users to various AWS services, including EC2 instances, Route 53 also enables AWS customers to route users to non-AWS infrastructure and to monitor the health of their application and its endpoints. Route 53's servers are distributed throughout the world. Amazon Route 53 supports full, end-to-end DNS resolution over IPv6. Recursive DNS resolvers on IPv6 networks can use either IPv4 or IPv6 transport to send DNS queries to Amazon Route 53.

<span class="mw-page-title-main">Amazon DynamoDB</span> NoSQL database service

Amazon DynamoDB is a fully managed proprietary NoSQL database service that supports key–value and document data structures and is offered by Amazon.com as part of the Amazon Web Services portfolio. DynamoDB exposes a similar data model to and derives its name from Dynamo, but has a different underlying implementation. Dynamo had a multi-leader design requiring the client to resolve version conflicts and DynamoDB uses synchronous replication across multiple data centers for high durability and availability. DynamoDB was announced by Amazon CTO Werner Vogels on January 18, 2012, and is presented as an evolution of Amazon SimpleDB.

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems via Kafka Connect, and provides the Kafka Streams libraries for stream processing applications. Kafka uses a binary TCP-based protocol that is optimized for efficiency and relies on a "message set" abstraction that naturally groups messages together to reduce the overhead of the network roundtrip. This "leads to larger network packets, larger sequential disk operations, contiguous memory blocks [...] which allows Kafka to turn a bursty stream of random message writes into linear writes."

Enduro/X is an open-source middleware platform for distributed transaction processing. It is built on proven APIs such as X/Open group's XATMI and XA. The platform is designed for building real-time microservices based applications with a clusterization option. Enduro/X functions as an extended drop-in replacement for Oracle Tuxedo. The platform uses in-memory POSIX Kernel queues which insures high interprocess communication throughput.

Firebase Cloud Messaging (FCM), formerly known as Google Cloud Messaging (GCM), is a cross-platform cloud solution for messages and notifications for Android, iOS, and web applications, which as of June 2022 can be used at no cost. Firebase Cloud Messaging allows third-party application developers to send notifications or messages from servers hosted by FCM to users of the platform or end users.

References

  1. "Amazon Simple Queue Service Released". Amazon Web Services. 2006-07-13. Retrieved 2021-10-29.
  2. Barr, Jeff (2014-08-19). "My First 12 Years at Amazon.com". jeff-barr.com. Retrieved 2021-01-11.
  3. "Amazon SQS introduces Free Tier and adds Support for Larger Messages and Longer Retention". aws.amazon.com. 2010-07-01. Retrieved 2021-01-11.
  4. "Amazon SQS and SNS Announce 256KB Large Payloads". aws.amazon.com. 2013-06-18. Retrieved 2021-01-11.
  5. An extension to the Amazon SQS client that enables sending and receiving messages up to 2GB via Amazon S3. on GitHub
  6. Granqvist, Hans (2011-04-18). ""More Like This…" Building a network of similarity". Netflix Tech Blog. Archived from the original on 2016-11-28.
  7. Fang, Wenbin (August 13, 2014). "Nextdoor Taskworker: Simple, Efficient & Scalable". Nextdoor Engineering.