Advanced Message Queuing Protocol

Last updated

Advanced Message Queuing Protocol
Communication protocol
AbbreviationAMQP
Purpose Message-oriented middleware
Developer(s)AMQP working group (OASIS)
Introduction2003;21 years ago (2003)
OSI layer application layer (Layer 7)
Website www.amqp.org

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security. [1]

Contents

AMQP mandates the behavior of the messaging provider and client to the extent that implementations from different vendors are interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous standardizations of middleware have happened at the API level (e.g. JMS) and were focused on standardizing programmer interaction with different middleware implementations, rather than on providing interoperability between multiple implementations. [2] Unlike JMS, which defines an API and a set of behaviors that a messaging implementation must provide, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of bytes. Consequently, any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.

Overview

AMQP is a binary application layer protocol, designed to efficiently support a wide variety of messaging applications and communication patterns. It provides flow controlled, [3] message-oriented communication with message-delivery guarantees such as at-most-once (where each message is delivered once or never), at-least-once (where each message is certain to be delivered, but may do so multiple times) and exactly-once (where the message will always certainly arrive and do so only once), [4] and authentication and/or encryption based on SASL and/or TLS. [5] It assumes an underlying reliable transport layer protocol such as Transmission Control Protocol (TCP). [6]

The AMQP specification is defined in several layers: (i) a type system, (ii) a symmetric, asynchronous protocol for the transfer of messages from one process to another, (iii) a standard, extensible message format and (iv) a set of standardised but extensible 'messaging capabilities.'

History

AMQP was originated in 2003 by John O'Hara at JPMorgan Chase in London. [1] [7] AMQP was conceived as a co-operative open effort. The initial design was by JPMorgan Chase from mid-2004 to mid-2006 and it contracted iMatix Corporation to develop a C broker and protocol documentation. In 2005 JPMorgan Chase approached other firms to form a working group that included Cisco Systems, IONA Technologies, iMatix, Red Hat, and Transaction Workflow Innovation Standards Team (TWIST). In the same year JPMorgan Chase partnered with Red Hat to create Apache Qpid, initially in Java and soon after C++. Independently, RabbitMQ was developed in Erlang by Rabbit Technologies, followed later by the Microsoft and StormMQ implementations.

The working group grew to 23 companies including Bank of America, Barclays, Cisco Systems, Credit Suisse, Deutsche Börse, Goldman Sachs, HCL Technologies Ltd, Progress Software, IIT Software, INETCO Systems Limited, Informatica (including 29 West), JPMorgan Chase, Microsoft Corporation, my-Channels, Novell, Red Hat, Software AG, Solace Systems, StormMQ, Tervela Inc., TWIST Process Innovations ltd, VMware (which acquired Rabbit Technologies) and WSO2.

In 2008, Pieter Hintjens, CEO and chief software designer of iMatix, wrote an article called "What is wrong with AMQP (and how to fix it)" [8] and distributed it to the working group to alert of imminent failure, identify problems seen by iMatix and propose ways to fix the AMQP specification. By then, iMatix had already started work on ZeroMQ. In 2010, Hintjens announced that iMatix would leave the AMQP workgroup and did not plan to support AMQP/1.0 in favor of the significantly simpler and faster ZeroMQ. [9]

In August 2011, the AMQP working group announced its reorganization into an OASIS member section. [10]

AMQP 1.0 was released by the AMQP working group on 30 October 2011, at a conference in New York. At the event Microsoft, Red Hat, VMware, Apache, INETCO and IIT Software demonstrated software running the protocol in an interoperability demonstration. The next day, on 1 November 2011, the formation of an OASIS Technical Committee was announced [11] to advance this contributed AMQP version 1.0 through the international open standards process. The first draft from OASIS was released in February 2012, [12] the changes as compared to that published by the Working Group being restricted to edits for improved clarity (no functional changes). The second draft was released for public review on 20 June (again with no functional changes), [13] and AMQP was approved as an OASIS standard on 31 October 2012. [14]

OASIS AMQP was approved for release as an ISO and IEC International Standard in April 2014. [15] AMQP 1.0 was balloted through the Joint Technical Committee on Information Technology (JTC1) of the International Standards Organization (ISO) and the International Electrotechnical Commission (IEC). The approved OASIS AMQP submission has been given the designation, ISO/IEC 19464.

Previous versions of AMQP were 0-8, published in June 2006, 0-9, published in December 2006, 0-10 published in February 2008 [16] and 0-9-1, published in November 2008. These earlier releases are significantly different from the 1.0 specification. [17] [18]

Whilst AMQP originated in the financial services industry, it has general applicability to a broad range of middleware problems.

Description of AMQP 1.0

Type system

AMQP defines a self-describing encoding scheme allowing interoperable representation of a wide range of commonly used types. It also allows typed data to be annotated with additional meaning, [19] for example a particular string value might be annotated so that it could be understood as a URL. Likewise a map value containing key-value pairs for 'name', 'address' etc., might be annotated as being a representation of a 'customer' type.

The type-system is used to define a message format allowing standard and extended meta-data to be expressed and understood by processing entities. It is also used to define the communication primitives through which messages are exchanged between such entities, i.e. the AMQP frame bodies.

The basic unit of data in AMQP is a frame. There are nine AMQP frame bodies defined that are used to initiate, control and tear down the transfer of messages between two peers. These are:

The link protocol is at the heart of AMQP.

An attach frame body is sent to initiate a new link; a detach to tear down a link. Links may be established in order to receive or send messages.

Messages are sent over an established link using the transfer frame. Messages on a link flow in only one direction.

Transfers are subject to a credit-based flow control scheme, managed using flow frames. This allows a process to protect itself from being overwhelmed by too large a volume of messages or more simply to allow a subscribing link to pull messages as and when desired. [20]

Each transferred message must eventually be settled. Settlement ensures that the sender and receiver agree on the state of the transfer, providing reliability guarantees. Changes in state and settlement for a transfer (or set of transfers) are communicated between the peers using the disposition frame. Various reliability guarantees can be enforced this way: at-most-once, at-least-once and exactly-once. [21]

Multiple links, in both directions, can be grouped together in a session. A session is a bidirectional, sequential conversation between two peers that is initiated with a begin frame and terminated with an end frame. A connection between two peers can have multiple sessions multiplexed over it, each logically independent. Connections are initiated with an open frame in which the sending peer's capabilities are expressed, and terminated with a close frame.

Message format

AMQP defines as the bare message, that part of the message that is created by the sending application. This is considered immutable as the message is transferred between one or more processes.

Ensuring the message as sent by the application is immutable allows for end-to-end message signing and/or encryption and ensures that any integrity checks (e.g. hashes or digests) remain valid. The message can be annotated by intermediaries during transit, but any such annotations are kept distinct from the immutable bare message. Annotations may be added before or after the bare message.

The header is a standard set of delivery-related annotations that can be requested or indicated for a message and includes time to live, durability, priority. [22]

The bare message itself is structured as an optional list of standard properties (message id, user id, creation time, reply to, subject, correlation id, group id etc.), an optional list of application-specific properties (i.e., extended properties) and a body, which AMQP refers to as application data. [23]

Properties are specified in the AMQP type system, as are annotations. The application data can be of any form, and in any encoding the application chooses. One option is to use the AMQP type system to send structured, self-describing data.

Messaging capabilities

The link protocol transfers messages between two nodes but assumes very little as to what those nodes are or how they are implemented.

A key category is those nodes used as a rendezvous point between senders and receivers of messages (e.g. queues or topics). The AMQP specification calls such nodes distribution nodes and codifies some common behaviors. [24]

This includes:

Though AMQP can be used in simple peer-to-peer systems, defining this framework for messaging capabilities additionally enables interoperability with messaging intermediaries (brokers, bridges etc.) in larger, richer messaging networks. The framework specified covers basic behaviors but allows for extensions to evolve that can be further codified and standardised.

Implementations

AMQP 1.0 broker implementations

Pre-1.0 AMQP broker implementations

Specification

AMQP protocol version 1.0 is the current specification version. It focuses on core features which are necessary for interoperability at Internet scale. It contains less explicit routing than previous versions because core functionality is the first to be rigorously standardized. AMQP 1.0 interoperability has been more extensively tested with more implementors than prior versions. [37]

The AMQP website contains the OASIS specification for version 1.0.

Earlier versions of AMQP, published prior to the release of 1.0 (see History above) and significantly different from it, include:

Comparable specifications

These open protocol specifications that cover the same or similar space as AMQP:

Java Message Service (JMS), is often compared to AMQP. However, JMS is an API specification (part of the Java EE specification) that defines how message producers and consumers are implemented. JMS does not guarantee interoperability between implementations, and the JMS-compliant messaging system in use may need to be deployed on both client and server. On the other hand, AMQP is a wire-level protocol specification. In theory AMQP provides interoperability as different AMQP-compliant software can be deployed on the client and server sides.

See also

Related Research Articles

The Jakarta Messaging API is a Java application programming interface (API) for message-oriented middleware. It provides generic messaging models, able to handle the producer–consumer problem, that can be used to facilitate the sending and receiving of messages between software systems. Jakarta Messaging is a part of Jakarta EE and was originally defined by a specification developed at Sun Microsystems before being guided by the Java Community Process.

The Organization for the Advancement of Structured Information Standards is a nonprofit consortium that works on the development, convergence, and adoption of projects - both open standards and open source - for Computer security, blockchain, Internet of things (IoT), emergency management, cloud computing, legal data exchange, energy, content technologies, and other areas.

In computer science, message queues and mailboxes are software-engineering components typically used for inter-process communication (IPC), or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content. Group communication systems provide similar kinds of functionality.

Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. MOM allows application modules to be distributed over heterogeneous platforms and reduces the complexity of developing applications that span multiple operating systems and network protocols. The middleware creates a distributed communications layer that insulates the application developer from the details of the various operating systems and network interfaces. APIs that extend across diverse platforms and networks are typically provided by MOM.

Web Services Security is an extension to SOAP to apply security to Web services. It is a member of the Web service specifications and was published by OASIS.

Microsoft Message Queuing (MSMQ) is a message queue implementation developed by Microsoft and deployed in its Windows Server operating systems since Windows NT 4 and Windows 95. Windows Server 2016 and Windows 10 also includes this component. In addition to its mainstream server platform support, MSMQ has been incorporated into Microsoft Embedded platforms since 1999 and the release of Windows CE 3.0.

Mule is a lightweight enterprise service bus (ESB) and integration framework provided by MuleSoft. It has a Java-based platform and can also act as broker for interactions between other platforms such as .NET using web services or sockets.

OpenJMS is one of the major providers of Java Message Service API, which implements Sun Microsystems' Java Message Service API 1.1 Specification. The current release is OpenJMS 0.7.7-beta-1.

<span class="mw-page-title-main">Pieter Hintjens</span> Belgian CEO, software developer, author

Pieter Hintjens was a Belgian software developer, author, and past president of the Foundation for a Free Information Infrastructure (FFII), an association that fights against software patents. In 2007, he was nominated one of the "50 most influential people in IP" by Managing Intellectual Property magazine.

Simple Text Oriented Message Protocol (STOMP), formerly known as TTMP, is a simple text-based protocol, designed for working with message-oriented middleware (MOM). It provides an interoperable wire format that allows STOMP clients to talk with any message broker supporting the protocol.

A message queueing service is a message-oriented middleware or MOM deployed in a compute cloud using software as a service model. Service subscribers access queues and or topics to exchange data using point-to-point or publish and subscribe patterns.

<span class="mw-page-title-main">Apache Qpid</span> Open-source messaging system

Apache Qpid is an open-source messaging system which implements the Advanced Message Queuing Protocol (AMQP). It provides transaction management, queuing, distribution, security, management, clustering, federation and heterogeneous multi-platform support. The Apache Qpid API supports multiple programming languages and comes with both C++ and Java brokers.

HornetQ is an open-source asynchronous messaging project from JBoss. It is an example of Message-oriented middleware. HornetQ is an open source project to build a multi-protocol, embeddable, very high performance, clustered, asynchronous messaging system. During much of its development, the HornetQ code base was developed under the name JBoss Messaging 2.0.

RabbitMQ is an open-source message-broker software that originally implemented the Advanced Message Queuing Protocol (AMQP) and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQ Telemetry Transport (MQTT), and other protocols.

ZeroMQ is an asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker; the zero in the name is for zero broker. The library's API is designed to resemble Berkeley sockets.

<span class="mw-page-title-main">Apache ActiveMQ</span> Software message broker

Apache ActiveMQ is an open source message broker written in Java together with a full Java Message Service (JMS) client. It provides "Enterprise Features" which in this case means fostering the communication from more than one client or server. Supported clients include Java via JMS 1.1 as well as several other "cross language" clients. The communication is managed with features such as computer clustering and ability to use any database as a JMS persistence provider besides virtual memory, cache, and journal persistency.

MQTT is a lightweight, publish-subscribe, machine to machine network protocol for message queue/message queuing service. It is designed for connections with remote locations that have devices with resource constraints or limited network bandwidth, such as in the Internet of Things (IoT). It must run over a transport protocol that provides ordered, lossless, bi-directional connections—typically, TCP/IP, but also possibly over QUIC. It is an open OASIS standard and an ISO recommendation.

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems via Kafka Connect, and provides the Kafka Streams libraries for stream processing applications. Kafka uses a binary TCP-based protocol that is optimized for efficiency and relies on a "message set" abstraction that naturally groups messages together to reduce the overhead of the network roundtrip. This "leads to larger network packets, larger sequential disk operations, contiguous memory blocks [...] which allows Kafka to turn a bursty stream of random message writes into linear writes."

Enduro/X is an open-source middleware platform for distributed transaction processing. It is built on proven APIs such as X/Open group's XATMI and XA. The platform is designed for building real-time microservices based applications with a clusterization option. Enduro/X functions as an extended drop-in replacement for Oracle Tuxedo. The platform uses in-memory POSIX Kernel queues which insures high interprocess communication throughput.

<span class="mw-page-title-main">Apache RocketMQ</span> Open-source stream processing platform

RocketMQ is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project.

References

  1. 1 2 O'Hara, J. (2007). "Toward a commodity enterprise middleware". ACM Queue. 5 (4): 48–55. doi: 10.1145/1255421.1255424 .
  2. ↑ Vinoski, S. (2006). "Advanced Message Queuing Protocol" (PDF). IEEE Internet Computing. 10 (6): 87–89. doi:10.1109/MIC.2006.116. S2CID   14211884.
  3. ↑ "OASIS AMQP version 1.0, sections 2.6.7-2.6.8". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  4. ↑ "OASIS AMQP version 1.0, sections 2.6.12-2.6.13". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  5. ↑ "OASIS AMQP version 1.0, section 5.1". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  6. ↑ "OASIS AMQP version 1.0, section 2.1". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  7. ↑ Hintjens, Pieter (7 February 2006). "Background to the AMQ Project, Authors". iMatix Corporation. Retrieved 18 February 2018.
  8. ↑ Hintjens, Pieter. "What is wrong with AMQP (and how to fix it)". iMatix. Archived from the original on 6 September 2011. Retrieved 1 March 2024.
  9. ↑ Hintjens, Pieter (30 March 2010). "iMatix will end OpenAMQ support by 2011". openamq-dev mailing list. Archived from the original on 2 April 2010. Retrieved 1 March 2024.
  10. ↑ "AMQP Working Group Transitions to OASIS Member Section". Archived from the original on 16 April 2012. Retrieved 28 January 2012.
  11. ↑ OASIS Forms AMQP Technical Committee to Advance Business Messaging Interoperability Within Middleware, Mobile, and Cloud Environments
  12. ↑ "30-day Public Review for OASIS AMQP Version 1.0". 21 February 2012. Retrieved 18 June 2012.
  13. ↑ "15 day Public Review for OASIS AMQP Version 1.0". 20 June 2012. Retrieved 20 June 2012.
  14. ↑ "AMQP 1.0 Becomes OASIS Standard". 31 October 2012. Retrieved 1 November 2012.
  15. ↑ "Information technology -- Advanced Message Queuing Protocol (AMQP) v1.0 specification". ISO/IEC 19464. ISO. Retrieved 1 May 2014.
  16. ↑ "AMQP PMC Meeting Minutes 13th February, 2008". AMQP PMC Minutes. AMQP Working Group. Archived from the original on 30 November 2012. Retrieved 15 June 2012.
  17. ↑ "RabbitMQ - AMQP 0-8 to 0-9-1". www.rabbitmq.com. Retrieved 12 January 2018.
  18. ↑ rabbitmq-amqp1.0: AMQP 1.0 support for RabbitMQ, RabbitMQ, 17 December 2017, retrieved 12 January 2018
  19. ↑ "OASIS AMQP version 1.0, section 1.1". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  20. ↑ "OASIS AMQP version 1.0, section 2.6.7-2.6.9". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  21. ↑ "OASIS AMQP version 1.0, section 2.6.12-2.6.14". AMQP WorkingGroup. Retrieved 18 June 2012.
  22. ↑ "OASIS AMQP version 1.0, section 3.2.1". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  23. ↑ "OASIS AMQP version 1.0, section 3.2". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  24. ↑ "OASIS AMQP version 1.0, section 3.3". OASIS AMQP Technical Committee. Retrieved 15 June 2012.
  25. ↑ "OASIS AMQP version 1.0, section 3.4". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  26. ↑ "OASIS AMQP version 1.0, section 3.5.2". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  27. ↑ "OASIS AMQP version 1.0, section 3.5.3". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  28. ↑ "OASIS AMQP version 1.0, section 3.5.4". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  29. ↑ "OASIS AMQP version 1.0, section 3.5.1". OASIS AMQP Technical Committee. Retrieved 18 June 2012.
  30. 1 2 axisc. "AMQP 1.0 in Azure Service Bus and Event Hubs protocol guide". docs.microsoft.com. Retrieved 21 May 2019.
  31. ↑ ShubhaVijayasarathy. "What is Azure Event Hubs? - a Big Data ingestion service". docs.microsoft.com. Retrieved 21 May 2019.
  32. ↑ axisc. "Overview of AMQP 1.0 in Azure Service Bus". docs.microsoft.com. Retrieved 21 May 2019.
  33. ↑ "IBM MQ". www.ibm.com. Retrieved 15 March 2024.
  34. ↑ "About IBM MQ". www.ibm.com. Retrieved 15 March 2024.
  35. ↑ "Solace PubSub+". solace.com.
  36. ↑ Qpid 0.22 - Apache Qpid™
  37. ↑ "Status of Pre-1.0 Versions of AMQP". AMQP site. Retrieved 28 May 2014.
  38. ↑ "AMQP 0-9-1 Model Explained". VMWare, Inc. Retrieved 1 November 2012.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.