Enduro/X

Enduro/X
Developer(s)	Mavimax SIA
Stable release	8.0.10 / 26 November 2023;10 months ago
Repository	github.com/endurox-dev/endurox ;
Written in	C, C++, Go, Java
Operating system	Linux, FreeBSD, IBM AIX, Oracle Solaris, Cygwin, macOS
Type	Application server, Message-oriented middleware, Enterprise messaging system, SOA
License	AGPL v3 or Commercial, LGPL v3 for Java and Go use
Website

Last updated October 05, 2024

Enduro/X is an open-source middleware platform for distributed transaction processing. It is built on proven APIs such as X/Open group's XATMI and XA. The platform is designed for building real-time microservices based applications with a clusterization option. Enduro/X functions as an extended drop-in replacement for Oracle Tuxedo. The platform uses in-memory POSIX message queues which insures high inter-process communication throughput.

Services

The platform allows runtime system patching, without service interruption. The system monitors running processes, and restarts them after a crash or hang. System configuration can be dynamically reloaded and it provides a cluster facility, allowing two or more instances to be joined in peer-to-peer manner. Enduro/X is written in C and provides native APIs for C/C++. Mavimax provides bindings are provided for Go, Java programming languages. Community provides bindings for Python, PHP and Perl. Thus effectively Enduro/X may be used as language interoperability tool between supported programming languages.

Technology

Currently Enduro/X works on Linux kernels starting from 2.6.12. The most critical Linux feature is epoll() on POSIX queues. This Linux functionality provides the mechanism for one queue - multiple servers, where Enduro/X servers are polling over the queues. Enduro/X developers pushed the EPOLLEXCLUSIVE flag inclusion into Linux kernel mainline (starting from version 4.5).^[1] The patch improved the Enduro/X response in case if hundreds of executables are providing the same service. Particularly kernel is not doing thundering herd type behavior anymore by waking up all binaries waiting on queue. Now only single receiver of the service call is woken. In some workloads the Enduro/X applications gets significant boost. The test case for the patch increased the performance of the workload from the 860 seconds to 24 sec.^[2]

Starting with Enduro/X version 3.1.2 production grade support for FreeBSD, IBM AIX (versions 6.1 and 7.1) and Oracle Solaris v11 is added. Version 3.1.2 introduced experimental support for Apple macOS and Cygwin. Under Unix platforms where POSIX queues are available, they are employed so that for every ATMI server/service pair there is separate queue open and caller service chooses queue in round-robin mode. As for macOS there is no support for POSIX queues in operating system, Enduro/X uses emulated POSIX queues which are based on memory-mapped files in tandem with POSIX thread's process shared mutexes. For FreeBSD POSIX queues can be polled too, thus Enduro/X on FreeBSD works in the say way as Linux with one queue-multiple servers. The polling on FreeBSD is done with help of kqueue.

With version 6.0 System V queues are preferred IPC mechanisms for Unix platforms other than FreeBSD and Linux. The System V queues also gives single queue - multiple servers (SQMS) message distribution approach for effective load balancing. To load balance in SQMS mode with System V queues, XATMI server must be configured to use request address (logical channel/queue) on which all XATMI servers advertises same services. This Queue mode is available for Oracle Solaris and IBM AIX. For macOS still emulated POSIX queue are used, due to fact that it is not possible to adjust queue capacity on particular operating system.

Features

Standard-based APIs - SCA, The Open Group XATMI
Communication types - Synchronous, Asynchronous, Conversational, Publish/subscribe
Typed buffers
- UBF (Unified Buffer Format) which provides emulation of Tuxedo's FML/FML32 format. UBF if high-performance binary protocol buffer format. Buffer is indexed by binary search on fixed data types. As from version v8.0, support for recursive UBF buffers, PTR and VIEW fields has been added.
- STRING buffer format.
- CARRAY (byte array) buffer format.
- JSON buffer format, automatic conversion between JSON and UBF available.
- VIEW buffer (starting from version 5.0+). This offer C structure sending between processes in cross platform way. Also this allows to map UBF fields to VIEW fields, thus helping developer quicker to develop applications, by combining UBF and VIEW buffers.
Transaction Management - Global Transactions - Two-phase commit protocol - X/Open XA
- XA interface emulation for PostgreSQL
Clustering - on peer-to-peer basis
Event broker (also called publish and subscribe messaging)
System process monitoring and self-healing (pings and restarts)
SOA Service cache. XATMI services can be cached to LMDB database. Resulting that the next call to service from any local client receives results directly from cache (mainly from direct memory read).
Dynamic re-configuration
Custom server polling extensions
XATMI subsystem is able to work without main application server daemon (ndrxd)
Main application server daemon (ndrxd) can be restarted (if crashed). When started back it enters in learning mode for some period of time, in which in gathers information about system, what services are running, etc. After learning period, it starts to do normal operations
Available tpforward() call
ATMI server threads may become clients, and can do tpcall()
ATMI call timeout configuration via tptoutset() API call
Extensive logging & debugging. Enduro/X logging can be configured per binary with different log levels. As ATMI servers can be started outside of appserver, it is possible to debug them from programming IDE or with tools like Valgrind.
For quality assurance project uses automated unit testing and integration testing
Built in ATMI service profiling.
Environment variables can be updated for XATMI server processes without full application reboot.
Generic client process monitor (cpm). Subsystem allows to start/stop/monitor client executables. At client process crashes, cpm will start it back. Massive client administration feature allows easy definition of many similar executables (with sub-section ranges) and in the same way it is very easy to manage them all together by regex masks.
Starting with version 5.2 Enduro/X provides configuration data encryption feature, so that software which is built on top of Enduro/X may comply with Payment Card Industry Data Security Standard (PCI DSS).
Starting with version 6.0.17 for XATMI server and client processes max resident and virtual memory limit can be set. Thus when limit is reached, the processes are gracefully restarted. This particular feature is welcome in environments where vendor have delivered binaries with memory leaks, thus in production environments these limits may help temporarily cope with the situation until resolved by developer.
TMIB API for system information gathering.
NetXMS agent for monitoring client and server processes, queues, services, machine and cluster.
Full support of build tools (buildserver, buildclient, buildtms).^[3]
XATMI service dispatch threading.
Data Dependent Routing support for UBF buffers.
Automatic transactions for XATMI servers.
Service priority handling.
On the fly log setting changing and log rotation for any Enduro/X related program.
Latent Command Framework for custom real-time command publishing to processes.

Performance

Several benchmarks have been made with the Enduro/X middleware and it shows that it competes with Oracle Tuxedo and ZeroMQ quite well. Enduro/X was faster than Tuxedo 30% in 56 KB data block range.^[4] Enduro/X was significantly faster than ZeroMQ in 10-20KB range.^[5] Also for test performed with RabbitMQ Enduro/X showed its strength and was ahead of the competitor. For the RPC calls, Enduro/X was several times ahead, for send only calls, RabbitMQ was bit closer to Enduro/X.^[6]

Subsystems

Enduro/X contains different sub-modules which extends the application server and middleware functionality.

Queuing subsystem

Enduro/X provides a queuing subsystem called TMQ (Transactional Message Queue). This facility provides persistent queues that allows applications to explicitly enqueue and dequeue messages from named queues. Queues can be ordered by message enqueue time in LIFO or FIFO order. Queues are managed by an XA-compliant resource manager allowing queue operations to participate in distributed transactions. An automated queue forwarding feature is provided that will remove entries from a queue and invoke an associated Enduro/X ATMI services, placing the reply message on an associated reply queue and failed messages to failure queue.

Configuration services

Enduro/X offers applications to use INI file based configuration reading via special XATMI server, provided in package, named "cconfsrv". Basically via standard tpcall() user application is able to read configuration files (can be separate ini file or folder with them). Technique allows to use the sub-sections with key/value inheritance from main section.

Trace logging services

Enduro/X introduced new set of API function named tplog. This allows user to use the same C based high-performance debug logging facilities which Enduro/X uses internally. It is possible to redirect all logging by thread bases files. Also there are APIs for "request logging", which means that user and Enduro/X can do logging in separate files for each request (e.g. log by session id, log by username, etc.)

Enduro/X Connect package

Enduro/X Connect package provides generic way for exposing XATMI services for outer world. Thus service can be invoked as REST normal services. The package provides functions for converting message internal messages to JSON format and vice versa. Web service module is extensible and provides several options for data conversion between formats. Module additionally supports file uploading, static content serving and orchestration of global transactions over the web service interface.

Enduro/X Connect package also provides module for accessing outer world web services via internal XATMI service call

Third connectivity component is in Enduro/X Connect package is generic TCP/IP connector which provides bidirectional XATMI service API to the TCP/IP sockets. Such functionality simplifies application development for IoT and banking industries, where the data for different reasons needs to be sent in custom binary protocols, such as ISO 8583.^[7]

Application Server for Go (ASG)

ASG is bindings project on top of the Enduro/X. Authors considers these binding as application server for Go, because Golang projects are linked statically and by default there is no such thing as components. Enduro/X allows to run separate executable binaries, which represents stateless services. These services can be reloaded on the fly without service interruption. Also platform allows to run distributed transaction processing by using Golang. Developers provides modified Oracle DB OCI8 drivers for Go, which supports XA transactions. Enduro/X ASG provides first ever distributed transaction facility for Golang.

Enduro/X Java

Enduro/X version 7.0 offers binding for Java. Full features XATMI servers and client processes may be written Java. Framework is based on multi-processing principle, so that XATMI servers and clients have its own copy of JVM. Requirements for XATMI server/service classes are minimal, thus framework may be considered as POJO application server. XA two-phase commit can be used too with standard JDBC drivers. Transactions are managed by standard Enduro/X Core transaction manager to which JDBC drivers are loaded for transaction orchestration. Binding package also offer Java Linker process called exjld. It offers option to link Java objects (JAR files) to standard executable process with dependency to libjvm and libjava. Thus all jars may be packed in executable process, making Java software delivery simpler, now each process may contain all its dependencies, thus allowing different executables to use different versions of common jar dependencies.

As server process both java executable with classpath or linked Java binaries are supported, ndrxd daemon can boot both of them.

Enduro/X-PHP

Full client bindings are available for PHP. Client PHP processes might compile a loadable module or compile the Enduro/X process inside the PHP. When PHP process is running as client, it must be present in Enduro/X Application environment, that could be done when application is hosted under the same user where PHP app runs (Apache or so) or for the PHP user creating a micro environment which further connects to main application server via tpbridge process. Module is available here .

Enduro/X-Perl

Client and server bindings are available for Perl scripting language. Perl module is available here .

Acquisition

The Enduro/X was acquired from ATR Baltic, Ltd by Mavimax Ltd in February 2017 with full rights to the source code and current agreements.[8]

History

Enduro/X was developed as closed source project by ATR Baltic. The development begun in December 2009 and first production customers (mainly banks) where put on the system in late 2012, at the time Enduro/X reached version 1.0. The project was open sourced in 2015, with version v2.3.2. From that event, all the development is made publicly available. Constantly new features are added with every major version release. Till version 6.0 Enduro/X was available under GNU GPLv2 license, starting from Enduro/X 6.0, project is moved to GNU AGPLv3 license. Starting with version 7.0, license is eased for middleware usage from Java and Go programming languages. The statement is that AGPL used by Enduro/X does not affect the Java or Go code which uses Enduro/X. Thanks to these changes, now Enduro/X Java and Enduro/X Go (ASG) modules now are released under LGPL v3, thus Java and Go projects may use middleware commercially with any extra license, while the support is provided commercially.

Related Research Articles

The Jakarta Transactions, one of the Jakarta EE APIs, enables distributed transactions to be done across multiple X/Open XA resources in a Java environment. JTA was a specification developed under the Java Community Process as JSR 907. JTA provides for:

The Jakarta Messaging API is a Java application programming interface (API) for message-oriented middleware. It provides generic messaging models, able to handle the producer–consumer problem, that can be used to facilitate the sending and receiving of messages between software systems. Jakarta Messaging is a part of Jakarta EE and was originally defined by a specification developed at Sun Microsystems before being guided by the Java Community Process.

<span class="mw-page-title-main">Inter-process communication</span> How computer operating systems enable data sharing

In computer science, inter-process communication (IPC), also spelled interprocess communication, are the mechanisms provided by an operating system for processes to manage shared data. Typically, applications can use IPC, categorized as clients and servers, where the client requests data and the server responds to client requests. Many applications are both clients and servers, as commonly seen in distributed computing.

In computer science, message queues and mailboxes are software-engineering components typically used for inter-process communication (IPC), or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content. Group communication systems provide similar kinds of functionality.

Message-oriented middleware (MOM) is software or hardware infrastructure supporting sending and receiving messages between distributed systems. Message-oriented middleware is in contrast to streaming-oriented middleware where data is communicated as a sequence of bytes with no explicit message boundaries. Note that steaming protocols are almost always built above protocols using discrete messages such as frames Ethernet), datagrams UDP, packets IP, cells ATM, et al.

IBM MQ is a family of message-oriented middleware products that IBM launched in December 1993. It was originally called MQSeries, and was renamed WebSphere MQ in 2002 to join the suite of WebSphere products. In April 2014, it was renamed IBM MQ. The products that are included in the MQ family are IBM MQ, IBM MQ Advanced, IBM MQ Appliance, IBM MQ for z/OS, and IBM MQ on IBM Cloud. IBM MQ also has containerised deployment options.

Tuxedo is a middleware platform used to manage distributed transaction processing in distributed computing environments. Tuxedo is a transaction processing system or transaction-oriented middleware, or enterprise application server for a variety of systems and programming languages. Developed by AT&T in the 1980s, it became a software product of Oracle Corporation in 2008 when they acquired BEA Systems. Tuxedo is now part of the Oracle Fusion Middleware.

Stratus VOS is a proprietary operating system running on Stratus Technologies fault-tolerant computer systems. VOS is available on Stratus's ftServer and Continuum platforms. VOS customers use it to support high-volume transaction processing applications which require continuous availability. VOS is notable for being one of the few operating systems which run on fully lockstepped hardware.

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing, reliability and security.

A message broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. Message brokers are elements in telecommunication or computer networks where software applications communicate by exchanging formally-defined messages. Message brokers are a building block of message-oriented middleware (MOM) but are typically not a replacement for traditional middleware like MOM and remote procedure call (RPC).

MessagePlus/Open is a communication platform for medium to large organizations designed specifically to handle high volumes of traffic within a high-availability environment. Currently supported communication channels include fax, fax over IP, e-mail and SMS, together withvarious options for integration with business applications. The product has been developed by INTERCOPE GmbH, Germany and is sold by both Intercope and IBM. It was previously known as FaxPlus/Open.

Open Message Queue is an open-source message-oriented middleware project by Oracle that implements the Java Message Service 2.0 API (JMS). It is the default JMS provider integrated into GlassFish.

A message queueing service is a message-oriented middleware or MOM deployed in a compute cloud using software as a service model. Service subscribers access queues and or topics to exchange data using point-to-point or publish and subscribe patterns.

OS 2200 supports all commonly used, and many not so commonly used, distributed processing protocols, APIs, and development technology.

The Application Interface Specification (AIS) is a collection of open specifications that define the application programming interfaces (APIs) for high-availability application computer software. It is developed and published by the Service Availability Forum and made freely available. Besides reducing the complexity of high-availability applications and shortening development time, the specifications intended to ease the portability of applications between different middleware implementations and to admit third party developers to a field that was highly proprietary in the past.

ZeroMQ is an asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker; the zero in the name is for zero broker. The library's API is designed to resemble Berkeley sockets.

Middleware is a type of computer software program that provides services to software applications beyond those available from the operating system. It can be described as "software glue".

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems via Kafka Connect, and provides the Kafka Streams libraries for stream processing applications. Kafka uses a binary TCP-based protocol that is optimized for efficiency and relies on a "message set" abstraction that naturally groups messages together to reduce the overhead of the network roundtrip. This "leads to larger network packets, larger sequential disk operations, contiguous memory blocks [...] which allows Kafka to turn a bursty stream of random message writes into linear writes."

References

↑ "Patchwork epoll: add EPOLLEXCLUSIVE flag". patchwork.kernel.org. Jason Baron.
↑ "EPOLLEXCLUSIVE Linux Kernel patch testing". mvitolin.wordpress.com. mvitolin.
↑ "Enduro/X 7.5 Release". mavimax.com. Mavimax.
↑ "Enduro/X VS Oracle Tuxedo performance benchmark". publicwork.wordpress.com. Publicwork.
↑ "High performance middleware wars: ZeroMQ vs Enduro/X benchmark". publicwork.wordpress.com. Publicwork.
↑ "Enduro/X vs RabbitMQ review and benchmark". publicwork.wordpress.com. Publicwork.
↑ "Enduro/X Connect". www.mavimax.com. Mavimax.
↑ "ATR Baltic's Enduro/X middleware Acquisition". www.mavimax.com. Mavimax Ltd.

External links

Official website

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[1] "Patchwork epoll: add EPOLLEXCLUSIVE flag". patchwork.kernel.org. Jason Baron.

[2] "EPOLLEXCLUSIVE Linux Kernel patch testing". mvitolin.wordpress.com. mvitolin.

[3] "Enduro/X 7.5 Release". mavimax.com. Mavimax.

[4] "Enduro/X VS Oracle Tuxedo performance benchmark". publicwork.wordpress.com. Publicwork.

[5] "High performance middleware wars: ZeroMQ vs Enduro/X benchmark". publicwork.wordpress.com. Publicwork.

[6] "Enduro/X vs RabbitMQ review and benchmark". publicwork.wordpress.com. Publicwork.

[7] "Enduro/X Connect". www.mavimax.com. Mavimax.

[8] "ATR Baltic's Enduro/X middleware Acquisition". www.mavimax.com. Mavimax Ltd.

[1]

[2]

[3]

[4]

[5]

[6]

[7]