Complex event processing

Last updated

Event processing is a method of tracking and analyzing (processing) streams of information (data) about things that happen (events), [1] and deriving a conclusion from them. Complex event processing (CEP) consists of a set of concepts and techniques developed in the early 1990s for processing real-time events and extracting information from event streams as they arrive. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) [2] in real-time situations and respond to them as quickly as possible.

Contents

These events may be happening across the various layers of an organization as sales leads, orders or customer service calls. Or, they may be news items, [3] text messages, social media posts, stock market feeds, traffic reports, weather reports, or other kinds of data. [1] An event may also be defined as a "change of state," when a measurement exceeds a predefined threshold of time, temperature, or other value.

Analysts have suggested that CEP will give organizations a new way to analyze patterns in real-time and help the business side communicate better with IT and service departments. [4] CEP has since become an enabling technology in many systems that are used to take immediate action in response to incoming streams of events. Applications are now to be found (2018) in many sectors of business including stock market trading systems, mobile devices, internet operations, fraud detection, the transportation industry, and governmental intelligence gathering.

The vast amount of information available about events is sometimes referred to as the event cloud. [1]

Conceptual description

Among thousands of incoming events, a monitoring system may for instance receive the following three from the same source:

  1. church bells ringing.
  2. the appearance of a man in a tuxedo with a woman in a flowing white gown.
  3. rice flying through the air.

From these events the monitoring system may infer a complex event: a wedding. CEP as a technique helps discover complex events by analyzing and correlating other events: [5] the bells, the man and woman in wedding attire and the rice flying through the air.

CEP relies on a number of techniques, [6] including:

Commercial applications of CEP exist in variety of industries and include the detection of credit-card fraud, business activity monitoring, and security monitoring. [7]

History

The CEP area has roots in discrete event simulation, the active database area and some programming languages. The activity in the industry was preceded by a wave of research projects in the 1990s. According to [8] the first project that paved the way to a generic CEP language and execution model was the Rapide project in Stanford University, directed by David Luckham. In parallel there have been two other research projects: Infospheres in California Institute of Technology, directed by K. Mani Chandy, and Apama in University of Cambridge directed by John Bates. The commercial products were dependents of the concepts developed in these and some later research projects. Community efforts started in a series of event processing symposia organized by the Event Processing Technical Society, and later by the ACM DEBS conference series. One of the community efforts was to produce the event processing manifesto. [9]

CEP is used in operational intelligence (OI) products to provide insight into business operations by running query analysis against live feeds and event data. OI collects real-time data and correlates against historical data to provide insight and analysis. Multiple sources of data can be combined to provide a common operating picture that uses current information.

In network management, systems management, application management and service management, people usually refer instead to event correlation. As CEP engines, event correlation engines (event correlators) analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events. [10]

Inference engines, e.g., rule-based reasoning engines, typically produce inferred information in artificial intelligence. However, they do not usually produce new information in the form of complex (i.e., inferred) events.

Example

A more systemic example of CEP involves a car, some sensors and various events and reactions. Imagine that a car has several sensors—one that measures tire pressure, one that measures speed, and one that detects if someone sits on a seat or leaves a seat.

In the first situation, the car is moving and the pressure of one of the tires moves from 45 psi to 41 psi over 15 minutes. As the pressure in the tire is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The car's Event Processor may detect a situation whereby a loss of tire pressure over a relatively long period of time results in the creation of the "lossOfTirePressure" event. This new event may trigger a reaction process to note the pressure loss into the car's maintenance log, and alert the driver via the car's portal that the tire pressure has reduced.

In the second situation, the car is moving and the pressure of one of the tires drops from 45 psi to 20 psi in 5 seconds. A different situation is detected—perhaps because the loss of pressure occurred over a shorter period of time, or perhaps because the difference in values between each event were larger than a predefined limit. The different situation results in a new event "blowOutTire" being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.

In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation the car is moving normally and suffers a blown tire which results in the car leaving the road and striking a tree, and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of "blowOutTire", "zeroSpeed" and "driverLeftSeat" within a very short period of time results in a new situation being detected: "occupantThrownAccident". Even though there is no direct measurement that can determine conclusively that the driver was thrown, or that there was an accident, the combination of events allows the situation to be detected and a new event to be created to signify the detected situation. This is the essence of a complex (or composite) event. It is complex because one cannot directly detect the situation; one has to infer or deduce that the situation has occurred from a combination of other events.

Integration with business process management

A natural fit for CEP has been with business process management (BPM). [11] BPM focuses on end-to-end business processes, in order to continuously optimize and align for its operational environment.

However, the optimization of a business does not rely solely upon its individual, end-to-end processes. Seemingly disparate processes can affect each other significantly. Consider this scenario: In the aerospace industry, it is good practice to monitor breakdowns of vehicles to look for trends (determine potential weaknesses in manufacturing processes, material, etc.). Another separate process monitors current operational vehicles' life cycles and decommissions them when appropriate. One use for CEP is to link these separate processes, so that in the case of the initial process (breakdown monitoring) discovering a malfunction based on metal fatigue (a significant event), an action can be created to exploit the second process (life cycle) to issue a recall on vehicles using the same batch of metal discovered as faulty in the initial process.

The integration of CEP and BPM must exist at two levels, both at the business awareness level (users must understand the potential holistic benefits of their individual processes) and also at the technological level (there needs to be a method by which CEP can interact with BPM implementation). For a recent state of the art review on the integration of CEP with BPM, which is frequently labeled as Event-Driven Business Process Management, refer to. [12]

Computation-oriented CEP's role can arguably be seen to overlap with Business Rule technology.

For example, customer service centers are using CEP for click-stream analysis and customer experience management. CEP software can factor real-time information about millions of events (clicks or other interactions) per second into business intelligence and other decision-support applications. These "recommendation applications" help agents provide personalized service based on each customer's experience. The CEP application may collect data about what customers on the phone are currently doing, or how they have recently interacted with the company in other various channels, including in-branch, or on the Web via self-service features, instant messaging and email. The application then analyzes the total customer experience and recommends scripts or next steps that guide the agent on the phone, and hopefully keep the customer happy. [13]

Integration with time series databases

A time series database is a software system that is optimized for the handling of data organized by time. Time series are finite or infinite sequences of data items, where each item has an associated timestamp and the sequence of timestamps is non-decreasing. Elements of a time series are often called ticks. The timestamps are not required to be ascending (merely non-decreasing) because in practice the time resolution of some systems such as financial data sources can be quite low (milliseconds, microseconds or even nanoseconds), so consecutive events may carry equal timestamps.

Time series data provides a historical context to the analysis typically associated with complex event processing. This can apply to any vertical industry such as finance [14] and cooperatively with other technologies such as BPM.

The ideal case for CEP analysis is to view historical time series and real-time streaming data as a single time continuum. What happened yesterday, last week or last month is simply an extension of what is occurring today and what may occur in the future. An example may involve comparing current market volumes to historic volumes, prices and volatility for trade execution logic. Or the need to act upon live market prices may involve comparisons to benchmarks that include sector and index movements, whose intra-day and historic trends gauge volatility and smooth outliers.

Internet of things and smart cyber-physical systems

Complex event processing is a key enabler in Internet of things (IoT) settings and smart cyber-physical systems (CPS) as well. Processing dense and heterogeneous streams from various sensors and matching patterns against those streams is a typical task in such cases. [15] The majority of these techniques rely on the fact that representing the IoT system's state and its changes is more efficient in the form of a data stream, instead of having a static, materialized model. Reasoning over such stream-based models fundamentally differs from traditional reasoning techniques and typically require the combination of model transformations and CEP. [16]

See also

Vendors and products

Related Research Articles

In connection-oriented communication, a data stream is the transmission of a sequence of digitally encoded signals to convey information. Typically, the transmitted symbols are grouped into a series of packets.

Business software is any software or set of computer programs used by business users to perform various business functions. These business applications are used to increase productivity, measure productivity, and perform other business functions accurately.

Business process automation (BPA), also known as business automation,distinguished from Business Process Management (BPM), is the technology-enabled automation of business processes. It can help a business in simplicity, to increase digital transformation, increase service quality, improve service delivery, or contain costs. BPA consists of integrating applications, restructuring labor resources, and using software applications throughout the organization. Robotic process automation is an emerging field within BPA.

Business activity monitoring (BAM) is software that aids the monitoring of business activities that are implemented in computer systems.

In computer science, event monitoring is the process of collecting, analyzing, and signaling event occurrences to subscribers such as operating system processes, active database rules as well as human operators. These event occurrences may stem from arbitrary sources in both software or hardware such as operating systems, database management systems, application software and processors. Event monitoring may use a time series database.

Oracle Fusion Middleware consists of several software products from Oracle Corporation. FMW spans multiple services, including Java EE and developer tools, integration services, business intelligence, collaboration, and content management. FMW depends on open standards such as BPEL, SOAP, XML and JMS.

Event-driven architecture (EDA) is a software architecture paradigm concerning the production and detection of events.

Real-time business intelligence (RTBI) is a concept describing the process of delivering business intelligence (BI) or information about business operations as they occur. Real time means near to zero latency and access to information whenever it is required.

Process mining is a family of techniques relating the fields of data science and process management to support the analysis of operational processes based on event logs. The goal of process mining is to turn event data into insights and actions. Process mining is an integral part of data science, fueled by the availability of event data and the desire to improve processes. Process mining techniques use event data to show what people, machines, and organizations are really doing. Process mining provides novel insights that can be used to identify the execution paths taken by operational processes and address their performance and compliance problems.

Operational intelligence (OI) is a category of real-time dynamic, business analytics that delivers visibility and insight into data, streaming events and business operations. OI solutions run queries against streaming data feeds and event data to deliver analytic results as operational instructions. OI provides organizations the ability to make decisions and immediately act on these analytic insights, through manual or automated actions.

Event correlation is a technique for making sense of a large number of events and pinpointing the few events that are really important in that mass of information. This is accomplished by looking for and analyzing relationships between events.

<span class="mw-page-title-main">Truviso</span>

Truviso is a continuous analytics, venture-backed, startup headquartered in Foster City, California developing and supporting its solution leveraging PostgreSQL, to deliver a proprietary analytics solutions for net-centric customers. Truviso was acquired by Cisco Systems, Inc. on May 4, 2012.

Event-driven SOA is a form of service-oriented architecture (SOA), combining the intelligence and proactiveness of event-driven architecture with the organizational capabilities found in service offerings. Before event-driven SOA, the typical SOA platform orchestrated services centrally, through pre-defined business processes, assuming that what should have already been triggered is defined in a business process. This older approach does not account for events that occur across, or outside of, specific business processes. Thus complex events, in which a pattern of activities—both non-scheduled and scheduled—should trigger a set of services is not accounted for in traditional SOA 1.0 architecture.

Business process management (BPM) is the discipline in which people use various methods to discover, model, analyze, measure, improve, optimize, and automate business processes. Any combination of methods used to manage a company's business processes is BPM. Processes can be structured and repeatable or unstructured and variable. Though not required, enabling technologies are often used with BPM.

Database activity monitoring is a database security technology for monitoring and analyzing database activity. DAM may combine data from network-based monitoring and native audit information to provide a comprehensive picture of database activity. The data gathered by DAM is used to analyze and report on database activity, support breach investigations, and alert on anomalies. DAM is typically performed continuously and in real-time.

The openPDC is a complete set of applications for processing streaming time-series data in real-time. The name stands for "open source phasor data concentrator" and was originally designed for the concentration and management of real-time streaming synchrophasors. Due to the system's modular design, the openPDC can be classified as a generic event stream processor.

SQLstream is a distributed, SQL standards-compliant plus Java stream processing platform. SQLstream, Inc. is based in San Francisco, California and was launched in 2009 by Damian Black, Edan Kabatchnik and Julian Hyde, author of the open source Mondrian Relational OLAP Server Engine.

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems via Kafka Connect, and provides the Kafka Streams libraries for stream processing applications. Kafka uses a binary TCP-based protocol that is optimized for efficiency and relies on a "message set" abstraction that naturally groups messages together to reduce the overhead of the network roundtrip. This "leads to larger network packets, larger sequential disk operations, contiguous memory blocks [...] which allows Kafka to turn a bursty stream of random message writes into linear writes."

Esper is an open-source Java-based software product for Complex event processing (CEP) and Event stream processing (ESP), that analyzes series of events for deriving conclusions from them.

Apama is a complex event processing (CEP) and event stream processing (ESP) engine, developed by Software AG. Apama serves as a platform for performing streaming analytics over a range of high volume/low latency inputs and applications, such as IoT devices, financial exchanges, fraud detection, social media and similar. Users can define data patterns to listen for and actions to take when these patterns are found, which are defined in the provided domain-specific language called the Event Processing Language (EPL). The core Apama engine is written in C++; the process can also optionally contain a JVM for interacting with user created Java code. Apama focuses on high throughput, low latency and memory efficient performance; used in both Intel benchmarks and smaller machines such as the Raspberry Pi, routers and other Edge/IoT devices. It is particularly noteworthy within the CEP space as being one of the earliest projects, a long term market leader, and innovator of many patents.

References

  1. 1 2 3 Luckham, David C. (2012). Event Processing for Business: Organizing the Real-Time Enterprise. Hoboken, New Jersey: John Wiley & Sons, Inc. p. 3. ISBN   978-0-470-53485-4.
  2. ↑ Bates, John (15 June 2011), John Bates of Progress explains how complex event processing works and how it can simplify the use of algorithms for finding and capturing trading opportunities, Fix Global Trading, retrieved May 14, 2012
  3. ↑ Crosman, Penny (May 18, 2009), Aleri, Ravenpack to Feed News into Trading Algos, Wall Street & Technology[ permanent dead link ]
  4. ↑ McKay, Lauren (August 13, 2009), Forrester Gives a Welcoming Wave to Complex Event Processing, Destination CRM
  5. ↑ D. Luckham, "The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems", Addison-Wesley, 2002.
  6. ↑ O. Etzion and P. Niblett, "Event Processing in Action", Manning Publications, 2010.
  7. ↑ Details of commercial products and use cases
  8. ↑ Leavit, Neal (April 2009), Complex-Event Processing Poised for Growth, Computer, vol. 42, no. 4, pp. 17-20 Washington
  9. ↑ Chandy, Mani K.; Etzion, Opher; Ammon, Rainer von (22 December 2017). Chandy, K. Mani; Etzion, Opher; Ammon, Rainer von (eds.). "10201 Executive Summary and Manifesto – Event Processing". Dagstuhl Seminar Proceedings. 10201. Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany: 1–60. doi:10.4230/DagSemProc.10201.1 – via Dagstuhl Research Online Publication Server.{{cite journal}}: Cite journal requires |journal= (help)
  10. ↑ J.P. Martin-Flatin, G. Jakobson and L. Lewis, "Event Correlation in Integrated Management: Lessons Learned and Outlook", Journal of Network and Systems Management, Vol. 17, No. 4, December 2007.
  11. ↑ C. Janiesch, M. Matzner and O. Müller: "A Blueprint for Event-Driven Business Activity Management", Lecture Notes in Computer Science, 2011, Volume 6896/2011, 17-28, doi : 10.1007/978-3-642-23059-2_4
  12. ↑ J. Krumeich, B. Weis, D. Werth and P. Loos: "Event-Driven Business Process Management: where are we now?: A comprehensive synthesis and analysis of literature", Business Process Management Journal, 2014, Volume 20, 615-633, doi : 10.1108/BPMJ-07-2013-0092
  13. ↑ Kobielus, James (September 2008), Really Happy in Real Time, Destination CRM
  14. ↑ "Time Series in Finance". cs.nyu.edu.
  15. ↑ "Balogh, Dávid, Ráth, Varró, Vörös: Distributed and Heterogeneous Event-based Monitoring in Smart Cyber-Physical Systems, In 1st Workshop on Monitoring and Testing of Cyber-Physical Systems, Vienna, Austria. 2016".
  16. ↑ I. Dávid, I. Ráth, D. Varró: Foundations for Streaming Model Transformations by Complex Event Processing, International Journal on Software and Systems Modeling, pp 1--28, 2016. doi : 10.1007/s10270-016-0533-1
  17. ↑ Apama Real-Time Analytics Overview Archived 2015-10-25 at the Wayback Machine . Softwareag.com. Retrieved on 2013-09-18.
  18. ↑ "Microsoft StreamInsight". technet.microsoft.com.
  19. ↑ "SAP ESP - Developers community". Archived from the original on 2015-01-05. Retrieved 2014-07-17.
  20. ↑ "Apache Flink 1.2 Documentation: FlinkCEP - Complex event processing for Flink". ci.apache.org.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.