Architecture

Using connectivity plug-ins to connect with external data sources

Connectivity plug-ins can be written in Java or C++, and run inside the correlator process to allow messages to be sent and received to/from external systems. Individual plug-ins are combined together to form chains that define the path of a message, with the correlator host process at one end and an external system or library at the other, and with an optional sequence of message mapping transformations between them.

Connectivity plug-ins allow plug-ins to transform and handle delivery of events.

Connectivity plug-ins perform a similar role to the Apama client library, which allows Java or C++ code in an external process to send/receive messages to/from the correlator. If Apama events need to be made available within an external system, then consider connectivity plug-ins if the external system has a protocol (such as JSON over HTTP). If the external system hosts plug-ins via an API, then the client library may be a better fit.

For detailed information, see Using connectivity plug-ins.

Using MQTT for communication between constrained devices

Apama provides a connectivity plug-in, the MQTT transport, which can be used to communicate between the correlator and an MQTT broker, where the MQTT broker uses topics to filter the messages. MQTT messages can be transformed to and from Apama events by listening for and sending events to channels such as *prefix*:*topic* (where the prefix is configurable).

For detailed information, see The MQTT Transport Connectivity Plug-in.

Using Cumulocity for communication with connected IoT devices

Apama provides several connectivity bundles which allow you to communicate with the IoT devices connected to Cumulocity. For example, you can receive events from the devices, send operations to the devices, and query the state stored in the platform.

For detailed information, see The Cumulocity transport connectivity plug-in.

Descriptions of Apama components

Apama’s event-driven architecture responds in real time to fast moving events such as measurements from IoT devices. Apama applications leverage a platform that combines analytic sophistication, flexibility, performance and interoperability. In addition to being an event processing engine, Apama provides development tools, a flexible testing environment, and an extensible connectivity framework. This makes Apama a comprehensive event processing platform for building real-time, event-driven applications.

Description of the Apama correlator

Apama’s correlator is the engine that executes an Apama application. Correlators execute the sophisticated event pattern-matching logic that you define in your Apama application. Apama applications track inbound event streams and listen for events whose patterns match defined conditions. The correlator’s patented architecture can monitor huge volumes of events per second

When an event or an event sequence matches an active event expression, the correlator executes the appropriate actions, as defined by the application logic.

• The correlator can concurrently search for and identify vast numbers of discrete event patterns with sub-millisecond responsiveness.
• The correlator can deliver low latency analytics on multiple inbound data streams by monitoring the event streams for patterns you specify.
• The correlator goes beyond simple event processing to deliver actionable responses.

See also How the correlator works.

Description of Apama EPL

EPL is Apama’s native event processing language. You can find complete information in Developing Apama Applications.

Before EPL can look for patterns in event streams, you must define the types of events you are interested in and inject their definitions in the correlator. An event definition informs the correlator about the composition of an event type. An example event definition for a stock exchange tick feed is as follows:

event StockTick {
   string symbol;
   float price;
   float volume;
}

Each field of the event has a type and a name. The type informs the correlator how to handle that field and what operations to allow on it. As you can see, the correlator can handle multiple types, such as numeric values and textual values, within the same event type. Apama can handle any number of different event types at one time.

External event sources such as connectivity plug-ins need to be able to send events into the correlator. For the correlator to be able to detect an event of interest, the event’s type definition must have been loaded into the correlator. An example of a StockTick event is as follows:

StockTick ("APAMA", 55.20, 250010)

Apama monitors

An EPL monitor defines:

• One or more listeners. EPL provides event listeners and stream listeners.
  • An event listener observes the correlator event stream analyzing each event in turn until it finds a sequence of events that match its event expression. When this happens the event listener triggers, causing the correlator to execute the listener action.
  • A stream listener passes stream query output to procedural code. A stream query operates on one or two streams to transform their contents into a single output stream. The type of the stream query output items need not be the same as the type of the stream query input items. The output for one stream query can be the input for another stream query. At the end of the chain of stream queries, a stream listener coassigns each stream query output item to a variable and executes specified code.
• One or more actions. An action is one or more operations that the correlator performs. An action might be to register a listener or it might be an operation to perform when the correlator finds a match between an incoming event or sequence and a listener.

The following EPL example illustrates these concepts in the form of a simple monitor called PriceRise. The monitor is composed of three actions. The first two actions declare listeners, which are indicated by the on keyword.

monitor PriceRise
{
   action onload() {
      on all StockTick("IBM",>=75.5,*) as firstTick {
         furtherRise (firstTick);
      }
      from tick in all StockTick(symbol="IBM")
         within 60.0 every 60.0
         select mean(tick.price) as f { average(tick.price); }
   }
   action average(float av) {
      log "60-second average for IBM: "+av.toString();
   }
   action furtherRise(StockTick tick) {
      on all StockTick("IBM",>=(tick.price*1.05),*) as finalTick {
         log "IBM has hit "+finalTick.price.toString();
         send Placeholder("IBM",finalTick.price,1000.0) to "PlaceholderChannel";
      }
   }
}

When a monitor starts running, the correlator executes the monitor’s onload() action. In the PriceRise monitor, the onload() action creates an event listener for all IBM stock ticks that have a price above 75.5 at any volume and a stream listener for all IBM stock ticks. Since the last field of the event (volume) is irrelevant to the event listener it is represented by an asterisk (*), which indicates a wildcard. This monitor effectively goes to sleep until the correlator detects an IBM stock tick.

If the correlator detects an IBM stock tick, the stream listener takes it as input and uses it to log 60-second averages for IBM stock prices. If the IBM stock tick also has a price that is greater than or equal to 75.5, the correlator copies the field values in that event to the firstTick variable and calls the furtherRise() action.

The furtherRise() action creates another event listener. This event listener is looking for the next part of the event pattern, which involves detecting if the IBM stock price goes up by more than 5% from its new value. The second listener uses the firstTick variable to obtain the price value in the event that caused the first listener to detect a match. If the price rise occurs, the correlator copies the values in the matching, incoming event to the finalTick variable, and executes the associated block of code.

The associated block of code logs the new price and sends a PlaceSellOrder event to a receiver that is external to the correlator. For example, an adapter can pick up this event, and translate it into a message that an order book can operate on. The PlaceSellOrder event causes placement of an order for 1000 units of IBM stock.

How the correlator works

The following figure shows the inner details of a running correlator.

Monitors identify event patterns of interest and the responses to take if those patterns are detected.

The correlator does not just execute loaded monitors in a sequential manner, as if they were traditional imperative programs. Instead, the correlator loads its internal components (the hypertree and the temporal sequencer) with the monitoring specifications of the monitors. The in-built virtual machines execute only the sequential analytic or action parts of the monitors.

The correlator contains the following components:

HyperTree multi-dimensional event matcher

The event matcher contains data structures and algorithms designed for high performance, multi-dimensional, event filtering. The correlator loads the event matcher with event templates. An event template identifies the event you are interested in. Logically, an event template is a multi-dimensional search. For example, a template for an IoT sensor might have values such as the following:

• deviceId: *
• deviceType: TemperatureSensor
• uptime: > 5
• temperature: 20.0 <- -> 22.5

This event template expresses a multi-dimensional search over sensor measurement events. The template will match events from devices of type TemperatureSensor which have been online for at least 5 seconds and have a temperature between 20.0 and 22.5 degrees. The individual deviceId is irrelevant to this search and so a * wildcard is used.

This kind of multi-dimensional, multi-type, ranged searching is what the event matcher was specifically designed for. In checking whether an incoming event matches any of the registered event templates, the event matcher exhibits logarithmic performance. This means that vast numbers of event templates can be queried against, with the minimum possible performance tail-off.

An event template is the basic unit of monitoring. A simple monitor might have one or a few event templates. A more complex monitor might have many. A monitor needs to load event templates only when events that match the specification are relevant to the monitor: in a multi-stage monitor, a monitor can insert and remove several event templates as the monitoring requirements change.

Temporal and stream sequencer

The temporal and stream sequencer builds upon the single event matching capabilities of the event matcher to provide multiple temporal event and stream correlations. With EPL, you can declare a temporal sequence such as “tell me when any news article event is followed within 5 minutes by a 5% fall in the price of the stock the news article was about”. This is a temporal sequence, with a temporal constraint. The sequence is a news article event, followed by the next stock price event, and then another stock price event with a price 5% less than the previous price event. The temporal constraint is that the last event occurs within 5 minutes of the first event.

The sequencer manages this temporal monitoring process, using the event matcher to monitor for appropriate event templates. This capability saves the programmer from having to encode such complex temporal logic through less intuitive imperative logic.

Monitors

The correlator provides the capability for monitors to be injected as EPL. The number of monitors that can be loaded into a single correlator are only limited by memory size. When loaded, a monitor configures the hypertree and temporal sequencer with event templates for monitoring. The correlator stores the monitor internally and executes actions in the appropriate virtual machine in response to event detection.

Each monitor instance has its own address space within the correlator for storage of variables and other state. Monitor temporary storage size is limited only by the memory size of the host machine.

Event input queue

External interfaces, such as connectivity plug-in chains, send events into the correlator. To start the monitoring process, the correlator injects each event, in the order in which it arrives, into the hypertree. Any matches filter through the temporal sequencer and invoke required actions in the virtual machines. Some actions might cause events to be queued for output. During peak event input flow, events might wait on an input queue for an extremely brief moment.

EPL virtual machine

In response to detected event patterns of interest, the EPL virtual machine executes EPL. The fact that the correlator behaves this way, rather than continuously executing imperative code, is another reason for its high performance. Also, you can implement parallel processing in your applications so that the correlator can concurrently execute code in multiple monitors.

Event output queue

Monitor actions can output events to be communicated to other monitors or to external systems. When a monitor routes an event, the event goes to the front of the input queue. This ensures that any monitors that are listening for that event immediately detect it. When a monitor generates an event for an external receiver the event goes to an output queue for transmission to the appropriate registered party.

When you use the correlator in conjunction with connectivity plug-ins, then an output event might represent an action on an external service. The connectivity plug-in transforms the output event into an invocation of the external service. An example is an event that places an order into the order book of a Stock Exchange.

EPL plug-ins

It is possible to extend the capabilities of the correlator through an EPL plug-in. An EPL plug-in is an externally-linked software module that registers with the correlator through the correlator’s extension API. EPL plug-ins are useful when programming libraries of useful real-time functions have been built up. These functions can be made available as objects that can be invoked by EPL actions.

Apama provides a number of standard EPL plug-ins, for example the “TimeFormat” plug-in helps you format dates and times.

Programming the correlator

You program the correlator by injecting monitors that you write in EPL.

When events are sent to the correlator, the correlator processes events by comparing the events to what listeners are active in the correlator. Each external event matches zero or more listeners. The correlator executes a matching event’s associated listeners in a rigid order. The correlator completes the processing related to a particular event before it examines the next event. If the processing of an event generates another event that is routed to the correlator, the correlator processes all routed events before moving on to the next event in its queue. If a listener action block does not route events, the next external event is considered.