Apama architecture has a modular, scalable design with core features that
Monitor inbound events typically delivered by a messaging infrastructure or market data feed.
Analyze those events in memory, either singly or in conjunction with other events whose attributes and temporal ordering represent a pattern.
Trigger outbound events that represent an action to be taken in response to the analysis.
As you can see, Apama’s architecture is designed to process events. Event processing requires an architecture that is fundamentally different from traditional data processing architectures. Because Apama’s architecture is event driven, an understanding of the distinctive qualities of this architecture is crucial to designing and building robust Apama applications.
Distinguishing architectural features
Apama inverts the paradigm of traditional data-centric systems. Rather than the “store > index > search” model of those architectures, Apama introduces the correlator — a real-time, event processing engine. An Apama application comprises monitors that specify the events or patterns of events that interest you. These specifications are the logical equivalent of database queries. After you load monitors into the correlator, incoming events flow over them and they monitor these event streams for the events and patterns you specified. When a matching event or pattern is found the correlator processes it according to the rules you specify.
Apama’s architecture is further distinguished by its ability to support huge numbers of monitors operating simultaneously. Each can have its own logic for monitoring the event streams, seeking out patterns and, upon detection, triggering specified actions.
EPL, Apama’s native event programming language, lets developers define rules for processing complex events. Such rules let the correlator find temporal and causal relationships among events.
Messages on a variety of transports carry events to and from correlators. Apama connectivity plug-ins translate application-specific data into Apama application event formats that the correlator can process.
Apama components can be connected to each other by executing the Apama engine_connect tool with specification of an explicit point-to-point connection.
How Apama integrates with external data sources
You can connect Apama to any event data source, database, messaging infrastructure, or application. There are several ways to do this:
Write transport and codec connectivity plug-ins.
Use the HTTP Server (REST) connectivity plug-in for general-purpose communication.
Use MQTT for communication between constrained devices, for example, devices with limited network bandwidth.
Use Cumulocity for communication with connected IoT devices.
Using connectivity plug-ins to connect with external data sources
Connectivity plug-ins can be written in Java or C++, and run inside the correlator process to allow messages to be sent and received to/from external systems. Individual plug-ins are combined together to form chains that define the path of a message, with the correlator host process at one end and an external system or library at the other, and with an optional sequence of message mapping transformations between them.
Connectivity plug-ins allow plug-ins to transform and handle delivery of events.
Connectivity plug-ins perform a similar role to the Apama client library, which allows Java or C++ code in an external process to send/receive messages to/from the correlator. If Apama events need to be made available within an external system, then consider connectivity plug-ins if the external system has a protocol (such as JSON over HTTP). If the external system hosts plug-ins via an API, then the client library may be a better fit.
Using MQTT for communication between constrained devices
Apama provides a connectivity plug-in, the MQTT transport, which can be used to communicate between the correlator and an MQTT broker, where the MQTT broker uses topics to filter the messages. MQTT messages can be transformed to and from Apama events by listening for and sending events to channels such as *prefix*:*topic* (where the prefix is configurable).
Using Kafka for communication with a Kafka distributed streaming platform
Apama provides a connectivity plug-in, the Kafka transport, which can be used to communicate with the Kafka distributed streaming platform. Kafka messages can be transformed to and from Apama events by listening for and sending events to channels such as *prefix*:*topic* (where the prefix is configurable).
Using Cumulocity for communication with connected IoT devices
Apama provides several connectivity bundles which allow you to communicate with the IoT devices connected to Cumulocity. For example, you can receive events from the devices, send operations to the devices, and query the state stored in the platform.
While traditional architectures can respond to events after they have happened, Apama’s event-driven architecture responds in real time to fast moving events of any kind. Apama applications leverage a platform that combines analytic sophistication, flexibility, performance and interoperability. In addition to being an event processing engine, Apama provides sophisticated development tools, a flexible testing environment, and an extensible connectivity framework. This makes Apama a comprehensive event processing platform for building real-time, event-driven applications.
Description of the Apama correlator
Apama’s correlator is the engine that powers an Apama application. Correlators execute the sophisticated event pattern-matching logic that you define in your Apama application. Apama applications track inbound event streams and listen for events whose patterns match defined conditions. The correlator’s patented architecture can monitor huge volumes of events per second
When an event or an event sequence matches an active event expression, the correlator executes the appropriate actions, as defined by the application logic.
The correlator can concurrently search for and identify vast numbers of discrete event patterns with sub-millisecond responsiveness.
The correlator can deliver low latency analytics on multiple inbound data streams by monitoring the event streams for patterns you specify.
The correlator goes beyond simple event processing to deliver actionable responses.
EPL is Apama’s native event processing language. You can find complete information in Developing Apama Applications.
Before EPL can look for patterns in event streams, you must define the types of events you are interested in and inject their definitions in the correlator. An event definition informs the correlator about the composition of an event type. An example event definition for a stock exchange tick feed is as follows:
Each field of the event has a type and a name. The type informs the correlator how to handle that field and what operations to allow on it. As you can see, the correlator can handle multiple types, such as numeric values and textual values, within the same event type. Apama can handle any number of different event types at one time.
External event sources such as connectivity plug-ins and clients need to be able to send events into the correlator. For the correlator to be able to detect an event of interest, the event’s type definition must have been loaded into the correlator. An example of a StockTick event is as follows:
StockTick ("APAMA", 55.20, 250010)
Apama monitors
A monitor defines:
One or more listeners. EPL provides event listeners and stream listeners.
An event listener observes the correlator event stream analyzing each event in turn until it finds a sequence of events that match its event expression. When this happens the event listener triggers, causing the correlator to execute the listener action.
A stream listener passes stream query output to procedural code. A stream query operates on one or two streams to transform their contents into a single output stream. The type of the stream query output items need not be the same as the type of the stream query input items. The output for one stream query can be the input for another stream query. At the end of the chain of stream queries, a stream listener coassigns each stream query output item to a variable and executes specified code.
One or more actions. An action is one or more operations that the correlator performs. An action might be to register a listener or it might be an operation to perform when the correlator finds a match between an incoming event or sequence and a listener.
The following EPL example illustrates these concepts in the form of a simple monitor called PriceRise. The monitor is composed of three actions. The first two actions declare listeners, which are indicated by the on keyword.
monitor PriceRise
{
action onload() {
on all StockTick("IBM",>=75.5,*) as firstTick {
furtherRise (firstTick);
}
from tick in all StockTick(symbol="IBM")
within 60.0 every 60.0
select mean(tick.price) as f { average(tick.price); }
}
action average(float av) {
log "60-second average for IBM: "+av.toString();
}
action furtherRise(StockTick tick) {
on all StockTick("IBM",>=(tick.price*1.05),*) as finalTick {
log "IBM has hit "+finalTick.price.toString();
send Placeholder("IBM",finalTick.price,1000.0) to "PlaceholderChannel";
}
}
}
When a monitor starts running, the correlator executes the monitor’s onload() action. In the PriceRise monitor, the onload() action creates an event listener for all IBM stock ticks that have a price above 75.5 at any volume and a stream listener for all IBM stock ticks. Since the last field of the event (volume) is irrelevant to the event listener it is represented by an asterisk (*), which indicates a wildcard. This monitor effectively goes to sleep until the correlator detects an IBM stock tick.
If the correlator detects an IBM stock tick, the stream listener takes it as input and uses it to log 60-second averages for IBM stock prices. If the IBM stock tick also has a price that is greater than or equal to 75.5, the correlator copies the field values in that event to the firstTick variable and calls the furtherRise() action.
The furtherRise() action creates another event listener. This event listener is looking for the next part of the event pattern, which involves detecting if the IBM stock price goes up by more than 5% from its new value. The second listener uses the firstTick variable to obtain the price value in the event that caused the first listener to detect a match. If the price rise occurs, the correlator copies the values in the matching, incoming event to the finalTick variable, and executes the associated block of code.
The associated block of code logs the new price and sends a PlaceSellOrder event to a receiver that is external to the correlator. For example, an adapter can pick up this event, and translate it into a message that an order book can operate on. The PlaceSellOrder event causes placement of an order for 1000 units of IBM stock.
Description of Apama Plugin for Eclipse
Apama Plugin for Eclipse can be used for Apama development. When you are ready to start developing your Apama application, open Apama Plugin for Eclipse and create an Apama project to contain your application files.
Info
The Apama Plugin for Eclipse is deprecated and will be removed in a future release.
Complete information is in Using Apama with Apama Plugin for Eclipse.
Description of Dashboard Builder and Dashboard Viewer
Info
Apama dashboards are deprecated and will be removed in a future release.
Apama’s Dashboard Builder enables you to create end-user dashboards and prepare them for deployment. For applications written in EPL, you create DataViews and use Dashboard Builder to create a dashboard from the DataViews.
Dashboard Builder is a visual design environment. A primary goal of Dashboard Builder is to enable non-technical users to create sophisticated dashboards. Consequently, Dashboard Builder provides a complete design and deployment environment. With a wide range of visual objects and drag-and-drop development, Dashboard Builder provides the tools needed to create highly customized dashboards from which users can start/stop, parameterize and monitor Apama DataViews.
Dashboard Builder offers an extensive array of graphical widgets with which to build custom user dashboards. Meters, gauges, tables, graphs, and scales are available for creating highly customized dashboards. You can further personalize the interface through addition/deletion of panels or modification of graphics and color schemes.
Dashboard Viewer is the tool that end-users run to access dashboards.
Apama is highly extensible with a range of APIs provided at the client and correlator levels. You can use these APIs to integrate with other environments, such as Java, JavaBeans, C++, or.NET. You can extend correlator behavior with plug-ins that can call external function libraries from within an application scenario.
Note: Apama’s Data Player and the Apama Database Connector (ADBC) are deprecated and will be removed in a future release.
Data Player, which runs in Apama Plugin for Eclipse, accelerates the development/deployment cycle of EPL applications by letting you pre-test (via simulation) your applications on event streams captured in Apama. It also supports flexible event processing replay features.
Data Player provides analysis tools for the Apama environment. It enables Apama users to investigate the likely behavior of Apama applications prior to deployment, as well as analyze the actual performance of those applications already in production.
Data Player operates on data captured by the Apama Database Connector (ADBC). ADBC provides Apama standard adapters that allows access to JDBC/ODBC compliant databases as well as to Apama Sim files. Analysis can include all events received by Apama or only selected event streams. Likewise, you can choose specific segments of time from the past (for example, an entire day, a specific 30 minute period, or any user chosen time slice). Additionally, you can accelerate replay speeds many times the actual live speeds, or slow them down or pause for more careful exploration of event processing operations.
The following figure shows the inner details of a running correlator. After the figure, there is a detailed discussion of how the correlator works.
Monitors identify event patterns of interest and the responses to take if those patterns are detected. You use EPL to write monitors directly. Apama uses the Apama Plugin for Eclipse development environment for writing source code for monitors.
The correlator does not just execute loaded monitors in a sequential manner, as if they were traditional imperative programs. Instead, the correlator loads its internal components (the hypertree and the temporal sequencer) with the monitoring specifications of the monitors. The in-built virtual machines execute only the sequential analytic or action parts of the monitors.
The correlator contains the following components:
HyperTree multi-dimensional event matcher
The event matcher contains data structures and algorithms designed for high performance, multi-dimensional, event filtering. The correlator loads the event matcher with event templates. An event template identifies the event you are interested in. Logically, an event template is a multi-dimensional search. For example, a template for an IoT sensor might have values such as the following:
deviceId: *
deviceType: TemperatureSensor
uptime: > 5
temperature: 20.0 <- -> 22.5
This event template expresses a multi-dimensional search over sensor measurement events. The template will match events from devices of type TemperatureSensor which have been online for at least 5 seconds and have a temperature between 20.0 and 22.5 degrees. The individual deviceId is irrelevant to this search and so a * wildcard is used.
This kind of multi-dimensional, multi-type, ranged searching is what the event matcher was specifically designed for. In checking whether an incoming event matches any of the registered event templates, the event matcher exhibits logarithmic performance. This means that vast numbers of event templates can be queried against, with the minimum possible performance tail-off.
An event template is the basic unit of monitoring. A simple monitor might have one or a few event templates. A more complex monitor might have many. A monitor needs to load event templates only when events that match the specification are relevant to the monitor: in a multi-stage monitor, a monitor can insert and remove several event templates as the monitoring requirements change.
Temporal and stream sequencer
The temporal and stream sequencer builds upon the single event matching capabilities of the event matcher to provide multiple temporal event and stream correlations. With EPL, you can declare a temporal sequence such as “tell me when any news article event is followed within 5 minutes by a 5% fall in the price of the stock the news article was about”. This is a temporal sequence, with a temporal constraint. The sequence is a news article event, followed by the next stock price event, and then another stock price event with a price 5% less than the previous price event. The temporal constraint is that the last event occurs within 5 minutes of the first event.
The sequencer manages this temporal monitoring process, using the event matcher to monitor for appropriate event templates. This capability saves the programmer from having to encode such complex temporal logic through less intuitive imperative logic.
Monitors
The correlator provides the capability for monitors to be injected as EPL. The number of monitors that can be loaded into a single correlator are only limited by memory size. When loaded, a monitor configures the hypertree and temporal sequencer with event templates for monitoring. The correlator stores the monitor internally and executes actions in the appropriate virtual machine in response to event detection.
Each monitor instance has its own address space within the correlator for storage of variables and other state. Monitor temporary storage size is limited only by the memory size of the host machine.
Event input queue
External interfaces, such as connectivity plug-in chains, send events into the correlator. To start the monitoring process, the correlator injects each event, in the order in which it arrives, into the hypertree. Any matches filter through the temporal sequencer and invoke required actions in the virtual machines. Some actions might cause events to be queued for output. During peak event input flow, events might wait on an input queue for an extremely brief moment.
EPL virtual machine
In response to detected event patterns of interest, the EPL virtual machine executes EPL. The fact that the correlator behaves this way, rather than continuously executing imperative code, is another reason for its high performance. Also, you can implement parallel processing in your applications so that the correlator can concurrently execute code in multiple monitors.
Event output queue
Monitor actions can output events to be communicated to other monitors or to external systems. When a monitor routes an event, the event goes to the front of the input queue. This ensures that any monitors that are listening for that event immediately detect it. When a monitor generates an event for an external receiver the event goes to an output queue for transmission to the appropriate registered party.
When you use the correlator in conjunction with connectivity plug-ins, then an output event might represent an action on an external service. The connectivity plug-in transforms the output event into an invocation of the external service. An example is an event that places an order into the order book of a Stock Exchange.
EPL plug-ins
It is possible to extend the capabilities of the correlator through an EPL plug-in. An EPL plug-in is an externally-linked software module that registers with the correlator through the correlator’s extension API. EPL plug-ins are useful when programming libraries of useful real-time functions have been built up. These functions can be made available as objects that can be invoked by EPL actions.
Apama provides a number of standard EPL plug-ins:
The MemoryStore plug-in lets monitors share in-memory data.
The TimeFormat plug-in helps you format dates and times.
State persistence
When the correlator shuts down the default behavior is that all state is lost. When you restart the correlator no state from the previous time the correlator was running is available. You can change this default behavior by using correlator persistence. Correlator persistence means that the correlator automatically periodically takes a snapshot of its current state and saves it on disk. When you shut down and restart that correlator, the correlator restores the most recent saved state.
To enable persistence, you indicate in your EPL code which monitors you want to be persistent. Optionally, you can write actions that the correlator executes as part of the recovery process. When code is injected for a persistence application, the correlator that the code is injected into must have been started with a persistence option. Persistent monitors are written in EPL. State in chunks, with a few exceptions, cannot be persistent.
Programming the correlator
You program the correlator by injecting monitors that you write in EPL.
When events are sent to the correlator, the correlator processes events by comparing the events to what listeners are active in the correlator. Each external event matches zero or more listeners. The correlator executes a matching event’s associated listeners in a rigid order. The correlator completes the processing related to a particular event before it examines the next event. If the processing of an event generates another event that is routed to the correlator, the correlator processes all routed events before moving on to the next event in its queue. If a listener action block does not route events, the next external event is considered.
