This section describes how to use the Apama Integration Adapter Framework (IAF). The IAF is a middleware-independent and protocol-neutral adapter tailoring framework designed to provide for the easy and straightforward creation of software adapters to interface Apama with middleware buses and other message sources. It provides facilities to generate adapters that can communicate with third-party messaging systems, extract and decode self-describing or schema-formatted messages, and flexibly transform them into Apama events. Vice-versa, Apama events can be transformed into the proprietary representations required by third-party messaging systems. It provides highly configurable and maintainable interfaces and semantic data transformations. An adapter generated with the IAF can be re-configured and upgraded at will, and in many cases, without having to restart it. Its dynamic plug-in loading mechanism allows a user to customize it to communicate with proprietary middleware buses and decode message formats.
There are two ways of integrating with Apama through software. The first is to use the low-level Client Software Development Kits (SDKs) to write your own custom software interface (see Developing custom clients). The second is to instantiate an adapter with the higher-level Integration Adapter Framework (IAF). This is described in the topics below.
Note: The IAF architecture is superseded by connectivity plug-ins. Therefore, it is strongly recommended to choose connectivity plug-ins over the IAF when creating new adapters and connectivity. Non-capital markets use of the IAF is deprecated.
Overview
The IAF is a middleware-independent and protocol-neutral adapter tailoring framework. It is designed to allow easy and straightforward creation of software adapters to interface Apama with middleware buses and other message sources. It provides facilities to generate adapters that can communicate with third-party messaging systems, extract and decode self-describing or schema-formatted messages, and flexibly transform them into Apama events. In the opposite direction, Apama events can be transformed into the proprietary representations required by third-party messaging systems. It provides highly configurable and maintainable interfaces and semantic data transformations. An adapter generated with the IAF can be re-configured and upgraded at will, and in many cases, without having to restart it. Its dynamic plug-in loading mechanism allows a user to customize it to communicate with proprietary middleware buses and decode message formats.
On the other hand, the SDKs provide lower-level client application programming interfaces that allow one to directly connect to Apama and transfer Apama Event Processing Language (EPL) code and events in and out. The SDKs provide none of the abstractions and functionality of the IAF, and hence their use is only recommended when a developer needs to write a highly customized and very high performance software client, or wishes to integrate existing client code with Apama in process.
The IAF is available on all platforms supported by Apama, although not all adapters will work on all platforms. For the most up-to-date information about supported platforms and compilers, see the Supported Platforms.
Architecture
The first step in integrating Apama within a user environment is to connect the correlator to one or more message/event sources and/or sinks. In the majority of cases the source or provider of messages will be some form of middleware message bus although it could also be a database or other storage based message source, as well as an alternative network-based communication mechanism, like a trading system. The same applies for the sink or consumer of messages.
These message sources and/or sinks vary extensively. Typically each comes with its own proprietary communications paradigm, communications protocol, message representation and programming interfaces. Interfacing any software like Apama with a source/sink of messages like a message bus therefore requires the writing of a specialized software adapter or connector, which needs to be maintained should the messaging environment or the message representation change. The adapter needs to interface with the messaging middleware or message source, receive messages from it and decode them, and then transform them into events that Apama can understand and process. The latter transformation is not always straightforward, as the message representation might vary from message to message and require semantic understanding.
Conversely, the events generated by Apama need to be processed in the inverse direction and eventually end up back on the message bus.
Note:
In the Apama documentation, a message traveling from a message source through the IAF and into Apama is described as traveling downstream, whereas a message output as an alert from Apama and progressing back out through the IAF towards a message sink is described as traveling upstream.
In order to facilitate development of software adapters, Apama provides the IAF. In contrast to the SDKs, the IAF is not a programming library. The IAF is effectively a customizable, middleware independent, generic adapter that can be adapted by a user to communicate with their middleware and apply their specific semantic transformations to their messages.
As illustrated above, the IAF acts as the interface between the messaging middleware and the correlator. There are four primary components to the IAF:
The Transport Layer. This is the layer that communicates with the user’s message source/sink. Its functionality is defined through one or more Apama or user-provided message source/sink specific transport plug-ins, written in C, C++ or Java.
The Codec Layer. The codec layer translates messages from any custom representation into a normalized form and vice-versa. The transformation is carried out by one of its codec plug-ins. These can be provided by Apama or by the user, and may be written in C, C++ or Java. Note that Java codec plug-ins may only communicate with Java transport plug-ins, and C/C++ codec plug-ins with C/C++ transport plug-ins.
The Semantic Mapper. This layer provides functionality to transform the messages received from the message source into Apama events. The Semantic Mapper is a standard component that is configured for use with a particular adapter by means of a set of semantically rich translation mapping rules. These rules define both how to generate Apama events from externally generated messages and how user-custom messages for an external destination may be generated from Apama events.
An adapter can be configured to bypass this kind of mapping in the Semantic Mapper. Used this way, the Semantic Mapper converts the string form of an Apama event directly to a normalized form and vice-versa.
The Apama Interface layer. This layer abstracts away communication with Apama’s correlator. It injects EPL definitions and event instances into the correlator and asynchronously receives back events from it.
Additionally, the engine_send and engine_receive tools can be run against the IAF simply by supplying the port on which the IAF is running. For example, running
engine_receive -p 16903
connects to the IAF running on the default port and receives all event emitted by it.
The next sections explore the transport, codec and Semantic Mapper layers in more detail.
The transport layer
The transport layer is the front-end of the IAF. The transport layer’s purpose is to communicate with an external message source and/or sink, extracting downstream messages from the message source ready for delivery to the codec layer, and sending Apama events already encoded by the codec layer on to the message sink.
This layer interfaces with the middleware message bus or message source/sink through the latter’s custom programming interface. It receives and dispatches messages from and to it as well as carrying out other proprietary operations. Depending on the nature of the message bus or message source in use, these operations could include opening a database file and running SQL queries on it, registering interest in specific message topics with a message bus, or providing security credentials for authentication. Note that if the IAF is also being used to output messages back to a message sink, then it must also carry out the operations required to enable this; for example, opening and writing to a database file, or dispatching messages onto a message bus.
As this functionality depends entirely on the message source and/or sink the IAF needs to interface with, the transport layer’s functionality is loaded dynamically through a custom transport plug-in.
Although Apama provides a set of standard transport plug-ins for popular messaging systems, the user may develop new transport plug-ins. See C/C++ transport plug-in development and Transport plug-in development in Java for the Transport plug-in Development Specifications for C/C++ and Java, which describe how custom transport plug-ins may be developed in the C, C++ and Java programming languages.
The transport layer can contain one or more transport layer plug-ins. These are loaded when the adapter starts, and the set of loaded plug-ins can be changed while the adapter is running. In addition, a loaded plug-in may be re-configured at any time using the IAF Client tool. If a transport plug-in requires startup or re-configuration parameters, these need to be supplied in the IAF configuration file as documented in The IAF configuration file.
Because a transport layer plug-in effectively implements a custom message transport, this manual uses the terms transport layer plug-inand event transportinterchangeably.
The codec layer
While the transport layer communicates with the custom message sources and sinks and extracts messages, such as stock trade data, from them, the responsibility of the codec layer is to correctly interpret and decode each message into a ‘normalized’ format on which the semantic mapping rules can be run; similarly in the upstream direction a codec may be responsible for encoding a normalized message in an appropriate format for transmission by particular transport(s).
Message sources like middleware message buses typically use proprietary representation of messages. Messages might appear as strings (possibly human readable or otherwise encoded) or sequences of binary characters. Messages might also be self-describing (possibly in XML or through some other proprietary descriptive format) or else be structured according to a schema available elsewhere.
Producing a universal generic normalized format from these messages requires the codec layer to understand the particular format of the messages. In the upstream direction the codec layer needs to encode the messages correctly according to the destination message sink.
As with the transport layer, in order to enable this custom functionality, the IAF is designed to dynamically load codec plug-ins that are capable of decoding and encoding the messages being received, when supplied with any required configuration properties. Apama provides some generic codec plug-ins, such as the StringCodec codec. This can decode most string based name-value representations of messages once it is configured with the syntactic elements used to delimit the elements in a message. In addition the user may develop proprietary codec plug-ins. See C/C++ codec plug-in development and Java codec plug-in development for the Codec plug-in Development Specifications for C/C++ and for Java, which describe how custom codec plug-ins may be developed using the C, C++ and Java programming languages.
An adapter can load multiple codec plug-ins (to deal with different message types). These are loaded at startup and the set of loaded codecs can be changed while the adapter is running. Individual codec plug-ins may also be re-configured at any time. If a codec plug-in requires startup or re-configuration parameters, these need to be supplied in the IAF configuration file as documented in The IAF configuration file.
This manual uses the terms codec layer plug-in and event codec interchangeably.
The Semantic Mapper layer
The Semantic Mapper maps and transforms incoming messages into Apama events that can be passed into the correlator. Conversely, it can accept incoming Apama events and map them into messages that can be sent upstream on the user’s message sink.
Apama events are rigidly defined. Every event must be structured according to a well-defined type definition. Therefore all events are of a specific named type, where this defines the number of fields (or parameters) in the event, their order, the name of each field, and its type. Furthermore it is possible to define which fields are relevant for querying in EPL event expressions, and which are not. See Developing Apama Applications for further information on event type definitions and EPL event expressions. This rigorous format permits the correlator to be highly optimized and contributes towards Apama’s scalable performance.
The source messages that are to be passed into Apama as events (or the sink messages that Apama needs to generate) might match this specification, in which case the mapping will be straightforward. However, they might also differ in several ways, some of which are listed here:
The messages might be self-describing, and need to be parsed in order to deduce what fields they contain.
The fields contained in every message might appear in varying order.
Some messages of different types and with differing sets of fields might reflect the same information but in a different format (e.g. trade events from different markets or news headlines from different sources).
The set of fields contained in messages might differ even if the messages are all of the same type.
The messages might not be of an obvious type, and their nature (e.g. a trade event or a news headline) might need to be deduced from their contents.
The set of fields might be enhanced over time to capture additional information.
Some messages might have fields that are completely irrelevant.
Some messages might have fields that are irrelevant for matching on but might be useful otherwise.
In order to address these conditions and allow meaningful Apama events to be created from external messages, the Semantic Mapper supports a semantically rich set of translation and transformation rules. These need to be expressed in the IAF configuration file.
You can configure an adapter so that some events bypass this kind of mapping in the Semantic Mapper. Instead of mapping each field in an incoming event to a field in an Apama event or the converse, the entire event is treated as a string in a single field.
Contents of the IAF
The Integration Adapter Framework contains the following components:
Core files – these include the IAF Runtime, the management tools and the libraries they require.
Example adapters written in C and Java – this includes the complete sources of the FileTransport/JFileTransport transport layer plug-ins and the StringCodec/ JStringCodec plug-ins, sample configuration files, a file with a set of input messages, an EPL file with a sample application, and a set of reference result messages.
A suite of development materials – these include the C/C++ header files and Java API sources required to develop transport and codec layer plug-ins for both languages. Also included is a skeleton transport and codec plug-in in C, the IAF configuration file XML Document Type Definition (DTD), a makefile for use with GNU Make on UNIX, and a ‘workspace’ file for use with Microsoft’s Visual Studio.NET on Microsoft Windows.
