Getting started with connectivity plug-ins

Concepts

A connectivity chain is used to get events of a certain shape from a particular external source and convert them from a given format into Apama events and back. There will be at least one chain for each event source. There could also be one chain for each format of events you want to receive from that source. There can be multiple instances of a given chain definition in the configuration file, corresponding to multiple connections to that source. The choice of which chain to use for a given event may vary between different sources.

All chains begin with a host plug-in, which defines how events are converted into Apama events. All chains end with a transport, which determines the external source we are connecting to. Between these you have an ordered sequence of any number of codecs. These are used to convert messages from the format produced by the transport to that which the host plug-in will consume and vice versa. Messages are passed between the elements in a connectivity chain using an abstract message format made up of maps, lists, byte arrays and various primitive formats. As well as the (arbitrary) payload, there is also a (nested) map of metadata with string keys and any of the supported types as values.

The following diagram shows a typical connectivity chain:

Illustration of a typical connectivity chain

The connectivity chain in the above diagram uses several plug-ins and codecs:

  • The host plug-in is the connection between the correlator and the chain of codecs that manipulate and transform the messages as they pass through the chain. There are two host plug-ins to choose from:

    • apama.eventMap. This is the most common choice for a host plug-in. It produces and consumes Apama events as a map with the keys being the names of the fields in the event and the metadata.sag.type element set to the name of the event.
    • apama.eventString. This host plug-in consumes events in Apama’s proprietary string format. It is usually used for interaction with legacy systems such as the IAF which use Apama’s string format. For detailed information, see Host plug-ins and configuration.

  • Any message is transformed into a compatible form for either the host or the transport as it flows through the codec pipeline toward its destination.

  • The transport is the plug-in that can actually retrieve or send the message to the external resource. This is can be one of the standard plug-ins such MQTT or HTTP (see Standard connectivity plug-ins for detailed information) or a plug-in written by the user.

Any message passing through from a transport plug-in to the host has common features that will be similar in most implementations. The following diagram describes the common features (metadata and payload) and indicates some of the common values they can take.

Illustration showing metadata and payloads

Codecs may be of various types. The following diagram describes an incoming message transformation:

Illustration of an incoming message transformation

Apama provides a selection of codecs by default for converting from common formats and for use with customer-provided codecs to provide standard transformations. The above diagram uses the following standard codecs:

  • The String codec transforms an inbound message to an Apama string. It converts a buffer (transport side) to a string (host side) by decoding it as UTF-8.

    The outbound flow performs the reverse mapping in the other direction (see the diagram below) and places a string representation of the message into an Apama string.

  • The JSON codec transforms the inbound message to a compatible representation. It converts a string (transport side) to a structured type (usually map) by parsing it as a string JSON object.

    The outbound flow performs the reverse mapping in the other direction (see the diagram below) and transforms a JSON-compatible representation to a JSON string in the payload.

  • Messages to the host are identified as a particular event type by matching patterns on the metadata or message contents. The Classifier codec defines what the metadata.sag.type will be and uses rules that are triggered by values held by a field or fields in the inbound message.

    This does not apply to outbound messages (see the diagram below).

  • The Mapper codec sets up the map fields ready for either the correlator or transport depending upon the direction of the flow. It moves fields in the event around and adds default values to missing fields. It can have different rules for different event types, and it can have symmetric rules or different rules depending on direction. towardsHost is the inbound direction, and towardTransport is the outbound direction. The metadata and payload keys provide the source or destination for the values.

For detailed information (and more codecs), see Codec Connectivity Plug-ins.

The following diagram describes an outgoing message transformation:

Illustration of an outgoing message transformation

Adding the connectivity bundles

You can easily add connectivity bundles, either to a new Apama project or to an existing Apama project. You can do this in Apama Plugin for Eclipse or using the apama_project tool. For more information, see Adding connectivity and adapter bundles to projects and Creating and managing an Apama project from the command line.

To get started quickly, we recommend that you import the genericsendreceive sample as a project into Apama Plugin for Eclipse. This is a simple generic EPL application for sending and receiving test messages, for use with any connectivity plug-in. You can find it in the samples/connectivity_plugin/application directory of your Apama installation. See the README.txt file in that directory for detailed information on how to import and run this sample.

With the above sample, you can add one or more connectivity bundles to the project.

You configure a connectivity plug-in by editing the properties and YAML files that come with the connectivity bundle.

Specfiying the main settings in the properties file

Edit the properties file (with the file extension .properties) which comes with the connectivity bundle to specify the main settings such as hosts, ports, and credentials.

The properties defined in this file are used to replace specific substitution values in the YAML file. In many cases, all the configuration that is required is in the .properties file.

See also Using properties files.

Specifying the settings for the connectivity chains in the YAML file

The configuration file for a connectivity plug-in describes both the chains and the plug-ins making up each chain. It is written in the YAML markup language. See also Using YAML configuration files.

Edit the YAML configuration file (with the file extension .yaml) which comes with the connectivity bundle and specify all required information for creating the connectivity chains.

Find out from the documentation for the standard connectivity plug-ins how connectivity chains are created for the transport you are using. See Standard connectivity plug-ins.

  • If the transport provides a dynamic chain manager, then the chains are created by the transport. In this case, you have to provide one or more dynamicChains definitions in the YAML configuration file so that the chains are then created from these definitions.

  • If the transport does not provide a dynamic chain manager, you have to choose between

    • defining the chains statically in your YAML configuration file using startChains, or
    • creating the chains dynamically from EPL using dynamicChains definitions in the YAML configuration file. Your decision on which transport to use affects which sections of the YAML configuration file you have to edit. See also Static and dynamic connectivity chains.

Sometimes it is necessary to make more changes in the YAML configuration file. For example:

  • You can add custom connectivity transport plug-ins or codec plug-ins that you have built yourself in Java or C++, or have downloaded from the Apama community. See also Configuration file for connectivity plug-ins.
  • You can change the configuration settings for the transport. This depends on whether your transport has a dynamic chain manager that will be either in the dynamicChainManager section and/or under the transport’s name in the startChains or dynamicChains sections of the YAML configuration file. For an example, see Configuring the HTTP server transport.

Controlling how the correlator interacts with a chain

A host plug-in controls how the correlator interacts with a chain. In the YAML configuration file, the host plug-in is the first element in the configuration of a chain. You can configure the apama.eventMap host plug-in, for example, to specify the following:

  • A default channel to send to.
  • A channel to subscribe to for events that are sent towards the transport.
  • A default event type for events that are sent towards the correlator (host).

See also Host plug-ins and configuration and Translating EPL events using the apama.eventMap host plug-in.

Using codecs

You can add one or more codec plug-ins to each chain which is defined in a YAML configuration file.

For example, you can add or configure the Classifier codec to have rules that determine which Apama event types to use for each message that comes from the transport and is sent towards the correlator (host). Alternatively you can use the Mapper codec for more advanced cases, or for simple cases where there is only one incoming event type for which you can set a defaultEventType in the apama.eventMap.

Or you can add or configure the Mapper codec rules to have rules that customize the mapping between the fields in your Apama event definitions and the payload and metadata of the transport messages, and that set default values in case any fields are missing. In some cases, you may wish to write rules to get or set the metadata.sag.type which specifies the Apama event type, or the metadata.sag.channel which specifies the correlator (EPL) channel name.

You can add, remove or configure any other standard codecs you wish to use, such as the JSON codec. For more information, see Codec Connectivity Plug-ins.

Writing EPL

After you have edited the properties file for the connectivity plug-in (and maybe also the YAML configuration file), you have to write some EPL to cover the following main steps:

  • Define the Apama event types for the messages you wish to send or receive.

  • Use monitor.subscribe to subscribe to the correlator channel(s) from which you wish to receive messages from the transport. Add an event listener for these events, perhaps logging the incoming events to check that everything is working.

  • Send events to any correlator channels to which the connectivity chain is subscribed. Keep in mind that the channel names depend on the transport and how it is configured.

  • Call ConnectivityPlugins.onApplicationInitialized once your EPL is ready to receive incoming messages.

    For simple applications, this can be done in the onload() action.

    For real applications, we recommend the following:

    1. Define an event to indicate when the application is fully injected.
    2. Send that event by providing an event (.evt) file, which is always sent by default after all EPL has been injected.
    3. Call ConnectivityPlugins.onApplicationInitialized once that event has been received.

  • If your transport does not have a dynamic chain manager and you wish to create chains dynamically from EPL (rather than statically in YAML), you also have to create those chains using com.softwareag.connectivity.ConnectivityPlugins.createDynamicChain.

Note:

If you use the genericsendreceive sample as recommended in Adding the connectivity bundles, all required EPL code is already available in the SendReceiveSample.mon file.