An IAF configuration file is an essential part of any adapter generated with the IAF.
The configuration file must be formatted in XML, and the Document Type Definition (DTD) for it is iaf_4_0.dtd, which is located in the etc directory of your installation. You may wish to use this DTD in conjunction with your XML editor to assist you in writing a correctly formatted configuration file.
The configuration file is loaded and processed when the adapter process starts. A running adapter can by signaled to reload and reprocess the configuration file at any time, by running the iaf_client tool with the –r or --reload option.
On UNIX platforms a SIGHUP signal sent to the IAF process re-opens the log file.
The root element in the configuration file is <adapter-config>. This must always be defined. Within it a single instance of the following elements must exist:
<transports> - This element defines the transport layer plug-in(s) to be loaded.
<codecs> - Defines the codec layer plug-in(s) to be loaded.
<mapping> - Defines the mapping rules for the Semantic Mapper layer, which are used in the conversion between codec layer normalized messages and correlator events.
Following those elements, there must be at least one of the following elements. It is also possible to specify one of each of these elements:
<apama> - Defines how the IAF connects to the Apama correlator(s).
There are also three optional elements that can appear before these required elements (in order):
<logging> - Defines the log file and logging level used by the IAF.
<plugin-logging> - Defines the log file and logging level used by the transport and codec layer plug-ins.
<java> - Defines the environment of the embedded Java Virtual Machine (JVM) in which any Java codec and transport plug-ins will run.
Each of these elements is discussed in more detail in the following sections.
Including other files
The adapter configuration file supports the XML XInclude extension so you can reference other files from the configuration file. This makes it possible, for example, to keep the transport properties in one file and the mapping properties in another. For more information on XML Inclusions, see https://www.w3.org/TR/xinclude/.
In order to match the DTD, the xmlns:xi attribute must be placed either on the <adapter-config> element (as the name xmlns:xi) or on the <xi:include> element. Apama strongly recommends that you use only relative filenames instead of URLs to remote servers.
The adapter configuration file requires both a <transports> and a <codecs> element.
The <transports> element defines the transport layer plug-in(s) to be loaded, and contains one or more nested <transport> elements, one for each plug-in.
The syntax of the <codecs> element mirrors the <transports> element precisely, and contains one or more nested <codec> elements, each of which defines a codec layer plug-in to be loaded.
The transport and codec elements
The transport or codec layer plug-in that should be loaded is defined by the attributes of the <transport> or <codec> elements:
To load a C or C++ plug-in, there must be a library attribute, whose value is the filename of the library in which the plug-in is implemented. The extension and library name prefix will be deduced automatically based on the platform the IAF is running on. For example, on Windows library="FileTransport" would reference a file called FileTransport.dll; on a UNIX system the library filename would be libFileTransport.so.
To load a Java plug-in, instead provide a className attribute, whose value is the fully qualified name of the Java class that implements the plug-in.
If the optional jarName attribute is also provided, the plug-in class will be loaded from the Java archive (.jar) that it specifies; otherwise the IAF will use the usual classpath searching mechanism to locate the class. See Java configuration (optional) for more information about setting a classpath for use with the IAF.
All <transport> and <codec> elements must also have a name attribute. The name is an arbitrary string used to reference the plug-in within the IAF, and must be unique within the configuration file. Even if the same plug-in was to be loaded more than once inside the same IAF, the corresponding <transport> or <codec> elements would still need to have different names.
The <transport> and <codec> elements can include an optional recordTimestamps attribute. This attribute supports the latency framework feature. The value of the attribute determines the values of the recordUpstream and recordDownstream members of the IAF_TimestampConfig object (C/C++ plug-ins) or TimestampConfig object (Java plug-ins) object passed to the semantic mapper. The attribute takes one of the following values:
none — Do not record any timestamps
upstream — Record timestamps for upstream events only
downstream — Record timestamps for downstream events only
both — Record timestamps for both upstream and downstream events
The default, if the recordTimestamps attribute is not present, is none.
The <transport> and <codec> elements can include an optional logTimestamps attribute. This attribute supports the latency framework feature. The attribute takes a space- or comma-separated list of keywords for its value. Supported keywords are:
upstream — Log latency for upstream events
downstream — Log latency for downstream events
roundtrip — Log roundtrip latency for all events, in a plug-in-specific way
*logLevel* — Set the logging level for timestamp logging. Any of the standard Apama log levels are accepted for this keyword.
The value of this attribute determines the values of the logUpstream, logDownstream, logRoundtrip and logLevel members of the IAF_TimestampConfig object (C/C++ plug-ins) or TimestampConfig object (Java plug-ins) passed to the semantic mapper. If the logTimestamps attribute is not present, the log level defaults to INFO and the other timestamp logging parameters default to false.
Note: Although C/C++ and Java transport and codec layer plug-ins may coexist in the same IAF, using a C/C++ codec plug-in with a Java transport plug-in or vice-versa is not permitted.
Plug-in property elements
<transport> and <codec> elements may also contain any number of <property> elements, which are the mechanism by which plug-in-specific options are configured.
Each <property> element has two attributes: name and value. The syntax of the name and value is entirely determined by the plug-in author. Typically the name, value pairs are not ordered, and there is no constraint on the uniqueness of the names. Most plug-ins treat the name attribute in a case-sensitive manner.
In most (though not all) cases, plug-in authors allow properties to be changed dynamically without restarting the IAF, by using the IAF Client tool to request a reload of properties from the IAF configuration file. See IAF Management – Managing a running adapter I for more information about using the IAF Client.
Example
The transport/codec definition section of an IAF configuration file might look as follows for a C/C++ transport plug-in implemented on Windows by FileTransport.dll with a codec in StringCodec.dll:
The adapter configuration file requires a <mapping> element, which configures the adapter’s Semantic Mapper layer.
The <mapping> element may contain the following optional attributes to support the latency framework:
An optional recordTimestamps attribute. The value of the attribute determines the values of the recordUpstream and recordDownstream members of the IAF_TimestampConfig object (C/C++ plug-ins) or TimestampConfig object (Java plug-ins) object passed to the transport or codec. The attribute takes one of the following values:
none — Do not record any timestamps
upstream — Record timestamps for upstream events only
downstream — Record timestamps for downstream events only
both — Record timestamps for both upstream and downstream events
The default, if the recordTimestamps attribute is not present, is none.
An optional logTimestamps attribute. This attribute takes a space- or comma-separated list of keywords for its value. Supported keywords are:
upstream — Log latency for upstream events
downstream — Log latency for downstream events
roundtrip — Log roundtrip latency for all events
log level — Set the logging level for timestamp logging. Any of the standard Apama log levels are accepted for this keyword.
The value of this attribute determines the values of the logUpstream, logDownstream, logRoundtrip and logLevel members of the IAF_TimestampConfig object (C/C++ plug-ins) or TimestampConfig object (Java plug-ins) passed to the transport or codec. If the logTimestamps attribute is not present, the log level defaults to INFO and the other timestamp logging parameters default to false.
The <mapping> element may contain the following (in order):
An optional <logUnmappedDownstream> element, which specifies a file to which unmapped downstream messages should be logged.
An optional <logUnmappedUpstream> element, which specifies a file to which unmapped upstream Apama events should be logged.
One or more <event> elements, specifying the mapping between Apama correlator events and external messages. Setting up the correct <event> elements is the main part of configuring the Semantic Mapper.
One or more <unmapped> elements, which specify events that will bypass the Semantic Mapper, using a string representation of the entire Apama event.
Note: The order in which <event> and <unmapped> elements appear can be mixed.
Each of these will be discussed in more detail below, after a brief explanation of the operation of the Semantic Mapper.
Semantic Mapper operation
The IAF Semantic Mapper takes as input a set of rules that specify when and how an Apama event can be generated from an external message, and similarly how suitable messages of the correct external format should be constructed from Apama events. These rules are termed an event mapping.
All Apama events must belong to a named event type that defines their structure. On startup, the IAF will parse each <event> element, derive the structure of the event being described, and optionally inject an EPL event definition for it into the correlator. The event mappings are therefore organized by Apama event type, as <event> elements.
When an external message is received from a codec plug-in, the Semantic Mapper will run it past each event mapping sequentially, in the order provided in the configuration file. First it checks whether it matches a set of conditions specified within that mapping. If it does, it proceeds to transform and translate it according to the mapping rules provided. If it does not match the conditions, the Semantic Mapper will move on to the next event mapping. If the message matches against several <event> mappings, only the first mapping is executed unless the breakDownstream attribute is set to false. When this attribute is set to false, all mappings that match are executed.
In the upstream direction, when the Semantic Mapper receives an Apama event, it will already know the type of the event, because this information is part of each event sent out by the correlator. However, it is possible to specify multiple upstream mappings from the same Apama event type; therefore, just as with downstream mappings, the Semantic Mapper will check the incoming Apama event against the conditions defined in each of the mappings for that event type. Just as for downstream mappings, the first matching mapping will be used, unless the breakUpstream attribute is set to false. When this attribute is set to false, all mappings that match are executed.
In both directions it is possible for an incoming event not to match against any event mapping, and in which case no mapping is executed. The <logUnmappedDownstream> and <logUnmappedUpstream> elements allow such messages and events to be logged.
The logUnmappedDownstream and logUnmappedUpstream elements
These optional elements enable the logging of Apama events and codec messages that were not matched by any of the configured mapping rules, before they are discarded by the adapter. This can be useful for debugging and diagnostics.
The <logUnmappedDownstream> element turns on logging of messages from an external event source that were not mapped onto an Apama correlator event, after being received from a codec plug-in; the <logUnmappedUpstream> element enables logging of events from the correlator that did not match the conditions necessary for mapping to an external message that could be passed on to a codec plug-in.
Both elements have a single attribute called file, which is used to specify the filename that the log should be written to.
Note: Due to buffering of files in the operating system, the contents of the log files on disk may not be complete until the IAF is shutdown or reconfigured.
Events
The event element
The <mapping> section contains one or more <event> elements, each of which specifies a mapping between an Apama correlator event type and a kind of external message. Setting up the correct <event> elements is the main part of configuring the Semantic Mapper.
Each <event> element can have the following attributes:
name – This is the name of this Apama correlator event type, and is required.
package – This optional attribute specifies the EPL package of the Apama event; if it is not provided, the default package is used. See Developing Apama Applications for information about packages.
direction – This optional attribute defines whether this event mapping is to be used solely for downstream mapping (from incoming external messages to Apama events), upstream mapping (from Apama events to outgoing messages) or for mapping in both directions. The allowed values for the attribute are upstream, downstream and both. The default value if the attribute is undefined is both.
encoder – Required for a mapping that can be used in the upstream direction (direction =``"upstream" or "both"), but ignored when processing downstream messages. In the upstream direction the attribute specifies the codec plug-in that should be used to process the message, once the translation process is complete. The name supplied here must match the name provided in the <codec> element.
copyUnmappedToDictionaryPayload – This optional Boolean attribute defines what the Semantic Mapper should do with any fields in the incoming messages that do not match with any field mapping. If copyUnmappedToDictionaryPayload is false, then any unmapped fields are discarded. If it is set to true however, they will be packaged into a special field called __payload, implicitly added as the last field of the Apama event type. Fields in normalized events with a value of null will be included in the dictionary with the value set to an empty string. If this attribute is undefined its value defaults to false.
Using copyUnmappedToDictionaryPayload puts all the payload fields in a standard EPL dictionary that is efficient and easy to access. See The event payload for more information about the payload field.
breakUpstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an upstream Apama event to an event mapping.
When set to true, the semantic mapper stops the evaluating process and begins executing the actions specified by that mapping and then goes on to evaluate the next event. This is the default behavior when the breakUpstream attribute is not specified.
When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same event against the other event mappings.
This attribute only affects upstream event processing.
breakDownstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an incoming downstream message to an event mapping.
When set to true, the semantic mapper stops the evaluating process and begins executing the actions specified by that mapping and then goes on to evaluate the next message. This is the default behavior, when the breakDownstream attribute is not specified.
When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same message against the other event mappings.
This attribute only affects downstream event processing.
inject – Determines whether the IAF will automatically inject the event definitions that are implicitly defined by this event mapping into the correlator. The default is false. Injecting events from the configuration files is deprecated. Instead, you should use the -e or --events option of the iaf executable to generate the EPL code for the event definitions and then inject the events (and monitors) during the application’s start-up sequence with the tool that you use to start the correlator, such as Apama Plugin for Eclipse or the Apama command-line tools.
copyTimestamps – This is an optional attribute. If set to true (the default is false), the semantic mapper will add an additional field to the generated event definition to hold timestamp information. The new field will be called __timestamps and will be of type dictionary<integer,float> where the dictionary keys are timestamp indexes and the values are the corresponding timestamps. The timestamp field is inserted before the payload field, so it may be the last or next to last field in the event definition. If timestamp copying is enabled for an event type, all timestamps present in the __timestamps field of a matching upstream event will be copied into a new AP_TimestampSet/TimestampSet object and passed to the upstream codec. Likewise, any timestamps passed to the semantic mapper by the codec will be copied into the __timstamps field of the outgoing downstream event and thus made available to the correlator.
transportChannel – optional. If present, then for upstream events (events leaving the correlator), the channel is put in the NormalisedEvent using the value of the transportChannel attribute.
If present, then for downstream events (events going into the correlator), if the value of the transportChannel attribute is in the NormalizedEvent, then that value from the NormalizedEvent is used as the channel name. It is possible that a subsequent <map> element with an identical transport attribute value could override it.
presetChannel – optional. If present, then for downstream events (events going into the correlator), if no channel has been set by the transportChannel attribute, then the value of presetChannel is used as the channel name.
If transportChannel is set, then that value in the NormalisedEvent can still be used for a normal <map> rule, but it will not appear in the unmappedDictionary (if present).
Thus, it is possible to define either a default channel name per type, or a NormalisedEvent field that the transport will send and receive, and this could be re-using a NormalisedEvent field used by a <map> element.
A typical bidirectional <event> element might look like the following. For downstream, if "CHANNEL" (from transportChannel) is in the NormalisedEvent, then the value of the "CHANNEL" entry is used as the channel name, otherwise "channelB" from presetChannel is used. For upstream, the channel name is placed in the "CHANNEL" entry in the NormalisedEvent.
The <id-rules> and <mapping-rules> elements are described below.
The event mapping conditions
The <id-rules> element defines a set of conditions that must be satisfied by an incoming message for it to trigger the mapping to an Apama event, or to decide how to map an incoming Apama event back to a normalized message. The <id-rules> element contains <upstream> and <downstream> sub-elements, which in turn contain the mapping conditions to be used when the Semantic Mapper is searching for a mapping to use in the upstream or downstream direction, respectively. Each condition is encoded in an <id> element.
Note: Conditions are only required for the directions that the mapping can operate in. For example, a mapping with direction="downstream" does not need any <upstream> id rules, while a mapping with direction="both" must specify both <upstream> and <downstream> id rules. The <id-rules>, <upstream> and <downstream> elements themselves must exist though.
<id> - Each <id> sets a condition on a set of fields contained in the normalized message or Apama event. This element takes up to three attributes; fields, which defines the fields that the condition must apply to; test, which specifies the condition; and value, which provides a value to compare the field value with. The value attribute is only required for relational tests. For example:
specifies that the Stock, Exchange and Price fields must exist if the condition is to be satisfied and the mapping proceed. No value is needed to perform the test in this case.
However, the following example:
<id fields="Exchange" test="==" value="LSE"/>
specifies that the Exchange field must exist and have the value “LSE” for the condition to be satisfied.
Note: The value for the fields attribute is a list of fields, delimited by spaces or commas. This means, for example that <id fields="Exchange EX,foo" test="==" value="LSE"/> will successfully match a field called “Exchange”, “EX” or “foo”, but not a field called “Exchange EX,foo”. You should keep this in mind when you assign field names for normalized events. While fields with spaces or commas in their names may be included in a payload dictionary in upstream or downstream directions, they cannot be referenced directly in mapping or id rules.
The following test conditions may be specified. The first four tests are unary operators that do not need a value attribute. These tests can be applied to multiple fields in the same <id> rule:
exists - The fields exist, but do not necessarily have a value
notExists - The fields do not exist
anyExists - One or more of the fields exist, not necessarily with values
hasValue - At least one of the fields exists and has a value. To test that multiple fields all exist and all have values, use multiple hasValue conditions, one for each field, such as
The remaining tests are binary relational operators that do require a value attribute. Furthermore, these tests can only be applied to single fields:
== - Case-sensitive string equality
!= - Case-sensitive string inequality
~== - Case-insensitive string equality
~!= - Case-insensitive string inequality
While the equality tests will fail on fields with missing values, the inequality tests will pass.
All <id> conditions in an <id-rules> element must be satisfied for the mapping to proceed.
Note: A mapping with no <id> elements will always match. This allows a catch-all mapping to be specified. This should be the last definition.
The id rules that test transport field values function in isolation from each other, that is, as soon as a test id rule fails, the mapper stops looking at subsequent rules. This means there is no way to group together tests against the same field name with an OR condition to see if any of them match. Any type of OR value testing needs to be implemented at the codec or EPL layers, or by creating copies of the entire <event> element for each value to test.
The following is an example of a valid <id-rules> element for a bidirectional mapping. Note that the upstream rules are empty, so this mapping will match any incoming Apama event of the appropriate type:
The <mapping-rules> element defines a set of mappings that describe how to create an Apama event from an incoming message. Conversely they define the mapping from an Apama event to an outgoing message. Each mapping must be defined in a <map> element, which has the following attributes:
apama – This is the Apama event field name to copy the value into (downstream) or to take the value from (upstream). This attribute is optional. In an upstream direction, if the apama attribute is not specified or is provided empty, a field will be created within the external message and set to the value specified by default. Not specifying the apama attribute has no significance in a downstream direction — this line of the mapping will be ignored.
Note:
The IAF does not know what types are injected into the correlator, and will drop events with a Failed to parse warning if the types and order of the elements with an apama= attribute do not match the event definition that was injected into the correlator. Mapping rules that do not specify an apama= attribute are not affected by this.
transport – This defines the external message’s field name to copy the value from (downstream) or to copy the value into (upstream). For a downstream mapping it is possible to define more than one value here; in which case the first encountered is used. This attribute is optional. In a downstream direction, if the transport attribute is not specified or is provided empty, a field will be created within the Apama event and set to the value specified by default. Not specifying the transport attribute has no significance in an upstream direction — this line of the mapping will be ignored.
type – The type of the field in the Apama event type. Any simple correlator type is valid here (string, integer, decimal, float, boolean and location); for complex correlator reference types such as sequence<...> and dictionary<...,...>, specify reference instead. If a field is of reference type, the referenceType attribute can be supplied if needed to define the type (see below). When a field is of reference type, the Semantic Checker passes its string form to and from the codec untouched, and performs no checking upon the validity of the value. Note that fields in the external message are always un-typed character strings, regardless of any type they may have had on the external transport that produced them. Furthermore, the Semantic Mapper does not perform any type “casting” or “coercion” when converting a character string in an external event field to the appropriate Apama type, meaning that the Apama event produced might be invalid and be rejected by the correlator. Conversions in the upstream direction, from the Apama field to a string in the external event, will always succeed. Codec and transport plug-ins should be aware of these rules when working with events that will be, or have been, processed by the Semantic Mapper.
referenceType – This is an optional attribute, but it must be supplied if the attribute type="reference" and the event of which this field is a member has the attribute inject="true" (which is now deprecated) or if the IAF will be run with the -e option in order to generate a EPL file with event definitions. This EPL file is then injected to the correlator (this is the recommended method of injecting events). The value of this attribute must be a valid correlator type. This attribute is only used for the process of constructing the event definitions that are to be injected into a correlator. Note that since this is an XML attribute value, some characters such as angle brackets or quotation marks must be correctly encoded using their XML entity name. For example, a sequence<string> must be written as referenceType="sequence <string>". Note also that when nesting sequences within sequences, a space must be present between the angle brackets to prevent the correlator from parsing this as a bitwise shift.
default – The default value to set the Apama field to if the external field specified in transport is missing. Note that the value provided must be of the type specified in type. For example, a valid string is test or “”, a valid integer is 0, a valid decimal is 0.0 or 0.0d, a valid float is 0.0, a valid boolean is true or false, and a valid location is (0.0, 0.0, 0.0, 0.0).
defaultIfEmpty – Optionally, sets the default value to assign to the field in the Apama event if the external field specified in transport is present but has no value defined. The same type considerations apply as for the default attribute. If this attribute is not defined the value specified by the default attribute will apply for this condition as well. Bearing in mind angled brackets and quotes have to be written as XML entities (see above), for a nested event you need to write default="InnerEvent("")" if you want the default value to be InnerEvent("").
Note that at least one of apama and transport must be specified in a mapping.
The following is an example of a valid <mapping-rules> element:
In a downstream direction, this specifies that the Stock field must be copied over into stockName, Price must be copied into stockPrice, and the first encountered of Volume, TradingVolume or CombinedVolume must be copied into stockVolume. First encountered means the first such instance when the event is parsed left to right.
In an upstream direction, the Apama field values would be copied into external message fields of the names specified. A given field in an upstream message can be generated from several different sources. These are evaluated in the following order:
A <map> rule mapping from a named Apama event field to the transport field.
A value for the transport field in the event payload. See The event payload for more details on using the event payload.
Any default value available from a <map> rule with no corresponding Apama event field.
You may have noticed that while an Apama event must always have its full complement of fields defined and with type-valid values, the same is not assumed of external events.
Note: If multiple upstream mappings for the same Apama type exist, they must all specify all of the fields in the type in the same order, with the same type values.
Tips for writing a codec when using reference types:
A codec is responsible for constructing the string form of any value. This means that if your event contains a sequence<string> then the codec must generate an entry in the normalized event whose value is of the form:
["string value 1", "string value 2", "Value with a \" and backslash \\"]
If the codec generates an event that the correlator cannot parse, the correlator will drop the event and the codec will have no way of knowing. Be careful constructing the event strings.
Similarly, events from the correlator will contain a normalized event entry whose value is the string from of the field’s value, as in the example above. The codec is responsible for parsing these strings.
When writing adapters in Java, Apama suggests you use the classes in the com.apama.event.parser package to parse and construct the strings to send to the Semantic Mapper. If you are writing a C/C++ adapter, the corresponding functions for parsing and constructing strings to send to the Semantic Mapper are found in the AP_EventParser.h and AP_EventWriter.h header files.
If nesting other events in the fields of an event, caution must be exercised regarding package namespaces. Always use the fully qualified event name when referencing it in the string form. Also always ensure that the correlator has the enclosed event type defined before the enclosing event type.
Unmapped events
The unmapped element
The <mapping> section may contain one or more <unmapped> elements, each of which specifies a mapping between the string representation of an Apama event type and a normalized event.
Each <unmapped> element can have the following attributes:
name – This optional attribute specifies the name of the Apama correlator event type to match. If omitted, matches all Apama event types.
package – This optional attribute specifies the EPL package of the Apama event. This attribute can be specified only if the name attribute is also supplied.
transport – This attribute is required; it specifies the field in the NormalisedEvent to map to.
direction – This optional attribute defines whether this event mapping is to be used solely for downstream mapping (from incoming external messages to Apama events), upstream mapping (from Apama events to outgoing messages) or for mapping in both directions. The allowed values for the attribute are upstream, downstream and both. The default value if the attribute is undefined is both.
encoder – Required for a mapping that can be used in the upstream direction (direction =``"upstream" or "both"), but ignored when processing downstream messages. In the upstream direction the attribute specifies the codec plug-in that should be used to process the message, once the translation process is complete. The name supplied here must match the name provided in the <codec> element.
breakUpstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an upstream Apama event to an event mapping.
When set to true, the semantic mapper stops the evaluating process and begins executing the actions specified by that mapping and then goes on to evaluate the next event. This is the default behavior when the breakUpstream attribute is not specified.
When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same event against the other event mappings.
This attribute only affects upstream event processing.
breakDownstream – This optional Boolean attribute defines what the Semantic Mapper should do when it matches an incoming downstream message to an event mapping.
When set to true, the semantic mapper stops the evaluating process and begins executing the actions specified by that mapping and then goes on to evaluate the next message. This is the default behavior, when the breakDownstream attribute is not specified.
When set to false, the semantic mapper executes the actions specified in the mapping and then keeps evaluating the same message against the other event mappings.
This attribute only affects downstream event processing.
transportChannel – optional. If present, then for upstream events (events leaving the correlator), the channel is put in the NormalisedEvent using the value of the transportChannel attribute.
If present, then for downstream events (events going into the correlator), if the value of the transportChannel attribute is in the NormalizedEvent, then that value from the NormalizedEvent is used as the channel name. It is possible that a subsequent <map> element with an identical transport attribute value could override it.
presetChannel – optional. If present, then for downstream events (events going into the correlator), if no channel has been set by the transportChannel attribute, then the value of presetChannel is used as the channel name.
If transportChannel is set, then that value in the NormalisedEvent can still be used for a normal <map> rule, but it will not appear in the unmappedDictionary (if present).
Thus, it is possible to define either a default channel name per type, or a NormalisedEvent field that the transport will send and receive, and this could be re-using a NormalisedEvent field used by a <map> element.
In the following example, for downstream, if "CHANNEL" (from transportChannel) is in the NormalisedEvent, then the value of the "CHANNEL" entry is used as the channel name, otherwise "channelB" from presetChannel is used. For upstream, the channel name is placed in the "CHANNEL" entry in the NormalisedEvent.
An <unmapped> element must have an <id-rules> element that defines a set of conditions an incoming must satisfy in order to trigger the mapping to an Apama event. If the value of the direction attribute of an <unmapped> element is “both” or “downstream,” the <id-rules> element must contain a <downstream> sub-element. The <downstream> sub-element contains conditions to be used by the Semantic Mapper when the message is moving downstream direction. Each condition is encoded in an <id> element.
Each <id> sets a condition on a set of fields contained in the normalized message or Apama event. This element takes up to three attributes; fields, which defines the fields that the condition must apply to; test, which specifies the condition; and value, which provides a value to compare the field value with. The value attribute is only required for relational tests.
The <unmapped> entries behave in the same way as <event> entries — the IAF processes <event> and <unmapped> entries in order, translating events with any that match, and ending at the first entry that has breakUpstream or breakDownstream set to true or not specified (they both default to true).
Connecting to the Apama Correlator
Apama correlator configuration
The adapter configuration file requires an <apama> element, which configures how the IAF connects to the Apama correlator(s). An <apama> element can contain the following elements in the following order:
<sinks> — This element lists the Apama correlators that the IAF needs to connect with in order to inject EPL event type definitions and events. You can specify the following attribute in a <sink> element:
parallelConnectionLimit — optional — The default behavior is that the IAF limits itself to an internally set number of connections with each specified sink. This number scales according to the number of CPUs that the IAF detects on the host that is running the IAF. While this number is usually sufficient, there are some situations in which you might want to change it. For example, if you are trying to conserve resources you might want to limit the number of connections to 1, or if you want to prevent multiple threads from sharing a connection you might allow a higher number of connections than the default allows. See the information below about multiple connections from IAF to correlator.
Each correlator is defined in its own <sink> element:
<sink> — This element defines a correlator that the IAF must send events to. You can define more than one <sink> element. All sinks specified will be injected with any EPL event type definitions that are defined in <event> elements in the configuration file. The following attributes are allowed in <sink> elements:
host — Required. Defines the name or address of the host machine where the correlator is running.
port — Required. Specifies the port that this correlator can be contacted on.
sendEvents — Optional. The default behavior is that all sinks receive all events generated by the Semantic Mapper. To prevent the Semantic Mapper from sending all events to a particular correlator, add sendEvents="false" to the <sink> element that defines that correlator. No events will be sent to that correlator regardless of any channel settings.
<sources> — This element lists the Apama components (usually correlators) from which the IAF can receive events. Each component is defined in its own <source> element:
<source> — This element defines an Apama component that the IAF needs to register with as an event consumer. This enables the IAF to receive any alerts generated by the specified component. The following attributes are allowed in <source> elements:
host — Required. Defines the name or address of the host machine where the Apama component is running.
port — Required. Specifies the port that this Apama component can be contacted on.
channels — Required. Specifies the channels that the IAF should listen on to receive events. An empty string indicates that the IAF receives all generated events. To receive events on only particular channels, specify a comma-separated list of channel names. Do not include any spaces. For example, channels="UK,USA,GER".
disconnectable — Specifies whether or not the IAF can be disconnected if it is slow. If set to yes or true (case insensitive), the IAF can be disconnected. Any other setting specifies that it cannot be disconnected.
It is possible to define the IAF as having only sinks, or only sources, or both, or neither. If the IAF has been started with no sinks and no sources, you would use the engine_connect tool to connect it to a correlator or another IAF.
To improve performance, an IAF transport might use multiple threads to send events to the codec and thus to the Semantic Mapper. If more than one thread is sending events downstream (IAF to correlator) then for each thread, the IAF creates a new connection to each <sink> defined in the configuration file, up to the defined limit. Thus, multiple threads can deliver events in parallel to the same sink. In combination with the channelTransport attribute on events (defined in <event> elements), threads can deliver events to different channels to be received by different contexts. For optimal parallel event delivery, each IAF transport thread should send events on a distinct set of channels. There are no ordering guarantees when different threads deliver events to the same sink.
There is a limit on how many connections to each sink the IAF can create. The IAF logs the limit for the number of connections in the startup stanza. If events are sent on more threads than the number of allowed connections, then the IAF re-uses existing connections, which means that some threads share connections. If a thread terminates, the connection it is using is not closed since it might be in use by another thread. See the information above for the parallelConnectionLimit attribute on the <sink> element.
Disabling Apama messaging and using Universal Messaging instead
Note: Use of Universal Messaging from the IAF is deprecated and will be removed in a future release. It is recommended that you now change any IAF-based adapter configurations using Universal Messaging with a <universal-messaging> element in the configuration file to use an <apama> element to talk directly to the correlator.
A deployed adapter can use the Universal Messaging message bus in place of connections specified in the <apama> element. When you want to use Universal Messaging instead of explicitly set connections do the following:
Add the enabled attribute to the <apama> element, if you have one, and set it to false. For example:
<apama enabled="false">.....</apama>
Alternatively, you can remove the <apama> element.
When the enabled attribute is set to "false" then the entire <apama> element is ignored. In other words, the deployed adapter does not use any connections specified in the <apama> element. Instead, the deployed adapter uses the Universal Messaging configuration specified in the <universal-messaging> element.
The default is that the enabled attribute is set to true. If the enabled attribute is not specified or if it is set to true then the connections specified in the <apama> element are used.
While specifying both an <apama> element and a <universal-messaging> element in an adapter configuration file is permitted, it is not recommended.
Configuring IAF adapters to use Universal Messaging
Note: Use of Universal Messaging from the IAF is deprecated and will be removed in a future release. It is recommended that you now change any IAF-based adapter configurations using Universal Messaging with a <universal-messaging> element in the configuration file to use an <apama> element to talk directly to the correlator. See Apama correlator configuration.
If you are configuring your Apama application to use Universal Messaging, you can configure an IAF adapter to use the Universal Messaging message bus to send and receive events. To do this, add a <universal-messaging> element to your adapter configuration file. A <universal-messaging> element can replace or follow the <apama> element.
A <universal-messaging> element contains:
Required specification of the realms attribute or the um-properties attribute.
Required specification of the <subscriber> element.
Optional specification of the defaultChannel attribute.
Optional specification of the enabled attribute, which indicates whether the deployed adapter uses the configuration specified in this <universal-messaging> element.
Specification of realms or um-properties attribute
Specification of the realms attribute or the um-properties attribute is required.
The realms attribute can be set to a list of Universal Messaging realm names (RNAME) to connect to. You can use commas or semicolons as separators.
Commas indicate that you want the adapter to try to connect to the Universal Messaging realms in the order in which you specify them here. Semicolons indicate that the adapter can try to connect to the specified Universal Messaging realms in any order.
If you specify more than one RNAME, each Universal Messaging realm you specify must belong to the same Universal Messaging cluster. Specification of more than one Universal Messaging realm lets you benefit from failover features. See the Universal Messaging documentation at https://docs.webmethods.io/on-premises/webmethods-universal-messaging for information on Communication Protocols and RNAMEs.
Specification of the <subscriber> element is required. The <subscriber> element must specify the channels attribute. Set the channels attribute to a string that specifies the names of the Universal Messaging channels this adapter receives events from. Use a comma to separate multiple channel names.
Specification of defaultChannel attribute
Specification of the defaultChannel attribute is optional. If specified, set the defaultChannel attribute to the name of a Universal Messaging channel. You cannot specify an empty string. In other words, the value of the defaultChannel attribute cannot be the default Apama channel, which is the empty string.
An adapter that uses Universal Messaging must send each event to a named channel. An adapter that is configured to use Universal Messaging identifies the named channel to use as follows:
If the transportChannel attribute is set for an event type (in an <event> or <unmapped> element), then this is the channel the adapter uses for that event type.
If the transportChannel attribute is not set for an event type but the presetChannel attribute is set, then this is the channel the adapter uses for that event type.
If neither transportChannel nor presetChannel is set for an event type, then the adapter uses the channel set by the defaultChannel attribute in the <universal-messaging> element.
If neither transportChannel nor presetChannel is set and you did not explicitly set defaultChannel and you used Apama Plugin for Eclipse to create the adapter configuration file, then the defaultChannel attribute is set to "*adapter_name* *adapter_instance_id*. For example: "File Adapter instance 3".
If none of transportChannel, presetChannel, or defaultChannel are set and if you did not use Apama Plugin for Eclipse to create the adapter configuration file, then the adapter fails if it tries to use Universal Messaging.
All events sent by the adapter on channels that are Universal Messaging channels are delivered to those channels.
Specification of a value for the defaultChannel attribute affects events that are sent from this adapter to Apama engine clients, and from this adapter to correlators when the adapter connects to that correlator by means of the engine_connect correlator tool.
Specification of the enabled attribute
Optional specification of the enabled attribute, which indicates whether the deployed adapter uses the configuration specified in this <universal-messaging> element. The default is that the enabled attribute is set to true. If the enabled attribute is not specified or if it is set to true, then the configuration specified in the <universal-messaging> element is used.
If enabled is set to false, then the deployed adapter ignores the <universal-messaging> element and does not use Universal Messaging. The deployed adapter uses only its explicitly set connections.
Subscribing to receive events from an adapter that is using Universal Messaging
In each context, in any correlator, that is listening for events from an adapter that is using Universal Messaging, at least one monitor instance must subscribe to the channel or channels on which events are sent from the adapter. For example, if you are using an ADBC adapter, you must include a monitor.subscribe("channelName") command for the corresponding instance of the ADBC adapter. Note that not all adapter service monitors support access from multiple correlators. If this is the case, then only one correlator should run the service monitors for that adapter.
Adapter configuration examples
Following are some examples of <universal-messaging> elements:
Defining Universal Messaging properties for the IAF
Note: Use of Universal Messaging from the IAF is deprecated and will be removed in a future release. It is recommended that you now change any IAF-based adapter configurations using Universal Messaging with a <universal-messaging> element in the configuration file to use an <apama> element to talk directly to the correlator. See Apama correlator configuration.
The IAF provides the UM-config.properties template file in the etc folder of your Apama installation directory. The template is for a standard Java properties file. When you use Apama in Apama Plugin for Eclipse to add Universal Messaging configuration to a project, Apama Plugin for Eclipse copies the UM-config.properties file to the config folder in your project.
A Universal Messaging properties file for the IAF can contain entries for the following properties:
Property name
Description
um.channels.escaped
Specifies whether channel names are escaped (true) or not (false). When set to false, the IAF passes channel names directly to Universal Messaging without escaping. In addition, when the slash (/) and backslash (\) characters are not escaped, they can be used to create nested channels. CAUTION:
The IAF treats slash (/) and backslash (\) as different characters while Universal Messaging treats them as identical characters (Universal Messaging generally changes a backslash to a slash). You must choose to use one of these characters in your application and standardize on this. Use of both characters as path separators will result in undefined behavior.
When escaping is disabled (false), you must be careful not to use characters which are not supported by Universal Messaging (see the Universal Messaging documentation for the most up to date list of supported characters and character sets).
Default: true.
um.channels.mode
Indicates whether Universal Messaging channels can be dynamically created. Specify one of the following: - autocreate
The IAF looks up only channels whose names begin with the specified prefix. If the channel does not exist, it is created. For example, if the default prefix is used, channel names must start with UM_ for the channel to be a Universal Messaging channel.
mixed
The IAF looks up each channel to determine if it is a Universal Messaging channel. If the channel does not exist, it is created only if it has the prefix specified by the um.channels.prefix property.
precreate
Requires Universal Messaging channels to be created by using Universal Messaging Enterprise Manager or Universal Messaging client APIs. The IAF looks up all channels (except the default "" channel) to determine whether they are Universal Messaging channels. If a channel does not exist as a Universal Messaging channel, it is not created.
Default: precreate.
um.channels.prefix
Specifies a prefix for channel names. Channel names must have this prefix to allow dynamic creation. Default: UM_.
um.realms
List of RNAME values (URLs). You can use commas or semicolons as separators. Commas indicate that you want the adapter to try to connect to the Universal Messaging realms in the order in which you specify them here. Semicolons indicate that the adapter can try to connect to the specified Universal Messaging realms in any order.
Every RNAME you specify must belong to the same Universal Messaging cluster.
Default: Required.
um.security.certificatefile
Security certificate used to connect to Universal Messaging. Default: None.
um.security.certificatepassword
Password for the specified security certificate file. Default: None.
um.security.truststorefile
Certificate authority file for verifying server certificate. Default: None.
um.security.user
User name supplied to the Universal Messaging realm. Default: Current user name from the operating system.
um.session.pool
Configures how many Universal Messaging sessions to use. More sessions can increase throughput by allowing events to be sent in parallel, but may consume more CPU. Note that if you are using the SHM protocol to communicate with the broker, you will probably want to limit the number of sessions to 1 or 2, as SHM connections will consume 2 CPU cores for each session.
Default: 8.
For example, a Universal Messaging properties file for an Apama installation running on Windows 64 might contain the following:
The Universal Messaging configuration file for the IAF is encoded in UTF-8.
Communicating with the correlator over Universal Messaging
Note: Use of Universal Messaging from the IAF is deprecated and will be removed in a future release. It is recommended that you now change any IAF-based adapter configurations using Universal Messaging with a <universal-messaging> element in the configuration file to use an <apama> element to talk directly to the correlator. See Apama correlator configuration.
The correlator must be using the Universal Messaging transport connectivity plug-in (see The Universal Messaging Transport Connectivity Plug-in), and this connectivity plug-in must be configured to be equivalent to the Universal Messaging properties you have configured for the IAF.
The optional <logging> and <plugin-logging> elements define the logging configuration used by the adapter. If present, they must appear as the first elements nested in the <adapter-config> element.
The <logging> element configures the logging for the IAF itself, whereas the <plugin-logging> element configures logging for the transport and codec layer plug-ins in the adapter.
Both elements have two attributes:
The level attribute sets the logging verbosity level; it must be one of the strings TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT, or OFF (with the same case).
The file attribute determines the file that logging messages will be written to. This should be a file path relative to the directory that the adapter was started from, or one of the special values stdout or stderr which to log to standard output or standard error, respectively.
If the IAF cannot write to the specified log file, the adapter will fail to start.
Note: If the --logfile and --loglevel options are passed to the IAF executable when this is run, the <logging> and <plugin-logging> elements are ignored.
If logging is not configured explicitly in the configuration file or with command-line options, logging defaults to the INFO level on the standard error stream.
The <logging> element accepts two optional sub-elements, <upstream-events> and <downstream-events> that can be configured to log details of all events sent to and received from a connected correlator, without needing to set the entire IAF to DEBUG level logging. These sub-elements each take a single attribute (level) whose values specifies the logging level to be used to log upstream and downstream events, respectively. If this log level is equal to or greater than the IAF logging level, details of the events will appear in the IAF log file. For example:
In this configuration, upstream events will be logged (because ERROR is greater than WARN) but downstream events will not (because INFO is less than WARN). If either of the upstream or downstream event logging levels is not explicitly set, it will default to DEBUG (so events will not be logged by default, unless the IAF is explicitly configured for DEBUG logging).
Java configuration (optional)
Transport and codec plug-ins written in Java are executed by the IAF inside an embedded Java Virtual Machine (JVM). The optional <java> element allows the environment of this JVM to be configured.
The <java> element may contain zero or more of the following nested elements:
<classpath> – This element adds a single entry onto the JVM classpath, which is the list of paths used by Java to locate classes. Each <classpath> element has a single path attribute that specifies a directory or Java Archive file (.jar) to add to the classpath.
The full classpath used by the IAF’s JVM is made up by concatenating (in order):
the contents of the APAMA_IAF_CLASSPATH environment variable if one is defined,
each of the path entries specified by <classpath> elements, in the order they appear in the configuration file, OR if there are none, the contents of the CLASSPATH environment variable,
the path of the lib/JPlugin_internal.jar file used internally by the IAF.
Additionally, if a jarName attribute is used in the <codec> or <transport> element that defines a plug-in (as in Transport and codec plug-in configuration), the plug-in will be loaded using a new classloader with access to the specified Java Archive in addition to the JVM classpath.
You should make sure that all shared classes are in a separate jar that is specified by a <classpath> element. The shared classes are then loaded by the parent classloader. This ensures that when a codec or transport references a shared class, they both agree it is the same class.
<jvm-option> – This element allows arbitrary JVM command-line options to be specified. The JVM option should be placed between the start and end jvm-option tags. For example:
<jvm-option>-Xmx256m</jvm-option>
See the usage screen of the JVM’s Java executable for a full list of supported options.
<property> – This element specifies a Java system property that should be passed to the JVM. It has name and value attributes, such that using:
<property name="propName" value="propValue"/>
is a shorthand equivalent to:
<jvm-option>-DpropName=propValue</jvm-option>
See The IAF runtime for a description of how the IAF selects the JVM library to use.
The properties specified in the <java> element cannot be changed once the JVM has been loaded by the IAF. This will occur when the IAF reads a configuration file that specifies a Java transport or codec plug-in. If the same IAF process is later reconfigured to use only C/C++ transports, the JVM will not be unloaded. The IAF will log a warning message if a reconfiguration of the IAF process attempts to change the previously configured JVM properties.
