Connecting correlators using the command line

To implement engine topologies where multiple correlators send events to each other requires a method of connecting correlators in “pipelined” configurations.

The recommended way to do this is by Setting up connections between correlators in a YAML configuration file. However it is also possible using the engine_connect command line tool described in this topic, or from EPL applications Using the Management interface.

Configuring pipelining with engine_connect

The engine_connect tool allows direct connection of running correlator instances. The executable for this tool is located in the Apama/bin directory.

To configure pipelining, run the following command:

engine_connect [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

engine_connect connects a source correlator (the sender) to a target correlator (the receiver). The target correlator will receive events from the specified channel(s) of the source correlator. Source and target correlators must already be running.

The connection between the source and target correlators is persistent. When one of the correlators stops running, then when that correlator restarts it automatically reconnects with the other correlator.

The tool is silent by default unless an error occurs. For verbose progress information, use the –v option.

Options

The engine_connect tool takes the options listed below.

Note:

Many of these options can also be specified as elements of a YAML configuration file (with different element names). If an option is specified in both the command line and a YAML configuration file, then the command line takes precedence. For further information, see Configuring the correlator and especially the topic Setting up connections between correlators in a YAML configuration file.

Exit status

The engine_connect tool returns the following exit values:

Comparison of legacy and parallel connection modes

Avoid mixing connection modes

Successive command lines that specify the same source/target hosts/ports build on each other. While this makes it possible to mix the legacy and parallel connection modes, you should avoid doing that. Mixing connection modes can cause an event to be delivered twice to the same channel. For example:

engine_connect -tp 15902 -sp 15903 -c channelA -c channelB
engine_connect -tp 15902 -sp 15903 -c channelA -c channelC -m legacy

The result of the first command is that there is one (legacy) connection for sending/receiving events on channelA and channelB. The result of the second command is that there is a dedicated connection for sending/receiving events on channelA and a dedicated connection for sending/receiving events on channelC. Events sent on channelA would be delivered twice: once on the legacy connection and once on the dedicated connection.

Examples

Because you can specify command lines that build on each other, you could set up a connection and add named channels later. You can also unsubscribe the channels you have added so that no events are sent or received. The connection remains and you can re-add channels at a later time. However, until you specify the -c option for a given connection, no events emitted by the source correlator are received by the target correlator. Consider the following command line:

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904

The correlators on host1 and host2 are connected but no channels have been subscribed and therefore no events are sent/received. To send and receive events, specify the -c option as in the following command line:

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -c CHAN1 -c CHAN2

Now the connected correlators can use CHAN1 and CHAN2 to send/receive events. To add another channel, execute this command:

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -c CHAN3

The correlators are now using CHAN1, CHAN2, and CHAN3 to send/receive events. To stop using CHAN2, execute the following command. The correlators continue to use CHAN1 and CHAN3.

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x -c CHAN2

To stop sending and receiving events, execute the following command. Note that the correlators remain connected until one of them stops. There is no penalty for this connection.

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x

In this example, the following command is equivalent to the previous command.

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x -c CHAN1 -c CHAN3

Connection configuration file

engine_connect can take connection information from a file for connecting and disconnecting correlators. A sample of such a configuration file is shown below, which defines the topology shown in Engine topologies.

##### comments are allowed prefixed by a '#' – the rest of the line
##### is ignored
generator:dopey.apama.com:1234
worker1:sleepy.apama.com:1234:generator{chan1,chan2,chan3}
worker2:grumpy.apama.com:1234:generator{chan2}
worker3:sneezy.apama.com:1234:worker1{w1_out}
worker4:bashful.apama.com:1234:worker2{w2_out_1,w2_out_2,w2_out_3}
tallier:happy.apama.com:1234:worker3{result},worker4{result}
watcher:doc.apama.com:1234:generator{chan1,chan2}

Connection configuration file format

Each entry in the configuration file specifies connection information for a single correlator in the deployment. Entries can be specified in any order. The general format of an entry is:

correlator_name[:host][:port][:connection[,connection...]]

where connection is defined as:

correlator_name[ {channel_name[,channel_name...]}]

correlator_name is a symbolic identifier for a correlator, used to identify source correlators in target correlator connection information. It can consist of any combination of characters other than whitespace, colon, comma or open/close brace characters, which are reserved as separators. host and port identify the specific correlator this entry applies to. They can be omitted, in which case the defaults of localhost and 15903 are used respectively.

Following this information are details of all connections to source correlators for the current (target) entry. This information is omitted if no correlators sit “upstream” of the current entry (as with the correlator generator, above). If there are multiple upstream source correlators, each name should be separated by a comma (as with tallier, above, which takes events from worker3 and worker4).

For each connection, it is possible to specify the channel(s) on which the target correlator will listen. If no channels are specified, the target correlator will register to receive all events emitted by the source correlator regardless of channel (as with correlators worker3 and worker4 which register for all events from worker1 and worker2 respectively). One can specify specific channel names by enclosing them in braces and separating multiple channels by commas (as with watcher which registers with generator for all events on channels chan1 and chan2).

In effect, the configuration file is a convenient way of grouping several calls to engine_connect. For example, to set up the connections for the correlator tallier would require two commands using engine_connect:

engine_connect -m parallel –sn sneezy.apama.com –sp 1234 –tn happy.apama.com
    -tp 1234 –c result
engine_connect -m parallel –sn bashful.apama.com –sp 1234 –tn happy.apama.com
    -tp 1234 –c result

Errors in the configuration file

The configuration file can be used to both establish and remove connections in a multi-correlator engine topology. For example, assuming the above file is saved as topology.dat, the following commands will first set up then tear down all the connections specified therein:

engine_connect -m parallel –f topology.dat
engine_connect -m parallel –x –f topology.dat

In each of these cases, engine_connect will exit with non-zero exit status on the first error it detects in the configuration file. An error message will be printed to standard error (stderr).

Re-playing the configuration file

The behavior of engine_connect without the -x option is additive. This means that successive calls to engine_connect will attempt to add the channels specified to any existing connection between the source and target correlator(s). For example, with reference to the configuration file above, these commands:

engine_connect -m parallel –sn dopey.apama.com –sp 1234 –tn sleepy.apama.com
    –tp 1234 –c foo
engine_connect -m parallel –f topology.dat

will first add a connection from correlator generator to worker1 on channel foo, then (from the configuration file) extend that connection so that worker1 also receives all events from generator emitted on channel chan1.

Once a connection is set up between two correlators on a channel, any further attempt to set up that connection on the same channel will have no effect. It is therefore possible to re-play the configuration file by invoking engine_connect without creating duplicate connections. This can be useful if there is an error in the configuration file signaled when engine_connect is called, as the error can be fixed and engine_connect re-run without requiring removal of connections that were successfully set up by the first call to engine_connect.