The File IAF adapter (JMultiFileTransport)

File adapter plug-ins

The Apama File adapter uses the following plug-ins:

• JMultiFileTransport.jar — The JMultiFileTransport plug-in manages the connections to the files opened for reading and writing.
• JFixedWidthCodec.jar or JCSVCodec.jar — These plug-ins parse lines of data in fixed-width format or comma separated value format (CSV) into fields. For details about using these codec plug-ins, see The CSV codec IAF plug-in.
• JNullCodec.jar

These plug-ins need to be specified in the IAF configuration file used to start the adapter. If you add this adapter to an Apama project in Apama Plugin for Eclipse, these plug-ins are automatically added to the configuration file. If you are not using Apama Plugin for Eclipse, you can use the File.xml.dist template file as the basis for the configuration file. See Configuring the File adapter for more information about adding the necessary settings to the adapter’s configuration file.

File adapter service monitor files

The File adapter requires the event definitions in the following monitors which are in your Apama installation directory. If you are using Apama Plugin for Eclipse, the project’s default run configuration automatically injects them. If you are not using Apama Plugin for Eclipse, you need to make sure they are injected to the correlator in the order shown before running the IAF.

1. monitors\StatusSupport.mon
2. adapters\monitors\IAFStatusManager.mon
3. adapters\monitors\FileEvents.mon
4. adapters\monitors\FileStatusManager.mon

For detailed information on the file adapter, see the com.apama.file package in the API reference for EPL (ApamaDoc).

Adding the File adapter to an Apama project

If you are developing an application with Apama in Apama Plugin for Eclipse, add the File adapter as described below.

To add the File adapter

1. In the Project Explorer, right-click the Connectivity and Adapters node and select Add Connectivity and Adapters.

2. Select File Adapter (File adapter for reading and writing to ASCII files). A default name is added to the Instance name field that ensures this instance of this adapter will be uniquely identified. You can change the default name, for example, to indicate what type of external system the adapter will connect to. You will be prevented from using a name already in use.

3. Click OK.

   A File adapter entry that contains the new instance is added to the project’s Connectivity and Adapters node and the instance’s configuration file is opened in Apama’s adapter editor.

   For the File adapter, the adapter editor’s Settings tab displays a listing of General Variables. When first created, it lists variables that are used in the Apama project’s default launch configuration. You can add variables by clicking the Add button and filling in the variable’s name and value.

   For editing other configuration properties for the File adapter, display the adapter editor’s XML Source tab and add the appropriate information.

Configuring the File adapter

Before using the File adapter, you need to add information to the IAF configuration file used to start the adapter. When you add an adapter to an Apama project, a configuration file for each instance of the adapter is automatically created. Double-click the name of the adapter instance to open the configuration file in the adapter editor.

If you are using the Apama adapter editor in Apama Plugin for Eclipse, you can edit or add variables to the General Variables section as displayed on the Settings tab. For other properties, you need to edit the XML code directly; to do this, select the adapter editor’s XML Source tab.

If you are not using Apama Plugin for Eclipse, the configuration file can be derived from the template adapters\config\File.xml.dist configuration file shipped with the Apama installation.

CAUTION:

Before changing any values, be sure to make a copy of the File.xml.dist file and give it a unique name, typically with an .xml extension instead of .xml.dist.

The template configuration file references the adapters\config\File-static.xml file using the XML XInclude extension. The File-static.xml file specifies the adapter’s codecs and mapping rules. Normally you do not need to change any information in this file. The default channel for File adapter events is FILE and the default for transportName is JMultiFileTransport. See The IAF configuration file for more information on the contents on an adapter’s configuration file.

In Apama Plugin for Eclipse, adapters are configured using Apama’s adapter editor. To open an adapter instance, in the Project Explorer, right-click on project_name > Adapters > File Adapter > instance_name and select Open Instance from the pop-up menu.

You can set the variables used by the File adapter in the main Settings tab. Values of the form ${...}, such as ${DefaultCorrelator:port} are set to the correct value automatically by the Apama project’s default launch configuration and do not need to be modified. To configure other properties used by the adapter, edit the XML code directly by selecting the XML Source tab.

If you are not using Apama Plugin for Eclipse, all adapter properties are configured by editing the adapter .xml file in an XML or text editor.

Customize the following properties:

• <logging level="INFO" file="logs/FileAdapter.log"/>

• Set the <classpath> elements:

  <classpath path="@ADAPTERS_JARDIR@/JNullCodec.jar" />
  <classpath path="@ADAPTERS_JARDIR@/JFixedWidthCodec.jar" />
  <classpath path="@ADAPTERS_JARDIR@/JCSVCodec.jar" />
  <classpath path="@ADAPTERS_JARDIR@/JMultiFileTransport.jar" />
  

  Replace @ADAPTERS_JARDIR@ with the actual path to the .jar files. Typically, this is the Apama_install_dir\adapters\lib directory.

  If you are using Apama Plugin for Eclipse, these jar files are automatically added to the classpath in the configuration file and you do not need to replace the @ADAPTERS_JARDIR@ token.

• In the <sink> and <source> elements, replace @CORRELATOR_HOST@ and @CORRELATOR_PORT@ with valid attribute values:

     <apama>
       <sinks>
          <sink host="@CORRELATOR_HOST@" port="@CORRELATOR_PORT@" />
       </sinks>
       <sources>
          <source host="@CORRELATOR_HOST@" port="@CORRELATOR_PORT@"
                  channels="FILE" />
       </sources>
     </apama>
  

  If you are using the adapter in an Apama project, the default launch configuration uses the default correlator host and port settings and you do not need to replace the @CORRELATOR_HOST@ and @CORRELATOR_PORT@ tokens.

• <property name="simpleMode" value="false" />

  Indicate whether or not to start the File adapter in simple mode. In simple mode, the File adapter reads lines from a single file or writes lines to a single file. In non-simple mode, you can use the fixed width or CSV codecs to decode/encode field data. Also, the File adapter can read/write to multiple files and additional controls are available for communication between the adapter and the correlator. Non-simple mode is recommended for most situations. Details about simple mode and non-simple mode are in the File.xml.dist file. If you are using Apama Plugin for Eclipse, switch to the adapter editor’s XML Source tab if you want to view these details or to edit the settings.

Overview of event protocol for communication with the File adapter

The adapters\monitors\FileEvents.mon file defines the event types for communication with the File adapter. The following event types in the com.apama.file package are defined in the FileEvents.mon file. These events enable I/O operations on files. See FileEvents.mon for details about the events that are not described in the subsequent topics.

• OpenFileForReading
• OpenFileForWriting
• FileHandle
• FileLine
• ReadLines
• NewFileOpened
• EndOfFile
• CloseFile
• FileClosed
• FileError
• LineWritten

See also the com.apama.file package in the API reference for EPL (ApamaDoc).

Opening files for reading

The File adapter can read from multiple files at the same time. Send an OpenFileForReading event for each file you want the File adapter to read. This involves sending an event to the channel specified in the adapter’s configuration file, typically FILE, for example:

send OpenFileForReading(...) to "FILE"

See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the OpenFileForReading event.

Opening files for reading with parallel processing applications

If your Apama application implements parallel processing, you may want to increase parallelism by processing the incoming events from the File adapter in a separate, private, context, rather than doing everything in the correlator’s main context. To request that events from the File adapter are sent to the private context your monitor is running in, the monitor should open the file using the com.apama.file.OpenFileForReadingToContext event instead of OpenFileForReading. The OpenFileForReadingToContext event has a field that contains a standard OpenFileForReading event (see Opening files for reading), in addition to a field specifying the context that file adapter events should go to for processing, (which is usually the context the monitor itself is running in, context.current()), and the name of the channel the File adapter is using. When using the OpenFileForReadingToContext event, the OpenFileForReadingToContext event and all other file adapter events must not be sent directly to the adapter, but rather sent to the correlator’s main context, where the adapter service monitor runs. The File adapter’s service monitor is responsible for sending the events that are sent from other contexts to the File adapter, and for sending the events received from the File adapter to whichever context should process them (as specified in the OpenFileForReadingToContext event).

See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the OpenFileForReadingToContext event.

Here is an example of how the OpenFileForReadingToContext event is used:

com.apama.file.OpenFileForReading openFileForReading :=
    new com.apama.file.OpenFileForReading;

... // populate the fields of the openFileForReading event as needed

  // Instead of sending openFileForReading to "FILE", wrap it in
  // the OpenFileForReadingToContext event and send it to the service
  // monitor in the main context.

send com.apama.file.OpenFileForReadingToContext(context.current(),
    "FILE", openFileForReading) to mainContext;

com.apama.file.FileHandle readHandle;
on com.apama.file.FileHandle(
  transportName=openFileForReading.transportName,
  requestId=openFileForReading.requestId):readHandle
{
  // Instead of sending to the "FILE" channel, send it to the main
  // context
  send com.apama.file.ReadLines(openFileForReading.transportName, -1,
    readHandle.sessionId, 20) to mainContext;
...
}

Specifying file names in OpenFileForReading events

In an OpenFileForReading event, the value of the filename field can be a specific file name or a wildcard pattern. However, the filename cannot have multiple wildcards.

Specific filename

When you specify a specific filename in an OpenFileForReading event, when the adapter receives requests to read lines from the file, the adapter reads till the end of the file and waits until more data is available. An external process, or the adapter itself, might write more data to the file if it is open for write at the same time that it is being read. If more data becomes available, the File adapter sends it. If the File adapter receives a CloseFile event, the File adapter closes the file against further reading.

Each time the File adapter reaches the end of the file it is reading, the File adapter sends an EndOfFile event to the correlator. If, during this process, more data was appended to the file, the file operations will continue as normal — that is, the File adapter will send more lines if they were requested. Thus, when reading specific files, file appends are acceptable and have a well defined behavior. However, any other modifications, such as changing the lines that have already been read, may have undefined results. An application can ignore or react to an EndOfFile event. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the EndOfFile event.

Wildcard filenames

Now suppose that in an OpenFileForReading event, the value of the filename field is a wildcard pattern. In this case, the adapter does the following:

1. Opens a new file that matches the pattern
2. Reads that file in its entirety
3. Sends back an EndOfFile event
4. Opens the next file that matches the pattern if one is available

For the application’s information, the File adapter sends back an event when it opens each new file. The NewFileOpened event contains the name of the file that was opened. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the NewFileOpened event.

The order of opening the files that match the wildcard pattern is not specified. Currently, the files are ordered by the modification date and then alphabetically by filename.

If a file that has been previously read is externally modified (while in the meantime, the File adapter is reading from other files that match the wildcard pattern), the file is read again in its entirety. That is, any files that are modified after reading from them will be read again (until the CloseFile is sent). Note that this includes file appends.

Opening comma separated values (CSV) files

An example of defining an OpenFileForReading event that opens a CSV file so that each field is automatically parsed appears below. The additional data required by the CSV codec is stored in the payload dictionary.

com.apama.file.OpenFileForReading openCSVFileRead :=
      new com.apama.file.OpenFileForReading;

   //matches transport in IAF config
   openCSVFileRead.transportName := JMultiFileTransport;

   //the request id to use
   openCSVFileRead.requestId := integer.incrementCounter("FileTransport.requestId");

   //read using JCSVCodec
   openCSVFileRead.codec := "JCSVCodec";

   //file to read
   openCSVFileRead.filename := "/usr/home/formby/stocktick.csv";

   //separator char is a ","
   openCSVFileRead.payload["separator"] := ",";

   //send event to channel in config.
   send openCSVFileRead to "FILE";

Subsequently, when the File adapter receives FileLine events, the adapter stores each field in the data sequence in order. You can access the ones you are interested in.

For details about using the CSV codec, see The CSV codec IAF plug-in.

Opening fixed width files

An example of defining an OpenFileForReading event that opens a fixed width file so that each field is automatically parsed appears below. The additional data required by the Fixed Width codec is stored in the payload dictionary.

com.apama.file.OpenFileForReading openFixedFileRead :=
      new com.apama.file.OpenFileForReading;

   //matches transport in IAF instance
   openFixedFileRead.transportName := JMultiFileTransport;

   //the request id to use
   openFixedFileRead.requestId := integer.incrementCounter("FileTransport.requestId");

   //read using CSV Codec
   openFixedFileRead.codec := "JFixedWidthCodec";

   //file to read
   openFixedFileRead.filename := "/usr/home/formby/stocktick.txt";
   //additional data required to interpret fixed width data

   //sequence of field lengths
   openFixedFileRead.payload["fieldLengths"] := "[6,4,9,9,9]";

   //it is left aligned
   openFixedFileRead.payload["isLeftAligned"] := "true";

   //the pad character
   openFixedFileRead.payload["padCharacter"] := "_";

   //send event to channel in config.
   send openFixedFileRead to "FILE";

Subsequently, when the File adapter receives FileLine events, the adapter stores each field in the data sequence in order. You can access the ones you are interested in.

For details about using the Fixed Width codec, see The Fixed Width codec IAF plug-in.

Sending the read request

After you construct an OpenFileForReading event, send it to the "FILE" channel. For example:

com.apama.file.OpenFileForReading openFileWeWantToRead :=
      new com.apama.file.OpenFileForReading;

   //populate the openFileWeWantToRead event
   //..
   //..
   send openFileWeWantToRead to "FILE";

Sending an OpenFileForReading event from EPL code signals the File adapter to open the file. If the open operation is successful, the File adapter returns a FileHandle event.

The sessionId is the most important field; all communication related to this file references this value.

If the open operation is unsuccessful, the File adapter returns a FileError event.

See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the FileHandle and FileError events.

Requesting data from the file

After your application receives a FileHandle event, it can send a ReadLines event, which signals the adapter to start reading lines from the file. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the ReadLines event.

The sessionId in the ReadLines event must be the same as the sessionId stored in the FileHandle event that the application received when the file was opened.

The adapter tries to read as many lines as specified in the ReadLines event. If the file does not contain that many lines, what the adapter does depends on whether the original OpenFileToRead event specified a specific file or a wildcard pattern. According to that setting, the adapter either waits until the file contains more data, or tries to open a new file to deliver the balance from.

Receiving data

As the File adapter reads the file, it returns FileLine events to your application. Each line is associated with a specific sessionId, and the data is stored within a sequence of strings. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the FileLine event.

Notice that the data field is a sequence of strings, rather than a string. However, when you use the null codec for reading, the sequence contains only one element, which contains the entire line read:

//the whole line is stored in the first element, we used null codec
string line := fileLine.data[0];

For specialized codecs, each field is in a discrete element in the sequence:

//The app knows which field contains the data we are interested in:
string symbol := fileLine.data[0];
string exchange := fileLine.data[1];
string currentprice := fileLine.data[2];
//and so on

After the File adapter opens a file for reading, the file remains open as long as the adapter is running. If you want to close a file, you must send a CloseFile event that specifies the sessionId of the file you want to close. For example, if you want to replace the contents of a file, you must close the file before you send an OpenFileForWriting event. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the CloseFile event.

If there is an error, the File adapter sends a FileError event. Otherwise, the File adapter closes the file and sends a FileClosed event, and then it is available to be opened again for writing or for reading.

Opening files for writing

To open a file for writing, send an OpenFileForWriting event. The definition of the OpenFileForWriting event is similar to the definition of the OpenFileForReading event. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the OpenFileForWriting event.

The procedure for opening a file for writing CSV or fixed width files is effectively the same as for reading. Specify the relevant fields in the payload to describe the format of the file you want to write. When subsequently sending FileLine events, populate the data sequence field with the data for each field.

Again, once constructed, send the OpenFileForWriting event to the "FILE" channel, for example:

send OpenFileForWriting(...) to "FILE"

For fixed width files, you can construct a more complex OpenFileForWriting event in a similar way to that described in Opening fixed width files.

Again, as with reading a file, the File adapter sends a FileHandle or FileError event (see Sending the read request), which your application should listen for, filtering on the requestId for the FileHandle event you are interested in.

Once a FileHandle event has been received, the file has successfully opened and the application can begin to send FileLine events to be written. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the FileLine event.

Notice that the data field is a sequence of strings, rather than a string. This allows you to have the fields you want to write as separate entries in the sequence, and it lets the File adapter format the sequence for writing according to the chosen codec. For the fixed width codec, the number of elements in the sequence should match the number of fields originally specified when opening the file. For the null codec, if the sequence contains more than one element, each field will be written out using a separator defined in the IAF configuration file. This separator can be blank, in which case each element will be written out immediately after the previous one, with a newline after the last element.

The FileLine event is exactly the same as the one received when reading; however, the requestId takes on a more important role. If you specify a positive requestId, your application receives an acknowledgment

When a file is already open for reading, you can write to that file only by appending new data. Of course, you must send an OpenFileForWriting event, and then the File adapter can process FileLine events for writing to that file. You receive a FileError event if the file is open for reading and for writing and you try to write data into the file but not by appending the new data.

Opening files for writing with parallel processing applications

If your Apama application implements parallel processing, you may want to increase parallelism by processing the incoming events from the File adapter in a separate, private, context, rather than doing everything in the correlator’s main context. To request that events from the File adapter are sent to the private context your monitor is running in, the monitor should open the file using the com.apama.file.OpenFileForWritingToContext event instead of OpenFileForWriting. The OpenFileForWritingToContext event has a field that contains a standard OpenFileForWriting event (see Opening files for writing), in addition to a field specifying the context that file adapter events should go to for processing, (which is usually the context the monitor itself is running in, context.current()), and the name of the channel the File adapter is using. When using the OpenFileForWritingToContext event, the OpenFileForWritingToContext event and all other File adapter events must not be sent directly to the adapter, but rather sent to the correlator’s main context, where the adapter service monitor runs. The File adapter’s service monitor is responsible for sending the events that are sent from other contexts to the File adapter, and for sending the events received from the File adapter to whichever context should process them (as specified in the OpenFileForWritingToContext event).

See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the OpenFileForWritingToContext event.

Using the OpenFileForWritingToContext event is similar to using the OpenFileForReadingToContext event. See Opening files for reading with parallel processing applications for an example use of the OpenFileForReadingToContext event.

LineWritten event

After the File adapter writes a line to a file, the adapter sends a LineWritten event. See the com.apama.file package in the API reference for EPL (ApamaDoc) for detailed information on the LineWritten event.

This is useful when you want your application to send FileLine events in a batch to control flow. If you need to do flow control, you would typically set all the requestIds to positive values and send the next FileLine events only after receiving the LineWritten notification for the previous FileLine event you sent. If you do not need to do flow control, you could set requestId=-1 for all but the last FileLine event, but set it to a positive value for the very last FileLine event so you get a single LineWritten notification when everything has been written.

The file remains open for the lifetime of the adapter unless you send a CloseFile event. See also Opening files for reading.

Monitoring the File adapter

You can use the File adapter status manager (FileStatusManager.mon in the adapters\monitors directory) to monitor the state of the File adapter.

The File adapter sends status events to the correlator, some of which are asynchronous (not requested) status messages. This occurs as a result of connection status changes, which happen in response to a file being closed or opened.

For single files, the File adapter sends an AdapterConnectionOpenedEvent when it opens a new file for reading or writing, and an AdapterConnectionClosedEvent when it closes a file. When the File adapter uses a wildcard pattern to open a series of files, in addition to those events, the File adapter sends an AdapterConnectionClosedEvent event after it has read everything in a file, and an AdapterConnectionOpenedEvent event when it opens the next file. This is an analogous pattern to the EndOfFile and NewFileOpened events sent by the adapter itself.