Replaying an input log to diagnose problems

When you start the correlator, you can specify that you want it to copy all incoming messages to a special file, called an input log. An input log is useful if there is a problem with either the correlator process or an application running on the correlator. If there is a problem, you can reproduce correlator behavior by replaying the messages captured in the input log. Incoming messages include the following:

• Events
• EPL
• Java
• Correlator deployment packages (CDPs)
• Connection, deletion, and disconnection requests

If you are unable to diagnose the problem, you can provide the input log to Cumulocity product support. A support engineer can then feed your input log into a new correlator to try to diagnose the problem.

The information in the following topics describes how to generate and use an input log. See also Examples for specifying log filenames.

Regularly rotating log files and storing the old ones in a secure location may be important as part of your personal data protection policy. For more information, see Handling personal data “at rest” in the correlator input log file.

Creating an input log file

To create an input log, specify the following option when you start a correlator:

--inputLog filename[${START_TIME}][${ROTATION_TIME}][${ID}][${PID}].log

You specify the name of an input log file in the same way that you specify the name of a main correlator log file. See Specifying log filenames.

In addition, specify any other options that you would normally specify when you start the correlator.

Rotating an input log file

While the input log can get rather large, most file systems can handle large input logs with no special action on your part. However, you might encounter one of the following situations:

• You want to archive your input logs.
• Your operating system enforces a limit on file size.
• The input log has become too large.

In these situations, you can rotate the input log. Rotating the input log means that the correlator closes the current input log and starts sending messages to a new input log.

You should rotate the input log only when you have a specific need to do so. You do not want to have thousands of input logs in a directory since file systems do not handle this efficiently.

If you plan to rotate input logs, specify the ${ID} tag when you specify the --inputLog option when you start the correlator. For examples, see Examples for specifying log filenames.

To rotate the input log, invoke the engine_management tool and specify the --rotateLogs option. The name of the new input log is the same as the name of the closed input log except that the correlator increments the ID portion of the input log filename by 1. See Rotating all correlator log files.

Performance when generating an input log

When the file system that hosts the input log is fast, generating an input log should not have any noticeable effect on correlator performance in most cases. It is possible to use the input log with connectivity plug-ins (see Using connectivity plug-ins), but the performance impact will be significant for chains using the apama.eventMap host plug-in and any chains that are using small batches of events. Consequently, the recommendation is to always run correlators that send information to input logs. Just make sure you have enough disk space for the input log. You need to monitor repeated use to determine how much space is required.

With the correlator generating an input log, you can implement your application so that it sends a minimum amount of information to the main correlator log file. You do not need to log application information because you can always recover application information from the input log. Implementing an application that sends large amounts of application information to the main correlator log file can negatively impact performance.

Reproducing correlator behavior from an input log

To use an input log to reproduce correlator behavior, you must do the following:

1. Run the extract_replay_log Python utility.
2. Run the replay_execute script that the extract_replay_log utility generates.

Invoking the extract script

The extract_replay_log.py script is in the utilities directory in your Apama/ directory. This should be run using the version of Python 3 installed with Apama.

The format for running the extract_replay_log utility is as follows:

extract_replay_log.py [options] inputLogFile

Replace inputLogFile with the path for the input log you want to extract. If you specify the first input log in a series, the subsequent input logs must be in the same directory as the first input log.

The options you can specify are as follows:

The extract_replay_log utility generates the following:

• A script whose execution duplicates the correlator activity captured by the input log.
• Event files where each one is prefixed with “replay_”.
• EPL and possibly JAR and correlator deployment package (CDP) files where each one is prefixed with “replay_”.

Invoking the replay script

Before you run the replay script, you can optionally edit the generated event files, EPL files, or JAR files to slightly modify the behavior you are about to replay. For example, you might add logging for debugging purposes. However, there are restrictions on what you can change:

• You cannot insert any of the following:
  • calls to integer.getUnique(), rand() or incrementCounter
  • send, emit, spawn...to, or enqueue...to statements
  • context constructors
• You cannot change the number of parseable events sent to the correlator.
• You cannot change the number of event definitions and monitors injected.

Making any of these changes can potentially alter the behavior of later operations.

If you are using the MemoryStore and the correlator reads or writes to a store on disk then to accurately play back execution you must have a copy of that store as it was before the correlator modified it. Also, if you are using the MemoryStore from multiple contexts it is unlikely to replay correctly because the order of interaction with the MemoryStore is not in the input log.

After you have optionally edited the generated files, you are ready to invoke the replay_execute script. The replay_execute script tries to replay the contents of the input log into the correlator running on the default port.

While the correlator exactly reproduces the activity captured in the input log, it can execute the same activity faster during replay than when it was executed originally. This is because the correlator already has all the events it needs to process; it does not have to wait for any events. Replaying a log is typically significantly faster than original correlator activity. It is possible that you will find that the time it takes to replay a log is not much less than the time it took for the original activity. In this case, it is possible you were running too close to capacity during the original run. If that is the case, you risk not being able to keep up with the event flow during regular correlator execution. If you anticipate higher event flow then you should investigate optimizing your application or running it on a faster computer.