Generating code coverage information about EPL files

The correlator can generate “code coverage” information about EPL files indicating which lines have been executed. This is useful for measuring the quality of test cases, discovering lines of EPL code which are not being exercised by any tests, as well as for helping diagnose bugs or understand complex interactions in the EPL.

Recording code coverage information

The recording of code coverage information can be enabled and written (dumped) to disk using management requests, or using an environment variable that automatically writes out a coverage file when the correlator is shut down or when code is deleted from the correlator.

The epl_coverage tool can then be used to merge together the coverage files that have been produced by the correlator and produce summary statistics about how much of each source file is covered, as well as an HTML report where each source line is shown annotated with different colors to indicate which lines are not being covered. For detailed information, see Creating code coverage reports.

Enabling the code coverage feature will disable the compiled runtime, and it will also enable the debugger (Using the command-line debugger) and CPU profiler (see Using the CPU profiler).

Dumping code coverage information using management requests

One way to enable and dump code coverage information is via the -r codeCoverage option of the engine_management tool (see also Shutting down and managing components). You can send the following requests:

Automatically writing code coverage information using an environment variable

It is also possible to start the correlator in a mode where it automatically writes code coverage information to disk when it is shut down or is given an engine_delete --all request (see also Deleting code from a correlator).

This mode is enabled by setting the AP_EPL_COVERAGE_FILE environment variable to the path of a file to which coverage information is to be written. If you do this, the correlator starts in code coverage collection mode with debugging enabled, the compiled runtime disabled and optimizations disabled. On shutdown, it writes the code coverage information to the path specified in the environment variable.

The environment variable can contain replacement tokens in the same format as the correlator log file (see Specifying log filenames). Given that the coverage file is not subject to log rotation, only the ${PID} and ${START_TIME} tags are appropriate.

Example:

export AP_EPL_COVERAGE_FILE=~/mycorrelator.${PID}.eplcoverage
correlator <parameters for starting the application here>
engine_management --shutdown "Clean correlator shutdown from command line"

Of course, the correlator must be cleanly shut down for this to work, as no coverage information is written if the process is terminated without warning. If a dump is triggered by engine_delete --all and more EPL is then injected before the correlator is shut down, all coverage information written by engine_delete is overwritten by later coverage information and is thus lost. However, if engine_delete is immediately followed by a clean shutdown, there will be no new coverage information when the shutdown occurs. Therefore, the file will not be overwritten.

Code coverage and deletion of monitors

The code coverage information for transient monitors, monitors which have died, and monitors which were deleted by name is retained. This enables you to later call codeCoverage dump to get the coverage information.

An engine_delete --all, however, resets all the coverage information. When you set the AP_EPL_COVERAGE_FILE environment variable, the coverage information is automatically dumped during an engine_delete --all.

If you delete a monitor and then reinject the EPL file, then all coverage information for that EPL file is reset for the newly injected file.

Common usage patterns

• Enable code coverage, inject your application and send typical events into the correlator. Then dump a coverage report. This gives you a complete list of code covered by initialization and events being processed in the system.
• Set the AP_EPL_COVERAGE_FILE environment variable before running your test suite. Then collate all the coverage reports. This lets you check that your tests exercise all the code paths.
• Enable code coverage and inject your application. Then disable and enable code coverage (to clear the reporting data). Then send a single event through and dump a coverage report. This lets you see what code is run by a single event.

Creating code coverage reports

The epl_coverage tool takes one or more coverage files that have been output by the correlator’s code coverage feature, merges them together to create a new combined .eplcoverage file (which can be used as input for the tool), and creates a CSV, XML and HTML report of the coverage of each source EPL file. The executable for this tool is located in the Apama/bin directory.

epl_coverage command line

To create code coverage reports, run the following command:

epl_coverage [ options ] file1.eplcoverage  [ file2.eplcoverage... ]

Example:

epl_coverage --output ~/mycoverage --source $APAMA_WORK/projects/myproject"
  --exclude "**/Apama/**/*.mon" *.eplcoverage

When you run this command with the –h option, the usage message for this command is shown.

The --output argument specifies the directory into which the tool writes the output files. If not specified, the current directory is used.

The output includes the following files:

• merged.eplcoverage. A single file containing the combined EPL code coverage information from all the input files. This can be used as input to another invocation of the epl_coverage tool.
• coverage_summary.csv. A summary of the percentage of lines and instructions covered in each source file in the standard “comma-separated values” text format (in the operating system’s local character encoding). This file may be useful for reviewing coverage information in a spreadsheet, or as input for an automated tool that records coverage information as part of a continuous integration build/test system. The file starts with a header line beginning with the hash (#) character which identifies the columns used in the rest of the file. It is recommended that any tool that reads this file should use the header line to identify the contents of each column; this is helpful in case columns are added or reordered in a later release.
• epl_coverage.xml. An XML representation of the combined code coverage information for all the input files. This file is written in a widely-used coverage file format that can be read by many third-party tools (the file format that was popularized by Cobertura, which is a code coverage utility for Java; see also https://cobertura.github.io/cobertura/).
• index.html (and associated .css and .html files). An HTML summary of coverage information, including annotated copies of the source files showing which executable lines are covered. These files are always written in the UTF-8 encoding.

Where possible, you should always specify the directories of the .mon source files using the --source option. The coverage tool uses the source directories to identify the canonical location of each EPL file, which may not be the same as the path it was injected from, especially when initializing a correlator from a correlator deployment directory. This also makes it easier for third-party coverage tools to recognize the paths from the XML report and allows the HTML report to show the source code even when the .mon file was injected from a test output directory that no longer exists.

By default, EPL files are assumed to be in the UTF-8 (or ASCII) encoding. If your EPL files are in a different encoding, you can provide the --source-encoding local option, in which case the EPL files are read in the local encoding instead. Files with a UTF-8 byte-order marker are read in UTF-8 in either case. This affects both locating the correct source file in the provided --source directories, and correctly rendering the source code into the HTML reports.

You can apply filters that remove information about unwanted EPL files (such as test .mon files) from all of the output files, or to restrict which parts of the source directories are to be searched. The --include and --exclude options can each be specified multiple times. They specify file patterns to include or exclude (for example “**/foo/Bar*.mon”). These patterns use the following characters:

• forward slashes (/) to indicate directory separators (on all platforms),
• a single asterisk (*) to indicate any number of non-directory separator characters,
• two asterisks (**) to indicate any number of characters potentially including directory separators, and
• a question mark (?) to indicate a single character.

If no --include argument is provided, the default is to include all file paths, except those that are removed by --exclude arguments.

When the number of coverage input files is large, you can avoid an extremely long command line (which some operating systems do not support) by putting the coverage file list into a newline-delimited UTF-8 text file and providing the path to that file on the command line instead, prefixed with an @ symbol. For example:

epl_coverage "@/foo/bar/coverage_file_list.txt"

Options

The epl_coverage tool takes the following options:

Interpreting the code coverage reports

Many lines in an EPL file do not contain any executable instructions, for example, comments, event definitions (except where they contain actions) and event expressions used to declare listeners. These lines are not marked up by the epl_coverage tool.

Lines that do contain executable code may have one or more executable elements (instructions), and the epl_coverage tool reports whether all or only some of those instructions have been executed. It may therefore be useful to split complex EPL constructs (such as multi-part if statements) over multiple lines as much as possible to make the output clearer as to what is covered. The exact details of how many instructions are on any given line is subject to change and therefore not documented, but information on partial coverage may sometimes be useful for identifying branching constructs where not all branches are covered.

Tip:

Every executable line that is not fully covered has the (!) string in the margin, which makes it possible to jump backwards and forwards between these lines using the Find functionality provided by most web browsers.

The purpose of the coverage information is to provide insight into areas of user EPL that are being missed by test cases. Although it is worth aiming for a high percentage of lines and instructions being covered, it is not always possible to write tests that cover every line. However, as long as someone looks at the lines that were missed, there is no need to worry about having less than 100 percent coverage.

Similarly, the information about partial line coverage can often be useful, particularly for control constructs where it might indicate a missed branch in an if statement, or a while loop condition that always returns false. But it will not always be possible for users to get 100 percent coverage of every line, or (since the internal instructions used by EPL are not documented and may be changed at any time between versions) even to understand the reason why a line was not fully covered in some cases. Cumulocity product support cannot provide explanations for why a given line of EPL was only partly covered.

Examples

The following code snippets illustrate some common cases.

• The following line is partially but not fully covered if a() returns true every time this line is executed, since the instructions for the value of b() are never checked in this case.

  if a() or b() {
  

• The following line is only partially covered unless the test is run with DEBUG logging enabled, since expressions in log statements are only evaluated if the log level is specified.

  log "Hello world" at DEBUG;
  

• Another common example is a stream query that uses an aggregate where nothing drops out of the window while the test is executed. For example, if less than 100 seconds pass after the first E() event, the following line is only partially covered:

  from a in all E() within 100.0 select com.apama.aggregates.sum(a.val) as i
  

  If the test does not have anything drop out of the within window, then you will get amber coverage, as no code to remove a value from the set being aggregated over (by sum) is being executed. This may happen if no events go through this query, or if only less than 20 seconds pass since the first event.

• Any code in an onunload action will never be covered at all, since it is only executed with engine_delete, which also removes the coverage information.

Using EPL code coverage with PySys tests

The Apama installation includes the Python-based PySys test framework and Apama helper classes for PySys.

The Apama helper classes for PySys can enable code coverage recording, and automatically run a coverage report from the .eplcoverage files at the end of test execution, which will help users to create better test cases and to find code paths in their EPL applications that do not have adequate test coverage. To use this feature, start your tests with -XcodeCoverage, for example:

pysys run -XcodeCoverage

For an example project, see the pysys/ directory of the Apama Samples.