Overview of Deploying Apama Applications

About Deploying Apama Applications with an Ant script

It is possible to deploy Apama applications an Ant script, making use of the Ant macro definitions provided in the apama-macros.xml file. You can find this file in the etc directory of your Apama installation. See the comments in that file for more detailed information about the available macros.

Note: Use of the Apama Ant macros is deprecated and will be removed in a future release.

Apama Plugin for Eclipse has an Ant export wizard that can generate a simple Ant script for deploying your Apama project. See Exporting to a deployment script for more information.

Instead of using Ant to inject the EPL, you can also do this using a YAML configuration file (see Deploying Apama applications with a YAML configuration file). If a YAML configuration file is used for this purpose, then Ant should not inject the EPL.

About Deploying Apama Applications with Docker

The recommended way to deploy Apama in production enviroments is using a containerized deployment using Docker, Kubernetes or other OCI-compatible systems. Information about deploying Apama in Kubernetes can be found in the About Deploying Apama Applications with Kubernetes section.

Introduction to Apama in Docker

Below is a brief overview of Docker, however, no familiarity with Docker commands is assumed in this part of the documentation. You need to install Docker before you can make use of the Apama images. See https://docs.docker.com/get-started/ for a more detailed overview of Docker and how to use it.

Images

Images are the base units of the Docker system, providing templates that are used to create the containers, which in turn form the running applications. Docker software runs on various operating systems and can run containers built from the Apama images.

Note: Apama supports building images on Linux only.

Docker

Docker is a technology that allows an organization to remove the complexity of configuring and deploying software applications within the infrastructure it uses. The Docker platform achieves this by running the applications in what are called containers, which isolate the environment of an application from the deployment environment and provide tools for portability and scalability. As long as the organization infrastructure can run the Docker software, it can run the containers and consequently any applications contained in them. Use cases for Docker containers include modernization of legacy applications, migration to a cloud infrastructure, services, and continuous integration and deployment.

Docker Compose

Docker handles single containers and getting them built and deployed. It is best practice for a container to perform one role, but often systems or services are composed of multiple applications interacting with each other. Docker Compose is the method for handling multiple containers and their interdependencies. It is an application that marshals the build, deployment, and runtime characteristics of one or more containers using a configuration file to control the process.

Docker swarm

Docker allows clusters of machines running Docker to be grouped into swarms. Swarms allow the containers to be distributed and load-balanced seamlessly using the Docker command line.

Docker stack

A Docker stack deals with deployment, runtime characteristics, containers and their dependencies. This is called orchestration. A Docker stack is a group of interrelated services that share dependencies and can be orchestrated and scaled together. Similar to the swarm, a Docker command line provides a way to manage and interact with stacks.

Apama and Cumulocity images

Cumulocity has a presence on the Amazon ECR Public Gallery image repository. Several images are available for Cumulocity products. The offering from Apama is an image which will run an instance of the correlator application. See https://gallery.ecr.aws/apama/ for the published images.

Alternatively, images for several products, including Apama, may be built from an installation using scripts included in the installation.

Published Apama container images

Cumulocity produces several different variations of the Apama product as Docker images, depending on your use case. Each Apama image has a corresponding builder image for use in multi-stage builds.

For more details of the content of each image, see the descriptions on the appropriate Docker Hub page. You must ensure that you select the appropriate image containing the components you require.

For information on how to use the builder images in a multi-stage build, see Building Apama projects during the Docker build.

Licensing Apama in Docker

For on-premise installations of Apama, license files are typically added at installation time. However, the Docker image does not come with a license file by default. You have to provide it by yourself. There are several ways to accomplish this, depending on your deployment architecture and whether you may have different license files or may need to update them while the correlator is running.

The Docker server mounts the license file at runtime

The simplest method is to provide the license file on the Docker server when it launches correlator containers. To do this, you mount a copy of the license file into the correlator container at the standard location of /apama_work/license/ApamaServerLicense.xml. You can either do this from a local file or using a cluster-wide shared file system.

To do this with a simple docker run command from a node-local file system, run:

docker run \
  -v/path/to/ApamaServerLicense.xml:/apama_work/license/ApamaServerLicense.xml \
  imagename

Mounts can also be provided in Docker Compose, Docker Stack or Kubernetes orchestration files. See the documentation for your orchestrator for more details.

The Docker server provides the license file via a configuration object

Both Docker Stack and Kubernetes support creating configuration objects in the stack, whose contents can be provided as a file mounted into the container at runtime.

Using Docker Stack, you need to create a stack configuration file which defines configuration. For example:

configs:
   apama_license:
      file:./ApamaServerLicense.xml

Then you need to load the configuration to a particular location in your container:

services:
   apama:
      image: imagename
      configs:
         - source: apama_license
           target: /apama_work/license/ApamaServerLicense.xml

The application image contains the license file

Alternatively, if you do not need to provide different license files at runtime (for example, while developing or testing), you can compile the license file into your application image. To do this, add a COPY line to your project Dockerfile which copies the license file to the standard location. For example:

COPY --chown=sagadmin:sagadmin ApamaServerLicense.xml \
  $APAMA_WORK/license/ApamaServerLicense.xml

If you ever need to change the license file, you will need to rebuild your application image with the new license file.

The project contains the license file

You can also store the license file inside your Apama Plugin for Eclipse project and load it via configuration, rather than having it picked up automatically by the correlator. This also has the disadvantage of not being able to have a different license file in production, but means that your license file is automatically available wherever you run the project, whether in Docker or not, even if you did not initially install Apama with the license file.

To do this, you need to add a section to the Apama configuration file in your project, with a relative path to the license file which has been added to your project. For example:

correlator:
   licenseFile: ${PARENT_DIR}/../ApamaServerLicense.xml

With this option, the license file and configuration will automatically be included in the application image using the default Dockerfile provided by Apama Plugin for Eclipse.

For more details, see YAML configuration file for the correlator.

Quick start to using an Apama image

The Apama image that we are using here is available on Amazon ECR Public Gallery (https://gallery.ecr.aws/apama/apama-correlator/). You must purchase it (free of charge) so that you can use it. After you have done this, proceed as described below.

1. Log in to Amazon ECR Public Gallery using your credentials:

   docker login
   

2. Obtain the image from the repository so that it will be available to use:

   docker pull public.ecr.aws/apama/apama-correlator:version
   

   where version is the current Apama version number (a two-digit number such as 10.15).

   Note:

   If you logged in successfully but get an error during the docker pull command, it is likely that there is a spelling mistake or that the image you specified is missing.

3. List the images available to you:

   docker images
   

   You should see the image that you pulled in the output of the above command (keep in mind there may be many images if you are using a shared machine).

4. Create a container using the image and run it in Docker:

   docker run -d --name container --rm public.ecr.aws/apama/apama-correlator:version
   

   The above command will “detach” after running the container. You can see it running using the following command:

   docker ps
   

   You can interact with the running image in many ways (see the Docker documentation for the appropriate commands), but we will now retrieve the logs from the running container and then stop the container.

5. To examine the log of the running container, enter the following command using the name that was set with the --name container argument of the docker run command.

   docker logs container
   

   When the above command is executed, the container identifier or the name which you can find in the output of the docker ps command is taken as a parameter.

6. You can now stop the container:

   docker stop container
   

   This also removes the container since it was started with the --rm option.

Apama image for Cumulocity microservices

If you are building a microservice for use within Cumulocity, then we provide a smaller base image designed for use in that environment (https://gallery.ecr.aws/apama/apama-cumulocity-jre). You can use the public.ecr.aws/apama/apama-cumulocity-jreversion image anywhere that you could use apama-correlator. Existing Dockerfiles which use apama-correlator but provide APAMA_IMAGE as a build argument can be built using apama-cumulocity-jre instead, using an argument to the docker build command:

docker build -t projectimage \
 --build-arg APAMA_IMAGE=public.ecr.aws/apama/apama-cumulocity-jre:version projectdir

The base image for Cumulocity only provides a JRE and not a full JDK. It does not have Python support built-in and it only provides connectivity with Cumulocity and not any other product.

Running as the sagadmin user within a container

Cumulocity images are configured not to run processes as root by default within containers. This is standard container practice. All images create a user sagadmin with the user identifier and group identifier of 1724. By default, all processes run in the container, and all RUN commands in Dockerfiles using these images as a base run with that user identifier.

COPY commands may need to specify writing as the sagadmin user via the --chown argument. It is recommended to continue using this user for all of your commands within Docker.

If you need to run as another user, you need to add USER statements to your Dockerfile or the appropriate options to your docker run command.

Building an Apama image from the current installation

As an alternative to using the image from Amazon ECR Public Gallery, the Apama installation provides a Dockerfile which enables you to build an image. You can find it in the samples/docker/image directory of your Apama installation. See the README.txt file in the samples/docker directory for detailed instructions.

1. To build the image, run the following command from your installation directory (by default, this is /opt/cumulocity):

   docker build --tag registrytag -f./Apama/samples/docker/image/Dockerfile.
   

   If you wish to publish the image to a registry, then registrytag should be of the following form:

   registryhost/organization/imagename:tag

   Note that the image will be stored locally, but it can be published to your registry as described below.

2. Enter the following command to publish the image:

   docker push registrytag
   

3. Enter the following commands to use the image:

   docker pull registrytag
   
   docker run -d --name container --rm registrytag
   
   docker logs container
   
   docker stop container
   

   You can now use the image in the same way as described in Quick start to using an Apama image, by referring to the registry name on the command line. You can also refer to it in the Dockerfile; see Apama samples for Docker for examples of how to refer to images in Dockerfiles.

Deploying an Apama application in Docker

The examples below reference the image in Amazon ECR Public Gallery. However, you can also use an image built from your Apama installation; the results will be the same.

The base image simply provides an empty running correlator, with the management port exposed. You can use the command line tools described in Correlator utilities reference to inject EPL and manage the correlator, or you can connect to the running container and use the management commands within the container. The expected use case is the deployment of a user application in an image derived from the base image.

1. Typically, the first step is to create the Dockerfile that references the required image of the Apama correlator. The following is an example Dockerfile:

   # Reference the Apama image at public.ecr.aws/apama/apama-correlator:<TAG>.
   # The tag refers to the version number of the Apama correlator.
   FROM public.ecr.aws/apama/apama-correlator:version
   
   # Copy files from the local app directory to /app in the resulting image.
   COPY --chown=sagadmin:sagadmin app/* /app/
   
   # This is the command we run - it references the internal directory.
   CMD ["correlator","--config","/app"]
   

   The above example file produces an image that contains a root level directory called /app that has copies of local files in it. These files will be owned by the user the correlator runs as, called sagadmin.

2. When the correlator image runs, it is directed to read the configuration file init.yaml that is present within the directory. An example of this from the Simple sample application is shown below. It can be found in the samples/docker/applications/Simple directory of your Apama installation, along with the HelloWorld.mon file that it references.

   Example init.yaml:

   correlator:
       initialization:
           list:
               - ${PARENT_DIR}/HelloWorld.mon
   

   At correlator startup, the above configuration file injects the monitor file HelloWorld.mon from the same directory. The image build process will copy the HelloWorld.mon and init.yaml files into the image. Thus, the application image can run without outside dependencies.

3. You build the image using the docker build command. The following example assumes that the command is run from the directory containing the Dockerfile. See the Docker documentation for details on the available options.

   docker build --tag organization/application.
   

4. Once the image is built, it is stored locally. You can view it using the docker images command. Note that the image is not running at this point.

   docker images | grep application
   

5. To run the image as a container, enter the following command:

   docker run -d --rm --name container organization/application
   

   The options -d and --rm are used to detach and remove the container after shutdown.

6. To examine the running process, enter the following command:

   docker ps
   

7. To examine the logs, enter the following command:

   docker logs container
   

8. To stop the container, enter the following command:

   docker stop container
   

   Now the docker ps command should no longer show your container.

9. To remove the image, enter the following command:

   docker rmi organization/application
   

   This untags and deletes the image.

Note that the application is “baked” into the image you create. The files copied into the image will not change and any output will not persist outside the container. The management port is exposed, and therefore the correlator can be manipulated remotely through the Management interface (see also Using the Management interface). However, you can interact with the running container in many ways, including starting a shell, examining logs, and running commands in the container. See the Docker documentation for details on the available options.

Developing an Apama application using the Docker image

The Dockerfile that you created in Deploying an Apama application in Docker fixes or bakes in the app directory and its contents. Thus, you would have to recreate the image every time the configuration file or the EPL files change, which is inconvenient during the development phase of the application where the EPL changes frequently. You can address that by mapping in a local directory when running the container and use that to hold your application. For example:

docker run -d --rm --name container -v /local/path:/app \
  public.ecr.aws/apama/apama-correlator:version correlator --config /app

Note:

The above command is broken over two lines via the \ escape character at the end of line. You can either copy this line verbatim or construct it to be a single line command.

Keep in mind that you are using the official image and not creating your own. Because you are mounting in a local directory containing the application, there is no need to build a custom image. It will be read from the /local/path which is mounted in instead.

Now you can make changes to the EPL or init.yaml file and simply restart the container to pick up the changes. When you have completed your changes and want to deploy, you can add a Dockerfile to bake the files into the image again. It is possible to define and map multiple volumes, allowing flexible usage of the image.

Building Apama projects during the Docker build

Some Apama applications, particularly those developed using Apama Plugin for Eclipse or those with custom plug-ins, require additional build steps when creating a Docker image, such as running the engine_deploy tool (see also Deploying a correlator). For those using the sample packaging kit to create base images from an installation, this can be done directly in a standard Dockerfile. For those using the official Amazon ECR Public Gallery image, a second builder image can be used for project build steps. This is in order to keep the runtime image as small as possible. The builder image can be used as part of a Docker multi-stage build.

A typical multi-stage Dockerfile looks like this:

FROM buildbase as builder

COPY source /source
RUN buildstep

FROM runtimebase

COPY --from=builder /buildoutput /buildoutput

CMD ["/buildoutput"]

For a typical Apama Plugin for Eclipse-based Apama project, your Dockerfile looks something like this:

ARG APAMA_VERSION=version
ARG APAMA_BUILDER=public.ecr.aws/apama/apama-builder:${APAMA_VERSION}
ARG APAMA_IMAGE=public.ecr.aws/apama/apama-correlator:${APAMA_VERSION}
FROM ${APAMA_BUILDER} as builder

COPY --chown=sagadmin:sagadmin MyProject ${APAMA_WORK}/MyProject
RUN engine_deploy --outputDeployDir ${APAMA_WORK}/MyProject_deployed \
${APAMA_WORK}/MyProject

FROM ${APAMA_IMAGE}

COPY --from=builder --chown=sagadmin:sagadmin ${APAMA_WORK}/MyProject_deployed \
${APAMA_WORK}/MyProject_deployed

WORKDIR ${APAMA_WORK}

CMD ["correlator", "--config", "MyProject_deployed"]

In Apama Plugin for Eclipse, you can add Docker support to your project as described in Adding Docker support to Apama projects. When you do this, a Dockerfile similar to the one above is automatically created in the project. Therefore, a project with Docker support can be built into an image using the following command:

docker build MyProject

For most projects, the provided Dockerfile will be sufficient. If you have additional build steps (such as building custom plug-ins), you can add them to the Dockerfile in your project. A default Dockerfile with the name Dockerfile.project is provided in the etc directory of your Apama installation. You can copy this file manually into the root of any project which can be deployed using the engine_deploy tool.

Also, note the use of build arguments in the Dockerfile. This allows you to use --build-arg to specify the name of an alternative builder or runtime image. If you want to use the automatically generated Dockerfile with your own image created from the packaging kit, you need to set the build arguments appropriately:

docker build -t appimage --build-arg APAMA_BUILDER=apamaimage
  --build-arg APAMA_IMAGE=apamaimage MyProject

Alternatively, you can just change the version of Apama that is used from Amazon ECR Public Gallery:

docker build -t appimage --build-arg APAMA_VERSION=version

Note:

Each time you import an Apama project from a previous version into the current version, you have to update the version in the Dockerfile, or you have to run docker build with the appropriate build arguments to override the version. Apama Plugin for Eclipse will warn you if your Dockerfile is not up to date with the current version.

By default, the Dockerfile added by Apama Plugin for Eclipse uses the public.ecr.aws/apama/apama-builder and public.ecr.aws/apama/apama-correlator images, which do not contain components for connection to JMS, Cumulocity, or other third party products. If your project uses this functionality, you may need to use one of the other pairs of published images instead, using the above-mentioned build arguments. For more details of the images available, see Published Apama container images. For exact details of the contents of each image, see the corresponding pages on Amazon ECR Public Gallery. However, in general, the builder images contain the following additional tools:

• engine_deploy - to convert a project (either from Apama Plugin for Eclipse, or a collection of monitors created outside of Apama Plugin for Eclipse) into a configuration directory which can be used to start a correlator. See also Deploying a correlator.
• engine_package - to create CDP (correlator deployment package) files from monitor and event files. See also Packaging correlator input files.
• apama_project - to manipulate an Apama project outside of Apama Plugin for Eclipse. See also Creating and managing an Apama project from the command line.
• PySys - to run automated tests on your application before deployment. See also the API reference for Python.

The image also contains a Java compiler. It does not by default contain a C++ compiler. If you want to compile C++ code, then you need to install a C++ compiler as part of your build step using a multi-stage build. This is only included while you are building, not in the final image, as long as you do it in the build part of a multi-stage build.

ARG APAMA_BUILDER=public.ecr.aws/apama/apama-builder:version
ARG APAMA_IMAGE=public.ecr.aws/apama/apama-correlator:version
FROM ${APAMA_BUILDER} as builder

COPY --chown=sagadmin:sagadmin MyProject ${APAMA_WORK}/MyProject
RUN engine_deploy --outputDeployDir ${APAMA_WORK}/MyProject_deployed \
${APAMA_WORK}/MyProject
RUN yum install gcc make && make -C MyProject

FROM ${APAMA_IMAGE}

COPY --chown=sagadmin:sagadmin --from=builder \
${APAMA_WORK}/MyProject_deployed ${APAMA_WORK}/MyProject_deployed
COPY --chown=sagadmin:sagadmin --from=builder \
${APAMA_WORK}/MyProject/libMyLib.so ${APAMA_WORK}/lib/libMyLib.so

WORKDIR ${APAMA_WORK}

CMD ["correlator", "--config", "MyProject_deployed"]

Using Docker Compose with Apama

Docker Compose uses a configuration file to define how to build and then how to run the one or more images and containers it defines. Where you have previously used individual commands for each image and container, you can now use Docker Compose to initiate the build or running of systems of containers.

A simple sample is provided in the samples/docker/applications/Simple directory of your Apama installation. Using that sample, you can see the basic workflow for all of the samples.

When run, the docker-compose command reads a configuration file that defines what it will build and how it should behave. This configuration file is named docker-compose.yml by default. It is recommended that you read the Docker documentation for the docker-compose command to get details on what the entries in the configuration file are.

It is important to note that the correlator may read all configuration with the extension of .yaml contained in a specified directory. You must take care over the placement and naming of any files in YAML format with an extension of .yaml to avoid unexpected behavior in the correlator. The following command uses the docker-compose.yml file in the current directory to build all the images needed for the containers defined in the docker-compose.yml file and then runs them as directed.

docker-compose up -d --rm

When run in the Simple sample directory, the above command builds and then runs the sample. You can use the following command to see output from the correlator running a simple “Hello world” application in the container:

docker-compose logs

To stop the container, enter the following command from the Simple sample directory:

docker-compose down

For more complex examples, see Apama samples for Docker. These examples demonstrate communications between containers, persisting container data, sharing between containers, and exposing the containers as services.

Apama samples for Docker

There are a number of samples that can be found in the samples/docker/applications directory of your Apama installation. The Simple sample that is referenced in Using Docker Compose with Apama contains a “Hello world” application. The other samples in the above directory cover more complex use cases. These samples build upon the base image and demonstrate how to use a dockerized Apama correlator to build your own application or service.

The README.txt files that are provided in the samples/docker/applications directory and in each of the individual sample subdirectories guide you through the process of building and running. The samples demonstrate various ways in which Docker can be used to deploy Apama. They also demonstrate interactions between Docker containers and normal applications such as dashboards.

Weather

This sample deploys Apama’s Weather demo. It demonstrates that various Apama components can be run in distinct containers. The sample creates a correlator container and a separate container for the dashboard server which connects to the correlator just as in the non-Docker Weather sample. This sample also creates a Compose network called “front” which is used to allow both containers to communicate.

You can view this network using the following command:

docker network ls

Adapter

This sample starts an IAF in a container, which connects to a correlator running in another container. The IAF is running the File transport. An EPL application is deployed into the correlator, which requests the contents of a file inputFile.txt from the File adapter, and then directs it to write those contents back out to another file outputFile.txt.

The output file is stored within the container, but can be retrieved via Docker:

docker cp adapter_iaf_1:/apama_work/Adapter/outputFile.txt.

MemoryStore

Earlier samples showed how to use Dockerfiles to create derived images that give Apama components from the base image access to configuration and EPL code. However, this approach only suffices for static data that can be reproduced by copying in files from a canonical source. For containers to share dynamic data, they need a live view on that data. This is provided by a Docker feature called “volumes”, which allows containers to share parts of their file system with each other.

This sample contains a toy application MemoryStoreCounter.mon that makes use of the MemoryStore to lay down persistent state on disk in the form of a number that increments each time the monitor is loaded. While the correlator container is the one reading and writing to the MemoryStore, the persistent file is on a Docker volume which is persisted between container restarts.

You can view the created volume using the following command:

docker volume ls

Docker volumes give you the ability to manage data that has a different lifecycle to the container that uses it. In an application like this, you can replace the other containers with equivalents that are based on a newer version of Apama. After bringing them up again, the correlator will still have access to the MemoryStore data that it wrote in a previous iteration, as this data is owned by the volume.

Universal Messaging

This sample demonstrates two Apama correlators communicating with each other via a Universal Messaging realm server, all in separate containers. This sample requires an installation of Universal Messaging.

Secrets

This sample demonstrates how to use Docker secrets to set variables in correlator configuration files. These can then be loaded at runtime into a correlator.

Using the Apama image with the Docker stack

Current versions of Docker include swarm mode for natively managing a cluster of Docker engines called a swarm. You can use the Docker command line interface to create a swarm, deploy application services, and manage behavior. See the Docker documentation for available commands and more detailed information regarding swarms and their management.

The top of the hierarchy of distributed applications is the stack. A stack is a group of interrelated services that share dependencies, and can be orchestrated and scaled together. A single stack is capable of defining and coordinating the functionality of an entire application (though very complex applications may want to use multiple stacks).

Using stacks is an extension of creating a Compose file and then using the docker stack deploy command. The majority of samples described in Apama samples for Docker use single service stacks running on a single host, which is not usually what takes place in production.

1. Use the following command to enable swarm mode:

   docker swarm init
   

   The init command outputs a token which can be used to add extra processing workers to the swarm using the following command:

   docker swarm join --token token workeraddress
   

2. The docker-compose.yml configuration file is the key element to creating the deployment. To use stack commands, the version of the configuration file must be greater than 3. The image specified in the configuration file must exist because (unlike Docker Compose) the stack commands ignore build sections in the configuration file. As a consequence, building the image is a separate step and must be done before attempting to start the application.

3. Use the following command to run the application:

   docker stack deploy stack --compose-file docker-compose.yml
   

   The stack element in the above command is a name which will be prefixed to the elements of the deployment that the above command creates. It is also used in the commands for interrogating the running system, for example:

   docker stack ps stack
   
   docker stack services stack
   

4. Run the following command to cleanly shut down the running application when you are finished:

   docker stack rm stack
   

About Deploying Apama Applications with Kubernetes

Introduction to Apama in Kubernetes

Kubernetes is an open-source system that provides an alternative for orchestrating containers. The Apama images as described in Deploying Apama applications with Docker can be used within Kubernetes, allowing an alternative to deploying and controlling a user application. See https://kubernetes.io/ for a more detailed overview of Kubernetes and how to use it.

Images

Much like Docker, images form the basis of the containers that are run and controlled by Kubernetes. Creating and obtaining images is identical to Docker. That is, you can either get the Apama image from Amazon ECR Public Gallery as described in Quick start to using an Apama image or you build your own image as described in Building an Apama image from the current installation.

The images are templates that are used to create the containers. Kubernetes software runs on various operating systems and can run containers built from the Apama images.

Note: Apama supports building images on Linux only.

Kubernetes

Kubernetes uses a different command line application and terminology from Docker. For example, the simplest unit is a pod. A pod corresponds to a running process on the cluster, but can be more than one container. The command line interface is kubectl and should be used instead of docker. See the Kubernetes documentation for more details on the various command-line options.

Quick start to using Apama in Kubernetes

For this quick start, you have to build the image from the Simple sample that can be found in the samples/docker/applications/Simple directory of your Apama installation. See the README.txt file in the samples/docker/applications directory for detailed instructions.

Once you have built the image, proceed as described below.

1. Define a deployment YAML file which references the image you wish to run as a container. An example of this is the kubernetes.yml file which can be found in the samples/docker/applications/Simple directory of your Apama installation.

   This example YAML file creates a pod. However, Kubernetes can be used to define much more complex objects and behaviors; see the Kubernetes documentation for details of the possible configurations.

2. Use the Kubernetes command line tool (kubectl) to create the container and run it (for example, using the above example YAML file):

   kubectl create -f kubernetes.yml
   

   The create command starts the container under Kubernetes control. You can now use the kubectl command line to interrogate the pod that has been created, see below.

3. To return the name of the pod that corresponds to running your Apama container:

   kubectl get pods
   

4. To examine the logs from the container in the pod:

   kubectl logs podname
   

5. Once you are finished, you can shut down everything and remove the containers. To do so, you use the same YAML file that has been used to start the process:

   kubectl delete -f kubernetes.yml
   

The power of Kubernetes comes with more complex setups involving multiple containers and hosts. Some of these features are covered in more complex samples; see Apama samples for Kubernetes for further information.

Deploying an Apama application using Kubernetes

You will either need to create an image or you will already have images for the application you wish to deploy using Kubernetes. To illustrate the concepts and basic process of getting the image running and interrogating the running containers, this topic describes how to use the Adapter sample which can be found the samples/docker/applications/Adapter directory of your Apama installation. The README.txt files that are provided in the samples/docker/applications directory and in the subdirectory for the Adapter sample guide you through the process of building and running.

1. To create the images, enter the following commands from the samples/docker/applications/Adapter directory (see also Deploying Apama applications with Docker for details on building images):

   cd deployment
   docker build -t registrytag-correlator.
   cd../iaf
   docker build -t registrytag-iaf.
   cd..
   docker push registrytag-correlator
   docker push registrytag-iaf
   

   You must replace the tags registrytag-correlator and registrytag-iaf with tags that are valid for a registry that you can write to, in the following form:

   registryhost/organization/repository:image

   To use Kubernetes, you must publish the images in a registry and not just in a local Docker server. You must also include these tags in the Kubernetes configuration file that is used to create the objects. If you are using the kubernetes.yml file from the Adapter sample (which is mentioned below), you must replace the correlator-image and iaf-image lines with your tags.

2. Once you have determined which images you want to use, and know their location and tag name, you can create the YAML configuration file for Kubernetes. This configuration file defines the state that you want from a running system, indicating the runtime characteristics you want Kubernetes to adhere to. This file makes no mention of where things run on a cluster, but it can be used to determine behavior like restarting, replica containers, load balancing and resource restrictions.

   Apama provides the example configuration file kubernetes.yml which can be found in the samples/docker/applications/Adapter directory of your Apama installation. Detailed descriptions of the possible contents of a Kubernetes configuration file can be found on the Kubernetes website (https://kubernetes.io/).

   The Adapter sample contains two pods and a service tying the running containers together. A service in Kubernetes is an abstraction which enables a loose coupling between dependent pods. Although each pod has a unique IP address, these IP addresses are not exposed outside the cluster without a service. Services allow your applications to receive traffic. Whenever you have a pod which depends on another service, it is recommended that you use an init container to ensure that the service is ready before starting the dependant pod. All the samples involving multiple pods use this pattern.

3. To create the Kubernetes objects (pods and service):

   kubectl create -f kubernetes.yml
   

   After you have created the objects defined in the kubernetes.yml file, you can list the pods and the service, and you can examine the details of the running objects, see below.

4. To list the pods:

   kubectl get pods
   

5. To list the service:

   kubectl get services
   

6. To examine the details of the running objects (image, container, volumes, and status events):

   kubectl describe pod engine
   

7. To examine the logs in the correlator instance:

   kubectl logs engine
   

8. To examine the IAF logs:

   kubectl logs iaf
   

9. You can run commands in the pod. For example, to run a single command:

   kubectl exec engine 'ls'
   

   Or to open a shell on the running container:

   kubectl exec -it engine bash
   

10. To shut down the application, you use the same YAML file that has been used to start the process:

    kubectl delete -f kubernetes.yml
    

For more samples of Kubernetes configurations applied to Apama images and applications, see Apama samples for Kubernetes.

Apama samples for Kubernetes

There are a number of samples that can be found in the samples/docker/applications directory of your Apama installation. The Simple sample has already been referenced in Quick start to using Apama in Kubernetes. The other samples in the above directory cover more complex use cases. These samples build upon the base image and demonstrate how to use an Apama correlator in a Kubernetes environment to build your own applications.

The README.txt files that are provided in the samples/docker/applications directory and in each of the individual sample subdirectories guide you through the process of building and running. The samples demonstrate various ways in which Kubernetes can be used to deploy Apama. They also demonstrate interactions between containers and normal applications such as dashboards.

Weather

This sample deploys Apama’s Weather demo. It demonstrates the use of separate pods and connecting services to allow communication between them. A correlator pod and a dashboard server pod connect just as in the non-Docker Weather sample. The engine and weather services expose the required ports allowing connections to be made.

Adapter

This sample starts an IAF pod which connects to a correlator pod connected by a service. The IAF is running the File transport. An EPL application is deployed into the correlator, which requests the contents of a file inputFile.txt from the File adapter, and then directs it to write those contents back out to another file outputFile.txt.

The configuration for the correlator pod defines a readinessProbe that checks that the correlator is running before the pod is marked ready. In a similar way, the IAF pod configuration defines an initContainer to make sure the service it uses to connect to the correlator is present before continuing.

See also Deploying an Apama application using Kubernetes which refers to this sample.

MemoryStore

For containers to share dynamic data, they need a live view on that data. This is provided by a Kubernetes feature called “volumes”, which allows containers to share parts of their file system with each other. We use the Kubernetes PersistentVolume, PersistantVolumeClaim and Deployment objects to implement the sample.

This sample contains a toy application MemoryStoreCounter.mon that makes use of the MemoryStore to lay down persistent state on disk in the form of a number that increments each time the monitor is loaded. While the correlator container is the one reading and writing to the MemoryStore, the persistent file is on a volume which is persisted between container restarts.

Kubernetes volumes give you the ability to manage data that has a different lifecycle to the container that uses it. In an application like this, you can replace the other containers with equivalents that are based on a newer version of Apama. After bringing them up again, the correlator will still have access to the MemoryStore data that it wrote in a previous iteration, as this data is owned by the volume.

Universal Messaging

This sample demonstrates two Apama correlators communicating with each other via a Universal Messaging realm server. Therefore, Universal Messaging must be installed to run. A service is created that allows the correlators to communicate with the Universal Messaging pod, and the correlator pod uses an initContainer element in the Kubernetes configuration to ensure that the Universal Messaging service exists before continuing.

Secrets

This sample demonstrates how to use Kubernetes secrets to set variables in correlator configuration files. These can then be loaded at runtime into a correlator.

About Apama command line utilities

Correlator Utilities Reference

The Apama correlator is at the heart of Apama applications. The correlator is Apama’s core event processing and correlation engine. This section provides information and instructions for using command-line tools to monitor and manage correlators.

For information about EPL, event definitions, monitors, namespaces and packages, see Getting started with Apama EPL.

Starting the correlator

The correlator executable starts the correlator. This is located in the bin directory of the Apama installation. Running the executable in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Synopsis

To start the correlator, run the following command:

correlator [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

By default, the correlator executable starts a correlator process on the current host, and configures it to listen on port 15903 for monitoring and management requests.

On start-up, the executable displays the current version number and configuration settings.

To terminate a correlator process, press Ctrl+C in the window in which it was started. Alternatively, you can issue the engine_management command with the --shutdown option. See Shutting down and managing components. Regardless of which technique you use to terminate the correlator, Apama first tries to shut down the correlator cleanly. If this is not possible, for example, perhaps because of a monitor in an infinite loop, Apama forces the correlator to shut down.

Note:

If a license file cannot be found, the correlator will run with reduced capabilities. See Running Apama without a license file.

Options

The correlator executable takes the options listed below.

Note:

Many of these options can also be specified as elements of a YAML configuration file (with different element names). If an option is specified in both the command line and a YAML configuration file, then the command line takes precedence. For further information, see Configuring the correlator and especially the topic YAML configuration file for the correlator which lists the available elements.

Exit status

The correlator executable returns the following exit values:

Specifying log filenames

A correlator can send information to the following log files:

• Main correlator log file. Upon correlator startup, the default behavior is that the correlator logs status information to stdout. To send this information to a file, specify the -f file or --logfile file option and replace file with a log filename. The format for specifying a log filename is described below.

  Before you specify a log filename, you should consider your log rotation policy, which can determine what you want to specify for the log filename. See Rotating correlator log files.

• EPL log files. You can create log files for packages, monitors, and events in your application. The format you use to specify these log filenames is the same as for the main correlator log file. For details about how to create EPL log files, see Setting EPL log files and log levels in a YAML configuration file and Setting EPL log files and log levels dynamically.

• Correlator input log file. When you start a correlator, you can specify the --inputLog file option so that the correlator maintains a special log file for all inputs. Again, before you specify a log filename, you should consider the rotation policy for your input log files. See Rotating an input log file.

  Note: The correlator input log file is slightly different from the other log files, and it is not intended to be read by a human.

• Correlator configuration log file. When you start a correlator, you can specify the --configLog file option so that the correlator maintains a special log file for all of the correlator’s configuration.

The format for specifying a log filename is as follows:

file[${START_TIME}][${ROTATION_TIME}][${ID}][${PID}].log

The following table describes each part of a log filename specification. You cannot include spaces. You can separate the parts of the filename specification with underscores. You can specify ${START_TIME}, ${ROTATION_TIME}, ${ID} and/or ${PID} in any order. For examples, see Examples for specifying log filenames.

If you plan to rotate log files then be sure to specify ${ROTATION_TIME} or ${ID}. You can also specify both.

Examples for specifying log filenames

This topic provides examples of specifying log filenames. The format for specifying a log filename is the same in the following cases:

• Starting the correlator and specifying a main correlator log file with the --logfile option.
• Starting the correlator and specifying a correlator input log file with the --inputLog option.
• Invoking engine_management --setLogFile to rotate the main correlator log.
• Invoking engine_management --setApplicationLogFile to create an EPL log file for a package, monitor or event.

The following specifies that the name of the main log is “correlator_status.log”:

--logfile correlator_status.log

Suppose that the correlator processes events for a while, sends information to correlator_status.log, and then you find that you need to restart the correlator. If you restart the correlator and specify the exact same log filename, the correlator overwrites the first correlator_status.log file. To avoid overwriting a log, specify ${START_TIME} in the log file name specification when you start the correlator. For example:

--logfile correlator_status_${START_TIME}.log

The above command opens a log with a name something like the following:

correlator_status_2015-03-12_15:12:23.log

This ensures that the correlator does not overwrite a log file. Now suppose that you want to be able to rotate the log, so you specify the ${START_TIME} and ${ID} tags:

correlator_status_${START_TIME}_${ID}.log

The initial name of the log file is something like the one on the first line below. If you then rotate the log file then the correlator closes that file and opens a new file with a name like the one on the second line:

correlator_status_2015-03-12_15:12:23_001.log
correlator_status_2015-03-12_15:12:23_002.log

To specify an EPL log filename for messages generated in com.example.mypackge, you can specify the log filename as follows:

mypackage_${ID}_${ROTATION_TIME}.log

With that specification, messages generated in com.example.mypackage will go to a file with a name such as the one on the first line below. The time in the initial EPL log filename is the time that the initial log file is created. If you rotate the logs every 24 hours at midnight then the names of subsequent EPL log files will be something like the names in the second and third lines below.

mypackage_001_2015-03-21_18:42:06.log
mypackage_002_2015-03-22_00:00:00.log
mypackage_003_2015-03-23_00:00:00.log

If you want to run multiple correlators with the same arguments but with separate log files, you can use the process ID to differentiate them:

--logfile correlator_${PID}.log

The above command will produce a log file with a name such as the following, where each correlator will have a unique log file:

correlator_23487.log

UNIX note

In most UNIX shells, when you start a correlator you most likely need to escape the tag symbols, like this:

correlator -l license --inputLog input_\${START_TIME}_\${ID}.log

Descriptions of correlator status log fields

The correlator sends information to the main correlator log file every five seconds (the default behavior) or at an interval that you specify with the --logQueueSizePeriod option when you start the correlator.

There are one or two lines in the log file, depending on whether persistence is enabled or not:

• The line starting with Correlator Status: is always shown. See Correlator status for detailed information on the log fields that are shown in this line.
• The line starting with Persistence Status: is only shown when persistence has been enabled. See Persistence status for detailed information on the log fields that are shown in this line.

Note:

Correlators with correlator-integrated messaging for JMS enabled send additional information to the main log file of the correlator. For details on this information, see Logging correlator-integrated messaging for JMS status.

Correlator status

When logging at INFO level, this information contains the following:

Correlator Status: sm=11 nctx=1 ls=60 rq=0 iq=0 oq=0 icq=0 lcn="<none>" lcq=0 lct=0.0 rx=5 tx=17 rt=31 nc=2 vm=325556 pm=81068 runq=0 si=915.3 so=0.0 srn="<none>" srq=0 jvm=0

Where the fields have the following meanings (see List of correlator status statistics for more information):

Persistence status

If persistence is enabled, information such as the following is also shown when logging at INFO level:

Persistence Status: numSnapshots=25 lastSnapshotTime=1516203192 snapshotWaitTimeEwmaMillis=0.029071 commitTimeEwmaMillis=3.459181 lastSnapshotRowsChangedEwma=8

Where the fields have the following meanings (see List of correlator status statistics for more information):

Text internationalization in the logs

The information given here applies for the correlator, and also for the IAF and dashboard servers.

The strings in the logs are encoded in two different ways, depending on whether they are written to a log file or to the console:

• Log file

  Apama log files are always encoded as UTF-8. Any non-ASCII strings in a log file (such as Chinese characters, German umlauts, some currency symbols, and so on) will only look correct if your text editor is configured to view the text as UTF-8. This is irrelevant on most Linux installations, where the default encoding is usually UTF-8.

• Console

  If the Apama component is logging to the console, the output is encoded according to the locale of your system. In practice, this means that any strings that are logged to the console (including non-ASCII strings) will look correct.

Determining whether to disconnect slow receivers

The correlator sends events to multiple receivers. Sometimes, a receiver cannot consume its events fast enough for the correlator to continue sending them. When this happens, the default behavior is that the correlator suspends processing until the receiver disconnects or starts consuming events fast enough. In other words, a slow receiver can prevent other consumers from receiving events. However, you might prefer to have the correlator disconnect a slow receiver and continue processing and sending events to other consumers. Information in this section can help you determine whether to disconnect slow receivers.

See also Handling slow or blocked receivers.

Description of slow receivers

The correlator uses two strategies to detect slow receivers: time-based, and size-based.

Time-based (maxOutstandingSecs) slow consumer detection

Events that the correlator sends to Apama clients, IAFs or other correlators are acknowledged by the receiver after the event has been delivered to the receiver. By default, if the correlator does not receive an acknowledgment within 10 seconds after the correlator sent the event, the correlator marks that receiver as being slow to consume events.

For most systems, and assuming that the underlying network connections are not prone to drop-outs, the default setting of 10 seconds is usually adequate.

You can control the length of time within which the receiver must acknowledge an event before it is marked as a slow receiver. To do so, you can specify maxOutstandingSecs in the YAML configuration file that is used when starting the correlator. See YAML configuration file for the correlator.

For example, if you specify maxOutstandingSecs: 15.0 in the YAML configuration file, the correlator marks a receiver as slow if the correlator does not receive an acknowledgment within 15 seconds. If you do not specify this element, the default is 10 seconds. You should not specify a value under 1 second because doing so raises the risk that the correlator might designate a receiver as slow when it is in fact not slow.

The mechanism that flags a receiver as slow is not precise. If a receiver does not acknowledge an event sequence after 10 seconds (the default setting), the correlator does not immediately designate the receiver as slow. Typically, the designation happens within the next 5 seconds. If you change the value of maxOutstandingSecs, the slow designation takes effect between 1 and 1.5 times the value of maxOutstandingSecs.

Size-based (maxOutstandingKb) slow consumer detection

maxOutstandingKb can also be specified in the YAML configuration file that is used when starting the correlator. See YAML configuration file for the correlator.

The correlator keeps track of the events that have been sent to each receiver but have not yet been acknowledged, based on the amount of memory taken up by those events. The correlator marks a receiver as being slow if the size of the events waiting to be acknowledged goes above the maxOutstandingKb limit, which is 10MB by default.

Note that the size-based slow consumer detection operates completely independently of the time-based (maxOutstandingSecs) slow consumer detection.

It is rare for the size-based limit to be exceeded unless the events being transmitted are very large.

Connectivity plug-ins slow consumer detection

Connectivity chains use time-based slow consumer detection similar to maxOutstandingSecs. The size of the time window is currently set at 10 seconds (not configurable). A chain that has not processed a message for more than 10 seconds is logged as slow, but is not disconnected.

How frequently slow receivers occur

In practice, sending acknowledgments should not be slow because a dedicated thread sends acknowledgments. Network interruptions are the most common cause of delayed acknowledgments. Of course, network interruptions affect events being sent as well.

Most receivers, including the engine_receive tool, normally send acknowledgments 0.1 seconds after the message was sent. Consequently, there is very little chance of a receiver being mistakenly designated as slow. In production, slow receivers should be rare as long as you have done the appropriate load testing before deployment.

If an engine library client blocks in the middle of a sendEvents call, the receiver cannot acknowledge messages while the client is blocked. As you know, a receiver is made up of an engine library and a client. Clients receive events by registering a sendEvents callback with the engine library. When the engine library gets an event from the correlator, it calls sendEvents. Problems that can cause a client to block are typically related to I/O, networking, or synchronization. The sendEvents call cannot complete until the problem is resolved. The receiver cannot send the acknowledgment until the sendEvents call completes. For example, the engine_receive tool is a receiver that is made up of an engine library and a client whose sendEvents callback writes events received to a disk file. If the client has to wait for the disk, then it is blocked. The likelihood of a sendEvents callback being blocked depends on what the client is doing. If the client is writing to a local disk, the process might block sometimes, but never more than a fraction of a second. However, sending the events over a slow or unreliable network might block for a while if the network, or the remote system cannot keep up with the event rate.

During development of event consumers, however, slow receivers are more likely. This can happen when a newly developed consumer receives an event from the correlator but cannot send the acknowledgment because of a deadlock. Another development problem might be that the event consumer cannot keep up with the load. If you have problems with slow receivers during development, you probably need to evaluate the design of your application.

Correlator behavior when there is a slow receiver

When the correlator has a slow receiver, it can behave in one of two ways:

• The default behavior is that the correlator blocks further processing. This is because a slow receiver causes the correlator’s event output queue to become full. When the queue is full, the correlator stops processing because it has no place to put events. The processing thread stops and does not execute any more EPL code. The transport thread does not send any more events to any of the correlator’s other receivers. The correlator resumes processing when the slow receiver disconnects or acknowledges the outstanding sequence number.
• The correlator disconnects the slow receiver, and continues processing events and sending them to its other receivers. To obtain this behavior, you specify the -x (or --qdisconnect) option when you start the correlator. The correlator sends a message to the receiver to indicate that the correlator is disconnecting the receiver. It is up to the receiver to reconnect.

To ensure that it has received an acknowledgment for every event sent, the correlator buffers each event that it sends until it receives the message’s corresponding acknowledgment. When there is a slow receiver, this can use a lot of memory if the correlator is sending a large number of messages.

Tradeoffs for disconnecting a slow receiver

When you specify the -x option when you start the correlator, it means that the correlator always disconnects a slow receiver. There are two main disadvantages to this:

• The correlator loses the events that it sent to that receiver.
• It is possible for the correlator to disconnect a receiver that is temporarily overloaded, and to therefore lose events unnecessarily.

Clearly, losing events can be a very serious problem. This is why the default is that the correlator does not disconnect slow receivers.

The advantage of disconnecting a slow receiver is that the correlator continues processing events.

The correlator always sends a warning message to its main log when it detects a slow receiver. This lets you see where there are potential problems.

If you cannot allow the correlator to lose events, do not specify the -x option when you start the correlator.

Determining whether to disable the correlator’s internal clock

When you start the correlator, you can specify the -Xclock option to disable the correlator’s internal clock. By default, the correlator uses internally generated clock ticks to assign a timestamp to each incoming event. When you specify the -Xclock option, you must send time events (&TIME) to the correlator. These time events set the correlator’s clock.

Use &TIME events in place of the correlator’s internal clock when you want to reproduce the historic behavior of an application. For example, Apama’s Data Player in Apama Plugin for Eclipse starts a correlator with a command that specifies the -Xclock option. The Data Player then sends &TIME events that let you play back events from the database.

A situation in which you might want to generate &TIME events is when you want to run experiments at faster than real time but still obtain correct timestamp behavior. In this situation, the correlator uses the externally generated time events instead of real time to invoke timers and wait events.

Disabling the correlator’s internal clock, and sending external time events, affects all temporal operations, such as timers and wait statements.

Regardless of whether the correlator generates internal clock ticks, or receives external time events, the correlator assigns a timestamp to each incoming event. The timestamp indicates the time that the event arrived on the context’s input queue. The value of the timestamp is the same as the last internally-generated clock tick or externally-generated time event. For example, suppose you have the following events and clock ticks:

&TIME(1)
A()
B()
&TIME(2)
C()

A and B receive a timestamp of 1. C receives a timestamp of 2.

See also Understanding time in the correlator.

Injection time of compiled runtime

Injection times for systems using the compiled runtime can be very long - significantly longer than if using only the interpreted runtime. Subsequent injection times can be improved by using the --runtime-cache dir option, which specifies a directory where the correlator can cache the compilation state (see Starting the correlator). This stores the results of compiling EPL code on disk to be used for subsequent injections of the same code.

The compiled EPL code is specific to the system on which it was compiled and the version of Apama that was used to compile it. This means that while a cache can be moved or shared between machines to improve startup on a new machine, it must be identical to the original. Otherwise, the cached version cannot be used and it must be recompiled

An injection is able to use a cached version of a previous injection if all of the following are the same as in a previous injection:

• The EPL source code.
• The source code of all files that contain any type that an injection depends on.
• The correlator version.
• The host operating system.
• The CPU model.

The results of injections can be affected by any of the above. Therefore, if any change occurs, the correlator will re-compile the EPL.

The cache is designed to be used to speed up re-injection on production systems and not for quicker development cycles, which should typically use the interpreter for faster injection times. If there are identical user acceptance testing (UAT) and production environments, then the UAT environment can prime a cache which can then be used by the production correlator to improve initial startup times. However, the two systems must be identical. The strings used to disambiguate systems are logged at correlator startup when using the compiled runtime and can be used to compare the systems.

The cache contents are never removed by the correlator. If you change your source, correlator versions or platform, then the cache may grow and contain stale items which are no longer needed. If cache sizes become a problem, then we either recommend deleting the entire cache, or just the cache files with the oldest timestamps. The correlator will transparently recompile any needed files which are missing from the cache.

Configuring the correlator

You can configure the correlator using YAML configuration files and properties files. See the topics below for detailed information on these types of files, and on how to configure certain aspects of the correlator.

YAML configuration files are also used to configure connectivity plug-ins. See Configuration file for connectivity plug-ins for further information.

Using YAML configuration files

You can specify one or more YAML files using the --config option when you start the correlator. See Starting the correlator.

For detailed information on YAML, see http://www.yaml.org/spec/1.2/spec.html. A quick overview is given below.

YAML configuration files can contain maps, lists or simple values:

• A map contains a string key, followed by a colon and space, followed by a value (which can be a map, a list or a simple value). Typically, an entry with a simple value is written on one line, and collections (maps and lists) are written on a following line with indentation.

• A list contains a number of values, each of which is written on a new line with a preceding dash and space ("- “).

• A simple value includes a string or number. It is typically written on a single line. A string may be enclosed in quotes if needed, but this is not mandatory.

  Some characters in YAML have special significance at certain positions. For example, a value ending with a colon (:) is treated as a key in a dictionary, so if you want a string value to end with a literal colon (:), you should quote it.

• Nesting is expressed using spaces to indent different levels of object. Tabs are forbidden in YAML files, all indentation must be performed using spaces.

For example:

myTopLevelMap:
  mykey: myvalue
  mylist:
    - item 1 # comment
    - item 2
    - "a quoted string value"

YAML is a superset of JSON. Thus, any valid JSON is also usable in the YAML configuration file. This is helpful if there is ambiguity in the way YAML expresses configuration.

YAML documents should be saved with the standard UTF-8 character encoding.

YAML configuration file for the correlator

The following sample shows the format of a YAML configuration file for the correlator.

correlator:
  javaApplicationSupport: false
  pythonSupport: true
  randomSeed: number
  initialization:
    list: [... ]
  persistence:
    enabled: false
    snapshotIntervalMillis: number
    adjustSnapshot: true
    storeLocation: path
    storeName: string
    clear: false
  runtime: interpreted
  runtimeCacheDir: path
  licenseFile: path
  jmsConfigDir: path
  distMemStoreConfigDir: path
  inputLogFile: path
  externalClock: false
  timerFrequency: 10
  truncateLogFile: false
  disableOptimizations: false
  includeDatabaseInInputLog: true
  statusLogIntervalSecs: 5.0
  inputQueueSize: 20000
  logFile: path
  shutdownTimeoutSecs: number

server:
  pidFile: path
  port: 15903
  name: string
  bindAddress: [...]
  permittedRootURIs: [...]
  allowClient: [...]
  maxOutstandingSecs: 10.0
  maxOutstandingKb: 10240
  disconnectSlowConsumers: false

jvmOptions: [...]

eplLogging:
.root:
    level: loglevel
    file: path
  string:
    level: loglevel
    file: path
correlatorLogging:
  string:
    level: loglevel

environment:
  string: string

engineConnect:
  - sourceHost: string
    sourcePort: integer
    channels:
      - string
    mode: string
    disconnectIfSlow: boolean

includes: [...]

### For configuring EPL plug-ins:
eplPlugins: [...]

### For configuring connectivity plug-ins:
connectivityPlugins: [...]
startChains: [...]
dynamicChainManagers: [...]
dynamicChains: [...]

For paths, we recommend using replacements like ${PARENT_DIR} or ${APAMA_WORK} to make the configuration files portable. See also the list of predefined properties in Using properties files.

[...] denotes that this can be a list.

You can use multi-line and single-line syntax for maps (for example, in the eplLogging case).

Many of the elements in the above sample configuration file correspond to command-line options of the correlator executable and have the same syntax and options. See the tables below, and see the descriptions of these command-line options in Starting the correlator for detailed information.

Note:

If an element in a YAML configuration file is defined more than once, the first definition that is loaded by the correlator takes precedence over all definitions that are loaded later. An INFO message is then logged for all later definitions that are being ignored. Any corresponding options that are specified directly on the command line, however, take precedence over any other later definitions.

The following table lists the elements that can be specified in the correlator section of the YAML configuration file (see the separate table below for the correlator/persistence section).

Note:

When developing using Apama Plugin for Eclipse, add any elements you need to set to the config/CorrelatorConfig.yaml file. See also the description of the Configuration option in Correlator arguments.

The following table lists the elements that can be specified in the correlator/persistence section of the YAML configuration file:

The following table lists the elements that can be specified in the server section of the YAML configuration file:

The following table lists the remaining sections that can be specified in the YAML configuration file:

Other elements are described in topics under Configuring the correlator, and this also includes additional information for some of the elements that correspond to the command-line options. These are:

• In the correlator section:
  • initialization. See Deploying Apama applications with a YAML configuration file.
  • persistence. See Configuring persistence in a YAML configuration file.
• In the server section:
  • bindAddress and permittedRootURIs. See Binding server components to particular addresses.
  • allowClient. See Ensuring that client connections are from particular addresses.
• jvmOptions. See Specifying JVM options.
• eplLogging. See Setting EPL log files and log levels in a YAML configuration file.
• correlatorLogging. See Setting correlator and plug-in log files and log levels in a YAML configuration file.
• environment. See Setting environment variables for Apama components.
• engineConnect. See Setting up connections between correlators in a YAML configuration file.
• includes. See Including YAML configuration files inside another YAML configuration file.

The eplPlugins element is used for configuring EPL plug-ins. There is no equivalent command-line option. See Writing EPL plug-ins in Python for detailed information.

The following elements are used for configuring connectivity plug-ins. There are no equivalent command-line options. See Using connectivity plug-ins for detailed information.

• connectivityPlugins
• startChains
• dynamicChainManagers
• dynamicChains

Including YAML configuration files inside another YAML configuration file

Instead of specifying a list of configuration files on the command line with the --config option, you can also specify these files in the includes section of another configuration file. For example:

includes:
   - myCodec.yaml
   - myTransport.yaml

When the configuration file that is specified with the --config option is processed, the list in the includes section is expanded recursively, and the information inside the included configuration files is merged into the main configuration. The same rules are used as for the --config option; for details, see the description of that option in Starting the correlator.

You do not need to worry about multiple file inclusions (such as cyclical or diamond references), but you must still be careful not to have duplicate keys in top-level maps.

Using an includes section can be useful for specifying a connectivity chain in a modular way. For example, you may have a main configuration file with the following contents:

startChains:
  myChain:
     - apama.eventMap
     - MyCodec
     - MyTransport
includes:
   - myCodec.yaml
   - myTransport.yaml

where the included files have the following contents:

• myCodec.yaml:

  connectivityPlugins:
    MyCodec:
      classpath: ${myJarVersion}
      class: com.example.my.Codec
  includes:
     - propDir
  

  You can include further files in an included file. Instead of a file name, you can also specify the name of a directory in the includes section. All files that can be found in this directory are then included.

  The includes section in the above example assumes that there is a directory named propDir which contains a jarVersions.properties file with the following content:

  myJarVersion=myJar316.jar
  

• myTransport.yaml:

  connectivityPlugins:
    MyTransport:
      classpath: myOther.jar
      class: com.example.my.Transport
  

Properties defined in included files are only valid for later files. For example, the deployment with the following files will work:

• 1.yaml:

  includes:
     - my.properties
     - 2.yaml
  

• my.properties:

  3=3.yaml
  

• 2.yaml:

  includes:
     - ${3}
  

The deployment with the following file, however, will not work:

• 1.yaml:

  includes:
     - my.properties
     - 2.yaml
     - ${3}
  

  In this case, you should specify the my.properties file as an argument with the --config option, in addition to the 1.yaml file.

See Using properties files for more information.

Using properties files

Properties files (with the file extension .properties) can be used to either specify values for substitution variables in YAML files (see also Using YAML configuration files) or to configure EPL. A properties file must be either in ISO-8859-1 encoding or in UTF-8 encoding if it begins with an UTF-8 byte order mark (BOM).

You can specify one or more properties files using the --config option when you start the correlator. See Starting the correlator. The properties files are processed in the order in which they appear on the command line. Each properties file can refer to properties that have already been defined by a previously processed properties file, using ${varname} syntax.

Note: If the same property is defined more than once, the first definition that is loaded by the correlator takes precedence over all definitions that are loaded later. Messages are then logged for all later definitions that are being ignored.

The properties file format is the same as the standard Java .properties file format, with # for comments and \\ used to escape any occurrences of \. For example:

# my comment line
myplugin.mykey=c:\\my directory

Properties files are by default parsed using the ISO-8859-1 encoding (Latin-1). To use characters from other character sets, you have two options. You can use a \uXXXX escape sequence, which uses the UCS-2 character codes. This allows you to encode any character from the Basic Multilingual Plane (BMP). However, we do not support UTF-16 surrogate pairs to access characters outside the BMP. Alternatively, you can provide a UTF-8 BOM as the first characters of the file. In this case, you can use any Unicode sequence in UTF-8 directly in the file without any escaping.

You can use the following predefined properties:

All properties are applied to all YAML files, although conventionally, there is often a .properties file named the same as each .yaml file. Properties files, however, are not actually tied in to YAML files in any way. It is therefore recommended that you prefix each property key with a unique string, such as the identifier of the chain to which it applies.

Instead of using a properties file or in addition to using a properties file, you can also use the -D option of the correlator executable. See Starting the correlator.

Properties defined either through properties files or on the command line are available to EPL via the Management interface. For more details, see Using the Management interface and the API reference for EPL (ApamaDoc).

Correlator-integrated messaging for JMS and the distributed MemoryStore have their own properties files, which are Spring files. Keep in mind that any properties that are specified with the --config file or -Dkey=value option of the correlator executable take precedence and override the properties defined in a Spring properties file. See also Configuration files for JMS and Configuration files for distributed stores.

Runtime parameterization of configuration

Some components allow parameterization of configuration at runtime, for example, connectivity plug-in chains created from a manager or from EPL. Variables which must be replaced at runtime are denoted with @{varname} instead of ${varname}. Strings using this syntax may need to be escaped with double quotes in the YAML configuration file. Instead of being supplied via the command line or via properties files, these substitutions are provided within the subsystem using that part of the configuration. See also Creating dynamic chains from EPL.

Specifying the correlator port number

You can specify the port on which the correlator should listen for monitoring and management requests. To do so, you specify a port definition in the server section of the YAML configuration file. For example:

server:
  port: 15903

This is identical to specifying the --port option when starting the correlator. See also Starting the correlator.

If the port is specified with the --port option when starting the correlator, this value is used. Else, if it is specified in the YAML configuration file, that value is used. If a port is not specified at all, the default value 15903 is used.

Binding server components to particular addresses

To bind Apama server components to a particular address or set of addresses, specify a bindAddress definition for each address. Specify this in the server section of the YAML configuration file. For example:

server:
  bindAddress:
    - 127.0.0.1:15903
    - 10.0.0.1

You can specify as many bindAddress definitions as you want. Clients can connect to any of the listed addresses.

An IP address is required. If you do not specify a port, the Apama server components use the port that is specified when the correlator is started. This makes it possible to share YAML configuration files if you want to restrict connections according to only addresses.

If you do not specify a YAML configuration file when you start the correlator, or there are no bindAddress definitions in the YAML configuration file, the Apama components bind to the wildcard address (0.0.0.0).

Using the correlator web interface with non-default addresses

The correlator web interface sometimes uses the client’s host header for redirects. For security, this is checked against a list of accepted host:port aliases for this component. By default, all of the addresses of the machine on the default or overridden correlator port are part of this list.

If the correlator has a custom bind address configuration, or is being hidden behind another component which is redirecting the web interface, the default list of permitted addresses may need to be overridden. This can be done with the permittedRootURIs configuration entry:

server:
  permittedRootURIs:
     - ${EXTERNAL_HOSTNAME}:${EXTERNAL_PORT}

This can either be a single string or a list of multiple permitted URIs. This list replaces the automatically calculated defaults, it does not append.

Ensuring that client connections are from particular addresses

To ensure that client connections are from particular addresses, add one or more allowClient definitions to the YAML configuration file in the server section. For example:

server:
  allowClient:
    - 127.0.0.1
    - 192.168.128.0/17

An allowClient definition takes an IP address, as in the first example above, or a CIDR (Classless Inter-Domain Routing) address range, as in the second example above. With these example entries in the YAML configuration file, the Apama components allow connections from either the localhost (127.0.0.1) or IP addresses where the first 17 bits match the first 17 bits of 192.168.128.0. The Apama components do not accept connections from any other IP addresses.

If you specify a YAML configuration file when you start the correlator, and if there are any allowClient definitions in the YAML configuration file, then the Apama components do not allow connections from any IP address that does not fall within one of the allowClient ranges specified. If you do not specify a YAML configuration file when you start the correlator, or there are no allowClient definitions in a YAML configuration file that you do specify, the Apama components accept connections from any client.

Setting environment variables for Apama components

You can use the YAML configuration file to set environment variables. Put environment variable declarations in the environment section. For example:

environment:
  MY_ENV_VAR: myvalue

If you specify a YAML configuration file when you start the correlator, and if there are any environment variable entries in the YAML configuration file, then the Apama components use those environment variable settings. If you do not specify a YAML configuration file when you start the correlator, or there are no environment variable entries in a YAML configuration file that you do specify, the Apama components use the environment variable settings specified elsewhere.

Note:

You cannot use this feature to set variables such as LD_PRELOAD and LD_LIBRARY_PATH because UNIX dictates that they are set before the affected process starts execution. These environment variables should therefore be considered read-only.

Setting EPL log files and log levels in a YAML configuration file

This topic describes how to configure logging for individual EPL packages. For information about configuring the log level of the whole correlator and plug-ins running inside it, see Setting correlator and plug-in log files and log levels in a YAML configuration file.

You can configure per-package logging in several ways:

• Statically, in the YAML configuration file when starting the correlator, as described in this topic.

• Dynamically, using the following options of the engine_management tool:

  • --setApplicationLogFile
  • --setApplicationLogLevel See Setting EPL log files and log levels dynamically for detailed information.

• Dynamically from within EPL. See Using the Management interface for detailed information.

To set log files and/or log levels for EPL root, packages, monitors, or events, specify entries in the eplLogging section of the YAML configuration file.

To set the default log file and level for the EPL root package, specify this in the following format:

eplLogging:
.root:
     file: rootLogFilename
     level: ROOTLOGLEVEL

Replace rootLogFilename with the name of the log file for the EPL root package. No spaces are allowed in the log file name. Replace ROOTLOGLEVEL with TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT or OFF. For example:

eplLogging:
.root:
     file: apama/root.log
     level: ERROR

You do not need to specify both a log file and a log level; you can specify one or the other. If you do not specify a log file or log level for the root package, it defaults to the correlator’s log file/log level.

To set the default log file and level for an EPL package, monitor or event, specify this in the following format:

eplLogging:
  node:
     file: nodeLogFilename
     level: NODELOGLEVEL

Replace node with the name of the EPL package, monitor, or event for which you are setting the logging attribute. If a monitor or event is in a named package and not the root package, be sure to specify the fully qualified name. Replace nodeLogFilename with the name of the default log file for the specified EPL package, monitor or event. No spaces are allowed in the log file name. Replace NODELOGLEVEL with TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT or OFF. This is the default log level for the specified node. For example:

eplLogging:
  com.myCompany.Client:
    file: apama/Client.log
    level: DEBUG
  com.myCompany.Internal: { level: ERROR }

The above example shows both multi-line and single-line syntax. The single-line syntax is more compact when you are just setting either the log level or the file, but not both.

For a particular node, you do not need to specify both a log file and a log level; you can specify one or the other. If you do not specify a log file or log level for a particular node, it defaults to the settings for a parent node. See Tree structure of packages, monitors, and events.

When you set log attributes in the YAML configuration file, the rules for hierarchical logging apply. See Setting EPL log files and log levels dynamically.

If you pass a YAML configuration file to a correlator when you start that correlator and the configuration file contains an eplLogging section, the correlator uses the logging settings in that section. If you do not pass a configuration file when you start a correlator, or there are no settings in the eplLogging section, then correlator initialization does not include any log settings except for the default correlator log.

Whether or not you specify a YAML configuration file when you start a correlator, any log settings you specify can be overwritten after initialization by invoking the engine_management tool and specifying the --setApplicationLogFile and/or --setApplicationLogLevel options. See Managing EPL log levels and Managing EPL log files.

Note:

Regularly rotating log files and storing the old ones in a secure location may be important as part of your personal data protection policy. For more information, see Recommendations for logging by Apama application code.

Setting correlator and plug-in log files and log levels in a YAML configuration file

In a YAML configuration file, you can configure the log file and/or log level individually for plug-ins, as well as for components of the correlator itself. You can do this either for the whole correlator or for individual categories. This includes setting the log level for individual connectivity plug-ins and EPL plug-ins.

Note:

You can also set the log levels individually on the command line (see Starting the correlator). If a log level is specified on the command line, it overrides any setting in the YAML configuration file.

To set the log levels in the YAML configuration file, specify entries in the correlatorLogging section. The syntax for this section is:

correlatorLogging:
.root:
      level: ROOTLOGLEVEL
      file: ROOTLOGFILE
   category: CATEGORYLOGLEVEL

You can specify either the log level directly, or as a map with the key level. These are synonymous. To set the log file, you must use the map syntax.

.root is used to specify the default log file/level for the whole correlator. We do not recommend specifying a log level higher than INFO for the default log level, since important information may be lost from the log files.

Valid log levels are TRACE, DEBUG, INFO, WARN, ERROR, FATAL, CRIT or OFF.

You can use replacement tokens in a log file. See Specifying log filenames for more information.

The categories for which the logging can be configured are listed in the table below. Note that log categories are regarded as a hierarchy. For example, setting the log level for apama will be inherited by apama.status.

Example - the default log level for the whole correlator is set to INFO, and the log level for the connectivity plug-ins framework is set to DEBUG:

correlatorLogging:
.root:
      level: INFO
   apama.connectivity: DEBUG

Note:

It is not possible to dynamically change the correlator and plug-in log levels. Only EPL log levels can be dynamically changed (see also Setting EPL log files and log levels dynamically).

Configuring persistence in a YAML configuration file

You can enable and configure correlator persistence in the following ways:

• Using a YAML configuration file as described here.
• On the command line, using the persistence options of the correlator executable. See Starting the correlator for more information on these options.

On the command line, the persistence options are given as -Poption=value.

In a configuration file, they are given as follows:

correlator:
  persistence:
    option: value

All of persistence options for the command line can also be specified in a configuration file. Special notations are required for the following options:

• -P without further options or -Penabled=true enables persistence. In a configuration file, this is specified as follows:

  correlator:
    persistence:
      enabled: true
  

• -Pclear does not have a value; it is implicitly set to true. In a configuration file, this is specified as follows:

  correlator:
    persistence:
       clear: true
  

The following is a list of all the options that you can specify in a configuration file:

• enabled: boolean
• snapshotIntervalMillis: interval
• adjustSnapshot: boolean
• storeLocation: path
• storeName: filename
• clear: boolean

The following sample shows the format of a YAML configuration file that is used to specify the persistence options:

correlator:
  persistence:
    enabled: true
    snapshotIntervalMillis: 12000
    storeLocation: ${PARENT_DIR}/store
    storeName: mystore.db
    clear: false

For detailed information on correlator persistence, see Using correlator persistence and especially its subtopic Enabling correlator persistence which provides more information on the different persistence options.

Setting up connections between correlators in a YAML configuration file

Rather than separately invoking the engine_connect tool, you can also define connections between correlators during correlator startup. These connections are defined as a list of entries in the engineConnect section of the YAML configuration file. For example:

engineConnect:
  - sourceHost: localhost
    sourcePort: ${FIRST_PORT}
    channels:
      - myChannel
    mode: parallel
    disconnectIfSlow: false
  - sourceHost: localhost
    sourcePort: ${SECOND_PORT}
    channels:
      - myChannel
    mode: parallel
    disconnectIfSlow: true

The above elements correspond to command-line options of the engine_connect tool and have the same syntax and options. See the descriptions of these command-line options in Configuring pipelining with engine_connect for detailed information, but keep in mind that only the options shown in the above example are supported.

On startup, the configuration specified in the engineConnect section always uses the current correlator as the target correlator. Unlike the engine_connect tool, you cannot explicitly specify an arbitrary target correlator with engineConnect.

On startup, engineConnect is not available to correlators running in persistent mode.

Unlike the engine_connect tool, when the mode is not specified for an entry in the engineConnect YAML, the default mode is parallel, meaning that there is one connection for each specified channel.

Specifying JVM options

In a YAML configuration file, you can specify JVM options which the correlator is to pass to the embedded JVM. To do so, you provide a list of JVM options with the jvmOptions key. If an option has a leading hyphen, you have to enclose the option in quotes. But you can also specify the option without quotes by leaving out the leading hyphen; in this case, the correlator will automatically add the hyphen. Thus, you can specify the JVM options as shown in the following example:

jvmOptions:
 - "-Dkey1=value1"
 - Dkey2=value2
 - "-Xms100m"
 - Xmx500m

You can specify JVM options in multiple configuration files. The options from all the files are then appended together in the order in which they have been specified. If the same JVM option is specified in the command line as well as in a configuration file, the command line takes precedence. For more details, see the description of the -J option in Starting the correlator.

Deploying Apama applications with a YAML configuration file

Instead of having another process inject code and send events into a correlator at startup, it is also possible to use a YAML configuration file to list files to be loaded by the correlator at startup. This is useful for Docker containers or other minimal environments where only part of an Apama installation is available or it is not practical to run Java tools to perform the injections. It is also a better fit for Docker use cases as the correlator does not require any other coordination process for startup. For typical installations not using such environments, use of Ant macros is recommended instead, which perform the injections after starting the correlator.

The YAML configuration file for the correlator is specified using the --config option when starting the correlator (see also Starting the correlator). The YAML file itself contains the following:

correlator:
    initialization:
        list:
            - ${PARENT_DIR}/bin/myPlugin.jar
            - ${PARENT_DIR}/eventdefinitions/evtdef.mon
            - ${APAMA_HOME}/monitors/ConnectivityPlugins.mon
            - ${PARENT_DIR}/monitors/app.mon
            - ${PARENT_DIR}/events/start.evt
        encoding: UTF8

It is recommended to use a substitution variable such as ${APAMA_HOME} or ${PARENT_DIR} rather than absolute or relative paths. This makes the configuration independent of the correlator’s current working directory.

The list entries must have one of the following extensions:

• .mon for EPL monitor, aggregate and event definition source.
• .jar for EPL plug-ins written in Java.
• .cdp for correlator deployment packages.
• .evt for event files.

Apama queries (.qry) are not supported in source form.

Note:

If you use the engine_deploy tool, EPL code is automatically generated from query files. For further information, see Deploying a correlator.

Apama queries are deprecated and will be removed in a future release.

The encoding entry is optional. If UTF8 is specified, all of the text input files (.mon, .evt) are read as UTF-8. If local is specified or if the encoding entry is not specified at all, the text files are assumed to be in the local encoding unless they start with a UTF-8 byte order mark (BOM) in which case they are treated as UTF-8.

This mechanism separates the build-time (calculating injection order, generating EPL) steps from the deployment-time steps, so no build steps are required in the environment where the correlator is running. This does mean that any changes to the project potentially require rewriting the YAML list and then redeploying, however, it allows separation of these concerns.

Note:

The correlator port is opened before the injections have completed. This allows monitoring tools to connect while the injections occur, but this also means that the correlator may not be entirely ready when a client connects. You can use the flushAllQueues request (see Shutting down and managing components) to wait for the injections to complete.

Injecting code into a correlator

To inject EPL files, EPL plug-ins in Java, JMon applications, or correlator deployment packages (CDPs) into the correlator, invoke the engine_inject tool. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Note: Apama’s in-process API for Java (JMon) is deprecated and will be removed in a future release.

Synopsis

To inject applications into the correlator, run the following command:

engine_inject [ options ] [ file1 [ file2... ] ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_inject tool reads application definitions from the specified file(s) and injects them into a correlator. If you do not specify a filename, or if you specify a hyphen (-) as the filename, the correlator reads data from the standard input device (stdin) until you indicate the end of the file: Ctrl+D on UNIX and Ctrl+Z on Windows.

Application definitions can be monitors scripted in Apama’s Event Processing Language (EPL). For more information on EPL, see Introduction to Apama Event Processing Language. Alternatively, you can specify the –j or -c options. The -j option specifies that you will inject an application or plug-in written in Java. The -c option specifies that you will inject a correlator deployment package file.

When you specify the -j option, each file you inject must be a Java archive file (JAR) that contains a single EPL plug-in written in Java or JMon application. For more information, see Writing EPL plug-ins in Java and Overview of Apama JMon applications.

When you specify the -c option, the file you inject must be an Apama correlator deployment package (CDP). For more information on preparing a CDP, see Packaging correlator input files.

By default, the engine_inject tool is silent unless an error occurs. To view information about engine_inject execution, specify the --verbose option.

If you try to inject invalid EPL files or invalid JMon applications, the correlator generates an error. None of the application data in the invalid file is loaded. The engine_inject tool terminates. If you specify multiple EPL or Java files for injection the engine_inject tool injects all of them or terminates when it reaches the first file that contains an error. For example:

engine_inject 1.mon 2.mon 3.mon

If the file 2.mon contains an error, then engine_inject successfully injects 1.mon and then terminates when it finds the error in 2.mon. The tool does not operate on 3.mon.

If you try to inject a CDP, the correlator processes each EPL file packaged in the CDP separately. If one file in a CDP contains an error, then the correlator reports an error for that file and does not run it but it does run the other files in the CDP (if they have no errors). It does not matter which file in the CDP contains the error. That is, the first file in the CDP that the correlator processes can contain an error and the correlator still runs the other files in the CDP if they contain no errors.

Note:

If a license file cannot be found, the correlator does not allow the injection of user-generated CDPs. See Running Apama without a license file.

Options

The engine_inject tool takes the following options:

Operands

The engine_inject tool takes the following operands:

Exit status

The engine_inject tool returns the following exit values:

Text encoding

By default, the engine_inject tool uses the default system encoding to determine the local character set. The engine_inject tool then translates all submitted EPL text from the local character set to UTF-8. Consequently, it is important to correctly set the machine’s locale.

However, some input files might start with a UTF-8 Byte Order Mark. The engine_inject tool treats such input files as UTF-8 and does not do any translation. Alternatively, you can specify the -u option when you run the engine_inject tool. This forces the tool to treat each input file as UTF-8.

Creating and managing an Apama project from the command line

The apama_project tool can be used to create and manage an Apama project outside of Apama Plugin for Eclipse. It lets you perform the following actions:

• create an Apama project,
• list all supported bundles that can be added to a project,
• list all bundles/instances that have already been added to a project,
• add connectivity, adapter and EPL bundles and their instances to a project,
• remove bundles/instances from a project.

Each Apama project generated by this tool is compatible with Apama Plugin for Eclipse and can seamlessly be imported into Apama Plugin for Eclipse as an Apama project. For more information, see Working with Projects.

The executable for this tool is located in the bin directory of the Apama installation. It is recommended that you run this tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt); otherwise you have to provide the full path the apama_project executable each time you run it.

Synopsis

To use this tool, run the following command:

apama_project command [options]

When you run this command with the help (or --help, -help or –h) option, the usage message for this command is shown. For example:

• To display help for all commands:

  apama_project help
  

• To display help for a specific command:

  apama_project command help
  

Description

You can only specify one command at a time.

Apart from the command for creating a project, all other commands require you to run the tool from the project directory.

The apama_project tool takes the following commands and options:

Examples

The following examples show the different ways in which the apama_project tool can be started.

Note: Keep in mind that you have to change to the project directory if you want to run the commands for listing, adding or removing bundles.

• Create an Apama project that is named “MyApamaProject” in the current directory:

  apama_project create MyApamaProject
  

• List all connectivity and EPL bundles in APAMA_HOME that can be added to the current Apama project:

  apama_project list bundles
  

• Add a bundle to the current Apama project, with a user-defined instance name:

  apama_project add bundle "HTTP Client" --instance "MyHTTP"
  

• Add a bundle to the current Apama project using the index number:

  apama_project add bundle 34
  

• Remove a bundle from the current Apama project, including all of its instances:

  apama_project remove bundle "HTTP Client"
  

Deploying a correlator

The engine_deploy tool lets you perform the following actions with an Apama project that has been created with Apama Plugin for Eclipse or with the apama_project tool:

• generate an initialization file list,
• generate a correlator deployment directory,
• create a Zip file with the contents of the correlator deployment directory,
• generate a correlator deployment package (CDP),
• perform the initialization in a running correlator.

This tool can also be used with a directory of Apama files if you are not using Apama Plugin for Eclipse.

The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

You can also use Ant to generate a correlator deployment directory or a correlator deployment package. The apama-macros.xml file includes the generate-correlator-deploy-dir and project-to-cdp macros for this purpose. See also About Deploying Apama Applications with an Ant script.

Note:

If a license file cannot be found, the correlator cannot read user-generated CDPs. See Running Apama without a license file.

Synopsis

To use this tool, run the following command:

engine_deploy action [options] path1 [path2...]

When you run this command with the –h option, the usage message for this command is shown.

Description

The action that you have to specify when you run this tool can be one of the following:

Note: Only one action can be specified at a time.

Deployment directory

When you use this tool to generate a deployment directory, it copies all required files from the project into a deployment directory. It also generates all required YAML configuration files and properties files using the information that is currently defined in the project’s launch configuration (see Launching Projects for more information on how to set up a launch configuration).

The deployment directory includes all of the project artifacts, except for log files, along with following generated files:

correlator --config C:/MyDeployDir

See Starting the correlator for detailed information on the available options.

If you want to override one or more property values that are defined in the generated properties files, you have to send an additional properties file containing these overrides (or you can set the property directly by specifying it on the command line) to the correlator before sending the generated properties files. For example, when you specify the following, the property values defined in the file myOverrides.properties will take precedence over all of the properties defined in other files.

correlator --config myOverrides.properties --config deployDir/

Options

The engine_deploy tool takes the following options:

Operands

path1 and other optional paths that you can specify when you run the engine_deploy tool can point to the following:

• A project directory. This is the directory which contains the .project file and, if defined, the .dependencies file.

  If exactly one deployment (.deploy) file exists in the project directory, it is automatically used. If the directory contains more than one deployment file, an exception is thrown. In this case, you have to select the deployment file to be used by specifying the path to it (instead of specifying the path to a project directory). You also have to specify the path to a deployment file explicitly if your launch configuration contains multiple correlators, followed by an exclamation mark (!) and the correlator name as described below.

  Note: You can only specify one project at a time. If a project references additional projects, then the generated injection order might not be accurate.

  If you are not using Apama Plugin for Eclipse or the apama_project tool, you can specify any directory containing Apama files (for example, .mon files, etc.).

• A deployment (.deploy) file. This file is automatically generated by Apama Plugin for Eclipse for each launch configuration that has been defined for a given project. It is located under project_dir/config/launch. See also Defining custom launch configurations.

  If more than one correlator is defined in the deployment file, you have to add an exclamation mark (!) followed by the correlator name (otherwise, the tool will give an error message and fail). For example:

  MyDeployFile.deploy!myCorrelator

  If your launch configuration has multiple correlators, it is recommended that you generate a separate correlator deployment directory for each correlator.

  If a deployment file is used which contains environment variables, you have to explicitly specify these variables when you start the correlator with the correlator executable. These variables are not captured from the launch configuration. This is important when using EPL plug-ins and connectivity plug-ins that are written in C++.

• Zero or more .properties files which contain substitution variables that have been defined in the specified project. The properties files are used when you specify a project directory or deployment file. You can specify these paths in any order.

• A text (.txt) file. This is the initialization file list which lists the project artifacts to be included.

• One or more correlator deployment packages (.cdp files) to be injected into the correlator.

Examples

The following examples (for Windows) show the different ways in which the engine_deploy tool can be started.

• Create an initialization list by pointing to a project directory containing the EPL files:

  engine_deploy --outputList C:/initialization_list.txt MyProject
  

• Create a correlator deployment directory by pointing to a deployment file within the project, and using the substitution variables that have been defined in a properties file that is also available in the project:

  engine_deploy --outputDeployDir C:/MyDeployDir
     MyProject/config/launch/MyDeployFile.deploy!myCorrelator
     C:/MyProjects/environment.properties
  

  The name of the correlator that is to be used is given after the exclamation mark (!).

  Files coming from external variables are also copied into C:/MyDeployDir. In addition, the file initialization.properties is generated which contains information on these variables.

• Create a correlator deployment package by pointing to a deployment file within the project:

  engine_deploy --outputCDP C:/output.cdp
     C:/MyDeployDir/MyProject/config/launch/MyDeployFile.deploy!myCorrelator
  

  The correlator deployment package can then be injected into the correlator using engine_initalize or engine_inject.

• Perform the initialization into a running correlator by pointing to a deployment file within the project and excluding specific files from the injection:

  engine_deploy --inject C:/MyDeployDir
     --exclude MyProject/dashboards/**
     MyProject/config/launch/MyDeployFile.deploy!myCorrelator
  

  The --exclude option specifies that the generated deployment directory does not contain the files from the MyProject/dashboards directory, and that these files are also not to be used during injection.

Deleting code from a correlator

The engine_delete tool removes EPL code and JMon applications from the correlator. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Note: Apama’s in-process API for Java (JMon) is deprecated and will be removed in a future release.

Synopsis

To remove applications from the correlator, run the following command:

engine_delete [ options ] [ name1 [ name2... ] ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_delete tool deletes named applications, monitors and event types from a correlator. Names are the full package names as previously assigned to an application monitor or event type when injected into the correlator.

To specify the items you want to delete, you can specify any one of the following in the engine_delete command line:

• Names of the items to delete.
• The -f option with the name of a file that contains the names of the items you want to delete. In this file, specify each name on a separate line.
• Neither of the above. In this case, the engine_delete tool reads names from stdin until you type an end-of-file signal, (Ctrl+D on UNIX and Ctrl+Z on Windows). If you want, you can specify a hyphen (-) in the command line to indicate that input will come from stdin.

The tool is silent by default unless an error occurs. To receive progress information, specify the –v option.

The tool permits two kinds of operations: delete and kill. These cause different side-effects. Therefore, you must use them carefully.

• When you delete a monitor, the correlator tries to terminate all of that monitor’s instances. If they are responsive (not in some deadlocked state), each one executes its ondie() action, and when the last one exits the correlator calls the monitor’s onunload() action. This assumes that the monitor you are deleting defines ondie() and onunload() actions.

  If a monitor instance does not respond to a delete request, the correlator cannot invoke the monitor’s onunload() action. In this case, you must kill, rather than delete, the monitor instance.

• When you kill a monitor, the correlator immediately terminates all of the monitor’s instances, without invoking ondie() or onunload() actions.

Time taken to delete code

Deleting code from a correlator can require scanning the state of the correlator to ensure that the types being deleted are no longer in use. Thus, the deletion will run at least as slowly as it takes the slowest context in the correlator to respond to external events, and will depend on how many objects there are live in the correlator.

If a type is found to be in use and you are not using the -F or -a option, then the deletion will fail with an error message, reporting what is still using the type that was requested to be deleted. If events of the type being deleted are sent to the correlator, they will fail to be parsed and the correlator will report errors.

Options

The engine_delete tool takes the following command line options:

Operands

The engine_delete tool takes the following operands:

Exit status

The engine_delete tool returns the following exit values:

Packaging correlator input files

The engine_package tool assembles EPL files, JAR files and event files into a correlator deployment package (CDP). You can inject a CDP file into the correlator just as you inject an EPL file. CDP files use a proprietary, non-plaintext format that treats files in a manner similar to the way a JAR file treats a collection of Java files. In addition, using a CDP file guarantees that all files, assuming no errors, are injected and are injected in the correct order. See Injecting code into a correlator for details about how the correlator handles an error in a file that is in a CDP. See also Deploying a correlator for alternative ways to deploy the correlator or for creating CDPs from Apama Plugin for Eclipse projects.

While the names of events, monitors, aggregates, and JAR files that are contained in a CDP file are visible to the correlator utilities engine_inspect, engine_manage, and engine_delete, the code that defines them is not.

The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Synopsis

To package files into a CDP file, run the following command:

engine_package [ options ] [ file1 [ file2... ] ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_package tool creates a correlator deployment package (CDP). A CDP file contains one or more files. You specify the name of the CDP file to create as an argument to the -o option.

You can specify the files you want to include on the command line, or you can use the -m option and specify a manifest file that contains the names of the files. The manifest file is a text file; each line in the file specifies a relative or absolute path to a file. Files should be listed in the order in which you want them to be injected into the correlator.

You can also specify another CDP file to include in this package. The files from the original CDP are injected in the specified place in the order within this package.

Options

The engine_package tool takes the following options:

Operands

The engine_package tool takes the following operands:

Exit status

The engine_package tool returns the following exit values:

Example

The following example describes how to create a correlator deployment package file with multiple monitor files and inject the CDP file into a running correlator.

1. Create a manifest file containing a list of files to include in the CDP. For this example, the file is named “manifest.txt” and each line contains the full path name of an EPL file or JAR file:

   c:\dev\sample\monitor1.mon
   c:\dev\sample\monitor2.mon
   C:\dev\sample\java-plugin.jar
   

2. To create the CDP file, call the engine_package tool stating the output filename and the manifest file to include in the CDP. (Note, instead of using a manifest file, you can list the files individually in the engine_package arguments.)

   engine_package.exe -o c:\sample.cdp -m c:\dev\sample\manifest.txt
   

3. To inject the CDP file, call the engine_inject tool with -c (or --cdp). This injects each file that is included in the CDP file into the correlator.

   engine_inject.exe -c c:\sample.cdp
   

   Sample output from the correlator:

   2012-07-11 13:51:33.156 INFO  [3852] - Injected CDP from file
           c:\sample.cdp (b2f097b02791e5dd4ac73cda38e153e9),
           size 313 bytes, decoding and compile time 0.00 seconds
   

Sending events to correlators

The engine_send tool sends Apama-format events into a correlator or IAF adapter. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

If the events you want to send are not in Apama format, you must use an adapter that can transform your event format into Apama event format.

Note:

You can also send events using Apama Plugin for Eclipse. For more information, see Sending an event from the Engine Information view.

Synopsis

To send Apama-format events to a correlator or IAF adapter, run the following command:

engine_send [ options ] [ file1 [ file2... ] ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_send tool sends Apama-format events to a correlator. In Apama-format event files, you can specify whether to send the events in batches of one or more events or at set time intervals.

The correlator reads events from one or more specified files. Alternatively, you can specify a hyphen (-) or not specify a filename so that the correlator reads events from stdin until it receives an end-of-file signal (Ctrl+D on UNIX and Ctrl+Z on Windows).

For details about Apama-format events, see Event file format.

By default, the engine_send tool is silent unless an error occurs. To view progress information during engine_send execution, specify the -v option when you invoke engine_send.

You can also use engine_send to send events directly to the Integration Adapter Framework (IAF). To do this, specify the port of the IAF. By default, this is 16903.

Options

The engine_send tool takes the following options:

Operands

The engine_send tool takes the following operands:

Exit status

The engine_send tool returns the following exit values:

Operating notes

To end an indefinite cycle of sending events, press Ctrl+C in the window in which you invoked the engine_send tool.

You might want to indefinitely cycle through and send events in the following situations:

• In test environments. For example, you can use engine_send to simulate heartbeats. If you then kill the engine_send process, you can test your EPL code that detects when heartbeats stop.
• In production environments. For example, you can use the engine_send tool to initialize a large data table in the correlator.

Text encoding

By default, the engine_send tool checks the environment variable or global setting that specifies the locale because this indicates the local character set. The engine_send tool then translates EPL text from the local character set to UTF-8. Consequently, it is important to correctly set the machine’s locale.

However, some input files might start with a UTF-8 Byte Order Mark. The engine_send tool treats such input files as UTF-8 and does not do any translation. Alternatively, you can specify the -u option when you run the engine_send tool. This forces the tool to treat each input file as UTF-8.

Receiving events from correlators

The engine_receive tool lets you connect to a running correlator and receive events from it. Events received and displayed by the engine_receive tool are in Apama event format. This is identical to the format used to send events to the correlator with the engine_send tool. Consequently, it is possible to reuse the output of the engine_receive tool as input to the engine_send tool.

The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Synopsis

To receive Apama-format events from a correlator, run the following command:

engine_receive [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_receive tool receives events from a correlator and writes them to stdout or to a file that you specify. The correlator output format is the same as that used for event input and is described in Event file format.

You can specify one or more channels on which to listen for events from the correlator. The default is to receive all output events. For more information, see Subscribing to channels.

To view progress information during engine_receive execution, specify the –v option.

You can also use engine_receive to receive events emitted by the Integration Adapter Framework (IAF) directly. To do this, specify the port of the IAF. By default, this is 16903.

Options

The engine_receive tool takes the following options:

Exit status

The engine_receive tool returns the following exit values:

Text encoding

The engine_receive tool translates all events it receives from UTF-8 into the current character locale. It is therefore important that you correctly set the machine’s locale. To force the engine_receive tool to output events in UTF-8 encoding, specify the -u option.

Watching correlator runtime status

The engine_watch tool lets you monitor the runtime operational status of a running correlator. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Synopsis

To monitor the operation of a correlator, run the following command:

engine_watch [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_watch tool periodically polls a correlator for status information, writing the standard status messages to stdout (see List of correlator status statistics for more information on the standard status messages). When you also specify the -a option, any user-defined status values are appended to the standard status messages. For additional progress information, use the –v option.

Options

The engine_watch tool takes the following options:

Exit status

The engine_watch tool returns the following exit values:

List of correlator status statistics

This topic gives a detailed list of the status values that can be monitored for a correlator. The descriptions below show where the status values are used. The status is available through the following mechanisms:

• REST API: The name of the key in the REST API. See also Managing and monitoring over REST and the descriptions of /correlator/status and /info/stats in the API reference for Component Management REST APIs.

• Java API: The name of the method in the EngineStatus Java API (and also the equivalent, where available, in the C++,.NET and EPL APIs). See the com.apama.engine.EngineStatus interface in the API reference for Java (Javadoc).

  Note:

  In the C++ API, the status statistics names are defined as constants inside the client_status_names.hpp file. See also the API reference for C++ (Doxygen).

• Log field: The name of the status log field in the Status: log lines in the main correlator log file. See also Descriptions of correlator status log fields.

• Prometheus metric name: The name used to expose internal correlator statistics to the Prometheus monitoring system. See also Monitoring with Prometheus.

• Display name: The standard status message that the engine_watch tool writes to stdout (see Watching correlator runtime status). The same status message is also shown in the Engine Status view of Apama Plugin for Eclipse (see Using the Engine Status view).

The descriptions below also indicate the typical trend. This can be one of the following:

• Steady: After any start-up phase, this number would typically be steady. It may increase as bursts of events come in, or if there is a change in the size of the application (for example, the number of items the application is tracking). Typically, if these numbers are continually trending upwards when there is no more being asked of the application, that indicates an application leak of monitor instances, listeners or objects. This will eventually lead to an out of memory condition.
• Increasing: This may be increasing in normal usage. Depending on deployment, some statistics may not be increasing, though if they normally are and have stopped increasing, this may indicate that something is preventing events being delivered or processed correctly.
• Low: This number is typically 0 or near 0. If this number increases, this typically indicates that the correlator is not keeping up with processing events. For queues, it is normal that during bursts of activity, these may be non-zero for some time. Steadily increasing queue sizes can be a sign of back-pressure due to a slow receiver, or the system is not keeping up and may eventually block senders due to not processing the events at the rate they arrive.
• Varies: Will typically vary. 0 may indicate a problem with events being delivered.
• None: Typically, all contexts and receivers should be keeping up, so none are reported as slow (in which case, the empty string will be returned from the API).

The term “receiver” which is used in the descriptions below refers to any of the following:

• EPL, Java or C++ plug-ins using the Correlator.subscribe method.
• Connectivity plug-ins for “towards” transport events.
• JMS connections sending events out of the correlator.
• Client library connections, including other correlators that have been connected with the engine_connect, iaf or engine_receive tools.

Time since the correlator was started

The time in milliseconds since the correlator was started.

Typical trend: increasing.

• REST API: uptime
• Java API: getUptime
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_uptime_seconds
• Display name: Uptime (ms)

Number of contexts

The number of contexts in the correlator, including the main context.

Typical trend: steady.

• REST API: numContexts
• Java API: getNumContexts
• Log field: nctx=n
• Prometheus metric name: sag_apama_correlator_contexts_total
• Display name: Number of contexts

Number of monitors

The number of EPL monitor definitions injected into the correlator. This number changes on injections, deletions or if the last instance of a monitor terminates.

Typical trend: steady.

• REST API: numMonitors
• Java API: getNumMonitors
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_monitors_total
• Display name: Number of monitors

Number of monitor instances

The number of monitor instances, also known as sub-monitors.

Typical trend: steady.

• REST API: numProcesses
• Java API: getNumProcesses
• Log field: sm=n
• Prometheus metric name: sag_apama_correlator_monitor_instances_total
• Display name: Number of sub-monitors

Number of Java applications and Java EPL plug-ins

The number of Java applications and Java EPL plug-ins loaded in the correlator. This number changes on injections and deletions.

Typical trend: steady.

• REST API: numJavaApplications
• Java API: getNumJavaApplications
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_java_applications_total
• Display name: Number of Java applications

Number of listeners

The number of listeners in all contexts. This includes on statements and active stream source templates.

Typical trend: steady.

• REST API: numListeners
• Java API: getNumListeners
• Log field: ls=n
• Prometheus metric name: sag_apama_correlator_listeners_total
• Display name: Number of listeners

Number of sub-listeners

The number of sub-event-listeners that are active across all contexts. Stream source templates will have one sub-event-listener. An on statement can have multiple sub-event-listeners. See also Evaluating event listeners for all A-events followed by B-events.

Typical trend: steady.

• REST API: numSubListeners
• Java API: getNumSubListeners
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_sub_listeners_total
• Display name: Number of sub-listeners

Number of event types

The number of event types defined within the correlator. This number changes on injections and deletions.

Typical trend: steady.

• REST API: numEventTypes
• Java API: getNumEventTypes
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_event_types_total
• Display name: Number of event types

Number of executors on input queues

The number of executors on the input queues of all contexts. As well as events, this can include clock ticks, spawns, injections and other operations. A context in an infinite loop will grow by 10 per second due to clock ticks. Every context has an input queue, which by default is a maximum of 20,000 entries.

Typical trend: low.

• REST API: numQueuedInput
• Java API: getNumQueuedInput
• Log field: iq=n
• Prometheus metric name: sag_apama_correlator_queued_input_total
• Display name: Events on input queue

Number of received events

The number of events that the correlator has received from external sources since the correlator started. This includes connectivity plug-ins, correlator-integrated JMS, engine_send, other correlators connected with engine_connect, dashboard servers, the IAF, and events that are not parsed correctly. This number excludes events sent within the correlator from EPL monitors or EPL plug-ins.

Typical trend: increasing.

• REST API: numReceived
• Java API: getNumReceived
• Log field: rx=n
• Prometheus metric name: sag_apama_correlator_input_total
• Display name: Events received

Number of processed events

The number of events processed by the correlator in all contexts. This includes external events and events routed to contexts by monitors. An event is considered to have been processed when all listeners and streams that were waiting for it have been triggered, or when it has been determined that there are no listeners for the event.

Typical trend: increasing.

• REST API: numProcessed
• Java API: getNumProcessed
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_processed_total
• Display name: Events processed

Sum of events on route queues

The sum of routed events on the route queues of all contexts.

Typical trend: low.

• REST API: numQueuedFastTrack
• Java API: getNumQueuedFastTrack
• Log field: rq=n
• Prometheus metric name: sag_apama_correlator_queued_route_total
• Display name: Events on internal queue

Number of routed events

The number of events that have been routed across all contexts since the correlator was started.

Typical trend: increasing.

• REST API: numFastTracked
• Java API: getNumFastTracked
• Log field: rt=n
• Prometheus metric name: sag_apama_correlator_route_total
• Display name: Events routed internally

Number of external consumers/receivers

The number of external consumers/receivers connected to receive emitted events. This includes connectivity plug-ins, correlator-integrated JMS, engine_receive, or correlators connected using engine_connect.

Typical trend: steady.

• REST API: numConsumers
• Java API: getNumConsumers
• Log field: nc=n
• Prometheus metric name: sag_apama_correlator_consumers_total
• Display name: Number of consumers

Number of events on output queues

The number of events waiting on output queues to be dispatched to any connected external consumers/receivers.

Typical trend: low.

• REST API: numOutEventsQueued
• Java API: getNumOutEventsQueued
• Log field: oq=n
• Prometheus metric name: sag_apama_correlator_queued_output_total
• Display name: Events on output queue

Number of events created for sending to external channels

The number of events that have been sent (see The send… to statement) or emitted (see The emit statement) to channels which have at least one external consumer/receiver subscribed (see also Number of external consumers/receivers). This excludes events sent to channels with no external consumers/receivers. This counts each event once, even if delivered to multiple external consumers/receivers.

Typical trend: increasing.

• REST API: numEmits
• Java API: getNumOutEventsCreated
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_created_output_total
• Display name: Output events created

Number of events delivered to external consumers/receivers

The number of events that have been delivered to external consumers/receivers. This counts for each external consumer/receiver an event is sent to. It counts the number of deliveries of events.

Note:

This status indicator counts every event that was delivered, whereas the previous status indicator counts every event that was sent. For example, sending one event to a channel with two external consumers/receivers would be counted as one event sent (numEmits), but two events delivered (numOutEventsSent).

Typical trend: increasing.

• REST API: numOutEventsSent
• Java API: getNumOutEventsSent
• Log field: tx=n
• Prometheus metric name: sag_apama_correlator_output_total
• Display name: Output events sent

Number of events on input queues of all public contexts

The number of events on the input queues of all public contexts. See also About context properties for information on the receiveInput flag.

Typical trend: low.

• REST API: numInputQueuedInput
• Java API: getNumInputQueuedInput
• Log field: icq=n
• Prometheus metric name: sag_apama_correlator_queued_input_public_total
• Display name: Events on input context queues

Name of slowest context

The name of the slowest context. This may or may not be a public context.

Typical trend: none.

• REST API: mostBackedUpInputContext
• Java API: getMostBackedUpInput
• Log field: lcn=name
• Prometheus metric name: The name of the slowest context is given as a Prometheus label on the Prometheus metric sag_apama_correlator_slowest_input_queue_size_total
• Display name: Slowest context name

Number of events on queue for slowest context

The number of events on the slowest context’s queue, as identified by the name of the slowest context.

Typical trend: low.

• REST API: mostBackedUpICQueueSize
• Java API: getMostBackedUpQueueSize
• Log field: lcq=n
• Prometheus metric name: sag_apama_correlator_slowest_input_queue_size_total
• Display name: Slowest context queue size

Time difference in seconds for slowest context

For the context identified by the slowest context name, this is the time difference in seconds between its current logical time and the most recent time tick added to its input queue.

Typical trend: low.

• REST API: mostBackedUpICLatency
• Java API: getMostBackedUpICLatency
• Log field: lct=seconds
• Prometheus metric name: sag_apama_correlator_slowest_input_queue_latency_seconds
• Display name: not applicable

Name of slowest consumer/receiver of events

The name of the consumer/receiver with the largest number of incoming events waiting to be processed. This is the slowest non-context consumer/receiver of events, which can be an external receiver or an EPL plug-in.

Typical trend: none.

• REST API: slowestReceiver
• Java API: getSlowestReceiver
• Log field: srn=name
• Prometheus metric name: The name of the slowest consumer/receiver of events is given as a Prometheus label on the Prometheus metric sag_apama_correlator_slowest_output_queue_size_total
• Display name: Slowest receiver name

Number of events on queue for slowest consumer/receiver

The number of events on the slowest consumer’s/receiver’s queue, as identified by the name of the slowest consumer/receiver.

Typical trend: low.

• REST API: slowestReceiverQueueSize
• Java API: getSlowestReceiverQueueSize
• Log field: srq=n
• Prometheus metric name: sag_apama_correlator_slowest_output_queue_size_total
• Display name: Slowest receiver queue size

Number of events per second

The number of events per second currently being processed by the correlator across all contexts. This value is computed with every status refresh and is only an approximation.

Typical trend: varies.

• REST API: not applicable
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: not applicable
• Display name: Event rate over last interval

Number of enqueued events

The number of events queued from the enqueue statement (not the enqueue...to statement). The enqueue statement is deprecated.

Typical trend: low.

• REST API: enqueueQueueSize
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: not applicable
• Display name: not applicable

Virtual memory

Virtual memory. For the REST API, the value is in megabytes. For the log field, the value is in kilobytes. For Prometheus, the value is in bytes.

Typical trend: steady.

• REST API: virtualMemoryMB
• Java API: not applicable
• Log field: vm=kB
• Prometheus metric name: sag_apama_correlator_virtual_memory_bytes
• Display name: not applicable

Physical memory

Physical memory. For the REST API, the value is in megabytes. For the log field, the value is in kilobytes. For Prometheus, the value is in bytes.

Typical trend: steady.

• REST API: physicalMemoryMB
• Java API: not applicable
• Log field: pm=kB
• Prometheus metric name: sag_apama_correlator_physical_memory_bytes
• Display name: not applicable

Peak physical memory usage

The highest amount of physical memory used by the correlator at any measurement point since startup, given in units of megabytes. This is the highest measured amount of memory, measured when a status line is logged or status is requested from the correlator.

Typical trend: steady.

• REST API: peakPhysicalMemoryMB
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: not applicable
• Display name: not applicable

Number of contexts on run queue

The number of contexts on the run queue. These are the contexts that have work to do but are not currently running.

Typical trend: low.

• REST API: not applicable
• Java API: not applicable
• Log field: runq=n
• Prometheus metric name: not applicable
• Display name: not applicable

Number of pages read from swap space

The number of pages per second that are being read from swap space. If this is greater than zero, it may indicate that the machine is under-provisioned, which can lead to reduced performance, connection timeouts and other problems. Consider adding more memory, reducing the number of other processes running on the machine, or partitioning your Apama application across multiple machines.

Typical trend: low.

• REST API: swapPagesRead
• Java API: not applicable
• Log field: si=n
• Prometheus metric name: sag_apama_correlator_swap_pages_read_hertz
• Display name: not applicable

Number of pages written to swap space

The number of pages per second that are being written to swap space. If this is greater than zero, it may indicate that the machine is under-provisioned, which can lead to reduced performance, connection timeouts and other problems. Consider adding more memory, reducing the number of other processes running on the machine, or partitioning your Apama application across multiple machines.

Typical trend: low.

• REST API: swapPagesWrite
• Java API: not applicable
• Log field: so=n
• Prometheus metric name: sag_apama_correlator_swap_pages_write_hertz
• Display name: not applicable

Number of snapshots

The number of persistence snapshots taken since the correlator started. These statistics will only exist if the correlator is running in persistent mode.

Typical trend: increasing.

• REST API: persistenceNumSnapshots
• Java API: not applicable
• Log field: numSnapshots=n
• Prometheus metric name: sag_apama_correlator_persistence_snapshots_total
• Display name: not applicable

Timestamp of last snapshot

The UNIX timestamp of the last completed snapshot. These statistics will only exist if the correlator is running in persistent mode.

Typical trend: increasing.

• REST API: persistenceLastSnapshotTime
• Java API: not applicable
• Log field: lastSnapshotTime=timestamp
• Prometheus metric name: sag_apama_correlator_persistence_last_snapshot_timestamp
• Display name: not applicable

Time waiting for a snapshot (EWMA)

An exponentially weighted moving average (EWMA) of the time in milliseconds taken to wait for a snapshot. These statistics will only exist if the correlator is running in persistent mode.

Typical trend: varies.

• REST API: persistenceSnapshotWaitTimeEwmaMillis
• Java API: not applicable
• Log field: snapshotWaitTimeEwmaMillis=milliseconds
• Prometheus metric name: sag_apama_correlator_persistence_snapshot_wait_ewma_seconds
• Display name: not applicable

Time to commit to database (EWMA)

An exponentially weighted moving average (EWMA) of the time in milliseconds taken to commit to the database. These statistics will only exist if the correlator is running in persistent mode.

Typical trend: varies.

• REST API: persistenceCommitTimeEwmaMillis
• Java API: not applicable
• Log field: commitTimeEwmaMillis=milliseconds
• Prometheus metric name: sag_apama_correlator_persistence_commit_time_ewma_seconds
• Display name: not applicable

Number of changed rows (EWMA)

An exponentially weighted moving average (EWMA) of the number of rows changed per snapshot. These statistics will only exist if the correlator is running in persistent mode.

Typical trend: varies.

• REST API: persistenceLastSnapshotRowsChangedEwma
• Java API: not applicable
• Log field: lastSnapshotRowsChangedEwma=n
• Prometheus metric name: sag_apama_correlator_persistence_snapshot_rows_changed_ewma_total
• Display name: not applicable

Total heap memory used by the JVM

The total heap memory used by the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.

Typical trend: steady.

• REST API: jvmMemoryHeapUsedMB
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_jvm_heap_used_bytes
• Display name: not applicable

Total free heap memory in the JVM

The total heap memory that is free in the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.

Typical trend: steady.

• REST API: jvmMemoryHeapFreeMB
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_jvm_heap_free_bytes
• Display name: not applicable

Total non-heap memory used by the JVM

The total non-heap memory used by the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.

Typical trend: steady.

• REST API: jvmMemoryNonHeapUsedMB
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_jvm_non_heap_used_bytes
• Display name: not applicable

Total memory used by all buffer pools in the JVM

The sum of memory used by all buffer pools in the Java virtual machine (JVM) which is embedded in the correlator. For the REST API, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.

Typical trend: steady.

• REST API: jvmMemoryBufferPoolUsedMB
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_jvm_buffer_pool_used_bytes
• Display name: not applicable

Total memory used by the JVM

The sum of all memory used by the Java virtual machine (JVM) which is embedded in the correlator (that is, the used heap memory, the used non-heap memory, and the used buffer pool memory). For the REST API and the log field, the value is in megabytes. For Prometheus, the value is in bytes. These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API and the log field will return 0 (zero) as the value, and Prometheus will not have this metric.

Typical trend: steady.

• REST API: jvmMemoryAllUsedMB
• Java API: not applicable
• Log field: jvm=MB
• Prometheus metric name: sag_apama_correlator_jvm_memory_all_bytes
• Display name: not applicable

Number of threads in use by the JVM

The total number of active threads in the Java virtual machine (JVM). These statistics will only exist if the embedded JVM has been enabled. If the JVM is disabled, the REST API will return 0 (zero) as the value, and Prometheus will not have this metric.

Typical trend: steady.

• REST API: jvmNumThreads
• Java API: not applicable
• Log field: not applicable
• Prometheus metric name: sag_apama_correlator_jvm_num_threads
• Display name: not applicable

Inspecting correlator state

The engine_inspect tool lets you inspect the state of a running correlator. This means you can review the applications loaded and running on a correlator. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Synopsis

To inspect applications on a running correlator, run the following command:

engine_inspect [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

The engine_inspect tool retrieves state information from a running correlator and sends it to stdout. By default, the tool outputs information on the monitors, JMon applications, event types and container types currently injected in a correlator.

Note: Apama’s in-process API for Java (JMon) is deprecated and will be removed in a future release.

You can filter this list by specifying command-line options. When you specify one or more of the -m, -j, -e, -t, -x, -P, or -R options, the engine_inspect tool displays only the information indicated by the option(s) you specify. See the table below for more information on these options.

Options

The engine_inspect tool takes the following options:

com.apama.samples.stockwatch.StockWatch 1
Tick 1

Exit status

The engine_inspect tool returns the following exit values:

Shutting down and managing components

All Apama components (correlator, IAF, dashboard data server, and dashboard display server) implement an interface with which they can be asked to shut themselves down, provide their process identifier, and respond to communication checks.

For historical reasons, there are several tools that all do the same thing. You can use any of these tools to manage any component:

• engine_management
• component_management
• iaf_management

When managing a correlator, the recommendation is to use the engine_management tool, which provides some additional correlator-specific options that are not available in the other tools. The only other differences in behavior among these tools are:

• engine_management and component_management default to the local correlator port (15903).
• iaf_management defaults to the default IAF port (16903).

The executable for the engine_management tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Note:

Some of the functions of the engine_management tool can be performed from within EPL. For more information, see Using the Management interface.

Synopsis

To use the correlator’s management tool, run the following command:

engine_management [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

Use the engine_management tool to connect to a running component. Once connected, the tool can shut down the component or return information about the component. The engine_management tool can connect to any of the following types of components:

• Correlator

• Adapter

• Dashboard data server and dashboard display server (using the management port, and not the data port)

  If you want to use the dedicated dashboard_management tool, see Managing and stopping the data server and display server.

The engine_management tool sends output to stdout.

Options

The engine_management tool takes the following options. These options are also available for the component_management and iaf_management tools. Keep in mind that all of these tools use different ports (see above). To obtain all information for a particular component, specify the -a option. All options are optional.

correlator.exe options
engine_inject some_EPL_files

engine_management -W 10

Management requests

The options in the tables below replicate -r (or --dorequest) request types of the same name.

The following options are specific to the correlator:

The following option is specific to the IAF:

Exit values

The engine_management tool returns the following exit values:

Viewing garbage collection and application events information

The information in this topic applies to the correlator only.

You can enable logging of verbose garbage collection and application events in different ways and at different times, as described below.

Using the engine_management tool to enable garbage collection and application event logging for a running correlator

A handy way to view garbage collection (GC) information for a running correlator is to execute the following command:

engine_management -r verbosegc on

This command enables logging of garbage collection events, and is particularly useful in production environments. The additional garbage collection information goes to the correlator log, where the garbage collection messages are logged at DEBUG level and are signposted by a prefix, <apama.verboseGC.MonitorName>, where MonitorName indicates the monitor where garbage collection occurred.

To disable logging of garbage collection information, execute the following command:

engine_management -r verbosegc off

The above commands provide an alternative to the following command, which provides a great deal of detailed output (such as context-state changes, event triggering, spawning, routing, etc.) in addition to garbage collection information:

engine_management -r applicationEventLogging on

Again, this output goes to the correlator log, with the messages being logged at DEBUG level. Any additional log messages to the garbage collection messages are signposted by the prefix <apama.applicationEvents>.

To turn this off, execute the following command:

engine_management -r applicationEventLogging off

See Shutting down and managing components for more information on the -r (or --dorequest) option of the engine_management tool.

Enabling garbage collection and application event logging at correlator startup

You can also enable garbage collection logging at correlator startup. To do so, execute the following command:

correlator --loglevel apama.verboseGC=DEBUG

This enables verbose garbage collection logging for all monitors.

If you only wish to enable garbage collection logging for a particular monitor, append the name of the monitor (for example, “MyMonitor”) to apama.verboseGC as follows:

correlator --loglevel apama.verboseGC.MyMonitor=DEBUG

This enables verbose garbage collection logging only for the monitor which has the name “MyMonitor”.

Similarly, you can also enable application event logging at start time by starting a correlator with the following command:

correlator --loglevel apama.applicationEvents=DEBUG

Note:

If you enable application event logging at start time in this fashion, this will not enable verbose garbage collection logging (as is the case when enabling it using the engine_management request). Furthermore, it is not possible to enable application event logging on a per-monitor basis, as it is for garbage collection logging.

See Starting the correlator for more information on the --loglevel (or -v) option of the correlator command.

Using a YAML configuration file to enable garbage collection and application event logging at correlator startup

Another way to enable garbage collection logging or application event logging at start time is through a YAML configuration file. For more information, see the descriptions of the following categories in Setting correlator and plug-in log files and log levels in a YAML configuration file:

• apama.verboseGC
• apama.verboseGC.MonitorName
• apama.applicationEvents

Using the EPL memory profiler

The information in this topic applies to the correlator only.

You use the EPL memory profiler to display information on monitors and monitor instances.

The EPL memory profiler is invoked using the -r (or --dorequest) option of the engine_management tool, followed by a request. Several requests are available for the EPL memory profiler, which are described below.

When a request is issued, the correlator execution is momentarily paused to gather statistics.

The information that is returned for a request can be viewed directly (for example, in the Apama Command Prompt), or it can be written to a comma-separated values (CSV) file which can easily be viewed in tabular form using a tool such as Microsoft Excel.

Note: All byte counts returned by a request are approximate values. The EPL memory profiler only shows memory usage that can be directly attributed to individual monitor instances. There are some parts of the correlator runtime that are not tracked, but these are typically small and fixed. Any memory used by Java or C++ plug-ins is not tracked. The profiler is useful in indicating the shape of the memory usage of an application - which monitors and event types are using more memory in proportion to the rest of the EPL runtime.

The values returned for the number of bytes and the number of EPL objects also include EPL objects that are no longer being used, and have not yet been garbage-collected. Therefore, the values will never be precise unless you are lucky enough to make this request just after garbage collection has run. See also Garbage collection.

The size of event expressions, including internal data structures associated with them, is excluded (and is typically small).

Each request returns the following string, in addition to the column headers described below:

"Version:version, Snapshot time:time, Component ID:id, Host:host-name, Port:port, EPL memory:bytes"

Returning information on all monitors

The following command returns information on all the monitors in the correlator:

engine_management -r eplMemoryProfileOverview

This request does not take arguments. If arguments are passed, they are ignored.

The output shows the following information in the following order:

Returning monitor instance details

The following command returns information for all EPL types across the monitor instances of a specific monitor in the correlator:

engine_management -r eplMemoryProfileMonitorInstanceDetail monitor-name

where monitor-name is the name of the monitor.

You can also specify all to list the instance details of all monitors. The results are returned using the following sort order:

1. monitor name (ascending)
2. monitor instance ID (ascending)
3. EPL type (ascending)

The output shows the following information in the following order:

The output for the context is a combination of EPL type and monitor instance. For example, if there are 10 monitor instances where each instance has lots of objects of 3 different types, then the output will have 30 rows.

Unlike other EPL objects which belong to a single monitor instance, some strings are shared between several monitor instances. When a string is only used by a single monitor instance, it is shown like any other object in the output of the request, that is, with an EPL type of “string”. However, if the same string is shared between multiple monitor instances, then each monitor or monitor instance that is using it will show the EPL type as “string (shared)”. This is a performance optimization which avoids unnecessary copying. For example, a string may be shared in the following cases:

• When a monitor containing a string spawns to another monitor instance.
• When a monitor has a string that it sends inside an event to a monitor in another context.
• When an input event containing strings is received by multiple monitor instances which then store these strings.

One of the implications of sharing is double-counting, for both the number of EPL objects and the number of bytes. If multiple monitor instances refer to the same shared strings, the output of the request will include these numbers against each monitor instance separately. However, the duplication is eliminated when object sizes are summed up for the “EPL memory” value, so it may end up being notably lower than the sum of the “Bytes” in each row.

See Handling of reference types for more information.

Returning aggregated monitor instance details

The following command is similar to eplMemoryProfileMonitorInstanceDetail, except that it aggregates the object count and size from each monitor instance, displaying data per monitor rather than per monitor instance.

engine_management -r eplMemoryProfileMonitorDetail monitor-name

where monitor-name is the name of the monitor. You can also specify all to list all monitors, sorted by the monitor name.

The output shows the following information in the following order:

This request also takes account of shared strings. See the description of the eplMemoryProfileMonitorInstanceDetail request for details.

See Handling of reference types for more information.

Handling of reference types

For reference types (such as sequence and dictionaries), the size is not reflected in the object referencing it. Instead, the size is associated with the actual object which is referenced. For example, if an event references/contains a sequence, the size of the sequence has no effect on the byte count of that event.

The any type does not show up in the EPL memory profiler output. Reference types held in an any value show up as if they were held in a value of their type. Boxed primitives show up as a separate type (with the first letter in capitals) in the EPL memory profiler output; for example:

monitorName,false,..Integer,main,1,1,2,240
monitorName,false,AnyContainer,main,1,1,2,240

In the above example, there are two AnyContainer objects, and there are two boxed primitives (the ..Integer type). For both, there are two objects in memory and are consuming 240 bytes for each type.

Visualizing the EPL memory profiler information in Microsoft Excel

You can write the output from each of the above requests to a comma-separated values (CSV) file which can easily be viewed in tabular form using a tool such as Microsoft Excel.

The example below shows how to visualize the output of the eplMemoryProfileMonitorInstanceDetail request in Microsoft Excel using a pivot table.

1. Save the output of the request in a CSV file and open this file with Microsoft Excel.

2. Create a new PivotTable in Microsoft Excel, and in the resulting dialog select the range of data for the pivot table (ideally, the information for the entire range is already provided in the corresponding text box).

3. For this example, add the following fields to the report:

   • Monitor

   • EPL type

   • Context name

   • EPL objects

   • Bytes If required, you can also add the following fields:

   • Monitor instance ID

   • Context ID The monitor instance ID is helpful if multiple monitor instances exist within the same context.

   After you have added the fields, you can see the following in the table:

   • row labels which include the monitor name, the context name and the EPL type, and
   • columns which sum up the number of EPL objects and the approximate number of bytes. For example:

   

4. If you want to get an overview of the context level in the row labels, just drag the Context name label above the Monitor label as shown in the example below, and then check the changes in the report:

   

5. Similarly, if you want to see how the EPL objects are distributed over the different contexts and monitors, just move the EPL type to the very top of the row labels, followed by the Monitor and Context name labels.

6. Once the data is shown as wanted in the table, you can conditionally format the table to highlight individual columns, for example, to show high values or values that are above the threshold or above the average. Detailed information on how to do this can be found in the Excel help.

   A basic use case is to highlight the values for the object count and byte count that are above the average. To do so, select the Sum of EPL objects column and then choose the following command: Conditional Formatting > Top/Bottom Rules > Above Average. You can then select a formatting option from a dialog, for example, red text. As a result, all values in the cells of the Sum of EPL objects column that are above the average are shown in red. If you want, you can do the same for the Sum of Bytes column.

   You can also use additional conditional formatting (for example, color scales) to highlight the cells with values above the average.

Using the CPU profiler

The information in this topic applies to the correlator only.

Using the CPU profiler, you can profile applications written with EPL. Data collected in the profiler allows you to identify possible bottlenecks in an EPL application. When testing an application, or after you deploy an application, you might find it handy to write a script that includes obtaining profile information. The CPU profiler that is described here allows you to obtain profile information without the overhead of Apama Plugin for Eclipse (see also Profiling EPL Applications).

The CPU profiler is invoked using the -r (or --dorequest) option of the engine_management tool:

engine_management -r cpuProfile argument

where argument can be one of the following:

If a context is executing, it is typically in the EPL interpreter. However, it might also be doing something such as matching events or collecting garbage. For EPL execution, there is a call stack for each context. For the purposes of the profiler, there is one entry at the top for the monitor name, then comes the listener/onload action, and then any actions that is calling, and so on. The only action that the correlator is actually executing is at the bottom of the stack.

A context can be in one or two of the following states:

• CPU. The correlator is executing code in this context.
• Runnable. The correlator has work to do in this context, but it has been rescheduled because the correlator is executing code in another context.
• Idle. The correlator has no work to do in this context.
• Non-Idle. The correlator has work to do in this context. When a context is in this state, it is also in one other state: CPU, Plug-in, Blocked, or Runnable.
• Plug-in. The correlator is executing a plug-in in this context.
• Blocked. The correlator cannot make progress in this context. It is blocked because of a full queue. The full queue might be the correlator output queue (the context is trying to emit an event) or another context’s input queue.

When the profiler takes a sample, it examines every context in the correlator. Every entry in each context’s call stack results in addition or modification of a line in the profiler output. The Cumulative column is incremented for all samples, and one or more of the other columns is incremented for the lowest (deepest) call stack element according to what states the context is in.

When the correlator is not executing EPL code, there is only one element in the stack, for example, when the correlator is processing an event.

The profiler’s resolution is to a EPL action. That is, the profiler does not distinguish between lines within an action. The line number in the output is the first line of the action that generates code. For example, variable declarations without initializers, and comments do not generate code, while statements, and declarations with initializers, do generate code. The profiler treats the body of a listener (the code the correlator executes when the listener fires) as an action with the name ::listenerAction::.

If you want to profile parts of a single large action, you need to split the action into multiple actions in order to determine where time is spent. Remember that action calls have some cost, so that could skew the results.

The cpuProfile get or cpuProfile gettotal request returns samples to stdout as lines of comma-separated values.

Output is sorted by context and then by CPU time. For example:

Context ID,Context name,Location,Filename and line number,Cumulative time,
CPU time,Empty,Non-Idle,Idle,Runnable,Plug-in,Blocked,Total ticks:573
3,3,processor:processor.::listenerAction::,create-state.mon:
50,556,293,0,556,0,0,263,0

In the above output, nearly all of the time of this context (3) is spent in the listener that starts on line 50 of create-state.mon. The time is spread between executing EPL code (293 samples) and executing a plug-in (263 samples). Each context spent similar amounts of time executing EPL and executing plug-ins but in different listeners (notice the different line numbers).

This output is intended to be imported to a spreadsheet, such as Microsoft Excel. If you do that, then the values in one sample (one row) provide the following information in the following order:

"Version:version, Snapshot time:time, Profile start time:time, Component ID:id, Host:host-name, Port:port"

Setting EPL log files and log levels dynamically

This topic describes how to configure logging for individual EPL packages. It applies to the correlator only. For information about configuring the log level of the whole correlator and plug-ins running inside it, see Setting correlator and plug-in log files and log levels in a YAML configuration file.

You can configure per-package logging in several ways:

• Dynamically, using the following options of the engine_management tool as described in this topic:
  • --setApplicationLogFile
  • --setApplicationLogLevel
• Statically, in a YAML configuration file when starting the correlator. See Setting EPL log files and log levels in a YAML configuration file for detailed information.
• Dynamically from within EPL. See Using the Management interface for detailed information.

In EPL code, you can specify log statements as a development or debug tool. By default, log statements that you specify in EPL send information to the correlator log file. If a log file was not specified when the correlator was started, and you have not executed the engine_management tool to associate a log file with the correlator, log statements send output to stdout.

In place of this default behavior, you can specify different log files for individual packages, monitors and events. This can be helpful during development. For example, you can specify a separate log file for a package or monitor you are implementing, and direct log output from only your development code to that file.

Also, you can specify a particular log level for a package, monitor, or event. The settings of log files and log levels are independent of each other. That is, you can set only a log level for a particular package, monitor or event, or you can set only a log level for a particular element. The topics below provide information for managing individual log files and log levels.

See also Rotating correlator log files.

Note:

Regularly rotating log files and storing the old ones in a secure location may be important as part of your personal data protection policy. For more information, see Recommendations for logging by Apama application code.

Tree structure of packages, monitors, and events

Packages, monitors and events form a tree as illustrated in the figure below. For each node in the tree, you can specify a log file and/or a log level. Nodes for which you do not specify log settings inherit log settings from their parent node.

The root of the tree is the default package, which contains code that does not explicitly specify a package with the package statement. Specified packages are intermediate nodes. Packages can nest inside each other. Monitors and events in specified packages are leaf nodes. If you specify an event type in a monitor, that event is a leaf node and its containing monitor is an intermediate node.

For example, suppose you specify packageA.log as the log file for packageA. The packageA.log file receives output from log statements in MonitorE and MonitorK. If EventF contains any action members that specify log statements, output would go to the packageA.log file.

Now suppose that you set ERROR as the log level for the default package and you set INFO as the log level for PackageB. For log statements in MonitorG, PackageH, and MonitorL, the correlator compares the log statement’s log level with INFO. For log statements in the rest of the tree, the correlator compares the log statement’s log level with ERROR. For details, see the table in Managing EPL log levels.

Managing EPL log levels

To set the log level for a package, monitor or event, invoke the engine_management tool as follows:

engine_management --setApplicationLogLevel [node=]logLevel

To obtain the log level for a particular node, invoke the tool as follows:

engine_management --getApplicationLogLevel [node]

If you do not specify a node, the tool returns the log level for the default package. To remove the log level for a node, so that it takes on the log level of its parent node, invoke the tool as follows. Again, if you do not specify a node, you remove the log level for the default package. The default package then takes on the log level in effect for the correlator. The default correlator log level is INFO.

engine_management --unsetApplicationLogLevel [node]

To manage the log level for an event that you define in a monitor, see Managing event logging attributes.

After the correlator identifies the applicable log level, the log level itself determines whether the correlator sends the log statement output to the appropriate log file. The following table indicates which log level identifiers cause the correlator to send the log statement to the appropriate log file.

See also Log levels determine results of log statements.

Managing EPL log files

To specify a log file for a package, monitor or event, invoke the engine_management tool as follows:

engine_management --setApplicationLogFile  [node=]logFile

To obtain the path of the log file for a particular node, invoke the tool as follows:

engine_management --getApplicationLogFile [node]

If you do not specify a node, the tool returns the log file for the default package. To disassociate a log file from its node, so that the node uses the log file of its parent node, invoke the tool as follows. Again, if you do not specify a node, you disassociate the log file from the default package. The correlator log file is then in effect for the default package. If a log file has not been specified for the correlator, the default is stdout.

engine_management --unsetApplicationLogFile [node]

Managing event logging attributes

If you specify an event type in a monitor, that event does not inherit the logging configuration from the enclosing monitor. It is expected that this will change in a future release. To explicitly set logging attributes for an event type defined in a monitor, invoke the engine_management tool and specify an unqualified event type name. Do not specify an enclosing scope, such as com.apamax.myMonitor.NestedEventType. For example:

engine_management --setApplicationLogFile NestedEventType=foo.log
engine_management --setApplicationLogLevel NestedEventType=DEBUG

Rotating correlator log files

Rotating a correlator log file refers to closing a log file being used by a running correlator and opening a new log file to be used instead from that point onwards. This lets you archive log files and avoid log files that are too large to easily view.

Each site should decide on and implement its own correlator log rotation policy. You should consider the following:

• How often to rotate log files.
• How large a correlator log file can be.
• What correlator log file naming conventions to use to organize log files.

There is a lot of useful header information in the main log file being used when the correlator starts. If you need to provide log files to Apama technical support, you should be able to provide the log file that was in use when the correlator started, as well as any other log files that were in use before and when a problem occurred.

Note:

Regularly rotating log files and storing the old ones in a secure location may be important as part of your personal data protection policy. For more information, see Protecting and erasing data from Apama log files.

To rotate the correlator log file and also rotate any other log file the correlator is using (input log file, EPL log files), see Rotating all correlator log files.

To rotate only the main correlator log file, see Rotating specified log files.

Note:

Some people use the term “log rolling” instead of “log rotation”.

Rotating all correlator log files

The information in this topic applies to the correlator only.

To invoke rotation of all log files that the correlator is using, you can do the following:

• Invoke the engine_management tool and specify --rotateLogs.

  This rotates the main correlator log file, the correlator input log file if it is being generated, and any EPL log files that are being generated. When you invoke this management request then the correlator closes each log file it was using.

  If the log filename specification declared ${START_TIME}, ${ROTATION_TIME} and/or ${ID}, then the correlator starts new log files with updated names according to the log filename specification; for example, if ${ID} was specified, then the ID portion of a log filename would be incremented by 1.

• In EPL, create a monitor that uses the Management interface EPL plug-in to trigger log rotation on a schedule. See Using the Management interface.

• On UNIX only, you can write a cron job that periodically sends a SIGHUP signal to Apama processes.

  The standard UNIX SIGHUP mechanism causes Apama processes to re-open their log files. If the log file names were specified with ${ROTATION_TIME} and/or ${ID}, then the re-opened files have names that contain the rotation time and/or the incremented ID.

If you want a log filename to always be the same and so did not declare ${START_TIME}, ${ROTATION_TIME} or ${ID} in the log filename specification, then the correlator starts new log files that have the same names as the log files it closed. On Windows, this would overwrite the closed log files, so you must move the log files before invoking rotation. On UNIX, log files are appended to if the names are the same.

Rotating specified log files

Run one of the following utilities to rotate a particular log file. On Windows, set up scheduled tasks that run the utilities. On UNIX, write a cron job that periodically runs the utilities. The behavior is the same on both Windows and UNIX, except as noted.

• The following command (which is available for both the correlator and the IAF) instructs the component to close its component-specific log file and start using a new log file that has the name you specify. If the name of the file contains blanks, be sure enclose it in quotation marks.

  engine_management --setLogFile log-filename
  

• The following command (which is only available for the correlator) instructs the correlator to use the specified file as the log file for the specified node, which can be a package, monitor, or event. See also Setting EPL log files and log levels dynamically.

  engine_management --setApplicationLogFile [node=]log-filename
  

  If you use separate log files for particular packages, monitors, or events you might want to rotate those log files at the same time that you rotate the main correlator log file. This keeps your Apama log files in sync with each other. See Rotating all correlator log files.

On Windows, when you rotate a log file, you must ensure that the new log filename is different from the name of the log file that was in use. Apama takes care of this for you if you specify ${ROTATION_TIME} and/or ${ID} in the log-filename specification. If the name is not different, the old file is overwritten. If you want to use the same log filename, then you must move the file before you rotate it.

On UNIX, a log file is never overwritten. If you rotate a log file and specify the same name, then Apama appends messages to the content already there.

Apama does not support automatic log file rotation based on log file size.

The only way to rotate the correlator input log is to rotate all correlator log files. See Rotating all correlator log files.

Using the command-line debugger

The engine_debug tool lets you control execution of EPL code in the correlator and inspect correlator state. This tool is a correlator client that runs a single command from the command line. It is not an interactive command-line debugger. The executable for this tool is located in the bin directory of the Apama installation.

In general, this tool is expected to be most useful when you are ready to deploy your application or after deployment. During development, the interactive debugger in Apama Plugin for Eclipse will probably be most useful to you.

Before you run the engine_debug tool, specify the -g option when you start the correlator. Specification of this option disables some correlator optimizations. If you run the engine_debug tool and you did not specify the -g option when you started the correlator, the optimizations hinder the debugging process. For example, the correlator might simultaneously execute multiple statements over multiple lines even if you are using debugger commands to step through the program line by line.

Synopsis

To debug applications on a running correlator, run the following command:

engine_debug [ [ global-options ] [command [options]]... ]

To obtain a usage message, run the command with the help option.

Description

Debugging a running correlator has some effect on the other programs that connect to that correlator. While you pause a correlator, the expected behavior of connected components is as follows:

• Sending events to the correlator continues to put events on the input queue of each public context. However, since the input queues are not being drained, if an input queue fills up, this will block senders, including the engine_send tool and adapters.
• The correlator sends out any events on its output queue. When the output queue is empty, receivers no longer receive events; no contexts are sending events.
• Other inspections of the correlator proceed as normal. For example, engine_watch, engine_management, and profiling data.
• You can shut down the correlator.
• You can inject monitors while the correlator is stopped. They will not run any of the onload() or similar code until the correlator resumes, but the inject call should succeed.
• Java applications continue to run completely independently of whether the correlator is stopped.
• All other requests block until the correlator resumes processing. This includes dumping correlator state, loading, and changing debug or profiling state.

The engine_debug tool is stateless. Consequently, during debugging, you can have multiple concurrent connections to the same correlator.

Debug commands

The ordering of arguments to engine_debug commands works as follows:

• All arguments before the first command apply to all commands in that command line. This is useful for setting the host and port if you are not using the local defaults.
• All arguments following a command apply to only that command and they override any applicable arguments specified before the first command.
• The arguments to a particular command can be in any order
• When there are multiple commands in a line, the debugger executes them in the order in which they are specified. Execution continues until either all complete, or one fails, which prevents execution of any subsequent commands.

The engine_debug tool takes the following commands as options:

Exit status

The engine_debug tool returns the following exit values:

Obtaining online help for the command-line debugger

The command-line debugger provides online help. To obtain general information, enter the following:

engine_debug help

To get help for a particular command, specify that command after the help keyword.

For example, if you want to find out what the status command does, enter the following:

engine_debug help status

Or to find out which options you can specify with the breakpoint add command, enter the following:

engine_debug help breakpoint add

Enabling and disabling debugging in the correlator

To use the debugger, you must enable debugging in the correlator. To enable debugging locally on the default port, enter the following:

engine_debug enable

When you are done debugging, you should disable debugging in the correlator. If you do not, the correlator runs more slowly and continues to pause when it hits a breakpoint. To disable debugging in the local correlator on the default port, enter the following:

engine_debug disable

You can also enable and disable the debugger in a remote correlator by specifying the host name and the port number. For example:

engine_debug enable --host foo.bar.com --port 1234

engine_debug disable --host foo.bar.com --port 1234

Working with breakpoints using the command-line debugger

You can use the command-line debugger to add, list and remove breakpoints.

Adding breakpoints

There are two ways to add a breakpoint. If you know the EPL file name and the line number, you can enter something like the following:

engine_debug breakpoint add --file filename.mon --line 27

When you specify a file name, you must specify the exact path you specified when you injected the monitor. For example, suppose you ran the following:

engine_inject foo.mon

You can then specify “foo.mon” for the file name. Now suppose you ran this:

engine_inject c:\foo\bar\baz.mon

You must then specify “c:\foo\bar\baz.mon” for the file name.

If you prefer to use the monitor and action name, along with the line number, enter something like this:

engine_debug breakpoint add --monitor package.monitor --action actionName
   --line 27

The debugger output indicates the line number where it added the breakpoint. In some cases, the debugger does not set the breakpoint on the line you specified, for example, when a statement runs over multiple lines.

Listing breakpoints

To obtain a list of the breakpoints currently set in the correlator, enter the following:

engine_debug breakpoint list

Removing breakpoints

To remove a breakpoint by specifying the file name and the line number, enter something like the following:

engine_debug breakpoint delete --file filename.mon --line 27

To use the monitor name to remove a breakpoint, enter something like this:

engine_debug breakpoint delete --monitor package.monitor --action actionName
   --line 27

To delete a breakpoint by using the breakpoint ID that appears in the breakpoint list returned by the debugger, enter something like this:

engine_debug breakpoint delete --breakpoint 1

Controlling execution with the command-line debugger

When the correlator stops at a breakpoint, you can use the debugger to step over the next line:

engine_debug stepover

However, you most likely want to step over the line, confirm that the correlator stopped, and learn about the current state of the debugger. You can do this by entering multiple commands in one line. For example:

engine_debug stepover wait --timeout 10 status

This is the equivalent of the following three commands:

• engine_debug stepover — Causes the debugger to step over one line of EPL.
• engine_debug wait --timeout 10 — Causes the debugger to pause until either a breakpoint is hit, or ten seconds pass.
• engine_debug status — Displays the debugger’s current status.

Following are more examples of entering multiple commands in one line.

engine_debug stepinto wait --timeout 10 status

engine_debug stepout wait --timeout 10 status

To instruct the correlator to continue executing EPL code, run the following command:

engine_debug run

You use the engine_debug run command regardless of how the correlator was stopped — a breakpoint was reached, a step operation, a wait command.

To stop the correlator, enter the following command:

engine_debug stop

The wait command

The wait command connects to the correlator to determine if the correlator has suspended processing. If the correlator is in suspend mode, the wait command returns immediately and debugging continues. If the correlator is not in suspend mode, the wait command remains connected to the correlator. The wait command returns when something else suspends the correlator or when the timeout is reached. Operations that can suspend the correlator include reaching a breakpoint, stepping into or over a line, or some other client explicitly stopping the correlator. If the wait command reaches the timeout, it suspends the correlator before it returns.

Stepping can take a variable amount of time. For example, suppose the debugger stops at the end of a listener and you execute a step command. The debugger is now outside the flow of execution until another event comes in. The time that the debugger has to wait for the step to finish is dependent upon when the next matching event arrives.

Command shortcuts for the command-line debugger

Putting multiple commands in the same command line can get verbose. For example, suppose you want to step out of an action on a remote machine. You would need to enter something like this:

engine_debug stepout --host foo.bar.com --port 1234 wait --timeout 10
   --host foo.bar.com --port 1234 status --host foo.bar.com --port 1234

The command-line debugger provides easier ways to invoke this.

• Any arguments that you specify before the first debugging command apply to the entire command line.
• All individual commands and their arguments have abbreviations.

For example, the following command does the same thing as the previous verbose command:

engine_debug -h foo.bar.com -p 1234 sot w -to 10 p

The following table lists the abbreviations you can use for command arguments. For abbreviations of commands, see Debug commands.

Examining the stack with the command-line debugger

When the correlator stops at a breakpoint, you can display the stack with the following command:

engine_debug stack

The results of this command show the number of the frame that contains each variable. In the following example, the frame number is the number before the right parenthesis:

0 )
        C:/dev/adbc/apama-test/system/correlator-debug/testcases/
           correctness/Corr_Debug_cor_002/Input/test.mon:35
        foo.baz.test.runtest[setupctx(2)/foo.baz.test(3)]
1 )
        C:/dev/adbc/apama-test/system/correlator-debug/testcases/
           correctness/Corr_Debug_cor_002/Input/test.mon:19
        foo.baz.test.::listenerAction::[setupctx(2)/foo.baz.test(3)]

You can use these frame numbers (frame IDs) as arguments to the engine_debug inspect command.

To see just the contents of the top frame, run this command:

engine_debug stack --frame 0

Displaying variables with the command-line debugger

To list all variables in the current stack frame, enter the following:

engine_debug inspect

To obtain the value for a variable in the current stack frame, enter the following:

engine_debug inspect -variable variableName

To obtain the value for a variable further down the stack, run the stack command to determine the frame number and then enter the following:

engine_debug inspect -variable variableName -frame frameid

Generating code coverage information about EPL files

The correlator can generate “code coverage” information about EPL files indicating which lines have been executed. This is useful for measuring the quality of test cases, discovering lines of EPL code which are not being exercised by any tests, as well as for helping diagnose bugs or understand complex interactions in the EPL.

Recording code coverage information

The recording of code coverage information can be enabled and written (dumped) to disk using management requests, or using an environment variable that automatically writes out a coverage file when the correlator is shut down or when code is deleted from the correlator.

The epl_coverage tool can then be used to merge together the coverage files that have been produced by the correlator and produce summary statistics about how much of each source file is covered, as well as an HTML report where each source line is shown annotated with different colors to indicate which lines are not being covered. For detailed information, see Creating code coverage reports.

Enabling the code coverage feature will disable the compiled runtime, and it will also enable the debugger (Using the command-line debugger) and CPU profiler (see Using the CPU profiler).

Dumping code coverage information using management requests

One way to enable and dump code coverage information is via the -r codeCoverage option of the engine_management tool (see also Shutting down and managing components). You can send the following requests:

It is also possible to start the correlator in a mode where it automatically writes code coverage information to disk when it is shut down or is given an engine_delete --all request (see also Deleting code from a correlator).

This mode is enabled by setting the AP_EPL_COVERAGE_FILE environment variable to the path of a file to which coverage information is to be written. If you do this, the correlator starts in code coverage collection mode with debugging enabled, the compiled runtime disabled and optimizations disabled. On shutdown, it writes the code coverage information to the path specified in the environment variable.

The environment variable can contain replacement tokens in the same format as the correlator log file (see Specifying log filenames). Given that the coverage file is not subject to log rotation, only the ${PID} and ${START_TIME} tags are appropriate.

Example (for Windows):

set AP_EPL_COVERAGE_FILE=c:\mypath\mycorrelator.${PID}.eplcoverage
start correlator
\(run application, etc.\)
engine_management --shutdown "Clean correlator shutdown from command line"

Of course, the correlator must be cleanly shut down for this to work, as no coverage information is written if the process is terminated without warning. If a dump is triggered by engine_delete --all and more EPL is then injected before the correlator is shut down, all coverage information written by engine_delete is overwritten by later coverage information and is thus lost. However, if engine_delete is immediately followed by a clean shutdown, there will be no new coverage information when the shutdown occurs. Therefore, the file will not be overwritten.

Code coverage and deletion of monitors

The code coverage information for transient monitors, monitors which have died, and monitors which were deleted by name is retained. This enables you to later call codeCoverage dump to get the coverage information.

An engine_delete --all, however, resets all the coverage information. When you set the AP_EPL_COVERAGE_FILE environment variable, the coverage information is automatically dumped during an engine_delete --all.

If you delete a monitor and then reinject the EPL file, then all coverage information for that EPL file is reset for the newly injected file.

Common usage patterns

• Enable code coverage, inject your application and send typical events into the correlator. Then dump a coverage report. This gives you a complete list of code covered by initialization and events being processed in the system.
• Set the AP_EPL_COVERAGE_FILE environment variable before running your test suite. Then collate all the coverage reports. This lets you check that your tests exercise all the code paths.
• Enable code coverage and inject your application. Then disable and enable code coverage (to clear the reporting data). Then send a single event through and dump a coverage report. This lets you see what code is run by a single event.

Creating code coverage reports

The epl_coverage tool takes one or more coverage files that have been output by the correlator’s code coverage feature, merges them together to create a new combined .eplcoverage file (which can be used as input for the tool), and creates a CSV, XML and HTML report of the coverage of each source EPL file. The executable for this tool is located in the bin directory of the Apama installation.

Synopsis

To create code coverage reports, run the following command:

epl_coverage [ options ] file1.eplcoverage  [ file2.eplcoverage... ]

Example (for Windows):

epl_coverage --output c:\mycoverage --source "%APAMA_WORK%\projects\myproject"
  --exclude "**/Apama/**/*.mon" *.eplcoverage

When you run this command with the –h option, the usage message for this command is shown.

Description

The --output argument specifies the directory into which the tool writes the output files. If not specified, the current directory is used.

The output includes the following files:

• merged.eplcoverage. A single file containing the combined EPL code coverage information from all the input files. This can be used as input to another invocation of the epl_coverage tool.
• coverage_summary.csv. A summary of the percentage of lines and instructions covered in each source file in the standard “comma-separated values” text format (in the operating system’s local character encoding). This file may be useful for reviewing coverage information in a spreadsheet, or as input for an automated tool that records coverage information as part of a continuous integration build/test system. The file starts with a header line beginning with the hash (#) character which identifies the columns used in the rest of the file. It is recommended that any tool that reads this file should use the header line to identify the contents of each column; this is helpful in case columns are added or reordered in a later release.
• epl_coverage.xml. An XML representation of the combined code coverage information for all the input files. This file is written in a widely-used coverage file format that can be read by many third-party tools (the file format that was popularized by Cobertura, which is a code coverage utility for Java; see also https://cobertura.github.io/cobertura/).
• index.html (and associated .css and .html files). An HTML summary of coverage information, including annotated copies of the source files showing which executable lines are covered. These files are always written in the UTF-8 encoding.

Where possible, you should always specify the directories of the .mon source files using the --source option. The coverage tool uses the source directories to identify the canonical location of each EPL file, which may not be the same as the path it was injected from, especially when initializing a correlator from a correlator deployment directory. This also makes it easier for third-party coverage tools to recognize the paths from the XML report and allows the HTML report to show the source code even when the .mon file was injected from a test output directory that no longer exists.

By default, EPL files are assumed to be in the UTF-8 (or ASCII) encoding. If your EPL files are in a different encoding, you can provide the --source-encoding local option, in which case the EPL files are read in the local encoding instead. Files with a UTF-8 byte-order marker are read in UTF-8 in either case. This affects both locating the correct source file in the provided --source directories, and correctly rendering the source code into the HTML reports.

You can apply filters that remove information about unwanted EPL files (such as test .mon files) from all of the output files, or to restrict which parts of the source directories are to be searched. The --include and --exclude options can each be specified multiple times. They specify file patterns to include or exclude (for example “**/foo/Bar*.mon”). These patterns use the following characters:

• forward slashes (/) to indicate directory separators (on all platforms),
• a single asterisk (*) to indicate any number of non-directory separator characters,
• two asterisks (**) to indicate any number of characters potentially including directory separators, and
• a question mark (?) to indicate a single character.

If no --include argument is provided, the default is to include all file paths, except those that are removed by --exclude arguments.

When the number of coverage input files is large, you can avoid an extremely long command line (which some operating systems do not support) by putting the coverage file list into a newline-delimited UTF-8 text file and providing the path to that file on the command line instead, prefixed with an @ symbol. For example:

epl_coverage "@c:\mypath\coverage_file_list.txt"

Options

The epl_coverage tool takes the following options:

Interpreting the code coverage reports

Many lines in an EPL file do not contain any executable instructions, for example, comments, event definitions (except where they contain actions) and event expressions used to declare listeners. These lines are not marked up by the epl_coverage tool.

Lines that do contain executable code may have one or more executable elements (instructions), and the epl_coverage tool reports whether all or only some of those instructions have been executed. It may therefore be useful to split complex EPL constructs (such as multi-part if statements) over multiple lines as much as possible to make the output clearer as to what is covered. The exact details of how many instructions are on any given line is subject to change and therefore not documented, but information on partial coverage may sometimes be useful for identifying branching constructs where not all branches are covered.

Tip:

Every executable line that is not fully covered has the (!) string in the margin, which makes it possible to jump backwards and forwards between these lines using the Find functionality provided by most web browsers.

The purpose of the coverage information is to provide insight into areas of user EPL that are being missed by test cases. Although it is worth aiming for a high percentage of lines and instructions being covered, it is not always possible to write tests that cover every line. However, as long as someone looks at the lines that were missed, there is no need to worry about having less than 100 percent coverage.

Similarly, the information about partial line coverage can often be useful, particularly for control constructs where it might indicate a missed branch in an if statement, or a while loop condition that always returns false. But it will not always be possible for users to get 100 percent coverage of every line, or (since the internal instructions used by EPL are not documented and may be changed at any time between versions) even to understand the reason why a line was not fully covered in some cases. Cumulocity product support cannot provide explanations for why a given line of EPL was only partly covered.

Examples

The following code snippets illustrate some common cases.

• The following line is partially but not fully covered if a() returns true every time this line is executed, since the instructions for the value of b() are never checked in this case.

  if a() or b() {
  

• The following line is only partially covered unless the test is run with DEBUG logging enabled, since expressions in log statements are only evaluated if the log level is specified.

  log "Hello world" at DEBUG;
  

• Another common example is a stream query that uses an aggregate where nothing drops out of the window while the test is executed. For example, if less than 100 seconds pass after the first E() event, the following line is only partially covered:

  from a in all E() within 100.0 select com.apama.aggregates.sum(a.val) as i
  

  If the test does not have anything drop out of the within window, then you will get amber coverage, as no code to remove a value from the set being aggregated over (by sum) is being executed. This may happen if no events go through this query, or if only less than 20 seconds pass since the first event.

• Any code in an onunload action will never be covered at all, since it is only executed with engine_delete, which also removes the coverage information.

Using EPL code coverage with PySys tests

The Apama installation includes the Python-based PySys test framework and Apama helper classes for PySys.

The Apama helper classes for PySys can enable code coverage recording, and automatically run a coverage report from the .eplcoverage files at the end of test execution, which will help users to create better test cases and to find code paths in their EPL applications that do not have adequate test coverage. To use this feature, start your tests with -XcodeCoverage, for example:

pysys run -XcodeCoverage

For an example, see the README.txt file in the samples/pysys directory of your Apama installation.

Replaying an input log to diagnose problems

When you start the correlator, you can specify that you want it to copy all incoming messages to a special file, called an input log. An input log is useful if there is a problem with either the correlator process or an application running on the correlator. If there is a problem, you can reproduce correlator behavior by replaying the messages captured in the input log. Incoming messages include the following:

• Events
• EPL
• Java
• Correlator deployment packages (CDPs)
• Connection, deletion, and disconnection requests

If you are unable to diagnose the problem, you can provide the input log to Cumulocity product support. A support engineer can then feed your input log into a new correlator to try to diagnose the problem.

The information in the following topics describes how to generate and use an input log. See also Examples for specifying log filenames.

Regularly rotating log files and storing the old ones in a secure location may be important as part of your personal data protection policy. For more information, see Handling personal data “at rest” in the correlator input log file.

Creating an input log file

To create an input log, specify the following option when you start a correlator:

--inputLog filename[${START_TIME}][${ROTATION_TIME}][${ID}][${PID}].log

You specify the name of an input log file in the same way that you specify the name of a main correlator log file. See Specifying log filenames.

In addition, specify any other options that you would normally specify when you start the correlator.

Rotating an input log file

While the input log can get rather large, most file systems can handle large input logs with no special action on your part. However, you might encounter one of the following situations:

• You want to archive your input logs.
• Your operating system enforces a limit on file size.
• The input log has become too large.

In these situations, you can rotate the input log. Rotating the input log means that the correlator closes the current input log and starts sending messages to a new input log.

You should rotate the input log only when you have a specific need to do so. You do not want to have thousands of input logs in a directory since file systems do not handle this efficiently.

If you plan to rotate input logs, specify the ${ID} tag when you specify the --inputLog option when you start the correlator. For examples, see Examples for specifying log filenames.

To rotate the input log, invoke the engine_management tool and specify the --rotateLogs option. The name of the new input log is the same as the name of the closed input log except that the correlator increments the ID portion of the input log filename by 1. See Rotating all correlator log files.

Performance when generating an input log

When the file system that hosts the input log is fast, generating an input log should not have any noticeable effect on correlator performance in most cases. It is possible to use the input log with connectivity plug-ins (see Using connectivity plug-ins), but the performance impact will be significant for chains using the apama.eventMap host plug-in and any chains that are using small batches of events. Consequently, the recommendation is to always run correlators that send information to input logs. Just make sure you have enough disk space for the input log. You need to monitor repeated use to determine how much space is required.

With the correlator generating an input log, you can implement your application so that it sends a minimum amount of information to the main correlator log file. You do not need to log application information because you can always recover application information from the input log. Implementing an application that sends large amounts of application information to the main correlator log file can negatively impact performance.

Reproducing correlator behavior from an input log

To use an input log to reproduce correlator behavior, you must do the following:

1. Run the extract_replay_log Python utility.
2. Run the replay_execute script that the extract_replay_log utility generates.

Invoking the extract script

The extract_replay_log.py script is in the utilities directory in your Apama installation directory. You must have at least Python 2.4 to run this utility. You can download Python from http://www.python.org. If you are using Linux, you probably already have Python installed.

The format for running the extract_replay_log utility is as follows:

extract_replay_log.py [options] inputLogFile

Replace inputLogFile with the path for the input log you want to extract. If you specify the first input log in a series, the subsequent input logs must be in the same directory as the first input log.

The options you can specify are as follows:

The extract_replay_log utility generates the following:

• A script whose execution duplicates the correlator activity captured by the input log.
• Event files where each one is prefixed with “replay_”.
• EPL and possibly JAR and correlator deployment package (CDP) files where each one is prefixed with “replay_”.

Invoking the replay script

Before you run the replay script, you can optionally edit the generated event files, EPL files, or JAR files to slightly modify the behavior you are about to replay. For example, you might add logging for debugging purposes. However, there are restrictions on what you can change:

• You cannot insert any of the following:
  • calls to integer.getUnique(), rand() or incrementCounter
  • send, emit, spawn...to, or enqueue...to statements
  • context constructors
• You cannot change the number of parseable events sent to the correlator. For example, you cannot attach a dashboard component to the input log because the dashboard components work by sending events to the correlator.
• You cannot change the number of event definitions and monitors injected.

Making any of these changes can potentially alter the behavior of later operations.

If you are using the MemoryStore and the correlator reads or writes to a store on disk then to accurately play back execution you must have a copy of that store as it was before the correlator modified it. Also, if you are using the MemoryStore from multiple contexts it is unlikely to replay correctly because the order of interaction with the MemoryStore is not in the input log.

After you have optionally edited the generated files, you are ready to invoke the replay_execute script. The replay_execute script tries to replay the contents of the input log into the correlator running on the default port.

While the correlator exactly reproduces the activity captured in the input log, it can execute the same activity faster during replay than when it was executed originally. This is because the correlator already has all the events it needs to process; it does not have to wait for any events. Replaying a log is typically significantly faster than original correlator activity. It is possible that you will find that the time it takes to replay a log is not much less than the time it took for the original activity. In this case, it is possible you were running too close to capacity during the original run. If that is the case, you risk not being able to keep up with the event flow during regular correlator execution. If you anticipate higher event flow then you should investigate optimizing your application or running it on a faster computer.

Event file format

You can use the engine_send tool to stream a sequence of events through the correlator. The engine_send tool accepts input from one or more data files to support tests or simulations, or from stdin to allow dynamic generation of events. In the latter case, you can generate events from user input or by piping output from an event generation program to engine_send. In all cases, engine_send requires event data formatted as described in this section. For detailed information on the engine_send tool, see Sending events to correlators.

The engine_receive tool outputs events in this same file format. This means you can use events generated by the engine_receive tool as input to a second correlator that is executing the engine_send tool. For detailed information on the engine_receive tool, see Receiving events from correlators.

Event representation

A single event is identified by the event type name and the values of all fields as defined by that type. Event type names must be fully-qualified by prefixing the package name into which the corresponding event type was injected, unless the event was injected into the default package.

The specific EPL types and how they map from the event representation are shown in an example in Event types, but there are certain basic types that can be included as shown in the following example:

// integer
MyEvent(-1,1)

// decimal and float
MyEvent(-2.0,2.0)

// decimal and float in exponential form (0.02,200)
MyEvent(-2.0e-2,2.0e2)

// string
MyEvent("three")

// boolean
MyEvent(true,false)

Both decimal and float types can be represented in scientific form if required, including when nested in optional or any types and inner events.

A string is a sequence of characters enclosed in double quotes (”). The backslash character (\) is used as an escape character to allow inclusion of special characters such as newlines and horizontal tabs.

Examples:

MyEvent("Hello, World!")
MyEvent("\ta\tstring\twith\ttabs\tbetween\twords")
MyEvent("a string on\n two lines")
MyEvent("a string with \\ a backslash and a \" quote")

Localization, such as different formats for decimals or quotation marks, is not supported.

Each event is given on a separate line. Only single-line comments are allowed. Start each comment line with // or #. Any blank lines are ignored.

For example, following are three valid events:

// This is an event file that contains some sample events.
// Here are three stock price events:
my.test.StockPrice("XRX",11.1)
my.test.StockPrice("IBM",130.6)
my.test.StockPrice("MSFT",70.5)

For those events, the following event type definition must be injected into the package test:

package my.test;

event StockPrice {
 string stockSymbol;
 float stockValue;
}

If the above events were saved in an .evt file, engine_send would send each event in turn, as soon as the previous event finished transmission. This behavior can optionally be modified in several ways:

• Specifying that batches of events should be sent at specified time intervals.
• Specifying that all events on all queues should be processed before sending the next event.

Event timing

In .evt files, it is possible to specify the following:

• Time intervals for sending batches of events to the correlator.
• Waiting for all events on all queues at that point in time to be processed before sending the next event.

Adding BATCH tags to send events at intervals

You can specify time intervals for sending batches of events to the correlator. This is achieved by specifying the BATCH tag followed by a time offset in milliseconds. For example, the following specifies two batches of events to be sent 50 milliseconds apart.

BATCH 50
StockPrice("XRX", 11.1)
StockPrice("IBM", 130.6)
StockPrice("MSFT", 70.5)
BATCH 100
StockPrice("XRX", 11.0)
StockPrice("IBM", 130.8)
StockPrice("MSFT", 70.1)

The addition of a “time” allows simulations of “bursts” of events, or more random distributions of event traffic. Times are measured as an offset from when the current file was opened. If only one file of events is being read and transferred, then this would be the same as since the start of a run (that is, from the time that the engine_send tool starts processing the event data). If multiple files are being read in, the timing starts all over again upon the (re)opening of each file.

If the time given for a batch is less than the current time, or if no time is given following a BATCH tag (or if no BATCH tag is provided), then the events are sent as soon as they are read in, immediately following the preceding batch.

Using &FLUSHING mode for more predictable event processing order

Sending events in flushing mode can help provide a more predictable event processing order. However, flushing mode is slower than the default behavior.

By default, events are delivered in an optimal way, not waiting for previously sent events to be processed before the next event is delivered to contexts (or other consumers of channels). When flushing mode is enabled the behavior is as follows:

1. The correlator sends an event.
2. The correlator processes all events on all queues at that point in time, repeating this as many times as specified in the flushing specification.
3. The correlator sends the next event.

To enable flushing mode, insert the following line in a .evt file:

&FLUSHING(n)

Replace n with an integer that specifies how many times to flush queues in between each event. Set this to the maximum length of a chain of send-to operations between contexts that could occur in your application. If you specify a number that is bigger than required the correlator simply repeats the flush operation, which incurs a small overhead. To disable flushing mode, insert the following line in the .evt file:

&FLUSHING(0)

Enabling or disabling flushing mode affects only the events sent on that connection or from that event file.

When sending &TIME events in to a multi-context application, the time ticks are delivered directly to all contexts. This can be different than the way in which events in the .evt file are sent in to the correlator and then sent between contexts in an application. This difference can result in processing events at an incorrect simulated time. In these cases, sending &FLUSHING(1), for example, before sending time ticks and events can result in more predictable and reliable behavior.

Event types

The following example illustrates how each type is specified in an event representation. Given the event type definitions:

event Nested {
  integer i;
}
event EveryType {
  boolean b;
  integer i;
  float f;
  string s;
  location l;
  sequence<integer> si;
  dictionary<integer, string> dis;
  Nested n;
  optional<Nested> opt;
  any anyValue;
}

the following is an expanded string representation for an EveryType event:

EveryType (
  true,                         // boolean is true/false (lower-case)
  -10,                          // positive or negative integer
  1.73,                         // float
  "foo",                        // strings are (double) quoted
  location(1.0,1.0,5.0,5.0),    // locations are 4-tuples of float values
  [1,2,3],                      // sequences are enclosed in brackets []
  {1:"a",2:"b"},                // dictionaries are enclosed in braces {}
  Nested(1),                    // nested events include event type name
  optional<Nested>(Nested(10)), // optional payload inside parentheses, may be
                                // empty, in which case it is represented by the
                                // string "optional()"
  any(integer,42)               // any names a type and the string form of that
                                // type; other examples include the following:
                                // any(sequence<Nested>,[Nested(1)])
                                // may also be empty, in which case the string
                                // form is: "any()"
);

Note: This example is split over several lines for clarity. In practice, this definition must all be written on the same line, and without the comments. Otherwise the correlator will fail to parse the event.

Types can of course be nested to create more complex structures. For example, the following is a valid event field definition:

sequence<dictionary<integer, Nested>>

and the following is a valid representation of a value for this field:

[{1:Nested(1)}, {2:Nested(2)}, {3:Nested(3)}]

You can also get an event’s string representation in EPL by using the toString() method.

Event association with a channel

The engine_send tool can send an event file that associates channels with events. Likewise, the engine_receive tool can output an event file that includes the channel on which an event was received. The event format is the same for both tools:

"channel_name",event_type_name(field_value1[, field_valuen]...)

For example, suppose you want to send Tick events, which contain a string followed by an integer, to the PreProcessing channel. The contents of the .evt file would look like this:

"PreProcessing",Tick("SOW", 35)
"PreProcessing",Tick("IBM", 135)

A channel name is optional. In a file being sent with the engine_send tool, you can mix event representations that specify channels with event representations that do not specify channels. Events for which a channel is specified go to only those contexts subscribed to that channel.

The default behavior is that events are sent on the default channel (the empty string) when a channel is not explicitly specified. Events sent on the default channel go to all public contexts. To change the default behavior for events sent by the engine_send tool, you can specify engine_send -c channel. If a channel is not explicitly specified for an event, then it is sent to the channel identified with the -c option. See Sending events to correlators.

Using the Data Player command-line interface

Note: The Data Player is deprecated and will be removed in a future release.

Apama’s Data Player in Apama Plugin for Eclipse lets you play back previously saved event data as you develop your application. During playback, you can analyze the behavior of your application. Or, if you modify the saved event data, you can analyze how your application performs with the altered data. Apama Plugin for Eclipse plays back event data that has been stored in standard data formats.

When you are ready to test your application, the command-line interface to the Data Player lets you write scripts and unit tests to exercise the API layers. Or, if you just want to play back events to the correlator, using the command-line interface might be easier than using the Data Player GUI in Apama Plugin for Eclipse.

To use the command-line interface to the Data Player, you must have already used the GUI interface in Apama Plugin for Eclipse. That is, you must have already defined queries and query configurations in Apama Plugin for Eclipse. When you use the command-line interface (that is, the adbc_management tool), you specify query names and query configurations that you created in Apama Plugin for Eclipse. The executable for this tool is located in the bin directory of the Apama installation.

The Data Player relies on Apama Database Connector (ADBC) adapters that are specific to standard ODBC and JDBC database formats as well as the comma-delimited Apama Sim format. Apama release 4.1 and earlier captured streaming data to files in the Sim format. These adapters run in the Apama Integration Application Framework (IAF), which connects the data sources to the correlator. The information here assumes that you are already familiar with the information in Using the Data Player.

Synopsis

To use the Data Player from the command line, run the following command:

adbc_management --query queryName --configFile file [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Options

The adbc_management tool takes the following options:

About Deploing Dashboards

About deploying dashboards

Dashboard deployment and administration involves the following activities:

• Deployment package installation and configuration. See Deploying Dashboards.
• Data server and display server management. See Managing the dashboard data server and display server.
• Security administration. See Administering Dashboard Security.

Before you perform these tasks, you should familiarize yourself with the deployment and administration concepts described in Dashboard Deployment Concepts.

Deployment options

Dashboards can be deployed as simple thin-client web pages or as files that can be loaded into a locally-installed desktop application, the Dashboard Viewer. These deployment options are described and compared in Deployment options.

Data server and display server

Scalability and security of dashboard deployment are supported by the use of the data server and display server, which mediate dashboard access to running DataViews. The data server and display server are introduced in Data server and display server.

Process architecture

Simple thin-client web-page dashboards communicate with the display server via servlets running on your application server. Locally-deployed dashboards communicate directly with the data server. The structure of deployed configurations is detailed in Process architecture.

Builders and administrators

Dashboard deployment involves the use of a dashboard deployment package. In some cases, the user that generated the deployment package is different from the user that installs and configures the deployment and administers the data server. The information that must be transferred between these two types of users is discussed in Builders and administrators.

About tuning applications for performance

Tuning Correlator Performance

Scaling up Apama

Apama provides services for real-time matching on streams of events against hundreds of different applications concurrently. This level of capacity is made possible by the advanced matching algorithms developed for Apama’s correlator component and the scalability features of the correlator platform.

Should it prove necessary, capacity can further be increased by using multiple correlators on multiple hosts. To facilitate such multi-process deployments, Apama provides features to enable connecting components to pass events between them. It is recommended that each correlator is run on a separate host, to assist in the configuration of scaled-up topologies. However, it is possible to run multiple correlators on a single host. There are two methods of configuration:

• Using the configuration tools from the command line or Apama macros for Ant.
• Programmatically through a client programming API.

This guide describes both approaches, but first discusses different ways in which Apama can be distributed and what factors affect the choice of the distribution strategy.

Note:

This topic focuses on scaling Apama for applications written in EPL. Java plug-ins can be used if invocation of Java code is required on multiple threads, either directly from EPL or by registering an event handler. See Using EPL plug-ins written in Java. Knowledge of aspects of EPL is assumed, specifically monitors, spawning, listeners and channels. Definitions of these terms can be found in Getting started with Apama EPL.

The core event processing and matching service offered by Apama is provided by one or more correlator processes. In a simple deployment, Apama comprises a single correlator connected directly to at least one input event feed and output event receiver. Although this arrangement is suitable for a wide variety of applications (the actual size depending on the hardware in use, networking, and other available resources), for some high-end applications it may be necessary to scale up Apama by deploying multiple correlator processes on multiple hosts to partition the workload across several machines.

Partitioning strategies

Using the patterns and tools described in this guide it is possible to configure the arrangement of multiple contexts within a single correlator or multiple correlators within Apama (the engine topology). It is important to understand that the appropriate engine topology for an application is firmly dependent on the partitioning strategy to be employed. In turn, the partitioning strategy is determined by the nature of the application itself, in terms of the event rate that must be supported, the number of contexts, spawned monitors expected and the inter-dependencies between monitors and events. The following examples illustrate this.

The stockwatch sample application (in the samples\epl folder of your Apama installation directory) monitors for changes in the values of named stocks and emits an event should a stock of interest fall below a certain value. The stocks to watch for and the prices on which to notify are set up by initialization events, which cause monitors that contain the relevant details to be spawned. In this example, the need for partitioning arises from a very high event rate (perhaps hundreds of thousands of stock ticks per second), which is too high a rate for a single context to serially process.

A suitable partitioning scheme here might be to split the event stream in the adapter, such that different event streams are sent on different channels. The illustration below shows how this can be accomplished:

This diagram shows an adapter sending events to different channels based on the symbol of the stock tick. The adapter transport configuration file would specify a transportChannel attribute for the stock event that named a field in the NormalisedEvent that specified the stock symbol. Either a thread per symbol or a single thread (which could become a bottleneck) managed by the transport, depending on what the system the transport is connecting to allows, is used to send NormalisedEvents to the semantic mapper to be processed. The IAF thus sends the events on the channel in the stock symbol value in the NormalisedEvent.

In this example, the stock symbol is either “IBM” or “XRX”. The IAF will send events to all sinks (typically one) that are specified in the IAF’s configuration file. In the correlator, all monitors interested in events for a given symbol would need to set up listeners in a context where a monitor has subscribed to that symbol. To achieve good scaling, the application is arranged so that each context is subscribed to only one symbol. For the stockwatch application, a separate context per symbol would be created, and the stockwatch monitor spawns a new monitor instance to each context. In each context, the monitor instance would execute monitor.subscribe(stockSymbol); where stockSymbol would have the value "IBM" or "XRX" corresponding to the stock symbol it is interested in. This application will scale well, as each event stream (for the different stock symbols) can run in parallel on the same host; this is referred to as scale-up.

Listeners in each context would listen for events matching a pattern, such as on all Tick(symbol="IBM", price < 90.0).

If the number of stock symbols is very large and the amount of processing for each stock symbol is large, then it may be required to run correlators on more than one host to use more hardware resources than are available in a single machine. This is referred to scale-out. To achieve scale-out, connections per channel need to be made between the Apama components using the engine_connect tool (or the equivalent call from Ant macros or the client API). The engine_connect tool can connect any two Apama components, either correlator to correlator, or IAF to correlator. For best scaling, multiple connections are required between components, which engine_connect provides in the parallel mode. The following image shows a scaled out configuration.

This configuration allows many contexts to run on two hosts and requires use of engine_connect to set up the topology.

Now consider a portfolio monitoring application that monitors portfolios of stocks, emitting an event whenever the value of a portfolio (calculated as the sum of stock price * volume held) changes by a percentage. A single spawned monitor manages each portfolio and any stock can be added to/removed from a portfolio at any time by sending suitable events.

This application potentially calls for significant processing with each stock tick, as values of all portfolios containing that stock must be re-calculated. If the number of portfolios being monitored grows very large, it may not be possible for a single context to perform the necessary recalculations for each stock tick, thus requiring the application to be partitioned across multiple contexts.

Unlike the stockwatch application, it is not possible to achieve scaling to larger numbers of portfolios by splitting the event stream. Each portfolio can contain multiple stocks, and stocks can be dynamically added and removed, thus one event may be required by multiple contexts. In this case, a suitable partitioning scheme is to partition the monitor instances across contexts (as with stockwatch) but to duplicate as well as split the event stream to each correlator. The following images shows the partitioning strategy for the portfolio monitoring application.

Again, each monitor instance is spawned to a new context and subscribes to the channels (stock symbols in this application) that it requires data for. Note that while the previous example would scale very well, this will not scale as well. In particular, if one monitor instance requires data from all or the majority of the channels, then it can become a bottleneck. However, there may be multiple such monitor instances running in parallel if they are running in separate contexts.

Similar to the stockwatch application, the portfolio monitoring application may require scale-out across multiple hosts, as shown below.

In summary, the partitioning strategy can be thought of as a formula for splitting and duplicating monitors and/or events between correlators while preserving the correct behavior of the application. In some circumstances, it may be necessary to re-write monitors that work correctly on a single correlator to allow them to be partitioned across correlators, as the following section describes.

Engine topologies

Once the partitioning strategy has been defined, in terms of which events and monitors go to which correlators, it is necessary to translate this into an engine topology. This is achieved by connecting source and target correlators on separate channels, such that events sent by a source correlator on a specific channel find their way to the correct contexts in the target correlator. A set of two or more correlators connected in this way is known as a correlator pipeline, as shown in the following image. This figure represents an example topology for a high-end application – the majority of applications use a single correlator only, or have far simpler topologies.

In this image, a correlator can perform the function of each of the 7 nodes (generator, worker, watcher, tallier). Each target correlator performs some processing before passing the results to a second worker correlator (worker3, worker4) in the form of events, sent on the channels as marked on the diagram. tallier collates the results from these correlators for forwarding to any registered receivers. A final correlator, watcher, monitors the events emitted by generator on chan1 and chan2 and emits events (possibly containing status information or statistical analysis of the incoming event stream) to any registered receivers.

To deploy an application on a topology like that shown above requires separating the processing performed into a number of self-contained chunks. In the previous figure, it is assumed that the core processing can be serialized into three chunks, with the first two chunks split across two correlators each (worker1/2 and worker3/4 respectively) and the third chunk residing on a single correlator (tallier). Intermediate results from each stage of processing are passed to the next stage as sent events, which contexts in the connected correlators receive by subscribing to the appropriate channels.

To realize this application structure requires coding each chunk of processing as one or more separate monitors, which send intermediate results as an event of known type on a pre-determined channel. These monitors can then be loaded onto the appropriate correlator. This may require an existing application that grows beyond the capacity of a single correlator, to be re-written as a number of (smaller) monitors to allow partitioning of the application processing into separate chunks in the manner described above.

Correlator pipelining

To implement engine topologies comprising multiple correlators requires a method of connecting correlators in pipelined configurations. This can be achieved in the following ways:

• Directly using the engine_connect tool. See Configuring pipelining with engine_connect.
• Indirectly using the Universal Messaging message bus. For complex deployments where parts of the application may be moved between Apama correlators, this is likely to be the best alternative. When using Universal Messaging, each correlator connects to the same Universal Messaging realm. See The Universal Messaging Transport Connectivity Plug-in.
• Programmatically via the client API, see Configuring pipelining through the client API.
• Using a custom launch configuration in Apama Plugin for Eclipse. See Connecting correlators.

Configuring pipelining with engine_connect

The engine_connect tool allows direct connection of running correlator instances. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Note:

Some of the functions of the engine_connect tool can be performed from within EPL. For more information, see Using the Management interface.

Synopsis

To configure pipelining, run the following command:

engine_connect [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

engine_connect connects a source correlator (the sender) to a target correlator (the receiver). The target correlator will receive events from the specified channel(s) of the source correlator. Source and target correlators must already be running.

Alternatively, if you specify the –f option, engine_connect reads connection information from the specified file and sets up each connection found therein (see Configuring pipelining through the client API for details of the file format). The engine_connect tool expects the specified file to be in the local character set. If the configuration file is in UTF-8, specify the -u option in addition to the -f option. If the filename provided to -f is a hyphen (-), then connection information is read from the standard input device (stdin) until end-of-file.

The connection between the source and target correlators is persistent. When one of the correlators stops running, then when that correlator restarts it automatically reconnects with the other correlator.

The tool is silent by default unless an error occurs. For verbose progress information, use the –v option.

Options

The engine_connect tool takes the options listed below.

Note:

Many of these options can also be specified as elements of a YAML configuration file (with different element names). If an option is specified in both the command line and a YAML configuration file, then the command line takes precedence. For further information, see Configuring the correlator and especially the topic Setting up connections between correlators in a YAML configuration file.

Exit status

The engine_connect tool returns the following exit values:

Comparison of legacy and parallel connection modes

Universal Messaging has no mechanism for enforcing ordering among events sent on different channels. However, Universal Messaging is the better alternative when you want to use a large number of channels to send events between components. Without Universal Messaging, the use of two TCP connections with threads on both ends of the connection might reach the limit of how many channels can have dedicated connections.

Avoid mixing connection modes

Successive command lines that specify the same source/target hosts/ports build on each other. While this makes it possible to mix the legacy and parallel connection modes, you should avoid doing that. Mixing connection modes can cause an event to be delivered twice to the same channel. For example:

engine_connect -tp 15902 -sp 15903 -c channelA -c channelB
engine_connect -tp 15902 -sp 15903 -c channelA -c channelC -m legacy

The result of the first command is that there is one (legacy) connection for sending/receiving events on channelA and channelB. The result of the second command is that there is a dedicated connection for sending/receiving events on channelA and a dedicated connection for sending/receiving events on channelC. Events sent on channelA would be delivered twice: once on the legacy connection and once on the dedicated connection.

Examples

Because you can specify command lines that build on each other, you could set up a connection and add named channels later. You can also unsubscribe the channels you have added so that no events are sent or received. The connection remains and you can re-add channels at a later time. However, until you specify the -c option for a given connection, no events emitted by the source correlator are received by the target correlator. Consider the following command line:

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904

The correlators on host1 and host2 are connected but no channels have been subscribed and therefore no events are sent/received. To send and receive events, specify the -c option as in the following command line:

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -c CHAN1 -c CHAN2

Now the connected correlators can use CHAN1 and CHAN2 to send/receive events. To add another channel, execute this command:

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -c CHAN3

The correlators are now using CHAN1, CHAN2, and CHAN3 to send/receive events. To stop using CHAN2, execute the following command. The correlators continue to use CHAN1 and CHAN3.

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x -c CHAN2

To stop sending and receiving events, execute the following command. Note that the correlators remain connected until one of them stops. There is no penalty for this connection.

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x

In this example, the following command is equivalent to the previous command.

engine_connect -sn host1 -sp 15903 -tn host2 -tp 15904 -x -c CHAN1 -c CHAN3

Connection configuration file

engine_connect can take connection information from a file for connecting and disconnecting correlators. A sample of such a configuration file is shown below, which defines the topology shown in Engine topologies.

##### comments are allowed prefixed by a '#' – the rest of the line
##### is ignored
generator:dopey.apama.com:1234
worker1:sleepy.apama.com:1234:generator{chan1,chan2,chan3}
worker2:grumpy.apama.com:1234:generator{chan2}
worker3:sneezy.apama.com:1234:worker1{w1_out}
worker4:bashful.apama.com:1234:worker2{w2_out_1,w2_out_2,w2_out_3}
tallier:happy.apama.com:1234:worker3{result},worker4{result}
watcher:doc.apama.com:1234:generator{chan1,chan2}

Connection configuration file format

Each entry in the configuration file specifies connection information for a single correlator in the deployment. Entries can be specified in any order. The general format of an entry is:

correlator_name[:host][:port][:connection[,connection...]]

where connection is defined as:

correlator_name[ {channel_name[,channel_name...]}]

correlator_name is a symbolic identifier for a correlator, used to identify source correlators in target correlator connection information. It can consist of any combination of characters other than whitespace, colon, comma or open/close brace characters, which are reserved as separators. host and port identify the specific correlator this entry applies to. They can be omitted, in which case the defaults of localhost and 15903 are used respectively.

Following this information are details of all connections to source correlators for the current (target) entry. This information is omitted if no correlators sit “upstream” of the current entry (as with the correlator generator, above). If there are multiple upstream source correlators, each name should be separated by a comma (as with tallier, above, which takes events from worker3 and worker4).

For each connection, it is possible to specify the channel(s) on which the target correlator will listen. If no channels are specified, the target correlator will register to receive all events emitted by the source correlator regardless of channel (as with correlators worker3 and worker4 which register for all events from worker1 and worker2 respectively). One can specify specific channel names by enclosing them in braces and separating multiple channels by commas (as with watcher which registers with generator for all events on channels chan1 and chan2).

In effect, the configuration file is a convenient way of grouping several calls to engine_connect. For example, to set up the connections for the correlator tallier would require two commands using engine_connect:

>engine_connect -m parallel –sn sneezy.apama.com –sp 1234 –tn happy.apama.com
    -tp 1234 –c result
>engine_connect -m parallel –sn bashful.apama.com –sp 1234 –tn happy.apama.com
    -tp 1234 –c result

Errors in the configuration file

The configuration file can be used to both establish and remove connections in a multi-correlator engine topology. For example, assuming the above file is saved as topology.dat, the following commands will first set up then tear down all the connections specified therein:

>engine_connect -m parallel –f topology.dat
>engine_connect -m parallel –x –f topology.dat

In each of these cases, engine_connect will exit with non-zero exit status on the first error it detects in the configuration file. An error message will be printed to standard error (stderr).

Re-playing the configuration file

The behavior of engine_connect without the -x option is additive. This means that successive calls to engine_connect will attempt to add the channels specified to any existing connection between the source and target correlator(s). For example, with reference to the configuration file above, these commands:

>engine_connect -m parallel –sn dopey.apama.com –sp 1234 –tn sleepy.apama.com
    –tp 1234 –c foo
>engine_connect -m parallel –f topology.dat

will first add a connection from correlator generator to worker1 on channel foo, then (from the configuration file) extend that connection so that worker1 also receives all events from generator emitted on channel chan1.

Once a connection is set up between two correlators on a channel, any further attempt to set up that connection on the same channel will have no effect. It is therefore possible to re-play the configuration file by invoking engine_connect without creating duplicate connections. This can be useful if there is an error in the configuration file signaled when engine_connect is called, as the error can be fixed and engine_connect re-run without requiring removal of connections that were successfully set up by the first call to engine_connect.

Configuring pipelining through the client API

Apama provides client software development kits (SDKs) that can be used to interface with a running correlator or group of correlators. You can use attachAsConsumerOfEngine of the Engine Client API in Java and.NET (or attachAsConsumerOfEngine of the lower-level Engine Management API in Java,.NET, C++ and C). For more information, see Developing custom clients.

Event partitioning

Using engine_connect or the Apama client library, it is possible to create topologies of correlators across which an application’s monitors can be partitioned. Use the engine_inject tool described in Injecting code into a correlator, or by means of the relevant functions of the client library, to load the relevant monitors directly on to the appropriate correlators, specifying the host and port for each correlator.

This scheme is suitable for most applications, as monitors can be loaded once when Apama is brought online. For some applications, however, there is a requirement for a dynamic routing mechanism that (depending on the requirements of the application) continually splits and/or duplicates the incoming event stream and sends it to two or more correlators. Use the IAF transportChannel attribute to specify the channel an event is sent to, and connect that channel to the appropriate correlators.

Using jemalloc to optimize memory usage

Apama supports the jemalloc memory allocator as an optional alternative to its standard memory allocator. See also http://jemalloc.net/.

Note: This option is only available on Linux platforms and is not available on Windows at this time.