Streaming analytics with Apama EPL

Using Apama streaming analytics, you can add your own logic to your IoT solution for immediate processing of incoming data from devices or other data sources. These user-defined operations can, for example, alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices.

The operation logic is based on Apama’s Event Processing Language (Apama EPL).

Important: Support for streaming analytics using CEL (Esper) is deprecated. All new Cumulocity IoT subscriptions use the Apama CEP engine. While using the Esper CEP engine is still supported for older installations, this will no longer be provided for new subscriptions and will not be invested into in the future.

For documentation on using the deprecated CEL functionality based on Esper, refer to the CEL analytics guide.

For details on migration, see Migrating from CEL (Esper) to Apama.

Typical real-time analytics use cases include:

The following sections describe how you can create your own analytics or other server-side business logic and automation. You can do this using smart rules, Analytics Builder models and EPL apps.

Info: This documentation assumes basic familiarity with Apama application development. Refer to the Apama documentation for further details.

Apama EPL Apps

Apama EPL Apps is a web application which is available from the application switcher. It allows you to develop EPL apps (that is, single *.mon files) directly within Cumulocity IoT, written in Apama EPL. You can also use it to import existing *.mon files as EPL apps into Cumulocity IoT. When you activate an EPL app via the Apama EPL Apps web application, you deploy it to Cumulocity IoT.

A quick way to get started is to explore the code of the EPL samples that can be accessed from the EPL editor in the Apama EPL Apps web application. See Developing apps with Apama EPL Apps for information on how to create an EPL app and access the samples. For a start, use one of the simpler samples with temperature measurements, such as “Raise Alarm if Measurement exceeds a threshold value”. You can immediately see results using this sample. Add your own EPL code to the sample and try out your changes.

You use the Apama API for interacting with Cumulocity IoT. For detailed information, see the com.apama.cumulocity package in the API Reference for EPL (ApamaDoc), which is part of the Apama documentation.

Apama Analytics Builder

Apama Analytics Builder is a web application which is available from the application switcher. It allows you to build analytic models that transform or analyze streaming data in order to generate new data or output events. The models are capable of processing data in real time.

You build the models in a graphical environment by combining pre-built blocks into models. The blocks in a model package up small bits of logic, and have a number of inputs, outputs and parameters. Each block implements a specific piece of functionality, such as receiving data from a sensor, performing a calculation, detecting a condition, or generating an output signal. You define the configuration of the blocks and connect the blocks using wires. You can edit the models, simulate deployment with historic data, or run them against live systems.

See the documentation for Apama Analytics Builder for Cumulocity IoT for detailed information.

Microservice runtime and applications

Custom EPL apps (individual *.mon files), Apama Analytics Builder models and smart rules are executed in an Apama-ctrl microservice. This has a per-tenant isolation scope, that is, each subscribed tenant has its own instance of an Apama container with dedicated resources (that is, memory and CPU usage). The container is isolated from other tenants, hence high CPU load or memory issues on other containers are tracked and resourced independently.

You can use predefined rules (see Smart rules), define your own custom rules with Apama EPL Apps, or use Apama Analytics Builder to build analytic models. This requires the following microservices and/or applications:

To do this you need the following
Use predefined rules Apama-ctrl microservice and Smartrule microservice (included in Cumulocity IoT’s Standard Tenant).
Define custom rules Apama-ctrl microservice and Apama EPL Apps web application.
Use Apama Analytics Builder Apama-ctrl microservice and Apama Analytics Builder web application.

See also the tables listing the available applications under Managing applications in the User guide.

If your tenant is subscribed to the Apama Starter microservice (instead of other Apama-ctrl microservices), then the following applies for Apama:

Contact product support to discuss adding more capabilities.

Info: If your tenant is subscribed to the Apama Smart Rules-only microservice (also called Apama-ctrl-smartrules), Apama EPL Apps and Apama Analytics Builder are not available.

Migrating from CEL (Esper) to Apama

Migrating from CEL when only using smart rules

If a tenant is only using smart rules and not any custom rules, you can simply unsubscribe the tenant from CEL and subscribe to Apama instead (or ask your operations team to do this for your tenant).

Any previously configured smart rules will be restarted. For smart rules which are stateful, this will, as with any restart of the microservice hosting the smart rules, lose state within the smart rule. In this case, the input (measurements, alarms, etc.) for the devices in question will need to be sent again before the smart rule will be functioning as before.

Smart rules will only work correctly if moving from CEL to Apama, not in the opposite direction.

The smart rules listed in the following table are stateful:

For this smart rule Check the following
On geofence create alarm Which devices are currently in or out of the geofence?
On geofence send email Which devices are currently in or out of the geofence?
On alarm escalate it Which alarms have been created?
On alarm duration increase severity Which alarms have been created?
Calculate energy consumption What are the last meter readings?
On missing measurements create alarm What is the previous time of measurement and which devices have sent the measurement before?

The following smart rules are stateless:

Migrating from CEL when also using custom rules

Migrating from custom rules written in CEL to Apama EPL requires rewriting and retesting the custom rules. If any of the CEL generated from smart rules has been modified, you will need to convert that to an Apama EPL app as well, and delete the smart rule when migrating. As with any scripting or programming, you should thoroughly test significant changes before deploying into a production environment. Thus, the recommended approach is to create a separate tenant for hosting Apama EPL apps as they are developed, and replicate any input data required in that tenant. The CEP rules can continue to run in your production tenant while you develop the new Apama EPL apps. To do this, follow these steps:

  1. Lock down the CEP custom rules on the existing tenant to prevent change.
  2. Make available a new tenant on which Apama has been enabled.
  3. Manually convert all old custom rules from the existing tenant into equivalent Apama EPL apps on the new tenant. Refer to the rest of this guide, in particular Best practices and guidelines. This includes smart rules where the CEL has been modified.
  4. Test the behavior of the new EPL apps by sending, for example, measurements or events into the new tenant and verifying that the new EPL apps respond appropriately.
  5. When all new EPL apps have been developed and tested, move your production tenant from CEP to Apama, that is: subscribe the new tenant to the Apama-ctrl microservice (and unsubscribe it from CEP).
    • Any unmodified smart rules will migrate automatically.
    • Delete any smart rules where the CEL version had been modified and a new EPL app has been implemented.
    • Activate your newly developed EPL apps in the production tenant.

You can also choose to work with Software AG Professional Services to help ensure the migration is as smooth as possible. Software AG Professional Services can help migrate CEL code into Apama EPL code and they can also provide training on using Apama in Cumulocity IoT.

Handling scheduled exports

Scheduled exports must be migrated to the report-agent microservice. This can be performed by opening a report in the Cockpit application. See also Exporting data in the User guide.

Info: To use the new export schedule feature and for the migration to work, the report-agent microservice needs to be subscribed. New tenants will be subscribed to it automatically. Existing tenants should make sure that they are subscribed to it.

Using the Esper-to-Apama EPL translation tool

The open-source Esper-to-Apama EPL translation tool assists you in the migration of Esper CEL to Apama EPL. It generates EPL that is compatible with the Apama microservice version 10.6.6 and above. The translation tool is available from GitHub at https://github.com/SoftwareAG/apama-streaming-analytics-esper2apama.

This tool focuses on reducing (though not eliminating) the amount of human involvement needed during migration by automating translation of some of the most commonly occurring Esper constructs. See the README in GitHub for more details.