Streaming analytics with Apama EPL
Using the Streaming Analytics application, you can add your own logic to your IoT solution for immediate processing of incoming data from devices or other data sources. These user-defined operations can, for example, alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices.
The operation logic is based on Apama’s Event Processing Language (Apama EPL).
Important: Support for streaming analytics using CEL (Esper) has ended. All new Cumulocity IoT subscriptions use the Apama CEP engine. Software AG terminated support for using CEL (Esper) in Cumulocity IoT on 31 Dec 2020 following its deprecation in 2018.
For documentation on using the unsupported CEL functionality based on Esper, refer to the CEL analytics guide.
For details on migration, see Migrating from CEL (Esper) to Apama.
Typical real-time analytics use cases include:
- Remote control: Turn a device off if its temperature rises over 40 degrees.
- Validation: Discard negative meter readings or meter readings that are lower than the previous.
- Derived data: Calculate the volume of sales transactions per vending machine per day.
- Aggregation: Sum up the sales of vending machines for a customer per day.
- Notifications: Send me an email if there is a power outage in one of my machines.
- Compression: Store location updates of all cars only once every five minutes (but still send real-time data for the car that I am looking at to the user interface).
The following sections describe how you can create your own analytics or other server-side business logic and automation. You can do this using smart rules, analytic models and EPL apps.
Info: This documentation assumes basic familiarity with Apama application development. Refer to the Apama documentation for further details.
Home screen
You access the Streaming Analytics application using the application switcher.
The home screen of the Streaming Analytics application lets you navigate to the different pages of the application. These are Analytics Builder and EPL Apps. You can access these pages using the corresponding Open button on the home screen or by using the navigator on the left.
If you need more space for a page, you can hide the navigator. Click the small arrow at the very left of the top bar to toggle the display of the navigator.
The home screen also provides information on smart rules. You create and maintain them with the Cockpit application. See Smart rules in the User guide for detailed information.
The home screen and navigator only show entries for the items that you are allowed to use, depending on the version of the Apama-ctrl microservice to which your tenant is subscribed. See also Microservice runtime and applications and Customizing the home screen of the Streaming Analytics application.
Analytics Builder
The Analytics Builder page of the Streaming Analytics application allows you to build analytic models that transform or analyze streaming data in order to generate new data or output events. The models are capable of processing data in real time.
You build the models in a graphical environment by combining pre-built blocks into models. The blocks in a model package up small bits of logic, and have a number of inputs, outputs and parameters. Each block implements a specific piece of functionality, such as receiving data from a sensor, performing a calculation, detecting a condition, or generating an output signal. You define the configuration of the blocks and connect the blocks using wires. You can edit the models, simulate deployment with historic data, or run them against live systems.
See the documentation for Analytics Builder for detailed information.
EPL Apps
The EPL Apps page of the Streaming Analytics application allows you to develop EPL apps (that is, single *.mon files) directly within Cumulocity IoT, written in Apama EPL. You can also import existing *.mon files as EPL apps into Cumulocity IoT. When you activate an EPL app from the Streaming Analytics application, you deploy it to Cumulocity IoT.
A quick way to get started is to explore the code of the EPL samples that can be accessed from the EPL editor. See Developing apps with the Streaming Analytics application for information on how to create an EPL app and access the samples. For a start, use one of the simpler samples with temperature measurements, such as “Raise Alarm if Measurement exceeds a threshold value”. You can immediately see results using this sample. Add your own EPL code to the sample and try out your changes.
You use the Apama API for interacting with Cumulocity IoT. For detailed information, see the com.apama.cumulocity package in the API Reference for EPL (ApamaDoc), which is part of the Apama documentation.
Microservice runtime and applications
Custom EPL apps (individual *.mon files), analytic models and smart rules are executed in an Apama-ctrl microservice. This has a per-tenant isolation scope, that is, each subscribed tenant has its own instance of an Apama container with dedicated resources (that is, memory and CPU usage). The container is isolated from other tenants, hence high CPU load or memory issues on other containers are tracked and resourced independently.
You can use predefined rules (see Smart rules), define your own custom rules on the EPL Apps page, or build analytic models on the Analytics Builder page. This requires the following microservices and/or applications:
| To do this | you need the following |
|---|---|
| Use predefined rules | Apama-ctrl microservice and Smartrule microservice (included in Cumulocity IoT’s Standard tenant). |
| Define custom rules | Apama-ctrl microservice and Streaming Analytics application. |
| Build analytic models | Apama-ctrl microservice and Streaming Analytics application. |
See also the tables listing the available applications under Managing applications in the User guide.
If your tenant is subscribed to the Apama Starter microservice (instead of other Apama-ctrl microservices), then the following applies for Apama:
- Unlimited number of smart rules.
- Limited number of at most 3 active models. Custom blocks written with the Analytics Builder Block SDK cannot be used.
- The EPL Apps page is not available in the Streaming Analytics application.
Contact product support to discuss adding more capabilities.
Info: If your tenant is subscribed to the Apama Smart Rules-only microservice (also called Apama-ctrl-smartrules), the EPL Apps and Analytics Builder pages are not available in the Streaming Analytics application.
Migrating from CEL (Esper) to Apama
Migrating from CEL when only using smart rules
If a tenant is only using smart rules and not any custom rules, you can simply unsubscribe the tenant from CEL and subscribe to Apama instead (or ask your operations team to do this for your tenant).
Any previously configured smart rules will be restarted. For smart rules which are stateful, this will, as with any restart of the microservice hosting the smart rules, lose state within the smart rule. In this case, the input (measurements, alarms, etc.) for the devices in question will need to be sent again before the smart rule will be functioning as before.
Smart rules will only work correctly if moving from CEL to Apama, not in the opposite direction.
The smart rules listed in the following table are stateful:
| For this smart rule | Check the following |
|---|---|
| On geofence create alarm | Which devices are currently in or out of the geofence? |
| On geofence send email | Which devices are currently in or out of the geofence? |
| On alarm escalate it | Which alarms have been created? |
| On alarm duration increase severity | Which alarms have been created? |
| Calculate energy consumption | What are the last meter readings? |
| On missing measurements create alarm | What is the previous time of measurement and which devices have sent the measurement before? |
The following smart rules are stateless:
- On alarm send email
- On alarm send SMS
- On alarm execute operation
- On measurement threshold create alarm
- On measurement explicit threshold create alarm
Migrating from CEL when also using custom rules
Migrating from custom rules written in CEL to Apama EPL requires rewriting and retesting the custom rules. If any of the CEL generated from smart rules has been modified, you will need to convert that to an Apama EPL app as well, and delete the smart rule when migrating. As with any scripting or programming, you should thoroughly test significant changes before deploying into a production environment. Thus, the recommended approach is to create a separate tenant for hosting Apama EPL apps as they are developed, and replicate any input data required in that tenant. The CEP rules can continue to run in your production tenant while you develop the new Apama EPL apps. To do this, follow these steps:
- Lock down the CEP custom rules on the existing tenant to prevent change.
- Make available a new tenant on which Apama has been enabled.
- Manually convert all old custom rules from the existing tenant into equivalent Apama EPL apps on the new tenant. Refer to the rest of this guide, in particular Best practices and guidelines. This includes smart rules where the CEL has been modified.
- Test the behavior of the new EPL apps by sending, for example, measurements or events into the new tenant and verifying that the new EPL apps respond appropriately.
Important: CEL allowed measurements with null values. This is no longer supported. Therefore, you must make sure that all of your measurements send numeric values.
- When all new EPL apps have been developed and tested, move your production tenant from CEP to Apama, that is: subscribe the new tenant to the Apama-ctrl microservice (and unsubscribe it from CEP).
- Any unmodified smart rules will migrate automatically.
- Delete any smart rules where the CEL version had been modified and a new EPL app has been implemented.
- Activate your newly developed EPL apps in the production tenant.
You can also choose to work with Software AG Professional Services to help ensure the migration is as smooth as possible. Software AG Professional Services can help migrate CEL code into Apama EPL code and they can also provide training on using Apama in Cumulocity IoT.
Handling scheduled exports
Scheduled exports must be migrated to the report-agent microservice. This can be performed by opening a report in the Cockpit application. See also Managing exports in the User guide.
Info: To use the new export schedule feature and for the migration to work, the report-agent microservice needs to be subscribed. New tenants will be subscribed to it automatically. Existing tenants should make sure that they are subscribed to it.
Using the Esper-to-Apama EPL translation tool
The open-source Esper-to-Apama EPL translation tool assists you in the migration of Esper CEL to Apama EPL. It generates EPL that is compatible with the Apama microservice version 10.6.6 and above. The translation tool is available from GitHub at https://github.com/SoftwareAG/apama-streaming-analytics-esper2apama.
This tool focuses on reducing (though not eliminating) the amount of human involvement needed during migration by automating translation of some of the most commonly occurring Esper constructs. See the README in GitHub for more details.