Real-time processing

Cumulocity allows developers and power users to run real-time IoT business logic inside Cumulocity based on a high-level real-time processing language.

This section introduces the basic concepts of real-time processing and shows how you can develop your own functional business logic at Cumulocity.

What is real-time processing in Cumulocity IoT?

On top of Cumulocity IoT you can use the Apama streaming analytics engine to define business operations for immediate processing of incoming data from devices or other data sources. These user-defined operations can for example alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices. The operation logic is implemented in Apama’s Event Processing Language (EPL).

Apama’s Event Processing Language covers statements, which are organized into actions and monitors. These can be deployed one file at a time, where a file may contain multiple monitors and event definitions. Monitor files can be edited with Software AG Designer - an Eclipse-based development environment, and can be deployed as Cumulocity applications, see Administration > Own applications in the User guide.

For further information on using Apama’s Event Processing Language in Cumulocity IoT refer to Using Apama Event Processing Language below and to the Analytics guide.

For more information about the interfaces for real-time processing also see Real-time Statements in the Reference guide.

Important: Support for streaming analytics using CEL (Esper) is deprecated. All new Cumulocity subscriptions use the Apama CEP engine. Software AG will terminate support for using CEL (Esper) in Cumulocity on 31 Dec 2020 following its deprecation in 2018. For documentation on using the deprecated CEL functionality based on Esper, refer to the CEL analytics guide.

What are the benefits of using real-time processing?

Cumulocity’s real-time processing feature has the following benefits:

Using the Apama Event Processing Language (EPL)

Overview

The Apama Event Processing Language has a syntax similar to Java. In addition to simple flow control statements such as if, while, for, users can write listeners with the on keyword to react to events.

Apama EPL is documented in the Apama documentation.

As an example, the following statement listens for new temperature sensor readings greater than a particular temperature:

on all Measurement(type="c8y_TemperatureMeasurement") as m {
    if m.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {
        Alarm alarm    := new Alarm;
        alarm.type     := "c8y_TemperatureAlert";
        alarm.source   := m.source;
        alarm.time     := currentTime;
        alarm.text     := "Temperature too high";
        alarm.status   := "ACTIVE";
        alarm.severity := "CRITICAL";
        send alarm to Alarm.CHANNEL;
    }
}

Here, Measurement is a pre-defined event containing the measurements. In this example, “m” is the “Measurement” event, the listener is filtering for measurements which are “c8y_TemperatureMeasurement” and the property is “c8y_TemperatureMeasurement.T.value” is in degrees Celsius of a temperature sensor (see the sensor library).

Listeners such as the above should be placed in a monitor in the onload statement, and the file will need to contain using statements for the types used by the listener - for most of the Cumulocity events, these are in the package com.apama.cumulocity. The full list is provided below - for the sake of brevity, we will omit these from further examples:

using com.apama.cumulocity.ManagedObject;
using com.apama.cumulocity.Operation;
using com.apama.cumulocity.Event;
using com.apama.cumulocity.Alarm;
using com.apama.cumulocity.Error;
using com.apama.cumulocity.Measurement;
using com.apama.cumulocity.MeasurementValue;
using com.apama.cumulocity.FindAlarm;
using com.apama.cumulocity.FindAlarmResponse;
using com.apama.cumulocity.FindAlarmResponseAck;
using com.apama.cumulocity.FindManagedObject;
using com.apama.cumulocity.FindManagedObjectResponse;
using com.apama.cumulocity.FindManagedObjectResponseAck;
using com.apama.cumulocity.FindMeasurement;
using com.apama.cumulocity.FindMeasurementResponse;
using com.apama.cumulocity.FindMeasurementResponseAck;
using com.apama.cumulocity.FindOperation;
using com.apama.cumulocity.FindOperationResponse;
using com.apama.cumulocity.FindOperationResponseAck;
using com.apama.cumulocity.FindEvent;
using com.apama.cumulocity.FindEventResponse;
using com.apama.cumulocity.FindEventResponseAck;
using com.apama.cumulocity.SendEmail;
using com.apama.cumulocity.SendSMS;
using com.apama.cumulocity.Util;
using com.apama.util.AnyExtractor;
using com.apama.correlator.timeformat.TimeFormat;
using com.softwareag.connectivity.httpclient.HttpOptions;
using com.softwareag.connectivity.httpclient.Request;
using com.softwareag.connectivity.httpclient.RequestType;
using com.softwareag.connectivity.httpclient.Response;

monitor ListenForHighTemperatures {
    action onload() {
        on all Measurement(type="c8y_TemperatureMeasurement") as e {
            if e.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {
                // handle the measurement
            }
        }
    }
}

How can I create derived data from EPL?

To create a new Alarm or Operation, create an instance of the relevant event type and use the send statement to send it to the relevant channel (defined with a constant on the event type). Assume that an alarm should be generated immediately if the temperature of a sensor exceeds a defined value. This is done with the following statement:

on all Measurement(type="c8y_TemperatureMeasurement") as m {
    if m.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {
        send Alarm("","c8y_TemperatureAlert",m.source,currentTime,"Temperature too high","ACTIVE","CRITICAL",1,new dictionary<string,any>) to Alarm.CHANNEL;
    }
}

Technically, this statement produces a new “Alarm” event each time a temperature sensor reads more than 100 degrees Celsius and sends it to Cumulocity.

How can I control devices from EPL?

Remote control with EPL is done by sending an Operation event. Remote operations are targeted to a specific device. The following example illustrates switching a relay based on temperature readings:

on all Measurement(type="c8y_TemperatureMeasurement") as m {
    if m.measurements.getOrDefault("c8y_TemperatureMeasurement").getOrDefault("T").value > 100.0 {
        send Operation("",m.source,"PENDING",{"c8y_Relay":<any>{"relayState":"CLOSED"}}) to Operation.CHANNEL;
    }
}

How can I query data from EPL?

It may be required to query information from the Cumulocity database as part of the ongoing event processing. This is supported by sending events and using listeners to wait for responses. Here is an example that shows how to summarize total sales for vending machines every hour. The sales report data created after a purchase is retrieved from the Cumulocity database.

using com.apama.aggregates.count;

monitor SalesReport {
    event SalesReport {
        Event e;
        ManagedObject customer;
    }
    event SalesOutput {
        integer count;
        string customerId;
    }

    action onload() {

        monitor.subscribe(Measurement.CHANNEL);

        on all Event() as e {
            monitor.subscribe(FindManagedObjectResponse.CHANNEL);
            integer reqId := integer.getUnique();
            on all FindManagedObjectResponse(reqId=reqId) as mor and not FindManagedObjectResponseAck(reqId=reqId) {
                route SalesReport(e, mor.managedObject);
            }
            on FindManagedObjectResponseAck(reqId=reqId) {
                monitor.unsubscribe(FindManagedObjectResponse.CHANNEL);
            }
            send FindManagedObject(reqId,"",{"childAssetId":e.source}) to FindManagedObject.CHANNEL;
        }

        from sr in all SalesReport() within 3600.0 every 3600.0
            group by sr.customer.id
            select SalesOutput(count(), sr.customer.id) as sales {
            send Measurement("", "total_cust_trx", "customer_trx_counterId", currentTime,
                {
                    "total_cust_trx":{
                        "total":MeasurementValue(sales.count.toFloat(), "COUNT", new dictionary<string,any>)
                    }
                }, {"customer_id":<any> sales.customerId}) to Measurement.CREATE_CHANNEL;
        }
    }
}

Above we create event definitions. These hold the SalesReport (the Event and ManagedObject that identifies a sale) and the information we want to derive from a set of sales: the count and customerId. We listen for Event objects, and send a FindManagedObject request to look up the ManagedObject that the event came from. These SalesReport objects are sent, via the route statement, into a stream query. The stream query fires every hour (3,600 seconds) and selects an aggregate of the sales data per customer, and sends a new Measurement representing the sales for that vending machine.

How is real-time processing implemented in Cumulocity?

There are two processing modes for API requests in Cumulocity: persistent and temporary. The “persistent” mode is the default: It will store data in the Cumulocity database as well as send the data to the real-time engine. After both is done, Cumulocity returns the result of the request.

Data marked as “temporary” is not stored into Cumulocity’s database but just handled by the real-time engine. This saves on storage and processing cost for example when tracking devices in real-time without requiring data to be stored.

The “temporary” mode will only send the data to the real-time engine and immediately return asynchronously and not store it in Cumulocity’s database. This mode saves storage and processing costs and is useful for example when tracking devices in real-time without requiring data to be stored.

CEP architecture

Examples

Assume that location updates from cars should be monitored every second while the car is driving, but only be stored once in a minute into the database for reporting purposes. This is done using the following Apama statement:

using com.apama.cumulocity.Event;
using com.apama.cumulocity.Measurement;

monitor SendEveryMinute {

    dictionary<string, Event> latestUpdates;

    action onload() {

        monitor.subscribe(Measurement.CHANNEL);
        on all Event() as e {
            if e.params.hasKey("c8y_LocationUpdate") {
                latestUpdates[e.source] := e;
            }
        }

        on all wait(60.0) {
            Event e;
            for e in latestUpdates.values() {
                send e to Event.CHANNEL;
            }
            latestUpdates.clear();
        }
    }
}

Another option is to output only every 60th update.

using com.apama.cumulocity.Event;
using com.apama.cumulocity.Measurement;

monitor SendEverySixtyEvents {

    event UpdateAndCount {
        Event latest;
        integer count;
    }

    dictionary<string, UpdateAndCount> latestUpdates;

    action onload() {
        monitor.subscribe(Measurement.CHANNEL);
        on all Event() as e {
            if e.params.hasKey("c8y_LocationUpdate") {
                UpdateAndCount updateCount := latestUpdates.getOrAddDefault(e.source);
                updateCount.latest := e;
                updateCount.count := updateCount.count + 1;
                if updateCount.count = 60 {
                    send e to Event.CHANNEL;
                    latestUpdates.remove(e.source);
                }
            }
        }
    }
}