Use Case: Anomaly Detection - Cumulocity IoT Guides

Overview

Anomaly detection (also outlier detection) is the identification of items, events or observations which do not conform to an expected pattern or other items in a data set.

Capturing anomalous events through the sensor data of a mobile device on an IoT platform can for instance serve the purpose of detecting accidents of elderly people living without a caretaker. Regular behavior sensor data of a person can be collected over a period of time. This data can then be used to train an anomaly detection model. As soon as an irregularity in behavior data is observed, e.g. the person falls down, an anomaly can be detected. For the purpose of showcasing this use case, we followed these steps:

Collect sensor data from a user performing regular everyday tasks.
With the collected data, train an anomaly detection model using the Jupyter Notebook and convert the model to PMML.
Deploy the model to Cumulocity IoT using the Machine Learning Workbench application.
Create and upload an EPL rule to Cumulocity IoT which does the following:
- Gathers specific measurements coming from the source device and conducts any necessary pre-processing steps.
- Sends the data via REST request to the Zementis microservice API for processing.
- Creates an anomaly detection alarm if the model predicts the input data to be anomalous.

Prerequisites

Download the AnomalyDetectionDemo.zip file which contains the project ZIP with name AnomalyDetectionDemoProject.zip and the EPL rule with name DetectAnomalies.mon.

Running the demo requires:

Prior experience and understanding of data science processes.
Familiarity with Cumulocity IoT and its in-built apps.
Subscription of the Zementis microservice (10.5.0.x.x or higher),the MLW microservice (10.13.0.x.x or higher),the Machine Learning application and the Machine Learning Workbench application on the tenant.
Subscription of the Apama-ctrl microservice and the Apama-epl application on the tenant.

Getting Started

A phone or a phone-like device is required for this use case so that the measurement data of that particular device can be captured and used for detecting anomalies.

The documentation provides instructions for the following devices:

Anomaly detection using a smartphone
Anomaly detection using a simulated demo device

Anomaly detection using a smartphone

This section deals with the basic data science steps of creating an anomaly detection model with self-collected data. First of all, you need to register your smartphone. Then follow the sections below for collecting data, training the model, and using the model to detect anomalies via the phone.

Info

The phone used for the entire workflow must be of the same type because the data and sensors may vary for different devices.

Register a smartphone in the platform

Follow the steps described in Cumulocity IoT Sensor App in the User guide and register a smartphone in Cumulocity IoT.

Info

Set “1 sec” as INTERVAL (secs) for Acceleration and Gyroscope sensors in the Cumulocity IoT Sensor App.

Once registered, note down the device ID by looking up your device on the All Devices page of your tenant’s Device management application.

In contrast to supervised classification models, labeled training data is not required for anomaly detection models. The model is trained with the regular data and any unseen behavior will later be detected as anomalous. The data can be collected by carrying around the registered device over a few days without any anomalous behavior. All data can then be accessed via Cumulocity IoT Machine Learning Workbench that automatically transforms the JSON data into the training data format.

Data collection with Machine Learning Workbench (MLW)

Follow the steps described in Machine Learning Workbench > Upload a project and upload the AnomalyDetectionDemoProject.zip project to MLW. This creates a new project named AnomalyDetectionDemoProject_{UUID}, where UUID is a system generated unique identifier. This project has a total of 6 resources. You will get 3 files in the Data folder and 3 files in the Code folder.
You can either download the recorded measurements of your smartphone or use the data provided within the project for model-building purposes.
- Follow the steps described in Machine Learning Workbench > Data pull > Cumulocity IoT and pull the measurements of the newly registered smartphone with “anomalyTrainingData” as File name, data interval (i.e. interval during which the data was created), “None” as Aggregation and select “c8y_Acceleration” and “c8y_Gyroscope” as Data points.
- Alternatively, use the anomalyTrainingData.csv file in the Data folder of the project.

Train the PMML model

For this use case, the “Isolation Forest”, an anomaly detection machine learning algorithm is applied. Isolation Forest is an approach that detects anomalies by isolating instances, without relying on any distance or density measure.

The logic argument goes: isolating anomaly observations is easier as only a few conditions are needed to separate those cases from the normal observations. On the other hand, isolating normal observations requires more conditions. Therefore, an anomaly score can be calculated as the number of conditions required to separate a given observation. See also Anomaly Detection Using Isolation Forests

The integrated Jupyter Notebook feature within Cumulocity IoT Machine Learning Workbench helps in writing the code that creates an Isolation Forest model in PMML format using the previously uploaded training data. The script uses the scikit-learn framework (https://scikit-learn.org) to train the Isolation Forest model.

Info

To obtain a robust and meaningful model, further cleaning of the training data and validating the best model parameters is required. This is not in the scope of this demo and presumes knowledge of data science best practices.

After the model is created, the scikit-learn object can be converted to PMML format using the Nyoka library https://github.com/nyoka-pmml/nyoka.

The following steps illustrate the training of an Isolation Forest machine learning model using the Jupyter Notebook.

Open the createModel.ipynb file in the Code folder of the project.
Follow the steps described in Machine Learning Workbench > Jupyter Notebook > Editing and executing a notebook and execute the existing code snippets in each cell of the createModel.ipynb to train an Isolation Forest PMML model.

Train Model using Jupyter Notebook

Click the refresh icon at the top of Tabs to list the newly created isolationForests.pmml in the Model folder.

Model deployment and predictions

Once the model is available in the Model folder, it can be deployed to Machine Learning Engine (MLE) for predictions.

Follow the steps described in Machine Learning Workbench > Automated ML > Model deployment and predictions and deploy the isolationForests.pmml model to Machine Learning Engine (MLE), and predict test_data.csv data available in the Data folder, using the isolationForest PMML model.

Create and upload Apama monitor file

For this anomaly detection scenario, we need to use Apama streaming analytics. With Apama streaming analytics, you can add your logic to your IoT solution for the immediate processing of incoming data from devices or other data sources. This user-defined logic can for example alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices.

We create an EPL-based monitor file and upload it to Cumulocity IoT. As mentioned earlier, the Apama EPL monitor file takes care of reading the measurements coming from the mobile device, sending it to the Zementis microservice, and raising an alarm when an anomaly is reported by our machine learning model.

Instead of creating a new monitor file, the attached DetectAnomalies.mon file can be used after making minor adjustments. Open DetectAnomalies.mon in a text editor and replace the deviceId variable with the ID of your registered device. Save your changes and upload this monitor file to your tenant. See EPL Apps > Basic functionality > Deploying EPL apps as single *.mon files with the Streaming Analytics application in the Streaming Analytics guide for details on uploading Apama monitor files.

using com.apama.correlator.Component;
using com.apama.cumulocity.Alarm;
using com.apama.cumulocity.CumulocityRequestInterface;
using com.apama.cumulocity.Measurement;
using com.apama.cumulocity.FindManagedObjectResponse;
using com.apama.cumulocity.FindManagedObjectResponseAck;
using com.apama.cumulocity.FindManagedObject;
using com.softwareag.connectivity.httpclient.HttpOptions;
using com.softwareag.connectivity.httpclient.HttpTransport;
using com.softwareag.connectivity.httpclient.Request;
using com.softwareag.connectivity.httpclient.Response;
using com.apama.json.JSONPlugin;

monitor DetectAnomalies {

    CumulocityRequestInterface cumulocity;

    action onload() {
        cumulocity := CumulocityRequestInterface.connectToCumulocity();
        // Replace yourDeviceId with the value of your device id
        listenAndActOnMeasurements("yourDeviceId", "IsolationForest");
    }

    action listenAndActOnMeasurements(string deviceId, string modelName) {
        monitor.subscribe(Measurement.SUBSCRIBE_CHANNEL);   
        on all Measurement(source = deviceId) as m {

            if( m.measurements.hasKey("c8y_Acceleration")){
            log "Received Measurement c8y_Acceleration from C8Y" + m.toString();

            dictionary <string, any> lastMeasurement := {};
                lastMeasurement["accelerationX"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationX").value;
                lastMeasurement["accelerationY"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationY").value;
                lastMeasurement["accelerationZ"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationZ").value;

                listener l := on all Measurement(source = deviceId) as n {
                    if n.measurements.hasKey("c8y_Gyroscope"){
                        lastMeasurement["gyroX"] := n.measurements.getOrDefault("c8y_Gyroscope").getOrDefault("gyroX").value;
                        lastMeasurement["gyroY"] := n.measurements.getOrDefault("c8y_Gyroscope").getOrDefault("gyroY").value;
                        lastMeasurement["gyroZ"] := n.measurements.getOrDefault("c8y_Gyroscope").getOrDefault("gyroZ").value;
                     string record := JSONPlugin.toJSON(lastMeasurement);

                       log "Sending record to zementis - " + record;
                       Request zementisRequest := cumulocity.createRequest("GET", "/service/zementis/apply/" + modelName, any());
                       zementisRequest.setQueryParameter("record", record);
                       zementisRequest.execute(ZementisHandler(deviceId).requestHandler);
                       log "EPL execution completed.";
                       l.quit();
                    }
                }
            }
        }
    }

   event ZementisHandler
   {
       string deviceId;
       action requestHandler(Response zementisResponse)
       {
           integer statusCode := zementisResponse.statusCode;
           boolean outlier := <boolean> zementisResponse.payload.getSequence("outputs")[0].getEntry("outlier");
           log "Zementis responded with status:" + statusCode.toString() + " result:" +  (outlier).toString();
           if (statusCode = 200 and outlier = true) {
               send Alarm("", "AnomalyDetectionAlarm", deviceId, currentTime,
                   "Anomaly detected", "ACTIVE", "CRITICAL", 1, new dictionary<string, any>) to Alarm.SEND_CHANNEL;
               log "Alarm raised";
           }
       }
   }

}

Trigger an anomaly alert

Now that you have all the pieces together, you can try to generate an anomaly. To generate an anomaly you could drop your mobile phone or throw it in the air and then catch it.

You should be able to see alarms being generated from your device which will be visible under the Alarms page of the Device management application.

Anomaly detection using a demo device

A fully functional demo can be prepared with the help of a demo device. For this, use the artifacts provided as part of the AnomalyDetectionDemo.zip file.

Register a demo device in the platform

A demo device can be registered instead of registering a real phone and used as a replica of an actual mobile phone.

Follow the steps described in Machine Learning Workbench > Upload a project and upload the AnomalyDetectionDemoProject.zip project to MLW. A new project is created with the name AnomalyDetectionDemoProject_{UUID}, where UUID is a system generated unique identifier. This project has a total of 6 resources. The project contains a script with name DemoDeviceCreator.py in the Code folder, that registers a demo device in Cumulocity IoT and also a CONFIG.json file in the Data folder of Machine Learning Workbench (MLW).
Select CONFIG.json in the Data folder and click the edit icon to edit the CONFIG.json.
Update the values of c_url, c_user and c_pass with your tenant credentials and click the save icon at the right of the top menu bar.
Select DemoDeviceCreator.py in the Code folder and click the execute icon at the right of the top menu bar and fill the form with “DemoDeviceCreator” as Task Name, “ONE TIME” as Recurrence and click the submit icon .

Click Tasks in the navigator and click the “DemoDeviceCreator” task name, to display the status of the Python execution in the Task History section at the center.

Script to add a device named “DemoDevice” to Cumulocity IoT.

DemoDeviceCreator.py

# Register a demo device with Cumulocity IoT
import requests
import json
from requests.auth import HTTPBasicAuth
import datetime

data = json.load(open('../Data/CONFIG.json',))

payload={'name': 'DemoDevice',
'c8y_IsDevice': [],
'c8y_SupportedMeasurements': ['RoboSensors'],
'c8y_SupportedOperations': ['c8y_Restart',
'c8y_Configuration',
'c8y_Software',
'c8y_Firmware',
'c8y_Command']}

url = data['c_url']+"/inventory/managedObjects"
headers = {
	'Content-Type': "application/json",
	'Accept': "application/json",
	'cache-control': "no-cache",
	'Postman-Token': "2dc79351-5a48-4b8b-b4f2-30b880732d01"
	}

response = requests.request("POST", url, data=json.dumps(payload), headers=headers,auth=HTTPBasicAuth(data['c_user'], data['c_pass']))

print("The device id is:" , json.loads(response.text)['id'])

Once registered, you can get the device ID by looking up your device on the All Devices page of your tenant’s Device management application. Now, update the c_device_source of the CONFIG.json file with the device ID of this demo device.

Upload the model and Apama monitor file

Upload the attached model iforest_model.pmml to Cumulocity IoT. To upload the model to Cumulocity IoT, follow the steps described in Machine Learning application > Managing models.
Download the attached DetectAnomalies.mon file, open it in a text editor and replace the value of c_device_source with the ID of your registered device.
Save your changes and upload this monitor file to your tenant. See EPL Apps > Basic functionality > Deploying EPL apps as single *.mon files with the Streaming Analytics application in the Streaming Analytics guide for details on uploading Apama monitor files.

Trigger an anomaly alert

A script AnomalySimulatorForDemoDevice.py has been attached which simulates sending of alternate anomalous and non-anomalous readings to Cumulocity IoT from our demo device. This script can be used to depict the generation of anomalies.

All you need to do is to run the script as explained before.

AnomalySimulatorForDemoDevice.py

# Simulate anamolous and non-anamolous data
import requests
import json
from requests.auth import HTTPBasicAuth
import datetime
import random
data = json.load(open('../Data/CONFIG.json',))

url = data['c_url']+"/measurement/measurements"

# Simulated anamolous data
tt=datetime.datetime.now()

headers = {
'Content-Type': "application/json",
'Accept': "application/vnd.com.nsn.cumulocity.measurement+json",
'cache-control': "no-cache",
'Postman-Token': "2d5fa27d-c8c8-428c-b2f9-0efe9490b716"
}

payload1={'type': 'anamoly',
	'time': str(tt.date())+'T'+str(tt.hour)+':'+str(tt.minute)+':'+str(tt.second)+'+05:30',
	'source': {'id': data['c_device_source']},
	"c8y_Acceleration":{"accelerationY":{"unit":"G","value":-0.2631993591785431},
						"accelerationX":{"unit":"G","value":5.769125938415527},
					"accelerationZ":{"unit":"G","value":8.193016052246094}},
	"c8y_Gyroscope":{"gyroX":{"unit":"°/s","value":-0.03604104742407799},
					"gyroY":{"unit":"°/s","value": 0.055571284145116806},
					"gyroZ":{"unit":"°/s","value":-0.0010122909443452952}}
}

response = requests.request("POST", url, data=json.dumps(payload1),
						headers=headers,auth=HTTPBasicAuth(data['c_user'], data['c_pass']))

# Simulated non-anamolous data
tt=datetime.datetime.now()
payload2={'type': 'non-anamoly',
	'time': str(tt.date())+'T'+str(tt.hour)+':'+str(tt.minute)+':'+str(tt.second)+'+05:30',
	'source': {'id': data['c_device_source']},
	"c8y_Acceleration":{"accelerationY":{"unit":"G","value":0.0971527099609375},
						"accelerationX":{"unit":"G","value":0.6249847412109375},
					"accelerationZ":{"unit":"G","value":-0.2371368408203125}},
	"c8y_Gyroscope":{"gyroX":{"unit":"°/s","value":-1.2540942430496216},
					"gyroY":{"unit":"°/s","value": -1.861748218536377},
					"gyroZ":{"unit":"°/s","value":-0.029031118378043175}}
}

response = requests.request("POST", url, data=json.dumps(payload2),
						headers=headers,auth=HTTPBasicAuth(data['c_user'], data['c_pass']))

This should send alternate anomalous and non-anomalous measurements to Cumulocity IoT on behalf of your demo device.

You should notice anomaly detection alarms for every anomalous measurement that it sends. These alarms generated from your device will be visible under the Alarms page of the Device management application.