Deprecation of Machine Learning Engine and Machine Learning Workbench
As of Cumulocity IoT 10.16, Machine Learning Engine and Machine Learning Workbench are deprecated, and we will end further development of these components. After Cumulocity IoT 10.17, both will be removed from the platform. If you have existing (production) projects which rely on MLE and/or MLW applications, then Software AG is committed to working together with you to find an appropriate transition path. If we have not yet been in contact for this matter, feel free to contact us at c8y_mlw_mle_sunsetting@softwareag.com.
Sensor-based activity recognition can be used to model a wide range of human activities. Mobile devices, such as smart phones provide sufficient sensor data and calculation power to enable physical activity recognition to provide an estimation of the energy consumption during everyday life.
Sensor-based activity recognition researchers believe that by empowering ubiquitous computers and sensors to monitor the behavior of agents (under consent), these computers will be better suited to act on our behalf (see Wikipedia).
In this use case, we would like to showcase the recognition of the human activities - sitting, jumping and walking. For the purpose of showcasing the same, we followed these steps:
Collect sensor data from a user performing activities (sitting, jumping and walking).
Train an activity recognition model with the collected data and generate the model in PMML format using Cumulocity IoT Machine Learning Workbench.
Deploy the model into Cumulocity IoT Machine Learning Engine.
Create and upload an EPL rule to Cumulocity IoT Streaming Analytics which does the following:
Gathers specific measurements coming from the source device and conducts any necessary pre-processing steps.
Sends the data via REST request to the Zementis microservice API for processing.
Creates an update alert once the user changes activities.
Prerequisites
Download the ActivityRecognitionDemo.zip file which contains the project ZIP named ActivityRecognitionDemoProject.zip and the EPL rules.
Running the demo scripts requires
Prior experience with Jupyter Notebook, Python, JSON, REST and understanding of data science processes.
Familiarity with Cumulocity IoT and its in-built apps.
Subscription of the Cumulocity IoT Machine Learning Workbench microservice (10.13.0.x.x or higher), Zementis microservice (10.5.0.x.x or higher), the Machine Learning Workbench application and the Machine Learning application on the tenant.
Subscription of the Apama-ctrl microservice and the Apama-epl application on the tenant.
Getting Started
Upload the project ZIP file to Cumulocity IoT Machine Learning Workbench. We have added a CONFIG.json file to the project. This file is meant to capture the tenant details and credentials which will be used by the demo scripts.
First, update the CONFIG.json with the appropriate values and save it. Replace c_url with your tenant URL, c_user with your tenant username and c_pass with your tenant password. Leave the c_device_source as is for now.
For this particular demo, a phone or a phone-like device needs to be used, so that the measurement data for that particular device can be captured and be used for recognizing activities.
Therefore, the documentation has been split up into two parts:
This section deals with the basic data science steps of creating an activity recognition model with self-collected data. First of all, you need to register your smartphone. Then follow the sections below for collecting data, training the model and using the model to recognize activities via the phone.
Info
The phone used for the entire workflow must be of the same type because the data and sensors may vary for different devices.
Register a smartphone in the platform
Follow the steps described in Cumulocity IoT Sensor App and register a smartphone in Cumulocity IoT.
Info
Set “1 sec” as INTERVAL (secs) for Acceleration and Gyroscope sensors in the Cumulocity IoT Sensor App.
Once registered, you can get the device ID by looking up your device on the All Devices page of your tenant’s Device Management application.
Data collection
Follow the steps described in Machine Learning Workbench > Upload a project and upload the ActivityRecognitionDemoProject.zip project to MLW. A new project is created with the name ActivityRecognitionDemoProject_{UUID}, where UUID is a system generated unique identifier. This project will have a total of 5 resources. You will get 2 files in the Data folder and 3 files in the Code folder.
Select the CONFIG.json in the Data folder and click the edit icon to edit the CONFIG.json.
Update the values of c_url, c_user and c_pass with your tenant credentials and click the save icon at the right of the top menu bar.
Recording training data for activity recognition is done by starting the Cumulocity IoT Sensor App, performing each of the activities over a few minutes, and noting the exact time.
The format of the JSON data might have changed in the meantime, or some sensors might not be available for some phone types, so check the exact format by viewing a current sample.
The following code block contains the data format of the JSON schema that was assumed for this demo.
Follow the steps described in Machine Learning Workbench > Data pull > Cumulocity IoT and pull the measurements of newly registered smartphone with “walkData.csv” as File name, data interval (i.e. interval during which the data was created), “None” as Aggregation and select “c8y_Acceleration” as Data points.
This file can be previewed to verify the downloaded data and can be used for model building exercise.
To collect the sitting data:
Follow the steps described in Machine Learning Workbench > Data pull > Cumulocity IoT and pull the measurements of the newly registered smartphone with “sittingData.csv” as File name, data interval (i.e. interval during which the data was created), “None” as Aggregation and select “c8y_Acceleration” as Data points.
This file can be previewed to verify the downloaded data and can be used for model building exercise.
To collect the jumping data:
Follow the steps described in Machine Learning Workbench > Data pull > Cumulocity IoT and pull the measurements of the newly registered smartphone with “jumpData.csv” as File name, data interval (i.e. interval during which the data was created), “None” as Aggregation and select “c8y_Acceleration” as Data points.
Once data is downloaded, this file can be previewed to verify the downloaded data and can be used for model building exercise.
The following steps illustrate the next steps to merge all these file and create a concatenated data with labels which can be used for the machine learning model building exercise using the Jupyter Notebook.
Select the data resource activityData.csv in the Data folder, and click the add icon at the right of the top menu bar to proceed with training the AutoML model on that data.
Select Classification as problem type at the top left, select Target Variable at the right for “label”, clear USE FOR MODEL for “time” and click BUILD.
In the Training Parameter section at the right, select the training parameters which include model evaluation criteria (Scoring), training iterations (Generation) and population size for each generation (Population Size) and click the submit icon .
This will create a new task in the Tasks section.
Click Tasks in the navigator and click the corresponding task name to display the status of the model training in the Task History section at the center.
Once the task is completed, all the trained models are listed along with the model evaluation score in descending order.
The hyper-parameters for each model can be viewed by clicking on the corresponding model name.
After the training is complete, the best model selected by the evaluation criteria will be saved in the Model folder of the respective Project in PMML format.
Deploy the model to the platform
Once the model is available in the Model folder, it can be deployed on Machine Learning Engine (MLE) for predictions.
Select the model from the Model folder and click the cloud icon (“Deploy”) at the right of the top menu bar to deploy the selected model on Machine Learning Engine (MLE).
Once the model is successfully deployed, the cloud icon will change to “Deployed”.
Create and upload Apama monitor file
For this active recognition scenario, we need to use Apama streaming analytics. With Apama streaming analytics, you can add your own logic to your IoT solution for immediate processing of incoming data from devices or other data sources. This user-defined logic can, e.g. alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices.
We create an EPL-based monitor file and upload it to Cumulocity IoT. As mentioned earlier, the Apama EPL monitor file takes care of reading the measurements coming from the mobile device, sending it to the Zementis microservice and raising an alarm when any change in activity is reported by our machine learning model.
Instead of creating a new monitor file, you can use the attached RecognizeActivitiesSmartPhone.mon file after making minor adjustments. Open RecognizeActivities.mon in a text editor and replace the deviceId variable with the ID of your registered device, same as c_device_source in the CONFIG.json file mentioned above. Save your changes and upload this monitor file to your tenant. See Deploying EPL apps as single *.mon files with the Streaming Analytics application in the Streaming Analytics guide for details on uploading Apama monitor files.
using com.apama.correlator.Component;
using com.apama.cumulocity.Alarm;
using com.apama.cumulocity.CumulocityRequestInterface;
using com.apama.cumulocity.Measurement;
using com.apama.cumulocity.FindAlarm;
using com.apama.cumulocity.FindAlarmResponse;
using com.apama.cumulocity.FindAlarmResponseAck;
using com.apama.cumulocity.FindManagedObjectResponse;
using com.apama.cumulocity.FindManagedObjectResponseAck;
using com.apama.cumulocity.FindManagedObject;
using com.softwareag.connectivity.httpclient.HttpOptions;
using com.softwareag.connectivity.httpclient.HttpTransport;
using com.softwareag.connectivity.httpclient.Request;
using com.softwareag.connectivity.httpclient.Response;
using com.apama.json.JSONPlugin;
monitor RecognizeActivities {
// Replace this value with your device id.
string deviceId := "";
// Model to be used for recognizing activities
string modelName := "activityRecognitionModel";
//counter to exclude first five readings
integer counter := 0;
// counter to include first five detections for the same activity
integer similarActivityCount := 0;
// threshold value for any activity to be predicted consecutively
integer thresholdValueForDeterminingActivity := 5;
//initializing activity tracker flags to 'idle state'
string lastActivity := "idle state";
string referenceActivity := "idle state";
CumulocityRequestInterface cumulocity;
constant string ALARM_TYPE := "ActivityRecognitionAlarm";
action onload() {
cumulocity := CumulocityRequestInterface.connectToCumulocity();
listenAndActOnMeasurements();
}
action listenAndActOnMeasurements() {
monitor.subscribe(Measurement.SUBSCRIBE_CHANNEL);
monitor.subscribe(FindAlarmResponse.SUBSCRIBE_CHANNEL);
on all Measurement(source = deviceId) as m {
if(m.measurements.hasKey("c8y_Acceleration")){
log "Received Measurement from C8Y.";
//Gather the data
string record := convertMeasurementToRecord(m);
log "Sending record to zementis - "+ record;
Request zementisRequest := cumulocity.createRequest("GET", "/service/zementis/apply/"+modelName, any());
zementisRequest.setQueryParameter("record", record);
zementisRequest.execute(responseHandler);
log "EPL execution completed.";
}
}
}
action convertMeasurementToRecord(Measurement m) returns string
{
dictionary<string, any> json := {};
json["c8y_Acceleration_accelerationX"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationX").value;
json["c8y_Acceleration_accelerationY"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationY").value;
json["c8y_Acceleration_accelerationZ"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationZ").value;
return JSONPlugin.toJSON(json);
}
action responseHandler(Response apiResponse) {
integer statusCode := apiResponse.statusCode;
log "Zementis responded with status -" + statusCode.toString();
// Ignore first 5 records and then starting checking the outputs for 200 responses.
// First 5 records need to be cached by Zementis Server. Hence send it out but ignore the incoming response.
if(counter >= 5 and statusCode = 200) {
string currentActivity := apiResponse.payload.getSequence("outputs")[0].getEntry("predicted_activity").valueToString();
log "Last activity was : " + lastActivity + ", Current activity is : " + currentActivity + ", Reference activity is : "+ referenceActivity;
if (currentActivity = referenceActivity) {
similarActivityCount := similarActivityCount + 1;
log "Similarity count is : "+ similarActivityCount.toString();
}
else {
referenceActivity := currentActivity;
similarActivityCount := 1;
log "Similarity count is : "+ similarActivityCount.toString();
}
// Hold on till you get 5 occurences of the same activity and for 6th activity onwards keep ignoring if its the same activity.
if (similarActivityCount = thresholdValueForDeterminingActivity and lastActivity != referenceActivity){
string alarmMessage := "User switched activity from '" + lastActivity + "' to '" + referenceActivity + "'.";
clearOldAlarmAndSendNewAlarm(alarmMessage);
// Overwrite the last activity with the last activity so that it doesn't generate duplicate alarms for the same activity.
lastActivity := referenceActivity;
}
}
counter := counter + 1;
}
action createNewAlarm(string alarmMessage) {
send Alarm("", ALARM_TYPE, deviceId, currentTime,
alarmMessage, "ACTIVE", "CRITICAL", 1, new dictionary<string,any>) to Alarm.SEND_CHANNEL;
log "Alarm added as - "+alarmMessage;
}
action clearOldAlarmAndSendNewAlarm(string alarmMessage) {
integer reqId:= integer.getUnique();
send FindAlarm(reqId, {"source": deviceId, "status": "ACTIVE", "type": ALARM_TYPE}) to FindAlarm.SEND_CHANNEL;
//only if old alarm is found, clear it
on FindAlarmResponse(reqId=reqId) as alarmResponse and not FindAlarmResponseAck(reqId=reqId) {
send Alarm(alarmResponse.id, ALARM_TYPE, deviceId, currentTime, alarmResponse.alarm.text,
"CLEARED", alarmResponse.alarm.severity, 1, new dictionary<string, any>) to Alarm.SEND_CHANNEL;
log "Old Alarm cleared: " + alarmResponse.alarm.text;
}
reqId:= integer.getUnique();
// now create a new alarm
send FindAlarm(reqId, {"source": deviceId, "status": "ACTIVE", "type": ALARM_TYPE}) to FindAlarm.SEND_CHANNEL;
on FindAlarmResponseAck(reqId=reqId){
//Now create new alarm
createNewAlarm(alarmMessage);
}
}
}
Classify activities
Now that you have all the pieces together, you can try to recognize change in activity patterns with your phone. You could sit down, start jumping or running along with your mobile phone.
You should be able to see alarms being generated from your device which will be visible under the Alarms tab of your device in the Device Management application.
Activity recognition using a demo device
A fully functional activity recognition demo can be prepared with the help of a demo device.
For this, use the artifacts provided as part of the ActivityRecognitionDemo.zip file.
Start with Machine Learning Workbench
Follow the steps described in Machine Learning Workbench > Upload a project and upload the ActivityRecognitionDemoProject.zip project to MLW. A new project is created with the name ActivityRecognitionDemoProject_UUID, where UUID is a system generated unique identifier. This project has a total of 5 resources. You will get 2 files in the Data folder and 3 files in the Code folder.
Register a demo device in the platform
Instead of registering an actual device for the activity recognition use case, a demo device can be registered. This device can be used as a replica of an actual to do human activity. Follow the below steps. We have added a script RegisterDevice.ipynb which registers a demo device in Cumulocity IoT.
Open it and click the edit icon .
Execute each cell one by one and you will have a registered device in Cumulocity IoT . Upon successful execution, a device named “DemoDevice” is registered in Cumulocity IoT .
Once registered, you can get the device ID by looking up your device on the All Devices page of your tenant’s Device Management application. The device ID is already updated by the code and is saved in the CONFIG.json file.
Upon successful execution, a device named DemoDevice is registered in Cumulocity IoT. Once registered, try to get the device ID by looking up your device on the All Devices page of your tenant’s Device Management application. The CONFIG.json file is already updated by the code with the device ID.
This device is capable of simulating readings of water flow to Cumulocity IoT in the form of a measurement named c8y_SignalStrengthWifi, c8y_Acceleration, c8y_Barometer, c8y_Gyroscope, c8y_Luxometer, c8y_Compass.
Upload the model and Apama monitor file
Upload the attached model ActivitiesDTreeJump.pmml to Cumulocity IoT. To upload the model to Cumulocity IoT, follow the steps described in Machine Learning application > Managing models.
Download the RecognizeActivitiesDemoDevice.mon file. Open it in a text editor and replace the deviceId variable with the ID of your registered device, same as c_device_source in the CONFIG.json file mentioned above.
Classify activities by simulating measurements for the demo device
Use SimulateDataforRealtimescoring.ipynb for simulating the measurements for the demo device.
Open it and click the edit icon .
Execute each cell one by one and you will have a registered device in Cumulocity IoT. Upon successful execution, c8y_Acceleration measurements are sent to Cumulocity IoT on behalf of the demo device. The measurements are simulated dynamically for the current date in gaps of 1 second.
This should now start sending measurements to Cumulocity IoT on behalf of your demo device. It would try to simulate transition of activities in the order sit → jump → run → jump → sit.
You should notice your device generating activity recognition alarms for every transition of activity. These alarms generated from your device will be visible under the Alarms tab of your device in the Device Management application. Click Show Cleared Alarms in the top bar to see the history of all the alarms.
to edit the CONFIG.json.
at the right of the top menu bar.
at the right of the top menu bar to proceed with training the AutoML model on that data.
.
(“Deploy”) at the right of the top menu bar to deploy the selected model on Machine Learning Engine (MLE).
“Deployed”.
