Use Case: Activity Recognition

Overview

Sensor-based activity recognition can be used to model a wide range of human activities. Mobile devices, such as smart phones provide sufficient sensor data and calculation power to enable physical activity recognition to provide an estimation of the energy consumption during everyday life.

Sensor-based activity recognition researchers believe that by empowering ubiquitous computers and sensors to monitor the behavior of agents (under consent), these computers will be better suited to act on our behalf (see Wikipedia).

In this use case, we would like to showcase the recognition of the human activities - sitting, jumping and walking. For the purpose of showcasing the same, we followed these steps:

Prerequisites

Download the ActivityRecognitionDemo.zip file which contains the project ZIP named ActivityRecognitionDemoProject.zip and the EPL rules.

Running the demo scripts requires

Getting Started

Upload the project ZIP file to Cumulocity IoT Machine Learning Workbench. We have added a CONFIG.json file to the project. This file is meant to capture the tenant details and credentials which will be used by the demo scripts.

First, update the CONFIG.json with the appropriate values and save it. Replace c_url with your tenant URL, c_user with your tenant username and c_pass with your tenant password. Leave the c_device_source as is for now.

CONFIG.json
{
	c_url:https://yourtenant.cumulocity.com
	c_user:user@company.com
	c_pass:password
	c_device_source:deviceID
}

For this particular demo, a phone or a phone-like device needs to be used, so that the measurement data for that particular device can be captured and be used for recognizing activities.

Therefore, the documentation has been split up into two parts:

Activity recognition using a smartphone

This section deals with the basic data science steps of creating an activity recognition model with self-collected data. First of all, you need to register your smartphone. Then follow the sections below for collecting data, training the model and using the model to recognize activities via the phone.

Info
The phone used for the entire workflow must be of the same type because the data and sensors may vary for different devices.

Register a smartphone in the platform

Follow the steps described in Cumulocity IoT Sensor App and register a smartphone in Cumulocity IoT.

Info
Set “1 sec” as INTERVAL (secs) for Acceleration and Gyroscope sensors in the Cumulocity IoT Sensor App.

Once registered, you can get the device ID by looking up your device on the All Devices page of your tenant’s Device management application.

Data collection

  1. Follow the steps described in Machine Learning Workbench > Upload a project and upload the ActivityRecognitionDemoProject.zip project to MLW. A new project is created with the name ActivityRecognitionDemoProject_{UUID}, where UUID is a system generated unique identifier. This project will have a total of 5 resources. You will get 2 files in the Data folder and 3 files in the Code folder.

  2. Select the CONFIG.json in the Data folder and click the edit icon Edit to edit the CONFIG.json.

  3. Update the values of c_url, c_user and c_pass with your tenant credentials and click the save icon Save at the right of the top menu bar.

Recording training data for activity recognition is done by starting the Cumulocity IoT Sensor App, performing each of the activities over a few minutes, and noting the exact time.

The format of the JSON data might have changed in the meantime, or some sensors might not be available for some phone types, so check the exact format by viewing a current sample.

The following code block contains the data format of the JSON schema that was assumed for this demo.

measurement.json

{
	"_id" : ObjectId("5c928ba9b524aca326d727fe"),
	"self" : "https://zdev.cumulocity.com/measurement/measurements/7668082",
	"time" : "2019-03-18T14:46:15.000-07:00",
	"id" : "7668082",
	"source" : {
		"self" : "https://zdev.cumulocity.com/inventory/managedObjects/7668076",
		"id" : "7668076"
	},
	"type" : "c8y_Acceleration",
	"c8y_Acceleration" : {
		"accelerationY" : {
			"unit" : "G",
			"value" : -0.2860107421875
		},
		"accelerationX" : {
			"unit" : "G",
			"value" : -0.1037750244140625
		},
		"accelerationZ" : {
			"unit" : "G",
			"value" : -0.9522247314453125
		}
	},
	"activity" : "none"
}

To collect the walking data:

To collect the sitting data:

To collect the jumping data:

The following steps illustrate the next steps to merge all these file and create a concatenated data with labels which can be used for the machine learning model building exercise using the Jupyter Notebook.

Follow the steps described in Machine Learning Workbench > Jupyter Notebook > Editing and executing a notebook and execute the existing code snippets in each cell of the MergeData.ipynb. This will perform the following actions:

Download

Train the PMML model

To train the model we will use the AutoML feature of Cumulocity IoT Machine Learning Workbench.

  1. Follow the steps described in Machine Learning Workbench > Projects > Resources.

  2. Select the data resource activityData.csv in the Data folder, and click the add icon Add at the right of the top menu bar to proceed with training the AutoML model on that data.

  3. Select Classification as problem type at the top left, select Target Variable at the right for “label”, clear USE FOR MODEL for “time” and click BUILD.

Download

  1. In the Training Parameter section at the right, select the training parameters which include model evaluation criteria (Scoring), training iterations (Generation) and population size for each generation (Population Size) and click the submit icon Submit.

Download

This will create a new task in the Tasks section.

Click Tasks in the navigator and click the corresponding task name to display the status of the model training in the Task History section at the center.

Once the task is completed, all the trained models are listed along with the model evaluation score in descending order.

The hyper-parameters for each model can be viewed by clicking on the corresponding model name.

After the training is complete, the best model selected by the evaluation criteria will be saved in the Model folder of the respective Project in PMML format.

Deploy the model to the platform

Once the model is available in the Model folder, it can be deployed on Machine Learning Engine (MLE) for predictions.

Select the model from the Model folder and click the cloud icon Deploy (“Deploy”) at the right of the top menu bar to deploy the selected model on Machine Learning Engine (MLE).

Once the model is successfully deployed, the cloud icon will change to Deployed “Deployed”.

Create and upload Apama monitor file

For this active recognition scenario, we need to use Apama streaming analytics. With Apama streaming analytics, you can add your own logic to your IoT solution for immediate processing of incoming data from devices or other data sources. This user-defined logic can, e.g. alert applications of new incoming data, create new operations based on the received data (such as sending an alarm when a threshold for a sensor is exceeded), or trigger operations on devices.

We create an EPL-based monitor file and upload it to Cumulocity IoT. As mentioned earlier, the Apama EPL monitor file takes care of reading the measurements coming from the mobile device, sending it to the Zementis microservice and raising an alarm when any change in activity is reported by our machine learning model.

Instead of creating a new monitor file, you can use the attached RecognizeActivitiesSmartPhone.mon file after making minor adjustments. Open RecognizeActivities.mon in a text editor and replace the deviceId variable with the ID of your registered device, same as c_device_source in the CONFIG.json file mentioned above. Save your changes and upload this monitor file to your tenant. See EPL Apps > Basic functionality > Deploying EPL apps as single *.mon files with the Streaming Analytics application in the Streaming Analytics guide for details on uploading Apama monitor files.

using com.apama.correlator.Component;
using com.apama.cumulocity.Alarm;
using com.apama.cumulocity.CumulocityRequestInterface;
using com.apama.cumulocity.Measurement;
using com.apama.cumulocity.FindAlarm;
using com.apama.cumulocity.FindAlarmResponse;
using com.apama.cumulocity.FindAlarmResponseAck;
using com.apama.cumulocity.FindManagedObjectResponse;
using com.apama.cumulocity.FindManagedObjectResponseAck;
using com.apama.cumulocity.FindManagedObject;
using com.softwareag.connectivity.httpclient.HttpOptions;
using com.softwareag.connectivity.httpclient.HttpTransport;
using com.softwareag.connectivity.httpclient.Request;
using com.softwareag.connectivity.httpclient.Response;
using com.apama.json.JSONPlugin;

monitor RecognizeActivities {

    // Replace this value with your device id.
    string deviceId := "";

    // Model to be used for recognizing activities
    string modelName := "activityRecognitionModel";

    //counter to exclude first five readings
    integer counter := 0;

    // counter to include first five detections for the same activity
    integer similarActivityCount := 0;

	// threshold value for any activity to be predicted consecutively
	integer thresholdValueForDeterminingActivity := 5;

	//initializing activity tracker flags to 'idle state'
	string lastActivity := "idle state";
	string referenceActivity := "idle state";

	CumulocityRequestInterface cumulocity;

	constant string ALARM_TYPE := "ActivityRecognitionAlarm";

    action onload() {
    	cumulocity := CumulocityRequestInterface.connectToCumulocity();
        listenAndActOnMeasurements();
    }

    action listenAndActOnMeasurements() {
        monitor.subscribe(Measurement.SUBSCRIBE_CHANNEL);
        monitor.subscribe(FindAlarmResponse.SUBSCRIBE_CHANNEL);

        on all Measurement(source = deviceId) as m {        

            if(m.measurements.hasKey("c8y_Acceleration")){
            	log "Received Measurement from C8Y.";
            	//Gather the data
                string record := convertMeasurementToRecord(m);
                log "Sending record to zementis - "+ record;
                Request zementisRequest := cumulocity.createRequest("GET", "/service/zementis/apply/"+modelName, any());
		        zementisRequest.setQueryParameter("record", record);
		        zementisRequest.execute(responseHandler);
		        log "EPL execution completed.";
            }
        }
    }

    action convertMeasurementToRecord(Measurement m) returns string
    {
        dictionary<string, any> json := {};
       	json["c8y_Acceleration_accelerationX"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationX").value;
    	json["c8y_Acceleration_accelerationY"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationY").value;
    	json["c8y_Acceleration_accelerationZ"] := m.measurements.getOrDefault("c8y_Acceleration").getOrDefault("accelerationZ").value;
        return JSONPlugin.toJSON(json);
    }

    action responseHandler(Response apiResponse) {  	
        integer statusCode := apiResponse.statusCode;
        log "Zementis responded with status -" + statusCode.toString();

        // Ignore first 5 records and then starting checking the outputs for 200 responses.
        // First 5 records need to be cached by Zementis Server. Hence send it out but ignore the incoming response.
        if(counter >= 5 and statusCode = 200) {
	        string currentActivity := apiResponse.payload.getSequence("outputs")[0].getEntry("predicted_activity").valueToString();
	        log "Last activity was : " + lastActivity + ", Current activity is : " + currentActivity + ", Reference activity is : "+ referenceActivity;

	        if (currentActivity =  referenceActivity) {
	        	similarActivityCount := similarActivityCount + 1;
	        	log "Similarity count is : "+ similarActivityCount.toString();
	        }
	        else {
	        	referenceActivity := currentActivity;
	        	similarActivityCount := 1;
	        	log "Similarity count is : "+ similarActivityCount.toString();
	        }

	        // Hold on till you get 5 occurences of the same activity and for 6th activity onwards keep ignoring if its the same activity.
			if (similarActivityCount = thresholdValueForDeterminingActivity and lastActivity != referenceActivity){
				string alarmMessage := "User switched activity from '" + lastActivity + "' to '" + referenceActivity + "'.";

				clearOldAlarmAndSendNewAlarm(alarmMessage);

				// Overwrite the last activity with the last activity so that it doesn't generate duplicate alarms for the same activity.
				lastActivity := referenceActivity;
			}
	    }
        counter := counter + 1;
    }

    action createNewAlarm(string alarmMessage) {
    	send Alarm("", ALARM_TYPE, deviceId, currentTime,
		           alarmMessage, "ACTIVE", "CRITICAL", 1, new dictionary<string,any>) to Alarm.SEND_CHANNEL;
		log "Alarm added as - "+alarmMessage;
    }

    action clearOldAlarmAndSendNewAlarm(string alarmMessage) {
	    	integer reqId:= integer.getUnique();
	        send FindAlarm(reqId, {"source": deviceId, "status": "ACTIVE", "type": ALARM_TYPE}) to FindAlarm.SEND_CHANNEL;
	        //only if old alarm is found, clear it
	        on FindAlarmResponse(reqId=reqId) as alarmResponse and not FindAlarmResponseAck(reqId=reqId) {
	            send Alarm(alarmResponse.id, ALARM_TYPE, deviceId, currentTime, alarmResponse.alarm.text,
	                        "CLEARED", alarmResponse.alarm.severity, 1, new dictionary<string, any>) to Alarm.SEND_CHANNEL;
	            log "Old Alarm cleared: " + alarmResponse.alarm.text;
	        }
	        reqId:= integer.getUnique();
	        // now create a new alarm
	        send FindAlarm(reqId, {"source": deviceId, "status": "ACTIVE", "type": ALARM_TYPE}) to FindAlarm.SEND_CHANNEL;
	        on FindAlarmResponseAck(reqId=reqId){
	        	//Now create new alarm
	            createNewAlarm(alarmMessage);
	        }
	}
}

Classify activities

Now that you have all the pieces together, you can try to recognize change in activity patterns with your phone. You could sit down, start jumping or running along with your mobile phone.

You should be able to see alarms being generated from your device which will be visible under the Alarms tab of your device in the Device management application.

Activity recognition using a demo device

A fully functional activity recognition demo can be prepared with the help of a demo device. For this, use the artifacts provided as part of the ActivityRecognitionDemo.zip file.

Start with Machine Learning Workbench

  1. Follow the steps described in Machine Learning Workbench > Upload a project and upload the ActivityRecognitionDemoProject.zip project to MLW. A new project is created with the name ActivityRecognitionDemoProject_UUID, where UUID is a system generated unique identifier. This project has a total of 5 resources. You will get 2 files in the Data folder and 3 files in the Code folder.

Register a demo device in the platform

Instead of registering an actual device for the activity recognition use case, a demo device can be registered. This device can be used as a replica of an actual to do human activity. Follow the below steps. We have added a script RegisterDevice.ipynb which registers a demo device in Cumulocity IoT.

  1. Open it and click the edit icon Edit.
  2. Execute each cell one by one and you will have a registered device in Cumulocity IoT . Upon successful execution, a device named “DemoDevice” is registered in Cumulocity IoT .
  3. Once registered, you can get the device ID by looking up your device on the All Devices page of your tenant’s Device management application. The device ID is already updated by the code and is saved in the CONFIG.json file.

Download

Upon successful execution, a device named DemoDevice is registered in Cumulocity IoT. Once registered, try to get the device ID by looking up your device on the All Devices page of your tenant’s Device management application. The CONFIG.json file is already updated by the code with the device ID.

This device is capable of simulating readings of water flow to Cumulocity IoT in the form of a measurement named c8y_SignalStrengthWifi, c8y_Acceleration, c8y_Barometer, c8y_Gyroscope, c8y_Luxometer, c8y_Compass.

Upload the model and Apama monitor file

  1. Upload the attached model ActivitiesDTreeJump.pmml to Cumulocity IoT. To upload the model to Cumulocity IoT, follow the steps described in Machine Learning application > Managing models.

  2. Download the RecognizeActivitiesDemoDevice.mon file. Open it in a text editor and replace the deviceId variable with the ID of your registered device, same as c_device_source in the CONFIG.json file mentioned above.

  3. Save your changes and upload this monitor file to your tenant. See EPL Apps > Basic functionality > Deploying EPL apps as single *.mon files with the Streaming Analytics application in the Streaming Analytics guide for details on uploading Apama monitor files.

Classify activities by simulating measurements for the demo device

Use SimulateDataforRealtimescoring.ipynb for simulating the measurements for the demo device.

  1. Open it and click the edit icon Edit.
  2. Execute each cell one by one and you will have a registered device in Cumulocity IoT. Upon successful execution, c8y_Acceleration measurements are sent to Cumulocity IoT on behalf of the demo device. The measurements are simulated dynamically for the current date in gaps of 1 second.

Download

This should now start sending measurements to Cumulocity IoT on behalf of your demo device. It would try to simulate transition of activities in the order sit → jump → run → jump → sit.

You should notice your device generating activity recognition alarms for every transition of activity. These alarms generated from your device will be visible under the Alarms tab of your device in the Device management application. Click Show Cleared Alarms in the top bar to see the history of all the alarms.