Notifications 2.0

Overview

The Notifications 2.0 API allows applications or microservices to receive and process notifications generated by the use of the Cumulocity IoT REST or MQTT APIs (device management, measurements, alarms, events and other platform APIs) in a reliable manner.

Notifications 2.0 improves upon the Real-time notification API by providing stronger delivery semantics and ordering guarantees. It is also intended to be simpler to use than the “Bayeaux” protocol used in the Real-time notification API. New capabilities are added to Notifications 2.0 in each release of Cumulocity IoT. However, it does not yet support all of the notifications available from the Real-time notification API so it is not yet a complete replacement for the older API. See the rest of this section and the detailed API documentation for full details of the notifications supported by this release of the Notifications 2.0 API.

To receive notifications over the Notifications 2.0 protocol, an application or microservice must subscribe to notifications, either for notifications about a particular managed object or in a wider context that is scoped to a tenant. Subscriptions are persistent, long lived, and allow notifications to be stored reliably until consumed and acknowledged by a consuming microservice or application.

Managed object context

For managed objects, an object’s global identifier must be used to subscribe in the managed object context (“mo”) in order to receive notifications. There can be multiple subscriptions for the same subscribed object, with different filters or just to fan out to multiple interested consumer parties.

This subscribed object is known as the source object for the notification/event and it is referenced in the notifications delivered to subscribers. Subscriptions are set up using the subscription method of the Notifications 2.0 API. This API requires the calling user to be an authenticated Cumulocity IoT user and to have the new ROLE_NOTIFICATION_2_ADMIN role.

When subscribing to notifications, a filter for notifications can be specified which determines the APIs (alarms, alarms with children, events, events with children, measurements, managed objects, operations, or any combination of these) to filter by. The alarms with children and events with children enable users to create explicit subscriptions that allow the delivery of child as well as parent managed object events and alarms. It is also possible to filter by presence of a specific JSON fragment or “fragment type”. When matched, either the whole notification content is forwarded, or one or more fragments can be specified to be copied over to the consumer. For usage, refer to the Cumulocity IoT OpenAPI Specification.

Receiving subscribed notifications

In order to receive subscribed notifications, a consumer application or microservice must obtain an authorization token that provides proof that the holder is allowed to receive subscribed notifications.

This token is in the form of a string conforming to the JWT (JSON Web Token) standard that is obtained from the token method of the Notifications 2.0 API. This API requires the calling user to be an authenticated Cumulocity IoT user and to have the new ROLE_NOTIFICATION_2_ADMIN role.

See the Cumulocity IoT OpenAPI Specification for both the Notification 2.0 Subscription and Token API.

Once subscribed, notifications are persisted and available to be consumed using a new WebSocket-based protocol. This protocol implements a reliable delivery with at-least-once semantics. The underlying Messaging Service will repeatedly attempt to deliver a notification until that notification is acknowledged as being received and processed by the consuming application or microservice. Notification order is preserved from the point of view of a device sending in REST and MQTT API requests. The protocol is text-based and described in detail in the next section.

Tenant context

The tenant context (“tenant”) is used for subscribing to and receiving notifications, in addition to the managed object context (“mo”) mentioned above. Creations of managed objects, which generate a new object identifier that can act as a source for notifications are reported in the tenant context. This allows an application to discover a new managed object, which can then choose to subscribe to in the managed object context. It is also possible to subscribe to all alarms that are generated in the tenant context.

See the Cumulocity IoT OpenAPI Specification on how to subscribe to these notifications, additionally filtering the notification of interest.

For the protocol consumer, both managed object creations and alarms subscribed under the tenant context are reported in the same way. There is no distinction between the two contexts for consumers, and notification ordering is maintained between the two contexts.

Building consuming microservices and applications

For Java developers, the API and the protocol have been wrapped up as an open Java API and a sample WebSocket client application. Any WebSocket library or programming language can be used as the protocol is text-based and relatively simple. Consumer can be either microservices or applications running externally from Cumulocity IoT and require only a JWT string when connecting.

There is a sample microservice available in the cumulocity-examples repository, so Java developers do not need to code to the protocol specification directly.

Token expiration

When creating a token, an expiration time must be given in minutes of validity from when the token was created. This security feature limits the potential damage due to leaking of a token. It requires tokens to be re-created or refreshed periodically. This can be done by calling the token create request with the same parameters as originally. If the parameters used are not available they can be extracted from the token. As the token string is a JWT (JSON Web Token), it can be decoded to extract the original information used to create the token by splitting it into 3 parts (on “.") and doing a base64 decode on the first substring. This way, information like the subscription name can be extracted and the create token REST point can be called again, all on the client side. The Cumulocity IoT microservice Java SDK TokenApi class contains a public refresh method which is implemented purely on the client side.

Deleting subscriptions and unsubscribing a subscriber

Once a subscription is made, notifications will be retained until consumed by all subscribers who have previously connected to the subscription. The normal workflow is to delete subscriptions when no longer interested in notifications and this is the resonsibility of the subscriber. The subscription API Cumulocity IoT OpenAPI Specification is used to delete subscriptions. After the subscription is deleted no more notifications will be saved. The consuming microservice or application can then drain down notifications and be removed when that is done.

However, unconsumed notifications will be retained, and for high throughput scenarios this can result in notifications remaining in storage if never consumed by the application. They will be deleted if a tenant is deleted but otherwise can take up considerable space in permanent storage for high frequency notification sources. It is therefore advisable to unsubscribe a subscriber that will never run again. A separate REST endpoint is available for this: /notification2/unsubscribe. It has a mandatory query parameter token. The token is the same as you would use to connect to the WebSocket endpoint to consume notifications. Note that there is no explicit “subscribe a subscriber” operation using a token. Instead this happens when you first connect a WebSocket with a token for the subscription name and subscriber. However, unsubscribing a microservice or an application is an explicit act using the original or a similar token. Unsubscribing should be infrequent, for example when deleting an application or during development when testing completes, as typically one wants messages to persist even when no consumer is running. Only if no consumer will ever run again to drain notifications should unsubscribing a subscriber be necessary.

It is also possible to unsubscribe a subscriber on an open consumer WebSocket connection. To do so, send unsubscribe_subscriber instead of a message acknowledgement identifier from your WebSocket client to the service. The service will then unsubscribe the subscriber and close the connection. It’s not possible to check if the unsubscribe operation succeeded as the connection always closes so this way of unsubscribing is mostly for testing.

It is always important to delete subscriptions (Delete operations on /notification2/subscriptions) even having unsubscribed, as otherwise notifications will be generated even if no subscriptions remain. While they would not persist, load and network traffic would still be incurred.

Consumer protocol

The Cumulocity IoT Notifications 2.0 API uses a secure WebSocket to consume notifications generated by the Cumulocity IoT API.

The new endpoint is accessible using the external Cumulocity IoT fully qualified domain name of your Cumulocity IoT environment and the standard SSL port 443 using a secure WebSocket connection. It is also available on the unsecured port 80 and to microservices using “cumulocity:8111” but in most cases a secure connection is preferred.

The URI scheme therefore is “wss” and consumers use URLs starting with “wss://” followed by the fully qualified domain name of the Cumulocity IoT environment or tenant, followed by a fixed URL path and a query string.

The fixed URL path is /notification2/consumer/ and there are only two query string arguments:

• token (required). Its value must be a valid token in the form of a JWT token string as returned by a create token request to the Tokens methods of the Notifications 2.0 API. Including the token as a query string parameter avoids having to set an HTTP header which can be an issue for some WebSocket clients or proxies.

• consumer (optional). Its value is a non blank unique name for the consumer.

In summary, the URLs used by consumers follow the following patterns:

wss://your.cumulocity.environment.fullqualifieddomainname/notification2/consumer/?token=yourJwtTokenRequestedFromNotification2TokenService

or

wss://your.cumulocity.environment.fullqualifieddomainname/notification2/consumer/?token=yourJwtTokenRequestedFromNotification2TokenService&consumer=aUniqueNameForThisConsumer

WebSocket timeouts

There is a timeout of 5 minutes set on idle WebSocket connections after which the connection will be closed by the server side. Therefore the consumer must be prepared to handle closed connections which is required for fault tolerant operation in any case. All consuming microservices or applications should handle the WebSocket being closed and re-connect as necessary.

Notification acknowledgements

The WebSocket established with such an URL is a textual bi-directional connection using UTF-8 encoding. The WebSocket service sends a sequence of notifications to the consumer and the consumer should send back a short acknowledgement over this connection for each notification received. This acknowledgement is for a particular notification and the server will periodically resend a notification until it is acknowledged. This will cease once the server has received and processed an acknowledgement for a particular notification.

Notification message header and content

A notification (transmitted service to client) consists of a header and a body (similar to an HTTP request). The header is one or more (in practice at least 3) lines of text, separated by a \n (newline) character. The end of the header is demarcated by a double new line \n\n.

The notification body follows the header. This also consists of UTF text - for example a JSON document. If the notification is binary data or includes binary data then it will be Base64 encoded.

The header lines for a notification are as follows (separated by \n newlines):

• Required message identifier for message acknowledgement. This opaque value is an encoded binary 64 bit value. It must be returned as part of the acknowledgement for the notification as described below.

• The notification description on the second header line. This is a string describing what type of notification this is and its source. Measurements (“measurements”), events (“events”) and alarms (“alarms”) are examples of notifications, as are inventory creates, updates and deletes (“managedObjects”). There is a direct correspondence with realtime notifications which features similar notification descriptions. These are not enumerated here and are expected to increase in number in the future. For REST API notifications, they follow a 3 part format, separated by “/”. More details on notification descriptions is given below.

• An action string is the third header. Examples are CREATE, UPDATE and DELETE. More actions may be added in the future. Together with the notification type they describe the logical event that generated the notification, such as a CREATE of an alarm or measurement.

Depending on the second header there may be further headers to follow but currently notifications only use the above three. In order to be future proof and forward compatible, we encourage consumer code to cope with more headers by parsing them out but ignoring them.

See the hello-world-notification-microservice example in the cumulocity-examples repository on how to do this.

After the headers, the notification body follows as UTF-8 text. Typically a JSON document is carried in this text.

Notification description header

The second header line is the notification description string in the form of a /-separated path. For API notifications descriptions have three parts: tenantId, type and sourceId.

• tenantId - this the identifier for the tenant under which the notification was generated.

• type - the platform type of notification generated. For example, event, measurement, alarm or managed object.

• sourceId - the identifier of the “source” object that generated or is the subject of the notification. Source is a very loose term here, much as in “event sourcing” but generally indicates which managed object the notification is about.

Some examples are provided in Traces and backwards compatibility to real-time notifications is provided for.

Also see the rest of the documentation, examples and experiment to get values for events that you are interested in.

Processing notification acknowledgements

The first header line in each notification consists of an opaque, encoded binary identifier that must be returned as is in a reply to the Notification 2.0 service in a message acknowledgement.

See the hello-world-notification-microservice example in the cumulocity-examples repository on how to do this. It involves sending the identifier back to the service in a self-contained WebSocket text message, that means, send back the first header without the trailing \n to the server.

Dealing with notification duplication

Until a notification is acknowledged, the WebSocket service will attempt to re-deliver it. It is therefore desirable that a notification is acknowledged as soon as possible (to help avoid duplicates). However, this should not be done until the notification has been successfully processed to make use of the at-least-once semantics that the Notification 2.0 service provides.

Simply process the message and only return the acknowledgement identifier when that processing completes successfully. If processing is longer than a minute or so, the service will resend the notification so the WebSocket client application must be prepared to deal with duplicates. Duplicates can also occur due to underlying network failures, consumer crashes or perceived failures (slow transmissions) and subsequent failure masking re-transmission attempts.

A duplicate can be delivered out of order if several notifications are unacknowledged but only after follow-on notifications so should be easy to deal with.

For example, in the logical sequence 1,2,3,4, the notification number 2 can be duplicated after 3 or even 4 as in 1,2,3,2,4 or even several times, as in 1,2,3,4,2,3,2.

The notifications don’t contain any unique identifier or timestamps to aid in de-duplication. Some events are easy to de-duplicate, such as inventory events where a unique source object is first CREATED and then DELETED. But inventory UPDATES or logically sequenced events such as alarms and measurements require application-specific sequencing.

This can be achieved by including unique identifiers, sequence numbers or timestamps in the notification JSON (body) as required. An alternative is to look up the current value in the Cumulocity IoT database, treating the notification as a signal only and ignoring the value carried.

As can be seen from the notification Traces, some notifications do carry timestamps. If the timestamps are not generated by the device client, then they may only be loosely synchronized between notifications and therefore should be used carefully or not at all for de-duplication.

Traces

The following is a sample of a trace of messages without showing the first header line which consists of the acknowledgement (encoded string that must be returned to acknowledge the notification).

------------------------
header /tenant-a170/managedobjects/111
header CREATE
notification {"additionParents":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/additionParents"},"owner":"admin","childDevices":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/childDevices"},"childAssets":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/childAssets"},"creationTime":"2021-09-03T12:28:58.692Z","lastUpdated":"2021-09-03T12:28:58.692Z","childAdditions":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/childAdditions"},"name":"a switch","assetParents":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/assetParents"},"deviceParents":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/deviceParents"},"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111","com_cumulocity_model_BinarySwitch":{"state":"ON"}}
------------------------
header /tenant-a170/measurements/111
header CREATE
notification {"self":"http://cumulocity.default.svc.cluster.local/measurement/measurements/117","time":"2021-09-03T12:29:01.664Z","id":"117","source":{"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111"},"type":"c8y_TenantUsageStatisticsMeasurement","resourcesUsage":{"memory":0,"cpu":0,"usedBy":[]}}
------------------------
header /tenant-a170/events/111
header CREATE
notification {"creationTime":"2021-09-03T12:29:01.932Z","source":{"name":"a switch","self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111"},"type":"com_cumulocity_model_DoorSensorEvent","self":"http://cumulocity.default.svc.cluster.local/event/events/118","time":"2021-09-03T12:29:01.664Z","id":"118","text":"Door sensor was triggered"}
------------------------
header /tenant-a170/eventsWithChildren/111
header CREATE
notification {"creationTime":"2021-09-03T12:29:01.932Z","source":{"name":"a switch","self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111"},"type":"com_cumulocity_model_DoorSensorEvent","self":"http://cumulocity.default.svc.cluster.local/event/events/118","time":"2021-09-03T12:29:01.664Z","id":"118","text":"Door sensor was triggered"}
------------------------
header /tenant-a170/managedobjects/111
header UPDATE
notification {"additionParents":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/additionParents"},"owner":"admin","childDevices":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/childDevices"},"childAssets":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/childAssets"},"creationTime":"2021-09-03T12:28:58.692Z","lastUpdated":"2021-09-03T12:28:58.692Z","childAdditions":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/childAdditions"},"name":"a switch","assetParents":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/assetParents"},"deviceParents":{"references":[],"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111/deviceParents"},"self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111","c8y_ActiveAlarmsStatus":{"major":1},"com_cumulocity_model_BinarySwitch":{"state":"ON"}}
------------------------
header /tenant-a170/alarms/111
header CREATE
notification {"severity":"MAJOR","creationTime":"2021-09-03T12:29:02.092Z","count":1,"history":{"auditRecords":[],"self":"http://cumulocity.default.svc.cluster.local/audit/auditRecords"},"source":{"name":"a switch","self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111"},"type":"com_cumulocity_events_TamperEvent","self":"http://cumulocity.default.svc.cluster.local/alarm/alarms/119","time":"2021-09-03T12:29:01.664Z","id":"119","text":"Tamper sensor triggered","status":"ACTIVE","com_mycorp_MyProp":{"key1":"value1"}}
------------------------
header /tenant-a170/alarms/111
header CREATE
notification {"severity":"MAJOR","creationTime":"2021-09-03T12:29:02.092Z","count":1,"history":{"auditRecords":[],"self":"http://cumulocity.default.svc.cluster.local/audit/auditRecords"},"source":{"name":"a switch","self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111"},"type":"com_cumulocity_events_TamperEvent","self":"http://cumulocity.default.svc.cluster.local/alarm/alarms/119","time":"2021-09-03T12:29:01.664Z","id":"119","text":"Tamper sensor triggered","status":"ACTIVE","com_mycorp_MyProp":{"key1":"value1"}}
------------------------
header /tenant-a170/alarmsWithChildren/111
header CREATE
notification {"severity":"MAJOR","creationTime":"2021-09-03T12:29:02.092Z","count":1,"history":{"auditRecords":[],"self":"http://cumulocity.default.svc.cluster.local/audit/auditRecords"},"source":{"name":"a switch","self":"http://cumulocity.default.svc.cluster.local/inventory/managedObjects/111","id":"111"},"type":"com_cumulocity_events_TamperEvent","self":"http://cumulocity.default.svc.cluster.local/alarm/alarms/119","time":"2021-09-03T12:29:01.664Z","id":"119","text":"Tamper sensor triggered","status":"ACTIVE","com_mycorp_MyProp":{"key1":"value1"}}