Device integration using MQTT

Integration life cycle

The basic life cycle for integrating devices into Cumulocity is discussed in Interfacing devices.

In this section, we will show how this life cycle can be managed using the MQTT implementation.

The life cycle consists of two phases, a startup phase and a cycle phase.

The startup phase can be as short as just checking the credentials:

• Step 0: Request device credentials, if they have not been requested yet.
• Step 1: Ensure that the device exists.
• Step 2: Ensure that the device children exist.
• Step 3: Subscribe to the topics.

The cycle phase consists of two kinds of actions:

• Step A: Send CSV data
• Step B: Receive CSV operations

Startup phase

Step 0: Request device credentials

In Cumulocity, every MQTT connection must be authenticated. You can use the device credentials topics in the MQTT implementation to generate new credentials for a device.

Once the device retrieved the credentials, it needs to store them locally for further connections.

To establish a connection you must configure the following parameters:

• Host: <your_cumulocity_url>
• User: <tenantID>/<username> (user alias is not supported)
• Password: <your_cumulocity_password>

For more information, refer to the Hello MQTT section.

The process works as follows:

• Cumulocity assumes that each device has some form of unique ID. For instance, a good device identifier can be the MAC address of the network adapter, the IMEI of a mobile device or a hardware serial number.
• When you take a new device into use, you enter this unique ID into Device registration in the Device Management application in Cumulocity, and start the device.
• The device will use this ID as part of the MQTT ClientId and static user credentials that can be enquired from product support.
• The device subscribes to the topic s/dcr.
• The device starts publishing continuous empty messages on the topic s/ucr to notify the server that it is ready to retrieve credentials.
• Next, you must accept the connection from the device in the Device Registration page.
• When the device sends the next empty message it should receive credentials in the format 70,<tenantID>,<username>,<password>.

After receiving the credentials, the device can close the MQTT connection and create a new one with the received credentials.

Step 1: Verify device

As MQTT supports an automatic device creation if the client sends data and there is no device present, this step is only required if you want to create the device manually.

The device creation can be achieved by employing the static template 100. This template can be blindly used on every boot of the device as it will only create the device if it is not already present.

The device will be linked automatically to the ID the client uses with its MQTT ClientId.

100,Device Name,Device Type

Step 2: Verify children

Like the root device, also its children are covered by the automatic device creation.

To handle this step manually you can send the static template 101 for creating a child device. The template will only create the child if it does not already exist.

101,Unique Child ID,Child Name,Child Type

Step 3: Subscribe topics

If the device supports operations, it should subscribe to all required topics (static templates and SmartREST 2.0).

Cycle phase

Step A: Send CSV data

While the device holds an active MQTT connection, it can publish either on the topics for static templates or on the topics for a SmartREST template to send data to the server.

Based on the MQTT ClientId, the physical device is directly connected to the device object in Cumulocity. Therefore, the data you send is automatically connected to the device.

To send data to a child device, publish the data to the topics described in Device hierarchies.

Step B: Receive CSV operations

By subscribing to a topic the device automatically tells Cumulocity that it wants to receive operations. Any operation created will be automatically parsed using either the static templates or the templates the device defines.

MQTT implementation

This section lists the implementation details for the MQTT protocol. The Cumulocity implementation supports MQTT Version 3.1.1.

Connecting via MQTT

Cumulocity supports MQTT both via TCP and WebSockets. As URL you can use the domain of the instance in the format mqtt.<instance_domain> (for example mqtt.cumulocity.com) or your tenant domain (for example mytenant.cumulocity.com/mqtt).

Available ports:

Port 8883 supports two types of SSL: two-way SSL using certificates for client authorization and one-way SSL using username and password for client authorization. The two-way SSL support is enabled by default. To disable it please contact product support.

SmartREST payload

The Cumulocity MQTT implementation uses SmartREST as a payload. SmartREST is a CSV-like message protocol that uses templates on the server side to create data in Cumulocity. It incorporates the highly expressive strength of the REST API but replaces JSON with comma-separated values (CSV) to avoid the complexity of JSON parsing for embedded devices. Additionally, the simple and compact syntax of CSV renders it highly efficient for IoT communication via mobile networks. It can save up to 80% of mobile traffic compared to other HTTP APIs.

SmartREST basics

A SmartREST message is a single row in which each parameter is separated by comma. The first parameter is an ID that defines the message. You can send multiple messages in a single publish by using a line break between messages.

SmartREST escaping

The CSV (comma-separated values) format is used for communication with the SmartREST endpoint. The following rules must be followed to ensure a frictionless communication.

• Every row must be terminated by the \n character sequence.
• Values are always separated by a comma (,).
• If a value contains double-quotes ("), commas (,), leading or trailing whitespaces, line-breaks (\n), carriage returns (\r) or tab stops, it must be surrounded by quotes ("). Contained double-quotes (") must be escaped by prepending a backslash (\).

The same escaping rules apply to messages that will be sent from the server to the client.

Publish example:

100,"This value, needs escaping",This value does not need escaping

Subscribe example:

511,myDeviceSerial,"execute this\nand this\nand \"this\""

Device hierarchies

MQTT sessions are linked to a single device, but this device can have a freely configurable device hierarchy below it.

All children require a unique ID defined when creating the device. We recommend you to use a combination of the unique ID of the root device and a unique ID within the hierarchy.

To create data for a child instead of the root device, the unique ID of the child is added as another section in the topic (for example s/us/myChildDeviceIdentifier).

The client will automatically receive operations for every child in the hierarchy by subscribing to the respective topic. It is not required to subscribe for each child.

Every operation received will contain the template ID followed by the ID of the device/child for which the operation was created (followed by other operation parameters).

MQTT features

MQTT authentication

The communication with Cumulocity employing MQTT supports authentication in two ways:

• Username and password. The MQTT username must include the tenant ID and username in the format <tenantID/username>.
• Device certificates. For secure communication, devices must contain the entire chain of certificates leading to the trusted root certificate, or if only the device certificate is provided, then the immediate issuer certificate must be uploaded to the platform’s truststore. Also, they must contain the server certificate in their truststore.

Troubleshooting

A device sends correct username and password, but incorrect certificate at the same time

If the platform is configured to support two-way SSL, your devices have a configured keystore with invalid certificates, and you want to use basic authorization, we recommend you to turn off sending certificates during connection. Certificates may be invalid because they expired or the root certificate is not uploaded to the platform. Turn off certificate sending in the device’s software. If that is not possible, to make the connection work, check the following:

• The platform’s trust store cannot be empty. At least one trusted certificate must be uploaded to the platform.
• The device’s MQTT client must be configured to not send certificates if it does not find its root certificate in the accepted issuers list returned by the server during handshake. In most cases this happens automatically. It is known that it’s not working with the MQTT client and Java 11. However, it works with Java 8.
• In order to support this situation, the platform must be configured accordingly. In case you experience issues please contact product support.
• If all of the cases above are met and the device connection is still rejected due to certificates validation, then probably some other tenant uploaded a certificate with the same ‘Common Name’ as one of those sent by your device. In this case the device will always try to authorize itself with certificates.

MQTT ClientId

The MQTT ClientId is a field to uniquely identify each connected client. The Cumulocity implementation also uses the ClientId to link the client directly to a device. Therefore, the following format should be used for the ClientId:

connectionType:deviceIdentifier:defaultTemplateIdentifier

For the simplest version of a client, the MQTT clientId can just be the deviceIdentfier. It will automatically be interpreted as device connection.

Examples of ClientIds:

mySerialNumber
d:mySerialNumber
d:mySerialNumber:myDefaultTemplate

The uniqueness of the MQTT ClientId is determined only by the deviceIdentifier. Therefore, from the above examples only one client can be connected at the same time.

During an SSL connection with certificates, the deviceIdentifier must match the ‘Common Name’ of the used certificate (first certificate in the chain, which is provided by the device).

MQTT Quality of Service (QoS)

The Cumulocity implementation supports all 3 levels of MQTT QoS:

• QoS 0: At most once
  • The client just sends the message once (fire and forget).
  • No reaction from the server.
• QoS 1: At least once
  • The client repeats the message until it receives a server acknowledgement.
• QoS 2: Exactly once
  • The client sends a message.
  • The server acknowledges (holds the message).
  • The client sends a release command.
  • The server processes the messages and acknowledges again.

For subscriptions to the operation or error topics, we will deliver all messages in the QoS which the client defined when subscribing to the topic.

MQTT clean session

Cumulocity requires clean session to be set to “1” (true). Currently we cannot guarantee that disabling clean session will work reliably, hence we recommend you to always enable clean session.

MQTT retained flag

In the current Cumulocity implementation, subscriptions to topics where devices publish data are not allowed. Publishing data with the retained flag on this topic is allowed but has no practical difference to sending it without the flag. Messages published by Cumulocity like operations and errors do not contain the retained flag.

MQTT last will

In MQTT, the “last will” is a message that is specified at connection time and that is executed when the client loses the connection. For example, using 400,c8y_ConnectionEvent,"Device connection was lost." as last will message and s/us as last will topic, raises an event whenever the device loses the connection.

MQTT return codes

When there is an MQTT error, the platform responds with a CONNACK message with a non-zero return code. This message is the first clue that there is a problem. Such a return code can be treated similarly to REST API HTTP codes, such as 401. They can be returned because of an unexpected error, lack of permissions, and so on.

CONNACK is not only a response to a CONNECT message, but also a way to signal errors that occurred in the platform. Therefore, it is possible to receive this message a second time during a normal connection, and without a direct action. It is also a way to signal a closing connection, as most MQTT clients treat CONNACK with a code other than 0 like the connection needs to be closed. See the details below.

The table below shows the list of errors returned by Cumulocity:

Refer to MQTT Version 3.1.1 > 3.2 CONNACK - Acknowledge connection request for details on the official MQTT connection return codes.

Debugging

To support developers during development, it is possible to subscribe to the topic s/e. On this topic the device can retrieve debug and error messages that occur during a publish from the device.

MQTT broker certificates

MQTT broker uses the certificates which are assigned to the main environment domain. MQTT broker always sends these certificates during TLS handshake to devices. Moreover, Enterprise tenants are not able to customize MQTT broker certificates via the SSL Management feature.

MQTT JWT session token retrieval

The code of the Cumulocity MQTT example client implemented in Java, which connects to the platform using x.509 certificates, is available here: https://github.com/Cumulocity-IoT//cumulocity-examples/tree/develop/mqtt-client. This example client uses the implementation of Eclipse Paho, which is described in detail on their website: https://www.eclipse.org/paho/index.php?page=documentation.php.

Here is an example that shows how to add the needed dependency in Maven to use the Eclipse Paho client:

<dependency>
    <groupId>org.eclipse.paho</groupId>
    <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
    <version>${paho.version}</version>
</dependency>

Then the instance of the MQTT client can be created with a single line:

MqttClient mqttClient = new MqttClient(BROKER_URL, "d:" + CLIENT_ID, new MemoryPersistence());

The BROKER_URL must contain the protocol, URL and port to which the client will connect, like this: ssl://<cumulocity url>:8883. The CLIENT_ID value must match the value of the common name of the device certificate that will be used. The certificate’s common name should not contain : characters, see MQTT ClientId for more information. The “d:” prefix is used in Cumulocity for device connections and it should not be removed or changed. Now the only thing that must be configured to establish the SSL connection is to fill paths in the code fragment:

sslProperties.put(SSLSocketFactoryFactory.KEYSTORE, getClass().getClassLoader().getResource(KEYSTORE_NAME).getPath());
sslProperties.put(SSLSocketFactoryFactory.KEYSTOREPWD, KEYSTORE_PASSWORD);
sslProperties.put(SSLSocketFactoryFactory.KEYSTORETYPE, KEYSTORE_FORMAT);
sslProperties.put(SSLSocketFactoryFactory.TRUSTSTORE, getClass().getClassLoader().getResource(TRUSTSTORE_NAME).getPath());
sslProperties.put(SSLSocketFactoryFactory.TRUSTSTOREPWD, TRUSTSTORE_PASSWORD);
sslProperties.put(SSLSocketFactoryFactory.TRUSTSTORETYPE, TRUSTSTORE_FORMAT);

• The certificate’s common name should not contain : characters, see MQTT ClientId for more information.
• KEYSTORE_NAME - The path to your keystore which contains the private key and the chain of certificates, which the device uses to authenticate itself.
• KEYSTORE_PASSWORD - The password created for keystore to use its private key.
• KEYSTORE_FORMAT - Either “JKS” or “PKCS12” depending on the file format. The path is provided by KEYSTORE_NAME.
• TRUSTSTORE_NAME - The path to your truststore which contains the certificate of the server.
• TRUSTSTORE_PASSWORD - The password to access the truststore.
• TRUSTSTORE_FORMAT - Either “JKS” or “PKCS12” depending on the file format. The path is provided by TRUSTSTORE.

After filling in this data, the example client will use the provided data to connect to the specified platform using certificates. The example also shows how to create the callback for the connection. First thing is to create the class which implements the interface MqttCallbackExtended. Then such a class can be created and an instance of it can be provided to the MQTT client: mqttClient.setCallback(this);.

In general, the MQTT Eclipse Paho client uses the Java Secure Socket Extension, which is part of the Java Development Kit, to provide secure connections via SSL. JSSE provides the Java implementation of the SSL and TLS protocol, which can be configured by developers using its classes. The documentation of the Java Secure Socket Extension shows how the SSL connection is established and provides some examples of customizing the implementation. The full document is available on the official Oracle website.