Using the microservice-based data broker
Notice of current storage limits and message TTL for the microservice-based data broker
Important
The microservice-based data broker stores messages persistently until they are successfully delivered to the destination tenant. To optimize resource usage, the storage limits for the microservice-based data broker have been adjusted as follows:
Message backlog quota:
- Each data broker connector can store a maximum of 50 MiB of unacknowledged messages in its backlog. This is equivalent to approximately 260,000 messages, assuming an average message size of 200 bytes.
Message time-to-live (TTL):
- A default message TTL of 36 hours has been introduced. Any unacknowledged messages remaining in the backlog for more than 36 hours will be automatically deleted.
These limits are configurable on a per-tenant basis. If your use case requires a different configuration, or if you have any concerns, please contact product support.
The microservice-based data broker is powered by the Cumulocity Messaging Service that enables reliable, scalable and high-performance movement of IoT data. The microservice-based data broker is similar to the existing data broker in its functionality, except that a microservice, the databroker-agent-server, must be enabled to make use of it.
Requirements
The Cumulocity Messaging Service is an optional component of the Cumulocity platform that may need to be enabled before the microservice-based data broker can be used. The original data broker will continue to operate alongside the microservice-based data broker for the time being, and users can choose which data broker to use on a per-tenant basis.
For the shared public cloud instances of the Cumulocity platform, the Messaging Service is enabled by default on release 10.13 and above, and the microservice-based data broker can be enabled on request for individual tenants that already have access to the original data broker. For dedicated and self-hosted instances, the Messaging Service and microservice-based data broker are available for release 10.10 and above, but will need to be explicitly enabled.
Please contact product support to inquire about using the Messaging Service and microservice-based data broker capabilities in your Cumulocity environment. See the Messaging Service - Installation & operations guide for further technical details of the configuration required, but note that these tasks can only be performed by a Cumulocity platform operator, not by a normal user.
In summary, to work with the microservice-based data broker, the following requirements must be met:
- The Cumulocity Messaging Service should be available on your Cumulocity platform.
- Your tenant must be subscribed to the application “feature-broker”.
- Your tenant must be subscribed to the microservice “databroker-agent-server”.
To enable the microservice-based data broker
The microservice-based data broker must be enabled from the Management tenant. Contact your Operations team for further support.
Data connectors
See Data connectors for details on how to manage data connectors.
Data subscriptions
See Data subscriptions for details on how to manage data subscriptions.
Migrating existing data connectors to the microservice-based data broker
After enabling the microservice-based data broker, your existing data connectors should continue to work without any additional configuration.
Troubleshooting
Subscription alert
The Management tenant cannot be used as a data broker source tenant and this alarm is raised when trying to subscribe a Management tenant to the data broker agent.
Data broker connection error
The data broker agent is pre-configured to monitor each connector for the number of failed forwarding requests sent. If this number reaches a pre-configured threshold a CRITICAL alarm is raised in the tenant. If this happens the data will be stored until the connection is restored and it can be forwarded again. Failed requests can happen in the event the data broker subscriber tenant becomes unreachable.
Data broker slow processing alert
The data broker agent is pre-configured to monitor the rate at which events are being delivered to their destination. If events cannot be delivered fast enough, slow processing alarms will be raised. A slow processing alarm includes a connector ID to help identify which destination tenant is affected.
Queue backlog
This alarm is raised when latency for message delivery crosses a specified threshold. This usually happens if there is a backlog of undelivered events to the destination tenant due to various factors.
Average request bytes sent per second
The data broker monitors the data rate at which events are being forwarded. If this rate is lower than a pre-configured threshold, a slow processing alert will be raised. This can occur due to a slow network.
Info
Refer to the Messaging Service Installation & operations guide to find out more about how these parameters can be configured.