Operating Cumulocity IoT DataHub

Checking system information

In the navigator, select Administration and then System status to get information about the system configuration and its status.

Under Microservice you will find the status of the microservice, which is either marked as green or red. This status reflects whether the microservice can be accessed from the web application. If the microservice is accessible, its current version is shown. If not, check the status of the microservice and its logs as described in section Administration > Managing applications in the Cumulocity IoT User guide.

Under Web application you will find the version of the web application.

Under Dremio you will find the status of Dremio, which is either marked as green or red. This status reflects whether Dremio can be accessed from the microservice. If Dremio is accessible, its current version is shown. If not, check the status of the microservice and its logs as described in section Administration > Managing applications in the Cumulocity IoT User guide.

Under Management you will find the setup of the system. If you expand that box by clicking on the arrow to the right, all relevant system properties and their values are listed. Note that these values cannot be modified for a running microservice. The tenant administrator needs to redeploy the microservice with corresponding new values.

Tracking usage statistics

If enabled, Cumulocity IoT DataHub tracks usage statistics on the amount of data being processed. These statistics are collected for offloading queries and track the amount of data these queries read from the Operational Store of Cumulocity IoT. The statistics are also collected for ad-hoc queries and track the amount of data these queries read from the data lake. The usage statistics can be utilized for a volume-based charging. They can also be utilized to pinpoint resource-intensive queries in terms of network load.

In the navigator, select Administration and then Usage statistics to view the usage statistics.

In the action bar, a date control allows you to select the month for which you want to see the usage statistics.

The three top panels show overall summary statistics as well as statistics separated for offloading and ad-hoc queries. If data from the month before the selected month is available, a tendency arrow illustrates whether the data volume of the selected month has decreased, increased, or stayed flat. The panels with the offloading and the ad-hoc query statistics additionally list the days with minimum/maximum volume as well as the daily average volume.

The table below the summary statistics shows the details on a per-day basis for the selected month. For each day, the volume offloaded and the volume queried are shown as well as their sum, which constitutes the daily volume. In addition the percentage of the monthly volume is shown, that is, how much did the daily volume contribute to the overall monthly volume. The date of each entry links to the Query log, which lists all queries for the respective day.

Viewing audit logs

Auditing shows in the query log the queries being executed and in the system log the operations that users have carried out.

Query log

In the navigator, select Auditing and then Query log to view the query log.

At the top of the page you can select either offload or ad-hoc queries, define a text filter on the offloading task/ad-hoc query string, and select a time period. Use the pagination buttons at the bottom of the page to navigate through the result list.

For each offloading query, the following information is provided:

For each ad-hoc query, the following information is provided:

System log

In the navigator, select Auditing and then System log to view the system log.

At the top of the page you can select log entries having status all/successful/errorneous/running, define a text filter on the log entries, and select a time period. Use the pagination buttons at the bottom of the page to navigate through the result list.

For each log entry, the following information is provided:

Endpoints for monitoring

ETL pipeline health

The Cumulocity IoT DataHub microservice exposes an endpoint to automatically monitor the health of active offloading configurations. The ETL pipeline health can be monitored with the endpoint GET /service/datahub/scheduler/health:

The endpoint examines the latest job executions of all jobs and classifies them:

• If the job has failed, it is reported as CRITICAL.
• If the job is still running, it is categorized as follows:
  • If it is running for up to one hour, its health is classified as STEADY.
  • If it is running for up to six hours, its health is classified as WARNING.
  • If it is running for more than six hours, its health is classified as CRITICAL.
• If the job has succeeded, it is checked whether it was the last job that should have been run for this configuration. If there should have been a new run of this job and the system is already 10 minutes behind the scheduled execution time, the job is classified as CRITICAL. Otherwise, the job is classified as STEADY.

If all jobs are classified as STEADY, the endpoint returns the HTTP status code 200 with the following message:

“HTTP 200 CDHCBEI0029 - Scheduler healthcheck succeeded.”

Otherwise, the endpoint returns the HTTP status code 500 with the following message:

“HTTP 500 CDHCBEE0031 - Scheduler healthcheck failed: There were failed or suspended jobExecutions.”

The response body indicates the jobs to be checked by an administrator:

{
"error" : "There were failed or suspended jobExecutions: \n\nCRITICAL: Job failed: uuid=0d2eb545-cae5-4718-b6c1-50c4169bac69, jobType=CTAS, jobRunId=NON_CLUSTERED1580741460697\n\n"
}

The endpoint can be accessed by any logged in Cumulocity IoT user who is authorized to access the Cumulocity IoT DataHub microservice.