Streaming Lake Ingestion

Overview

The Streaming Lake Ingestion feature in DataHub bridges the gap between real-time IoT data and large-scale business analytics and artificial intelligence needs. It takes the continuous stream of IoT data flowing into Cumulocity and automatically transforms it into a queryable and AI-ready format in your data lake. This fully managed process requires no configuration to start. When enabled, it processes your data transparently, continuously, reliably, and scalably, making it available for analysis within minutes.

From live stream to analytics

At a high level, the service consists of three parts:

• Data ingestion: The service subscribes to the real-time data feed from the Cumulocity Messaging Service (powered by Apache Pulsar).
• Processing and optimization: It intelligently batches, structures, and writes this data into Apache Iceberg tables within your data lake.
• Data access: The structured data is almost immediately available for high-performance querying from multiple endpoints, such as the built-in capabilities of Cumulocity Cockpit, Jupyter notebooks, AI/ML platforms like TensorFlow or PyTorch, or Business Intelligence tools.

A foundation for advanced analytics and AI

While powerful for business intelligence and ad-hoc queries, the true long-term value of your IoT data lies in its potential to fuel AI and Machine Learning (ML) models. However, raw device data alone is not enough. Technical data from your devices—such as PLC registers or sensor values—exists as cryptic identifiers (for example, register_012f or sensor_4a2b). These identifiers are typically associated with sensors and machine components, not with business-level assets. For AI and ML models to be useful, this raw data must be transformed into understandable, business-meaningful information.

Cumulocity and Streaming Lake Ingestion provide precisely this functionality:

• To automatically maintain cleaned, harmonized and historically complete data in a long-term data archive for any data science workflow, so that your data science team can focus on model development instead of data preparation.
• To contextualize it into business-relevant assets through Digital Twin Manager’s Data Service so that AI models can understand what the data represents and extract meaningful insights.

Reliability, performance, and an open standard for your data

Apache Iceberg is the strategic choice of Cumulocity to provide you with a reliable and future-proof foundation for your data lake. It is an open source table format governed by the Apache Software Foundation that has rapidly become an industry standard supported by major data platforms such as Dremio, Databricks, and Snowflake. It ensures that all your data assets remain portable and valuable for years to come.

Besides being a widely adopted industry standard, it provides the following other key advantages:

• Transactional reliability: Using Apache Pulsar and Apache Iceberg, Cumulocity manages your data with database-like guarantees for reliability, ensuring that your data is complete and consistent.
• Robust schema evolution: When your IoT data changes its structure, Cumulocity adapts your Iceberg tables automatically. Using Iceberg, new types of measurements or additional device properties are added without compromising previously stored data, and without breaking your queries and BI dashboards. This makes your data pipeline resilient to change.
• High performance analytics: The Iceberg catalog embedded into Cumulocity tracks detailed information about your data, allowing query engines and analytics tools to be much smarter. They can only read the files necessary to answer a query, returning results faster. This is further improved by running regular optimization on the data lake files to ensure that data is highly clustered for the many querying uses.

In summary, by building on Apache Iceberg and its standard REST catalog APIs, Cumulocity delivers IoT data as an open, vendor-neutral asset. Rather than forcing you into a proprietary ecosystem, your IoT data can be accessed and queried through the same tools and infrastructure you already use for your corporate data lake — whether that is Snowflake, Databricks, Dremio, or other platforms. This makes IoT data a true first-class citizen in your enterprise data initiatives. Your data science and analytics teams can seamlessly integrate IoT data with other business data sources without architectural compromises, breaking down the traditional barrier between operational IoT data and enterprise analytics to uncover insights that span your entire organization.

Using Streaming Lake Ingestion

Streaming Lake Ingestion is an optional service in Cumulocity. To subscribe to the service, contact the Cumulocity support. After subscription,

• Your current device and asset inventory is downloaded from the operational store into the lake.
• All new incoming data is stored in the lake.

Analyzing lake data

You can now analyze your data from any Apache Iceberg compliant tool. For example:

• Through the user interface of the embedded query engine.
• By using the secure and high performance querying APIs from applications and by using JDBC or ODBC with query engines and database tools.
• By connecting applications directly to the Cumulocity Iceberg catalog.

Using DataHub Query to analyze lake data

If your tenant includes a subscription to DataHub Query (Dremio), the system automatically sets up a data source with your tenant ID for querying the lake.

• Click the data source in the user interface to view the Iceberg folders (“namespaces”) that contain the tables of your data lake.
• Click a table to open the query editor and run SQL queries on the table. You see an example of a query as shown in the screenshot below.
• To refer to a table, use the pattern <tenant>.<namespace>.<table>. For example, if your tenant is “mytenant”, the inventory would be referred to as mytenant.cdc_inventory.inventory.
• To simplify your SQL statement, click the “Context:” link just above the query editor. You can select a data source and namespace that will be used as context for queries in the editor. For example, if you use “mytenant” as context, you can refer to the inventory using only cdc_inventory.inventory. If you use “mytenant.cdc_inventory” as context, you can refer to the inventory using only inventory.

Using DataHub Query APIs and drivers to analyze lake data programmatically

If your tenant includes a subscription to DataHub Query, use the DataHub APIs as well as the JDBC and ODBC drivers.

For example, use the REST API to submit a query job to DataHub Query.

$ curl -H "Content-Type: application/json" -u "<USER>:<PASS>" \
  https://<TENANT_DOMAIN>/service/datahub/dremio/api/v3/sql -d @- <<EOF \
{
  "sql": "SELECT * FROM inventory",
  "context": [ "<TENANT>", "cdc_inventory" ]
}
EOF

{"id":"12345678-1234-5678-1234-5678901234567"}

Then fetch the results of the query job once the job has finished.

$ curl -u "admin:$PASS" \
  https://<TENANT_DOMAIN>/service/datahub/dremio/api/v3/job/12345678-1234-5678-1234-5678901234567/results

{
  "rowCount": 7,
  "schema": [
    { "name": "id", "type": { "name": "VARCHAR" } },
    …
  ],
  "rows": [
    {
      "id": "99266201",
      "lastUpdated": "2026-03-30 15:27:35.781",
      "name": "Temperature #1",
      "owner": "service_device-simulator",
      "type": "c8y_MQTTDevice",
      "eventType": "MANAGED_OBJECT_UPDATE",
    …
  ]
}

Obtaining Iceberg catalog credentials

To connect to the Cumulocity Iceberg catalog directly — for example, from Apache Spark, Databricks, or a custom application — you need OAuth2 client credentials. As a tenant administrator, you can create and manage named catalog principals using the Manager API.

Prerequisites

• Your Cumulocity user must have the ROLE_TENANT_ADMIN permission.
• Your tenant must be subscribed to Streaming Lake Ingestion.
• Principal names must be strictly alphanumeric — letters and digits only, no dashes or underscores (for example, spark1 or dremioqa).

Creating a principal

Send a POST request to create a named principal. The response contains the clientId and clientSecret required to authenticate against the Iceberg catalog.

curl -s -X POST \
  "https://<TENANT_DOMAIN>/service/offloading/api/v1/principals/<NAME>" \
  -u "<USER>:<PASS>"

{
  "name":         "<NAME>",
  "clientId":     "<TENANTID>-<NAME>",
  "clientSecret": "<SECRET>"
}

Listing principals

To list the names of all principals for your tenant:

curl -s \
  "https://<TENANT_DOMAIN>/service/offloading/api/v1/principals" \
  -u "<USER>:<PASS>"

Rotating credentials

To replace a lost or compromised secret. The previous secret is immediately invalidated.

curl -s -X PUT \
  "https://<TENANT_DOMAIN>/service/offloading/api/v1/principals/<NAME>" \
  -u "<USER>:<PASS>"

Deleting a principal

To revoke a principal’s catalog access immediately:

curl -s -X DELETE \
  "https://<TENANT_DOMAIN>/service/offloading/api/v1/principals/<NAME>" \
  -u "<USER>:<PASS>"

Using the Iceberg catalog from Apache Spark

Apache Spark is a distributed computing framework that seamlessly integrates with the Cumulocity Iceberg catalogs to provide full SQL-based data processing through a standard Iceberg REST catalog interface.

Use OpenID Connect with a client credentials grant type to authenticate against the Cumulocity Iceberg REST catalog.

The URL of the OpenID Connect server is

https://iceberg.<INSTANCE>:19120/api/catalog/v1/oauth/tokens

For example, if your tenant is mytenant.cumulocity.com, the URL is

https://iceberg.cumulocity.com:19120/api/catalog/v1/oauth/tokens.

Run Spark SQL queries with the Cumulocity Iceberg catalog on an AWS S3 object store using the example below. In the example, replace

• The version number of the Iceberg Spark Runtime to match the Spark version that you use (here Spark 4.0.x).
• The <CLIENTID> and <CLIENT_SECRET> with the credentials obtained from support.
• <INSTANCE> and <TENANT_ID> with your instance URL and tenant ID.
• <REGION>, <AWS_ACCESS_KEY> and <AWS_SECRET_ACCESS_KEY> with credentials to access your AWS S3 store.

bin/spark-sql \
  --packages org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.1,org.apache.iceberg:iceberg-aws-bundle:1.10.1 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.polaris.credential="<CLIENT_ID>:<CLIENT_SECRET>" \
  --conf spark.sql.catalog.polaris.oauth2-server-uri="https://iceberg.<INSTANCE>:19120/api/catalog/v1/oauth/tokens" \
  --conf spark.sql.catalog.polaris.scope="PRINCIPAL_ROLE:ALL" \
  --conf spark.sql.catalog.c8y.rest-metrics-reporting-enabled=false \
  --conf spark.sql.catalog.c8y=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.c8y.type=rest \
  --conf spark.sql.catalog.c8y.uri=https://iceberg.<INSTANCE>:19120/api/catalog \
  --conf spark.sql.catalog.c8y.warehouse=<TENANT_ID> \
  --conf spark.sql.catalog.c8y.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
  --conf spark.sql.catalog.c8y.s3.endpoint=https://s3.amazonaws.com \
  --conf spark.sql.catalog.c8y.s3.region=<REGION> \
  --conf spark.sql.catalog.c8y.s3.access-key-id=<AWS_ACCESS_KEY_ID> \
  --conf "spark.sql.catalog.c8y.s3.secret-access-key=<AWS_SECRET_ACCESS_KEY>"

spark-sql (default)> show namespaces in c8y;
cdc_alarm
cdc_event
cdc_inventory
cdc_measurement
…
spark-sql (default)> select * from c8y.cdc_inventory.inventory;
99266201	2026-03-30 17:27:35.781	Temperature #1	service_device-simulator	c8y_MQTTDevice	MANAGED_OBJECT_UPDATE	["supportedMeasurements","com_cumulocity_model_Agent","c8y_IsDevice","c8y_SupportedOperations"]	NULL	NULL	NULL	["c8y_Temperature.T"]	2026-03-30 17:07:13.37
99266201	2026-03-30 17:27:44.838	Temperature #1	service_device-simulator	c8y_MQTTDevice	MANAGED_OBJECT_UPDATE	["supportedMeasurements","com_cumulocity_model_Agent","c8y_IsDevice","c8y_SupportedOperations"]	NULL	NULL	NULL	["c8y_Temperature.T"]	2026-03-30 17:07:13.37

Using the Iceberg catalog from other applications

Access the catalog from other applications using the curl example below. To obtain the <CLIENT_ID> and <CLIENT_SECRET> referenced in this section, see Obtaining Iceberg catalog credentials. First, get an access token to the catalog.

$  curl -X POST https://iceberg.<INSTANCE>:19120/api/catalog/v1/oauth/tokens \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials" \
  -d "client_id=<CLIENT_ID>" \
  -d "client_secret=<CLIENT_SECRET>" \
  -d "scope=PRINCIPAL_ROLE:ALL"

{"access_token":"ey…","token_type":"bearer","issued_token_type":"urn:ietf:params:oauth:token-type:access_token","expires_in":900}

Then use the token to access the catalog configuration and metadata:

$ curl "https://iceberg.<INSTANCE>:19120/api/catalog/v1/config?warehouse=<TENANT>" \
  -H "Authorization: Bearer <ACCESS_TOKEN>"

{
  "defaults": { "default-base-location": "s3://…>" },
  "overrides": { "prefix": "…" },
  "endpoints": [
    "GET /v1/{prefix}/namespaces",
    "GET /v1/{prefix}/namespaces/{namespace}",
    "HEAD /v1/{prefix}/namespaces/{namespace}",
    "POST /v1/{prefix}/namespaces",
    "POST /v1/{prefix}/namespaces/{namespace}/properties",
    "DELETE /v1/{prefix}/namespaces/{namespace}",
    "GET /v1/{prefix}/namespaces/{namespace}/tables",
    "GET /v1/{prefix}/namespaces/{namespace}/tables/{table}",
    …
}

$ curl https://iceberg.<INSTANCE>:19120/api/catalog/v1/<TENANT>/namespaces/cdc_inventory/tables/inventory' \
  -H "Authorization: Bearer <ACCESS_TOKEN>"

{
  "metadata": {
    "format-version": 2,
    …
    "schemas": [
      {
        "type": "struct",
        "schema-id": 0,
        "fields": [
          { "id": 1, "name": "id", "required": true, "type": "string" },
          { "id": 2, "name": "lastUpdated", "required": true, "type": "timestamptz" },
          { "id": 3, "name": "name", "required": false, "type": "string" },
    …

Understanding the data lake structure

Streaming Lake Ingestion provides three types of data:

• Change data capture (cdc): Includes the complete history of incoming IoT data. For example, the cdc_alarm.alarm table contains a complete, historical log of all alarm changes, while the cdc_inventory.inventory table captures all modifications made to your device master data over time.
• Latest data (latest): Provides the current representation of incoming inventory data. For example, the latest_inventory.inventory table contains a consolidated view on the latest state of your inventory without you needing to reconstruct the state from the historical logs.
• Views (view): Provides views for the change data capture tables. For example, the view_alarm.alarm view shows the same content as cdc_alarm.alarm with nested properties converted to top-level columns. This makes the data more easily browsable in tools like PowerBI.

For each type of data, folders (“namespaces”) are created to mirror the familiar Cumulocity domain model as shown in the screenshot. You will find folders with names ending in inventory, alarm, event, measurement and operation. Inside the folders, tables for standard data, device-specific data, and customer-specific data are created. Device- or customer-specific data in the form of fragments resides in separate, dedicated tables within the corresponding folders. For instance, if you send a measurement with a fragment type c8y_EngineMetric, it will be stored in a table named cdc_measurement.c8y_EngineMetric.

The following sections discuss the general table structure for each Cumulocity domain model class.

General table structure

All tables contain the following column:

• id: The record ID of the device, asset, alarm, event, measurement or operation associated with the change. In the inventory tables, this is the managed object ID.

The following columns are present in some tables, depending on the type of data:

• source: For alarms, events, or measurement, this column contains the managed object ID of the device or asset associated with the change.
• deviceId: For operations, this column contains the managed object ID of the device targeted for the operation.
• time: The timestamp of the change as sent by the device. This column is present for alarms, events, and measurements.
• type: The type property of the device, measurement and so on.

For change data capture tables and views, the following column is present:

• eventType: The type of change indicated by the record. For example, MANAGED_OBJECT_UPDATE indicates an update of a managed object.

Inventory

Creating inventory data

To understand how the service captures inventory data in the data lake, assume that the device user “device_123” creates a new tracking device “Tracking #1” with the following data:

{
    "name": "Tracker #1",
    "type": "sb_nano",
    "c8y_Position": {
        "alt": 67,
        "lng": 6.15173,
        "lat": 51.211977
    },
    "c8y_IsDevice": { },
    "someProperty": 10
}

This results in the following data in the data lake:

Table: cdc_inventory.inventory

The columns of the inventory table are:

• id contains “47635”, the managed object ID assigned to the new device by Cumulocity.
• lastUpdated contains “2025-08-20T13:41:39.678Z”, the timestamp of the most recent update to the managed object in Cumulocity.
• name contains “Tracker #1”, the name of the managed object as used in the request.
• owner contains the user who created the managed object.
• type contains “sb_nano”, the type of the managed object as used in the request to create the managed object.
• eventType contains “MANAGED_OBJECT_CREATE”, the event type for creating a new managed object.
• fragments contains a list of all fragments used in the managed object.
• childAdditions contains a list of IDs of services linked to a device.
• childAssets contains a list of IDs of child assets of an asset.
• childDevices contains a list of IDs of child devices of an asset or device.
• someproperty contains 10. Cumulocity creates additional columns in the inventory table when there are top-level properties of an atomic type in the managed object (character strings, numbers, booleans).

Additional columns may be visible depending on your use of Cumulocity. For example, once the device has emitted measurements, you will see a column supportedMeasurements with a list of the measurement fragments sent by the device.

Table: cdc_inventory.c8y_Position

For the c8y_Position fragment, a table c8y_Position is created with the following columns:

Generally, columns are set as follows:

• Key columns are stored as previously described.
• Properties of the fragment are stored in additional columns of the table.

Table: latest_inventory.inventory

In addition, the service stores the latest data in corresponding tables in the latest_inventory folder. You will find a table “inventory” with the following structure:

Since the table reflects the only latest state of the inventory and not the entire change history, no eventType column is provided.

Table: latest_inventory.c8y_Position

Accordingly, the latest custom data is visible in the c8y_Position table in latest_inventory:

Updating inventory data

Assume that the tracking device updates its location:

{
    "c8y_Position": {
        "alt": 69,
        "lng": 6.3213,
        "lat": 50.425
    }
}

This update appears in the inventory and c8y_Position tables as follows:

Table: cdc_inventory.inventory

Table: cdc_inventory.c8y_Position

Table: latest_inventory.inventory

The “latest data” versions of the tables reflect this update:

Table: latest_inventory.c8y_Position

Deleting inventory data

Currently, delete operations of inventory entries are only written for CDC tables. When a managed object is deleted, an update is written to cdc_inventory.inventory like the following:

Only the key columns are updated.

Alarms

Assume that our sample tracking device has low battery and sends an alarm:

{
  "source": { "id": "47635" },
  "type": "c8y_BatteryAlarm",
  "text": "Battery level below 5 percent.",
  "severity": "MAJOR",
  "status": "ACTIVE",
  "time": "2025-08-19T12:03:27.845Z"
}

This results in the following data in the data lake:

Table: cdc_alarm.alarm

Again, the columns of the table represent the properties of a Cumulocity alarm. The eventType property reflects the recorded change (in this case, a created alarm), and the count property is added by Cumulocity’s alarm de-duplication. The firstOccurrenceTime column is added to reflect the time of the first occurrence of the alarm, which stays the same for de-duplicated alarms.

In the example, there are no custom properties or fragments in the alarm data. The service treats such properties or fragments in the same way as described for the inventory.

Events

Now assume that the tracker moves and sends a location update event.

{
    "c8y_Position": { "alt": 67, "lng": 6.95173, "lat": 51.151977 },
    "time": "2025-08-20T13:41:39.678Z",
    "source": { "id": "47635" },
    "type": "c8y_LocationUpdate",
    "text": "Tracker location update"
}

This results in the following data in the data lake, consistent with the method described previously.

Table: cdc_event.event

Table: cdc_event.c8y_position

Measurements

To illustrate the storage of measurements, assume that the tracking device sends a battery measurement in the following form:

{
  "time": "2025-08-20T13:42:39.678Z",
  "source": { "id": "47635" },
  "type": "c8y_BatteryMeasurement",
  "c8y_Battery": {
    "voltage": { "value": 12.8, "unit": "V" },
    "stateOfCharge": { "value": 85.5, "unit": "%" },
    "temperature": { "value": 22.5, "unit": "C" }
  }
}

This results in the following data in the data lake:

Table: cdc_measurement.c8y_Battery

There is one important difference for measurements compared to other data types: There is no cdc_measurement.measurement table. All relevant data is present in the fragment tables.

This example also demonstrates how the service handles nested properties: Like the Cumulocity APIs and its operational store, Apache Iceberg supports nested data. Just like with the above JSON example, the property voltage is a nested structure containing two properties value and unit. You can query the contained properties from SQL using path expressions such as "voltage"."value". For more information on querying the data from SQL, see Example queries.

Operations

Finally, we would like to restart the tracker with a reset operation:

{
  "deviceId": "47635",
  "c8y_Restart": { }
}

This results in the following data in the data lake:

Table: cdc_operation.operation

The columns represent the properties of a Cumulocity operation. If you do not provide an optional property, the service stores it as a SQL “null” value.

Data types

Cumulocity relies on JSON as data exchange format in its APIs. JSON data types map to Iceberg data types as follows:

• Strings parsed as timestamps become the Iceberg timestamptz data type. Otherwise, they become the Iceberg string data type.
• Numbers are stored as Iceberg decimal with a maximum precision of 38 and scale of 9.
• Booleans (true/false) map to the equivalent Iceberg boolean data type.
• JSON lists become the equivalent Iceberg list data type.
• JSON objects become the equivalent Iceberg struct data type.

“Null” values are stored as Iceberg “null” values.

Empty JSON objects ({}) and empty arrays ([]) carry no data to store. They are omitted from the input and treated the same as an absent or “null” property, rather than being converted to a value of the column’s data type.

Schema evolution

The data lake storage service automatically discovers the structure of the incoming data and applies it to the data lake.

• When the service discovers a new fragment, it creates a table to store the data in the fragment.
• When the service discovers a new property, it creates a column to store the data in the associated table.

Handling of schema conflicts

Contrary to Apache Iceberg and SQL, Cumulocity does not mandate a consistent schema across all incoming data. Consistency issues are automatically discovered and resolved by the service. Revisiting our previous example, assume that “Tracker #1” sends data as follows:

{
    "name": "Tracker #1",
    …
    "someProperty": 10
}

Table: cdc_inventory.inventory

Now, “Tracker #2” sends data with a slightly different structure:

{
    "name": "Tracker #2",
    …
    "someProperty": "Some Value"
}

Storing the string value “Some Value” of someProperty in the numeric column with the corresponding name is not possible. When such a mismatch in type occurs, a new overflow column is created. The new column uses the same name as the property, but appends a double underscore followed by a short type suffix:

Table: cdc_inventory.inventory

Overflow columns use the following type suffixes:

If you send additional type variations for the same property, additional overflow columns are created accordingly. For example, if “Tracker #3” later sends "someProperty": [true], a new column someproperty__lb would be created for the list of booleans.

Naming

The Iceberg data lake supports names consisting of characters, numbers, underscores, and dashes — except as the first character, which must always be a letter (a-z, A-Z). Digits, underscores, dashes, and any other character are not permitted in the first position, even though they are permitted elsewhere in the name. Other characters are represented by the character string “_x” followed by the hexadecimal Unicode value of the character. This applies to table and property names, at any nesting level.

For example, if you use the property name “switch:status” in the data sent to Cumulocity, the corresponding Iceberg column name is “switch_x003Astatus”.

Characters are case-sensitive, but identifiers of tables and columns must be unique in a case-insensitive way. When the column or table name is not unique, a unique suffix is automatically appended to the name. The suffix is generated based on the original name as a binary code where uppercase letters are represented as 1 and lowercase letters as 0. This binary code is then converted to a hexadecimal number and appended to the name in the format {name}__{suffix}. For example:

Flat views

Streaming Lake Ingestion offers Apache Iceberg views on top of the data ingested into the data lake tables.

Integration with BI tools

As upstream reporting tools and query engines develop, there are still tools that do not support browsing structured data in database tables. The service solves this problem by flattening structured fields into “flat”, top-level columns. This makes it possible to browse, for example, the value and unit fields of measurements in such tools.

Take the following example of a measurement:

{
  "time": "2026-02-19T13:09:39.678Z",
  "source": { "id": "47635" },
  "type": "c8y_BatteryMeasurement",
  "c8y_Battery": {
    "voltage": { "value": 12.8, "unit": "V" },
    "stateOfCharge": { "value": 85.5, "unit": "%" },
    "temperature": { "value": 22.5, "unit": "C" }
  }
}

This results in a table like the following:

Table: cdc_measurement.c8y_Battery

Note that there are three columns which are of structured type: voltage, stateOfCharge and temperature.

Corresponding to the table, a view is created automatically as shown below. Note that the value property of voltage was converted to voltage\value, that is, all property names on the path up to the bottom-most property are concatenated with backslash as separator character.

View: view_measurement.c8y_Battery

This makes it possible to operate directly on nested fields and, for example, do analytics for measurement values in BI tools.

View representations

In Apache Iceberg, views can have multiple representations for different SQL dialects. Currently supported dialects are:

• DremioSQL
• spark (experimental)

Binning

To ensure robustness data storage and queryability of the data, the service enforces a number of limits beyond Cumulocity’s platform-wide limits. Data that exceeding these more specific limits is “binned”, that is, stored into a separate Iceberg table in a different schema for inspecting problematic cases and potentially reprocessing data. An example of “binned” data is shown below:

Table: cdc_rejected.trash

The following schema is used:

• source: Source/managed object ID of the device or asset
• eventType: Type of the originating payload
• dataType: Domain model type
• time: Timestamp from the payload
• path: Payload path of the affected field as json pointer
• data: Serialized value of the rejected field in binary (Base64) format
• cause: Type of violation
• reason: Description of the violation, providing context for the rejection
• tableName: Fragment name where the violation occurred
• id: Unique identifier for the record

Limits of Streaming Lake Ingestion

The following data is moved to the trash table.

Type limits

• Multidimensional arrays. (Note that arrays holding nested struct types are permitted.)
• Arrays with inconsistent types in the array fields.
• Numbers requiring larger precision than the maximum Iceberg decimal precision of (38, 9).

Structure limits

• Strings larger than 32,768 characters.
• Arrays larger than 128 entries
• Objects nested deeper than 16 levels.
• More than 1,000 total leaf properties per fragment.
• More than 1,000 fragments.
• Fragment or property names with more than 255 characters.

Examples

Illegal field name: The first character of a property or fragment name must be a letter (a-z, A-Z). Digits, underscores, dashes, or any other character in the first position cause the field to be rejected rather than escaped. See Naming for details.

{
    "event_fragment_2": {
        "nestedObject": {
            "_invalidFieldName": "test"
        }
    }
}

The same rule applies to fragment (table) names and to a number in the first position, for example:

{
    "3invalidFragmentName": {
        "test": "test"
    }
}

Maximum depth exceeded: Deeply nested structures exceeding the allowed depth of 16 levels.

{
    "inventory_fragment_2": {
        "level_1": {
            "level_2": {
                // ...
                    // ...
                        "level_17": {
                            "name": "The abyss",
                            "description": "You hit rock bottom, and then dug deeper."
                        }
                    // ...
                // ...
            }
        }
    }
}

Maximum number of columns exceeded: Adding columns to tables resulting in more than 1,000 columns.

{
    "fragment": {
        "property_0001": "value_0001",
        "property_0002": "value_0002",
        // ...
        "property_1000": "value_1000",
        // ...
    }
}

Maximum number of fragments exceeded: Creating more than 1,000 tables.

{
    "table_fragment_0001": {
        "test": "test"
    },
    "table_fragment_0002": {
        "test": "test"
    },
    // ...
    "table_fragment_1000": {
        "test": "test"
    },
    // ...
}

Size exceeded: Data exceeding the maximum permissible size of 32,768 characters for a string field or the maximum size of a list field (128 elements).

{
    "event_fragment_1": {
        "tooLargeProperty": "一石二鳥一石二鳥一石二鳥一石二鳥 ......",
        "tooLargeList": [
            "2025-01-1T01:00:11Z",
            "2025-01-1T01:05:02Z",
            "2025-01-1T02:00:11Z",
            // ...
            "2025-01-13T02:00:11Z"
        ]
    }
}

Unresolvable type conflict: Field has inconsistent data types across array elements.

{
    "fragment": {
        "listField": ["Some string", 5.1, false, null]
    }
}

Example queries

The following example queries demonstrate how to extract common metrics and insights from the data lake using the Dremio SQL dialect. To keep the queries concise, they assume that you set the query context to your tenant as described in Analyzing lake data. This allows you to omit the tenant prefix in the FROM clauses.

Basic inventory queries

To list all devices registered in your tenant, use the latest_inventory tables to get the most recent state and filter for the marker fragment c8y_IsDevice in the fragments array.

SELECT id, name
FROM latest_inventory.inventory
WHERE ARRAY_CONTAINS(fragments, 'c8y_IsDevice')

To find the current location of all devices within a specific geographic boundary, query the c8y_Position fragment in the latest_inventory folder.

SELECT id, lat, lng
FROM latest_inventory.c8y_Position
WHERE lat BETWEEN 50.0 AND 52.0
  AND lng BETWEEN 6.0 AND 8.0

Basic time series queries

To retrieve all events recorded for a specific device within a given timespan, query the cdc_event.event table. Note that time is a reserved word in Dremio SQL, so it must be quoted.

SELECT "time", type, text
FROM cdc_event.event
WHERE source = '52277201'
  AND "time" BETWEEN '2026-04-16 00:00:00' AND '2026-04-16 23:59:59'
ORDER BY "time" DESC

Parent and child queries

To find a parent device or asset linked to a specific device, you can search for the child’s ID within the childDevices array.

SELECT id, name
FROM latest_inventory.inventory
WHERE ARRAY_CONTAINS(childDevices, '102938')

Conversely, to retrieve all child devices belonging to a specific asset, use Dremio’s FLATTEN function to unroll the asset’s childDevices array and select the matching devices.

SELECT id, name
FROM latest_inventory.inventory
WHERE id IN (
  SELECT FLATTEN(childDevices)
  FROM latest_inventory.inventory
  WHERE id = '102938'
)

Measurement queries

To view a specific measurement series for a device over time, query the corresponding measurement fragment table. The following example fetches voltage measurements.

SELECT "time", b.voltage."value" AS voltage, b.temperature."value" AS temperature
FROM cdc_measurement.c8y_Battery b
WHERE source = '47635' AND
 "time" BETWEEN '2026-04-02 00:00:00' AND '2026-04-02 23:59:59'
ORDER BY "time" DESC

Note that Dremio SQL requires you to add a table name or table reference when accessing structured properties. In the example, it is not enough to only refer to voltage."value", b.voltage."value" has to be used. You can build on the previous query by adding a condition to filter for values that cross a certain threshold limit.

SELECT "time", b.voltage."value" AS voltage, b.temperature."value" AS temperature
FROM cdc_measurement.c8y_Battery b
WHERE source = '47635' AND
 "time" BETWEEN '2026-04-02 00:00:00' AND '2026-04-02 23:59:59' AND
 b.voltage."value" < 11.5
ORDER BY "time" DESC

To create reports that are easier to read, you can join the historical measurement data together with the latest inventory data. This allows you to display the current device name alongside the recorded measurement values.

SELECT b."time", i.name, b.voltage."value" AS voltage
FROM cdc_measurement.c8y_Battery b JOIN latest_inventory.inventory i ON b.source = i.id
WHERE source = '47635' AND
 "time" BETWEEN '2026-04-02 00:00:00' AND '2026-04-02 23:59:59' AND
 b.voltage."value" < 11.5
ORDER BY "time" DESC

Ensuring good query performance

While the data lake offers the flexibility of arbitrary SQL querying, performance can vary depending on the complexity of your query. Unlike the operational store, Cumulocity cannot provide a general response time guarantee for all possible SQL statements.

To ensure fast responses for common IoT use cases, performance is highly optimized for queries that select a specific set of devices, a particular time range, and, for measurements, a limited set of data points.

Conversely, other types of queries may exhibit slower responses. These include queries that process a very large base set of data with complex aggregations or execute multi-level joins with selections on non-indexed properties. In shared environments, the service may terminate very large or long-running queries to ensure fair resource usage.

To improve query performance for your specific applications, you have several options:

• Pre-process the data before it enters Cumulocity using Edge functionality or Data Preparation functions.
• Process the data within Cumulocity using Streaming Analytics or a custom microservice to create refined data streams.
• Post-process the data in the data lake by creating your own aggregated “gold layer” tables using external data lake tools.

Monitoring the data flow

The service automatically updates the data lake schema when the structure of the incoming data changes. You receive notifications in the form of Cumulocity events for schema changes and alarms for schema violations. These notifications help you stay informed about changes and be able to address any issues.

The following notifications appear in the system:

• Schema evolved event (c8y_StreamingLakeIngestion_SchemaEvolved): You see this event in the Cumulocity event viewer when the service detects a change in the structure of the incoming data. The event appears when new tables are created or new columns are added to existing tables. If several changes happen at once, the service combines them into a single event. The event lists all new tables and columns added during that period. This helps you track how your data structure grows over time.

  

• Schema violation alarm: This alarm appears in the alarm viewer when the service detects message payloads that do not match the expected schema (binned data). The alarm provides high-level information about the schema violation. For detailed information about the affected messages, query the trash table.

  

Migrating from DataHub Scheduled Offloading

You can run DataHub scheduled offloading jobs and Streaming Lake Ingestion in parallel. To migrate your postprocessing or reporting to Streaming Lake Ingestion incrementally, subscribe to Streaming Lake Ingestion and port your queries or scripts as needed. Contact Cumulocity support for questions on optimizing your data lake pipelines.

Current preview limitations

The current preview version includes a number of functional limitations as outlined in this section.

• latest data is only recorded for the inventory.
• Delete events are only recorded for the inventory change data capture, not for the latest inventory.
• Custom top-level properties in measurements are ignored.
• There is no automated cleanup of tables and columns that are not in use anymore.
• The automated views only expose change data capture tables.
• Arrays appear as-is in the views.
• You may see an internal folder with tables for internal service information. Do not modify these tables.
• The service does not offload any Cumulocity-internal inventory entries, events, measurements, and alarms.
• Iceberg credential vending is not yet supported. If you use the Iceberg catalog from another application, you have to provide credentials for the object store that you use.
• Users automatically get read permissions to the Dremio data source assigned. These read permissions cannot be manually revoked.

Frequently asked questions

How can I configure Streaming Lake Ingestion?

Streaming Lake Ingestion has no configuration options and stores all incoming realtime data into your lake. Depending on your use case, you can use a combination of options outside of Streaming Lake Ingestion to influence how data appears in your lake:

• Use Edge or Data Preparation to change data to more user friendly names, execute simple calculations or route data using processing modes.
• Use Digital Twin Manager to promote data points to asset level.
• Use Streaming Analytics to create new data in your desired output format.
• Use data broker to transfer data between tenants.
• Use a so-called “gold layer” to post-process IoT data together with your IT data into an joint and aggregated target format.

Is DataHub Query (Dremio) required for Streaming Lake Ingestion?

DataHub Query is optional if you want to post-process or query data using your own tools. Note that views are currently only generated for engines compatible with Dremio and Spark SQL. While we designed the service to be as compliant as possible with common tool restrictions, we cannot guarantee compliance with all Iceberg tools.

I see a device, alarm, measurement, event or operation in the operational store but not in the lake.

Only changes after subscription to Streaming Lake Ingestion are visible in the lake. If you have recently subscribed and the data was not changed since then, it will not be transferred to the lake.

If the data was changed and it is not visible, check the Alarms page in the Cockpit application for warnings. You may have breached a limit. In this case, data that breaches the limit is inserted into the trash table.

I see a device in the lake, but not in the operational store.

After you delete a device, the device is removed from the operational store but remains visible in the historic data in the lake for long-term reporting.

I see a device in latest_inventory but not in cdc_inventory. Is this expected?

Yes. The latest_inventory tables are pre-populated with all managed objects that existed at the time of subscription, while the cdc_inventory tables only record changes that occur after subscription.

My device, alarm … appears several times in the inventory in the lake.

The change data capture of the device inventory (cdc_inventory) records all changes to devices together with the kind of change that was applied. Inspect the eventType property and the lastUpdated to understand what change was applied when.