Overview
Thin-edge.io is an open-source framework to develop lightweight, smart, and secure connected devices.
The cloud-agnostic thin-edge.io provides the foundations for cloud connectivity and device management, a set of pre-packaged modules, plug & play connectors to cloud platforms, device certificate management, monitoring as well as built-in software and firmware management.
On top of these foundations, you can build telemetry applications using a combination of components provided by various IoT actors. The features provided by these components can include for example low-level connectivity to IoT protocols, event-stream analytics, machine-learning-powered systems, or application specific processors.
Thin-edge.io can be extended in various programming languages due to its extensible architecture. The following list shows the key aspects of the thin-edge.io architecture:
- The components are processes exchanging messages over an MQTT bus.
- The MQTT bus is connected to the cloud, forwarding the messages published on cloud-specific topics.
- A canonical data format lets the components exchange telemetry data independently of the connected cloud. This is an optional feature and the components are free to also use cloud-specific data formats.
- The mapper processes are responsible for translating the canonical data format into cloud-specific messages and vice versa.
Design principles
The primary goal of thin-edge.io is:
- to simplify the connection of edge devices to the cloud by providing a secure and reliable cloud connectivity as well as a device management agent
- the ability to build IoT applications around a large diversity of components provided by independent actors
For that purpose, thin-edge.io focuses on:
- Interoperability - Thin-edge.io lets you integrate components producing or consuming telemetry data: northbound with cloud platforms, southbound with sensors as well as for east-west communication between analytics components.
- Flexibility - Thin-edge.io lets you integrate components provided by different IoT actors, not even originally designed with thin-edge.io in-mind, using various technologies and programming languages.
- Security - Thin-edge.io provides a secure and stable foundation for cloud connections, software/firmware updates, and remote device management.
- Reliability - Thin-edge.io components can survive in chaotic environments as network outages and process restarts happen.
- Efficiency - Thin-edge.io lets you build applications that can run on constrained device hardware and with limited bandwidth networks.
- Multi-cloud - Thin-edge.io enables you to connect your edge devices with multiple clouds. Which cloud to use can be decided at runtime by the user.
Supported platforms
General
Common requirements for all systems are:
- minimum 16MB of RAM
- systemd (for production systems)
- mosquitto minimum version 1.6 (for security reasons we recommend the latest 1.x version)
- dpkg (if you want to use our prebuilt .deb packages)
Level 1
Level 1 supported platforms are officially supported and are actively tested in the CI/CD.
- ARMv7 Raspberry Pi OS 10
- ARMv8 Raspberry Pi OS 10
- AMD64 Ubuntu 20.04
Level 2
Level 2 platforms are not officially supported and tested yet, but the following systems have been used by some of our users.
If your OS is not listed here, this does not mean it is not working. We recommend trying your OS with our level 2 platforms.
- Ubuntu 20.04 in WSL: only for development, not for running thin-edge.io due to missing
systemd
. - AMD64 Debian 10
- ARMv6 Raspberry Pi OS 10 (needs to be built for this specific target, refer to Issue-161 at https://github.com/thin-edge/thin-edge.io/issues/161
Thin Edge JSON format
Thin Edge JSON is a lightweight format used in thin-edge.io to represent measurements data. This format can be used to represent single-valued measurements, multi-valued measurements or a combination of both along with some auxiliary data like the timestamp at which the single measurement or multiple measurements were generated.
Single-valued measurements
Simple single-valued measurements like temperature or pressure measurement with a single value can be expressed as follows:
{
"temperature": 25
}
The key represents the measurement type while the value represents the measurement value. The keys can only have alphanumeric characters and the “_” (underscore) character. However, the key cannot start with an underscore. Values can only be numeric, therefore string, boolean or other JSON object values are not allowed.
Multi-valued measurements
A multi-valued measurement is a measurement that is comprised of multiple values. Below you find the representation of a three_phase_current'measurement
that consists of L1
, L2
and L3
values, each representing the current on each phase, see example below:
{
"three_phase_current": {
"L1": 9.5,
"L2": 10.3,
"L3": 8.8
}
}
The key is the top-level measurement type and value is a JSON object having further key-value pairs which represent each aspect of the multi-valued measurement. Only one level of nesting is allowed, therefore the values of the measurement keys at the inner level can only be numeric values.
For example, a multi-level measurement as follows is NOT valid:
{
"three_phase_current": {
"phase1": {
"L1": 9.5
},
"phase2": {
"L2": 10.3
},
"phase3": {
"L3": 8.8
}
}
}
This example is not valid as the values at the second level, phase1
, phase2
, and phase3
, are not numeric values.
Grouping measurements
Multiple single-valued and multi-valued measurements can be grouped into a single Thin Edge JSON message, see example below.
{
"temperature": 25,
"three_phase_current": {
"L1": 9.5,
"L2": 10.3,
"L3": 8.8
},
"pressure": 98
}
The grouping of measurements allows you to represent measurements collected at the same instant of time.
Auxiliary measurement data
When thin-edge.io receives a measurement, it adds a generated timestamp to the measurement before any further processing. If you do not want to rely on this timestamp, an explicit timestamp can be provided in the measurement message itself by adding the time value as a string in ISO 8601 format using time
as the key name. See the example below:
{
"time": "2020-10-15T05:30:47+00:00",
"temperature": 25,
"location": {
"latitude": 32.54,
"longitude": -117.67,
"altitude": 98.6
},
"pressure": 98
}
The time
key is a reserved keyword and hence cannot be used as a measurement key. The time
field must be defined at the root level of the measurement JSON. It is not allowed at any other level, for example inside the object value of a multi-valued measurement. Non-numeric values like the ISO 8601 timestamp string are solely allowed for these reserved keys and not for regular measurements.
Below you find the complete list of reserved keys that serve a special role inside the thin-edge.io framework and hence must not be used as measurement keys:
Key | Description |
---|---|
time | Timestamp in ISO 8601 string format |
type | Internal to thin-edge.io |
Sending measurements to thin-edge.io
The thin-edge.io framework exposes some MQTT endpoints that can be used by local processes to exchange and to forward data to the cloud. It acts as an MQTT broker against which you can write your application logic. Other Thin Edge processes can use this broker as an inter-process communication mechanism by publishing and subscribing to various MQTT topics. Publish the respective data to a standard topic to forward data to the connected cloud provider.
Thin-edge.io reserves all topics with the prefix tedge/
for this purpose. To send measurements to thin-edge.io publish the measurements represented in Thin Edge JSON format to the tedge/measurements
topic. Other processes running on the Thin Edge device can subscribe to this topic to process these measurements.
If the messages published to this tedge/measurements
topic is not a well-formed Thin Edge JSON, thin-edge.io does not process this message. It publishes an appropriate error message including the reason why the validation failed to a dedicated tedge/errors
topic. The messages published to this topic will be highly verbose error messages and can be used for any debugging during development. Do not rely on the structure of these error messages to automate any actions as they are purely textual data and bound to change from time-to-time.
More topics will be added under the tedge/
topic in future to support more data types like events, alarms etc. We recommend you to avoid any sub-topics under tedge/
for any other data exchange between processes.
Below you find the complete list of topics reserved by thin-edge.io for its internal working:
Topic | Description |
---|---|
tedge/ |
Reserved root topic of thin-edge.io |
tedge/measurements |
Topic to publish measurements to thin-edge.io |
tedge/errors |
Topic to subscribe to receive any error messages emitted by thin-edge.io while processing measurements |
Sending measurements to the cloud
The thin-edge.io framework allows you to forward all the measurements generated and published to the tedge/measurements
MQTT topic in the Thin Edge device to any connected IoT cloud provider using a mapper component designed for that particular cloud.
The mapper is used to subscribe to the tedge/measurements
topic in order to receive all incoming measurements represented in Thin Edge JSON format and to format these measurements to a format understood by the connected cloud. Refer to Thin Edge Mapper for more details on the mapper component.
Thin Edge mapper
The tedge-mapper is a key concept to support multiple cloud providers. The purpose is to translate messages written using the cloud-agnostic Thin Edge JSON format, into cloud-specific messages.
The tedge-mapper is composed of multiple cloud-specific mappers, such as Cumulocity IoT mapper and Azure mapper. Each mapper is responsible for its dedicated cloud. These specific mappers are launched by the respective tedge connect
command. For instance, tedge connect c8y
establishes a bridge to Cumulocity IoT and launches a Cumulocity IoT mapper that translates the messages in the background.
A mapper subscribes to the reserved MQTT topic tedge/measurements
with the QoS level 1 at least once. The messages that arrive in the mapper must be formed in the Thin Edge JSON format. The mapper verifies if the arrived messages are correctly formatted. If the verification fails, the mapper publishes a corresponding error message on the topic tedge/errors
with the QoS level 1 at least once.
When the mapper receives a correctly formatted message, it translates the message into a cloud-specific format.
Platform mapper
The Cumulocity IoT mapper translates Thin Edge JSON into Cumulocity IoT’s JSON via MQTT.
The translated messages are published on the topic c8y/measurement/measurements/create
from where they are forwarded to Cumulocity IoT. This mapper is launched by the tedge connect c8y
command, and stopped by the tedge disconnect c8y
command.
Example in Thin-Edge JSON:
{
"temperature": 23
}
Translated into JSON via MQTT by the Cumulocity IoT mapper:
{
"type": "ThinEdgeMeasurement",
"time": "2021-04-22T17:05:26.958340390+00:00",
"temperature": {
"temperature": {
"value": 23
}
}
}
The Cumulocity IoT mapper added the three things which are undefined before the translation:
type
time
- Another hierarchy level. This is required by the Cumulocity IoT data model.
The string
temperature
is used as fragment and series.
(1) The type
is a mandatory field in the Cumulocity IoT JSON via MQTT manner. Therefore, the Cumulocity IoT mapper always adds ThinEdgeMeasurement
as a type. This value is not configurable by users.
(2) The time
is added by the mapper only when it is not specified in a received Thin Edge JSON message. In this case, the mapper uses the device’s local time zone. If you want another time zone, specify the time filed in Thin Edge JSON.
(3) The mapper uses a measurement name, for example “temperature”, as both a fragment type and a fragment series in Cumulocity IoT measurements.
After the mapper publishes a message on the topic c8y/measurement/measurements/create
, the message is transferred to the topic measurement/measurements/create
by the MQTT bridge.
Azure IoT Hub mapper
The Azure IoT Hub mapper takes messages formatted in the Thin Edge JSON as input. It validates if the incoming message is correctly formatted in Thin Edge JSON and outputs the message. The validated messages are published on the topic az/messages/events/
from where they are forwarded to Azure IoT Hub. This mapper is launched by the tedge connect az
command, and stopped by the tedge disconnect az
command.
The Azure IoT Hub mapper processes a message in the following way:
- Validates if it is a correct Thin Edge JSON message or not.
- Validates if the incoming message size is below 255 KB. The size of all device-to-cloud messages must be up to 256 KB, see the Azure IoT Hub Manager at https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-d2c-guidance for more information. The mapper keeps 1 KB as a buffer for the strings added by Azure.
- Adds a current timestamp if a timestamp is not included in an incoming message (default). To stop this behavior, refer to the following instruction:
If the input is the following,
{
"temperature": 23
}
the output of the mapper is
{
"temperature": 23,
"time": "2021-06-01T17:24:48.709803664+02:00"
}
Configure whether adding a timestamp or not
If you do not want to add a timestamp in the output of Azure IoT Hub mapper, you can change the behavior by running the following command:
sudo tedge config set az.mapper.timestamp false
After changing the configuration, you need to restart the mapper service by
sudo systemctl restart tedge-mapper-az.service
Error cases
When an error occurs in a mapper process, the mapper publishes a corresponding error message on the topic tedge/errors
with the QoS level 1 at least once.
Below you find an example for publishing invalid Thin Edge JSON messages on tedge/measurements
:
$ tedge mqtt pub tedge/measurements '{"temperature": 23,"pressure": 220'
$ tedge mqtt pub tedge/measurements '{"temperature": 23,"time": 220}'
You receive error messages from the mapper on the topic tedge/errors
:
$ tedge mqtt sub tedge/errors
[tedge/errors] Invalid JSON: Unexpected end of JSON: {"temperature":23,"pressure":220
[tedge/errors] Not a timestamp: the time value must be an ISO8601 timestamp string in the YYYY-MM-DDThh:mm:ss.sss.±hh:mm format, not a number.
Topics used by tedge-mapper
-
Incoming topics
tedge/measurements
-
Outgoing topics
tedge/errors
(for errors)c8y/measurement/measurements/create
(for Cumulocity IoT)az/messages/events/
(for Azure IoT Hub)
Software management with thin-edge.io
With thin-edge.io you can ease the burden of managing packages on your device. Software management operates end-to-end from a cloud to the OS of your device and reports the statuses accordingly.
Installation
tegde_agent
The tedge_agent is distributed as a Debian package and can be installed with following command:
sudo dpkg -i tedge_agent
The installation adds systemd service tedge-agent.service
and new user specific to the agent (tedge-agent). As some of the operations may require root permissions or sudo access we reccomend you to add the tedge-agent user to the sudo group which allows it to execute elevated commands.
This is executed using the following command:
sudo usermod -aG sudo tedge-agent
To start the agent, use the following command:
sudo systemctl restart tedge-agent
Plugins
SM plugins must be stored in the /etc/tedge/sm-plugins directory.
Architecture FAQ
Why is thin-edge.io an executable binary and not a library?
Interoperability of software components can be addressed along very different approaches. Thin-edge.io uses dynamic and loose Inter-Process Communication (IPC) using messages exchange over an MQTT bus.
In the past and even today, many clouds provide a library (SDK) to help you connect your code to the cloud.
In thin-edge.io we decided not to follow this approach for the following reasons:
- Libraries are programming language dependent.
Therefore, developing a library for a number of programming languages excludes developers using other
programming languages. Additionally the effort to support many libraries (C, C++, Rust, Python, etc) is extensive,
including adding new features, testing, documentation, examples, stack overflow, which would not serve a purpose.
- Using an IPC mechanism and not a library makes it easier to dynamically combine components during runtime instead of recompiling the software. For example, it is easier to add additional protocol stacks (OPC/UA, modbus, ProfiNet, IO-Link, KNX, etc.) to thin-edge.io during run-time.
- Linking libraries to existing code can be problematic for some developers, for example for licensing reasons. While thin-edge.io has a very user-friendly licensing (Apache 2.0), some developers prefer to reduce the number of libraries that they link to their software.
Why does thin-edge.io use MQTT for Inter-Process Communication (IPC)?
MQTT is a lightweight and flexible messaging protocol widely used by IoT applications.
We recommend you to use MQTT for the following reasons:
- The approach is used by other industrial IoT organizations and software, for example by Open Industry 4.0 Alliance.
- Existing components (like Node-RED or collectd) that support MQTT can be easily integrated. In this case, thin-edge.io acts as an MQTT proxy: existing components connect to the local MQTT bus of thin-edge.io, and thin-edge.io routes the messages to different clouds in a secure and reliable manner.
- MQTT is message-oriented and bi-directional, which matches well with the event-oriented programming model of industrial IoT.
- MQTT is available on many platforms, including Linux and Windows.
- MQTT client libraries are available for 25+ programming languages (see MQTT.org)
- MQTT overhead is relatively small in terms of client library size and network overhead.
- MQTT is message-payload-agnostic which enables sending not only JSON messages, but also text, CSV or binary data.
Why does thin-edge.io use MQTT for cloud communication?
MQTT is a lightweight and flexible messaging protocol widely used by IoT applications. Nearly all IoT cloud platforms provide an MQTT endpoint to consume and publish messages from a fleet of devices. Therefore, MQTT was an obvious choice for edge to cloud communication.
Using MQTT for cloud communication is not mandatory. You are free to add additional protocols beside MQTT. Since thin-edge.io has an internal bus, you can implement a bridge to another protocol (e.g. LWM2M or plain HTTPS). In that case, MQTT is used inside the edge devices, and another protocol is used for external communication.
Why is the thin-edge.io canonical format based on JSON?
Thin Edge JSON, the cloud-agnostic message format of thin-edge.io, is based on JSON.
Supported by nearly all programming languages, JSON provides a nice compromise between simplicity and flexibility. Notably, it features duck typing, a flexible way to group different data fields that can be read by consumers with different expectations over the message content. For instance, a consumer expecting a temperature can process messages where the temperature measurements are produced along with other kinds of measurements.
Additionally, JSON is supported by most (if not all) cloud vendors, which makes the transformation easier.
JSON is also used by other IIoT standards, including OPC/UA and LWM2M.
Why use Rust?
The command line interface and the daemon processes of thin-edge.io are implemented in Rust, a language empowering everyone to build reliable and efficient software.
The main reason to use Rust is the security aspect. Rust avoids many security vulnerabilities and threading issues at compile time. With the type system of Rust you write software that is free from typical security flaws: undefined behavior, data races or any memory safety issues.
The second motivation is efficiency. Rust software is typically as efficient as C/C++ software. One reason is that Rust does not have (by default) a garbage collector. Instead, memory lifetime is calculated at compile time.
Note that, even if the core of thin-edge.io is written in Rust, any programming language can be used to implement thin-edge.io components. For that, one just needs an MQTT library that lets them interact with the thin-edge.io MQTT broker.