General aspects

Introduction

Microservices use standard REST APIs with full authentication and authorization to communicate with Cumulocity. They are, in most cases, multi-tenant, i.e. they need to be able to strictly separate tenants and connect to multiple tenants at the same time. Microservices may offer their own endpoints that can be used by Cumulocity and Cumulocity-based applications, e.g. for system integration purposes. Examples of such microservices are the Jasper Control Center integration and the SMS integration for sending SMS notifications to end users.

You can extend the Cumulocity platform with customer-specific functionality by employing microservices. For instance, you can develop integrations to third-party software or provide server-side business logic. A microservice-based architecture introduces change that is often well received by those developing modern applications, and solutions can be delivered much more quickly to those requesting flexible and scalable applications. Microservices bring significant benefits such as deployability, reliability, availability, scalability, modifiability and management.

Cumulocity microservices have the following properties:

The following management features are supported:

Technically, microservices are Docker containers hosted by Cumulocity and they follow specific conventions. They are typically accessed using Cumulocity REST API available under /service/<microservice-name>.

Developers are not restricted to any programming language when developing a microservice for Cumulocity. However, a microservice must serve as an HTTP server working on port 80 and must be encapsulated in a Docker image. Refer to the relevant chapters in this guide for further information for the development of microservices.

Containerization and orchestration

Images and containers

Docker is a platform to develop, deploy and run applications with containers. An image is an executable package that includes everything needed to run an application (i.e. the code, a runtime, libraries, environment variables and configuration files). A container is a runtime instance of an image (i.e. what the image becomes in memory when executed). Refer to the Docker documentation for more information about Docker.

Cumulocity microservices are based on Docker. Hence, a microservice has to be packaged as a Docker image in order to run on the Cumulocity platform. A microservice is executed in a Docker container during runtime. The Docker container ensures that a microservice does not harm other microservices running in Cumulocity.

Containers have an upper thread limit of 4096 for microservices.

Pods

Kubernetes is the container orchestration engine for automating deployment, scaling and management of containerized applications. A Pod is the basic building block of Kubernetes and it represents a running process on your cluster. A Pod encapsulates an application container, storage resources, a unique network IP and options that govern how the container should run.

Docker is the most common container runtime used in a Kubernetes Pod. Moreover, Kubernetes is used to orchestrate Docker containers and it provides many enterprise-grade features for hosting Docker containers such as auto-scaling and load balancing. Refer to the Kubernetes documentation for more information about Kubernetes.

When Docker faces some issues, e.g. a Pod synchronization error, an alarm is created and can be seen in Alarms in the Cockpit application. Refer to Troubleshooting in this section to learn about common issues.

Requirements and interactions

The following requirements towards Cumulocity microservices must be met:

Microservices interact with Cumulocity and the outside world as shown in the following diagram:

Microservices interactions

The Microservices’ lifecycle is managed using the microservice subscription API. This allows registration and subscription of Microservices. External actors (e.g. web user interfaces, integrations or other microservices) can invoke a microservice by sending REST or Websocket requests to its endpoints /service/<microservice-name>/<path>. A microservice can issue requests to external endpoints or to the Cumulocity REST APIs. Microservices can store logs and metrics in the associated developer tenant.

Microservice manifest

The application manifest provides the required settings to manage microservice instances and the application deployment in the Cumulocity platform. The definition is provided within the cumulocity.json file in the binary uploaded to the Cumulocity platform.

Settings

Name Type Description Required
apiVersion String Document type format discriminator, for future changes in format. Yes
name String Application name Yes
contextPath String Microservice contextPath is used to define extension points.
Default: Microservice name
No
version String Application version. Must be a correct SemVer value but the "+" sign is disallowed._ Yes
provider Provider Application provider information. Simple name allowed for predefined providers e.g. c8y. Detailed object for external provider. Yes
billingMode Enum Values: RESOURCES, SUBSCRIPTION
Default: RESOURCES

In case of RESOURCES, the number of resources used is exposed for billing calculation per usage. In case of SUBSCRIPTION, all resources usage is counted for the microservice owner and the subtenant is charged for subscription.
No
isolation Enum Values: MULTI_TENANT, PER_TENANT
Default: MULTI_TENANT

Deployment isolation. In case of PER_TENANT, there is a separate instance for each tenant; otherwise, there is one single instance for all subscribed tenants. Should be overridable on subscription and should affect billing.
No
scale Enum Values: AUTO, NONE
Default: NONE

Enables scaling policy. In case of NONE, the platform guarantees that there is maximally one instance of the service per isolation level.
No
resources Resources Configuration for resources limits.
Guaranteed resources are CPU=0.25, Memory=256MB
Default limits are CPU=0.5, Memory=512MB
No
settings Option[ ] Set of tenant options available to define the configuration of a microservice.
Default: [ ] (empty list)
No
settingsCategory String Allows to specify custom category for microservice settings. By default contextPath is used. No
requiredRoles String[ ] List of permissions required by a microservice to work. Yes
roles String[ ] Roles provided by the microservice.
Default: [ ] (empty list)
No
livenessProbe Probe Defines the strategy used to verify if a microservice is alive or requires a restart. No
readinessProbe Probe Defines the strategy used to verify if a microservice is ready to accept traffic. No
extensions Extension[ ] Defines a set of extensions that should be enabled for a microservice.
Default: [ ] (empty list)
No

Version

The version has an impact on the microservice upload behavior:

The snapshot postfix means that the image build is a snapshot of your application at a given time and it is still under development. When your microservice is ready for production release, you can remove the postfix and just use the final version of your application.

Provider

Name Type Description Required
name String Company name of the provider Yes
domain String Website of the provider No
support Email Email of the support person No

Resources

Name Type Description Required
cpu String Limit for number of CPUs or CPU time
Default CPU: 0.5, min: 0.1
Default CPU time: 500m, min: 100m
No
memory String Limit for microservice memory usage
Default: 512M, Min: 10M
Possible units are: E, P, T, G, M, K, Ei, Pi, Ti, Gi, Mi, Ki
No

Option

Name Type Description Required
key String Key of the option Yes
defaultValue String Default value No
editable Boolean Defines if the option can be changed by a subscribed tenant on runtime
Default: false
No
valueSchema Schema Defines schema of value and follows the JSON schema defined here
Default: {type: string}
No

Probe

Name Type Description Required
exec ExecAction Commands to be executed on a container to probe the service No
tcpSocket TCPSocketAction TCP socket connection attempt as a probe No
httpGet HTTPGetAction HTTP request to be executed as a probe No
initialDelaySeconds Number Tells the platform for how long it should wait before performing
the first probe
Default: 200
No
periodSeconds Number Defines in which interval the probe should be executed
Default: 10
No
successThreshold Number Minimum consecutive successes for the probe to be considered
successful after having failed
Default: 1
No
timeoutSeconds Number Number of seconds after which the probe times out
Default: 1
No
failureThreshold Number Number of failed probes after which an action should be taken
Default: 3
No

ExecAction

Name Type Description Required
command String[ ] Commands to be executed on a container to probe the service Yes

TCPSocketAction

Name Type Description Required
host String Host to verify Yes
port Number Port to verify
Default:80
Yes

HTTPGetAction

Name Type Description Required
host String Host name to connect to Yes
path String Path to access on the HTTP server Yes
port Number Port to verify
Default: 80
No
scheme String Scheme to use for connecting to the host (HTTP or HTTPS)
Default: HTTP
No
headers HttpHeader HTTP headers to be added to a request No

HttpHeader

Name Type Description Required
name String Header name Yes
value String Header value Yes

Extension

Name Type Description Required
type String Type ID of the extension Yes
* any Configuration parameters No
Example Manifest
{
    "apiVersion": "v1",
    "name": "my-microservice",
    "version": "1.0.0",
    "provider": {
        "name": "New Company Ltd.",
        "domain": "http://new-company.com",
        "support": "support@new-company.com"
    },
    "isolation": "MULTI_TENANT",
    "scale": "AUTO",
    "resources": {
        "cpu": "1",
        "memory": "1G"
    },
    "requiredRoles": [
        "ROLE_ALARM_READ"
    ],
    "livenessProbe": {
        "httpGet": {
            "path": "/health"
        },
        "initialDelaySeconds": 60,
        "periodSeconds": 10
    },
    "readinessProbe": {
        "httpGet": {
            "path": "/health",
            "port": 80

        },
        "initialDelaySeconds": 20,
        "periodSeconds": 10
    },
    "settingsCategory": "my-ms",
    "settings": [
        {
            "key": "tracker-id",
            "defaultValue": "1234"
        }
    ]
}

Isolation and scaling

The following isolation levels are available for microservices:

The isolation level is set using the Microservice manifest.

If scaling is enabled, the microservice will be horizontally auto-scaled in case of high CPU usage. Auto-scaling monitors the microservices to make sure that they are operating at the desired performance levels, and it will automatically scale up your cluster as soon as you need it and scale it back down when you don’t.

The scale option is set using the Microservice manifest.

Security

Microservices typically provide a REST API and Cumulocity provides a light API gateway (“Proxy”) for inbound REST requests. Inbound WebSocket requests are supported. The API gateway – located between the client and the microservice container – provides:

Authentication and authorization

A request to a microservice can be authenticated using basic authentication or OAuth. In case of basic authentication the flow is fairly simple, as credentials can be read and utilized for further authentication to the platform.

Authentication with OAuth is based on cookies technology, so the access token has to be read from the request cookie header. There are two important parts of OAuth authorization: an access token stored in the authorization cookie and an X-XSRF-TOKEN header for XSRF attack prevention. Both must be forwarded with the request to the platform. If you use Java for development, we recommend using the Microservice SDK Version 9.12.6 or later for supporting OAuth in Java microservices.

OAuth

Refer to the OAuth Community Site for more details about the OAuth authorization framework.

Microservice authentication and multi-tenancy

In general, microservices use the standard Cumulocity authentication mechanisms. This is performed in two steps:

  1. The microservice can be created in any tenant that have feature-microservice-hosting enabled.
  2. The microservices access the Tenant API.

At installation time of the microservice, an application is created in the management tenant reflecting the new microservice. In addition, a service user is created in the management tenant that allows the microservice to retrieve subscriptions. Whenever required, a platform administrator will subscribe a customer to the new microservice. As part of the subscription, a service user in the customer tenant is created using random credentials.

Microservice authorization

Authorization is relevant on two levels:

  1. On the management tenant level, the only authorization of a microservice is to access its own subscriptions.
  2. For accessing customer tenants, the microservice installs a set of required permissions for being able to operate.

A microservice is associated with a service user in the management tenant, which will make sure that only its subscriptions are returned. A microservice is also associated with a set of permissions that it requires for carrying out its function on a customer tenant. These permissions are visualized in the Administration application. The permissions are associated with the service user that is created when a platform administration associates a microservice with a tenant.

Users and roles

There are three types of users:

The following role types are defined for users:

The roles are set in the Microservice manifest. For more details about users and roles, review Users in the Reference guide.

Microservice bootstrap

Each microservice receives a dedicated bootstrap user which ensures that a microservice can be identified by the platform and can have access only to the allowed resources. A microservice runtime provides bootstrap user and service user credentials as environment variables which can also be acquired via platform API. Note that depending on the isolation level, the environment variables differ.

Variable Description Per tenant scope Multi-tenant scope
C8Y_BOOTSTRAP_TENANT Application owner tenant ID x x
C8Y_BOOTSTRAP_USER Username of the bootstrap user x x
C8Y_BOOTSTRAP_PASSWORD Password of the bootstrap user x x
C8Y_TENANT Subscribed tenant ID x  
C8Y_USER Username of the service user of a subscribed tenant x  
C8Y_PASSWORD Password of the service user of a subscribed tenant x  

In multi-tenant scope, there is a single microservice deployment reused by multiple tenants and that is why service user credentials are not provided as hardcoded environment properties. However, a microservice running in multi-tenant isolation can retrieve all subscriptions via a GET request and using bootstrap credentials as follows:

GET /application/currentApplication/subscriptions
Host: ...
Authorization: Basic ...

Bootstrap user credentials can be retrieved with a GET request authorized with application owner credentials:

GET /application/applications/<APPLICATION_ID>/bootstrapUser
Host: ...
Authorization: Basic ...

An example of a typical user switching in multi-tenant isolation is presented below, where – in a hypothetical scenario – there is a need to send an alarm to each tenant subscribed to a microservice.

microservice_user_switch_example

The user wants to employ microservice capabilities to raise alarms to all subscribed tenants calls.

Steps:

  1. The user makes a request to the platform’s endpoint /service/<microservice>/createAlarms.
  2. The platform verifies the user credentials and redirects the request to the microservice.
  3. The microservice reads the bootstrap credentials (from environment variables) and uses them to retrieve the service user credentials for all subscribed tenants.
  4. The microservice iterates over the service user credentials and uses them to create alarms to each tenant.
  5. The microservice returns the result to the platform, and the platform to the invoking user.

Encryption

There is a mechanism to encrypt the tenant options that afterwards are automatically decrypted when injecting them into microservices requests. Refer to Option collection in the Tenants section of the Reference guide for more details.

If a tenant option is created at any category that starts with credentials., it is automatically encrypted and can be only fetched as unencrypted by “System users”. For instance, when you create a tenant option in “category” that matches to the application context path, the value is passed to the microservice by the microservice proxy on the platform as a header (key => value). All encrypted options are decrypted and passed. Moreover, the options can be fetched via REST using the options endpoint at microservice runtime.

Microservice runtime

Microservices deployed on the platform have a specific runtime environment and they need to understand certain details about the specific Cumulocity cluster they run in. For example, a microservice needs to know the endpoint address of the Cumulocity REST APIs. This information is provided by environment variables and they are injected by Cumulocity when the container is started.

Environment variables

The following environment variables are available for microservices:

Name Details
APPLICATION_NAME The name of the microservice application
SERVER_PORT Default open port (80)
MICROSERVICE_SUBSCRIPTION_ENABLED Default value: true
C8Y_BASEURL Platform address (contains port number)
C8Y_BASEURL_MQTT Platform address of the MQTT server (contains port number)
C8Y_MICROSERVICE_ISOLATION Isolation level (MULTI_TENANT or PER_TENANT)
C8Y_BOOTSTRAP_REGISTER Indicator whether the microservice should perform self registration or not.
Default value: false
C8Y_BOOTSTRAP_TENANT Bootstrap user tenant, for MULTI_TENANT - microservice owner
C8Y_BOOTSTRAP_USER Bootstrap user name
C8Y_BOOTSTRAP_PASSWORD Bootstrap user password
C8Y_TENANT Application user tenant (available only for PER_TENANT isolation)
C8Y_USER Application user name (available only for PER_TENANT isolation)
C8Y_PASSWORD Application user password (available only for PER_TENANT isolation)
MEMORY_LIMIT Max memory that can be used. Default value: 256M
TZ Timezone from the host machine or configurable tenant options
Example

Prerequisite: The microservice has been packed and deployed in the Docker repository. Get the microservice image name and tag with the following command:

$ docker images

Run the Docker container for the microservice providing the environment variables:

$ docker run -e C8Y_BASEURL=<URL> -e C8Y_BOOTSTRAP_TENANT=<BOOTSTRAP_TENANT> -e C8Y_BOOTSTRAP_USER=<BOOTSTRAP_USERNAME> -e C8Y_BOOTSTRAP_PASSWORD=<BOOTSTRAP_USER_PASSWORD> -e C8Y_MICROSERVICE_ISOLATION=MULTI_TENANT -i -t <DOCKER_REPOSITORY_IMAGE>:<TAG>

Use a backslash (\) before special characters such as &, !, ;, \.

Timezone variable

The timezone variable allows configuring a default timezone used by the microservice. The microservice installer injects the TZ environment variable into the microservice according to the following settings:

The tenant option has higher priority, i.e. if the parameter is set in both places, the value from the tenant option is taken.

Example

Assuming that the microservice owner has the tenant option:

{
    "category" : "microservice.runtime",
    "key" : "timezone",
    "value" : "Europe/Warsaw"
}

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment:

TZ=Europe/Warsaw

When using Java-based microservices this variable is automatically read and applied to the Java process, no additional work is required. Microservices developed with other programming languages may require some manual work, i.e. loading the TZ value from the environment and using it to configure the time zone on the language level programmatically.

Proxy variables

Proxy variables are used to set a proxy URL for different protocols. For the microservices written in Java, setting each variable will result in passing the corresponding parameter into the JVM runtime (for detailed information see the Java Networking and Proxies webpage).

Proxy variables are passed into the microservice environment during installation. The microservice installer passes the variables into the environment according to the following settings:

Tenant options have higher priority, i.e. if the parameter is set in both places, the value from the tenant option is taken.

The table below contains the options and proxy variables:

Tenant option Platform env variable Microservice env variable
proxy.http.host MICROSERVICE_RUNTIME_PROXY_HTTP_HOST PROXY_HTTP_HOST
proxy.http.port MICROSERVICE_RUNTIME_PROXY_HTTP_PORT PROXY_HTTP_PORT
proxy.http.non.proxy.hosts MICROSERVICE_RUNTIME_PROXY_HTTP_NON_PROXY_HOSTS PROXY_HTTP_NON_PROXY_HOSTS
proxy.https.host MICROSERVICE_RUNTIME_PROXY_HTTPS_HOST PROXY_HTTPS_HOST
proxy.https.port MICROSERVICE_RUNTIME_PROXY_HTTPS_PORT PROXY_HTTPS_PORT
proxy.socks.host MICROSERVICE_RUNTIME_PROXY_SOCKS_HOST PROXY_SOCKS_HOST
proxy.socks.port MICROSERVICE_RUNTIME_PROXY_SOCKS_PORT PROXY_SOCKS_PORT

All tenant options have the same category: microservice.runtime

For each protocol (HTTP, HTTPS, Socks), microservice environment variables are passed into runtime only if the HOST parameter is set. If the HOST parameter is missing, other parameters for the same protocol are not processed.

Examples

The microservice owner tenant has the tenant options:

{ "category" : "microservice.runtime", "key" : "proxy.http.host", "value" : "10.11.12.13" }
{ "category" : "microservice.runtime", "key" : "proxy.http.port", "value" : "8080" }

and there is an environment variable in the platform application:

MICROSERVICE_RUNTIME_PROXY_HTTP_PORT=8181

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment (notice PORT value):

PROXY_HTTP_HOST=10.11.12.13
PROXY_HTTP_PORT=8080

The microservice owner tenant has the tenant option:

{ "category" : "microservice.runtime", "key" : "proxy.https.host", "value" : "10.11.12.13" }

and there is an environment variable in the platform application:

MICROSERVICE_RUNTIME_PROXY_HTTPS_PORT=8181

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment:

PROXY_HTTP_HOST=10.11.12.13
PROXY_HTTP_PORT=8181        

The microservice owner tenant has the tenant options:

{ "category" : "microservice.runtime", "key" : "proxy.http.port", "value" : "8080" }
{ "category" : "microservice.runtime", "key" : "proxy.http.non.proxy.hosts", "value" : "localhost" }

and the proxy HOST is not set (neither in tenant option, nor env variable).

Deploying and running the microservice inside Docker will not pass any proxy environment variable.


The microservice owner tenant has the tenant option:

{ "category" : "microservice.runtime", "key" : "socks.http.host", "value" : "10.11.12.13" }  

Deploying and running the microservice inside Docker will result in passing the following variable into the microservice environment (only host parameter):

SOCKS_HTTP_HOST=10.11.12.13

Platform access and other microservices

To execute requests against the Cumulocity platform running a microservice, you have to send requests to the host specified by the C8Y_BASEURL variable.

A microservice does not have direct access to other microservices running on the platform. Instead, a microservice must use the platform as a proxy. The endpoint used to access other applications is <C8Y_BASEURL>/service/<OTHER_APPLICATION_NAME>/.

Request routing

The request is redirected to a microservice depending on the isolation level (auto-scaling is ignored at this moment for clarity), subscription and authorization. A typical request to the platform looks like

<METHOD>  /service/<MICROSERVICE>/<MICROSERVICE_ENDPOINT>
Host: ...
Authorization: Basic ...

Credentials are used to verify if a requesting user is authorized to access the microservice, and tenant subscription is verified afterwards. If both checks pass, the request is routed to a dedicated microservice deployment in case that the isolation level is per tenant, or to a shared deployment in case of multi-tenant isolation.

The routed request is stripped of /service/<MICROSERVICE> part. However, the Authorization header is not modified, thus a request is still executed as a tenant platform user.

<METHOD>  /<MICROSERVICE_ENDPOINT>
Host: ...
Authorization: Basic ...

Microservice utility tool

Cumulocity provides you a utility tool for easy microservice packaging, deployment and subscription. The script requires a local installation of Docker and jq, which is a lightweight and flexible JSON processor.

Prerequisites

Docker

Verify that you have a Docker installation. The Docker version must be >= 1.2.6

$ docker version
Client:
 Version:         1.12.6
 API version:     1.24
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 OS/Arch:         linux/amd64

JSON processor

Execute the following command to install the JSON processor on Linux systems:

$ sudo yum install jq

For macOS, use the following command:

$ brew install jq

Bash

The microservice utility tool (script) needs Bash version 4+ to run. Verify your Bash version with the following command:

$ bash --version

GNU bash, version 5.0.3(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.

macOS systems come with a preinstalled Bash version 3.x. Hence, you must update it in order to execute the microservice script. To do so, execute the following commands:

$ brew install bash
$ chsh -s /usr/local/bin/bash

If your Bash version has not changed while executing bash --version, you may need to restart your system. Note that the updated interpreter gets installed at /usr/local/bin/bash and you will have to modify the first line of the microservice utility tool (script) as follows:

#!/usr/local/bin/bash

or

#!

Configure the microservice utility tool

Execute the following command to download the script on Linux systems:

$ wget http://resources.cumulocity.com/examples/microservice

For macOS, use the following command:

$ curl -O http://resources.cumulocity.com/examples/microservice

Change the mode to allow the script to be executed:

$ chmod +x microservice

Use the help option to see all the available functions (goals) and options:

$ ./microservice help

Packing

A microservice has to be packed as a Docker image in order to be deployed. It requires a Docker image.tar and cumulocity.json files packed into a ZIP file.

The following directory structure is required to pack a microservice:

/docker/Dockerfile      # Instructions to build the Docker image
/docker/*               # All files within the directory will be included in the Docker build
/cumulocity.json        # The application manifest file

The script can be run in a parent folder holding such structure, or by passing the path to the directory using the -dir option. For instance, to pack a “Hello World” microservice application, execute:

$ ./microservice pack --name hello-world

It will create a ZIP file named hello-world.zip and an intermediate image.tar which is an exported Docker image.

Deploying

A microservice becomes available once it has been successfully deployed on the Cumulocity platform. This is done by uploading a ZIP file with the microservice packed as specified above. A user cannot directly push an image to the Docker registry.

Deploying your microservice application is rather easy, just execute the following command:

$ ./microservice deploy -n hello-world -d <URL> -u <username> -p <password> -te <tenant>

Note that you need to have a tenant and user credentials in order to deploy your microservice.
The successful execution will create an application on the Cumulucity platform with the specified name, if it does not exist yet. Then it will upload the hello-world.zip file into the platform. Once it has been uploaded, your application will be listed on Applications > Own Applications in the Administration application.

For further information on deploying microservices to Cumulocity, refer to Administration > Managing applications in the User guide.

Subscribing

You need to subscribe to the application in order to use it. Execute the following command to subscribe your tenant to the deployed microservice:

$ ./microservice subscribe -n hello-world -d <URL> -u <username> -p <password> -te <tenant> -id <APPLICATION_ID>

It will result in tenant subscription to an application specified by the ID parameter. If the user has already been subscribed, a warning message will be displayed.

Multiple goals

Goals can be executed together to pack, deploy and subscribe the application in a single line. In this case, the application ID will be automatically pulled by the script.

$ ./microservice pack deploy subscribe -n hello-world -d <URL> -u <username> -p <password> -te <tenant>

Operating microservices

Cumulocity manages microservices by monitoring the microservice instance and storing the metrics. In case a microservice exceeds the memory limit, it is restarted automatically. Also, microservices can be auto-scaled in case of high CPU usage. For more information, review the scaling details above.

Troubleshooting

Some common issues have been identified and documented below.

I deployed my microservice but requests to any endpoint returns an error message “Microservice not available Connection refused”

After uploading the microservice, the internal deployment and container run may take a couple of minutes. Once completed, the error message will disappear. Meanwhile, enjoy your coffee ☕️

An alarm was created with the message “Failed to pull image … rpc error: code = Canceled desc = context canceled”

This may happen occasionally when pulling large Docker images for containers within a pod. It will go to normal once it has been successfully pulled.

An alarm was created with the message “Pod synchronization error.”

This may happen when Kubernetes is performing the auto-scaling and is trying to restart the pods.

An alarm was created with the message “No nodes are available that match all of the predicates: [Insufficient cpu …]”

There are additional containers running besides yours and they are provisioned by default with Kubernetes. These additional containers might be taking up of the CPU quota of the single node. Kubernetes will manage this and change the pod status from Pending to Running.