Troubleshooting and diagnostics

Downloading diagnostics and logs

To download diagnostics information, you need READ permission for “CEP management”. See Managing permissions for more information.

If you have READ permission for “CEP management”, links for downloading diagnostics information are available when you click the User button in the Streaming Analytics application. This opens the right drawer which contains a Diagnostics section with the following links:

• A Basic diagnostics (ZIP) link for downloading basic diagnostics information. This should typically be a few megabytes and take about 5 seconds to generate.
• An Enhanced diagnostics (ZIP) link for downloading enhanced, more resource-intensive diagnostics information.

It may be useful to capture this diagnostics information when experiencing problems, or for debugging EPL apps. It is also useful to provide to product support if you are filing a support ticket. You can find the tenant ID in the Platform info section of the right drawer. If you want to find out the version numbers of the different components on your tenant, click the Download platform details button in the Platform info section and then open the downloaded JSON file. See User options and settings for more details.

Basic diagnostics information is provided in a ZIP file named diagnostic-overview<timestamp>.zip and includes the following information:

• The microservice log file contents, if available, including a record of the correlator’s startup logging and the last hour or maximum of 20,000 lines of logging.

  Info

  In case of Cumulocity Edge Applicance VM, since Apama-ctrl is not deployed as a microservice in the VM-based distribution, Apama logs can be retrieved using the diagnostic utility. For more details, see Apama log file locations and Diagnostic utility.

• Apama-internal diagnostics information (similar to the engine_watch and engine_inspect command-line tools available in Apama).

• A copy of all EPL apps, smart rules and analytic models.

• A copy of any alarms that the Apama-ctrl microservice has raised.

• CPU profiling (over a duration of 5 seconds).

• EPL memory profiler snapshots.

• Some information from the environment (tenant details, environment variables).

• Version information for the components.

Enhanced diagnostics information is provided in a ZIP file named diagnostic-enhanced<timestamp>.zip and includes the following information:

• Contains what is in the above-mentioned diagnostic-overview<timestamp>.zip file.
• In addition, it includes requests that are more resource-intensive and may significantly slow down the correlator. These include the contents of the queues, CPU usage, and so on.

What you can see or do depends on your permissions:

• If you have only READ permission for “CEP management”, you have read-only access to EPL apps and analytic models and you can access the diagnostics information.
• Without ADMIN permission for “CEP management”, you are not able to activate or edit EPL apps or analytic models.
• If you have both READ and ADMIN permissions for “CEP management”, you have read-write access and you can access the diagnostics information.
• If you have only ADMIN permission for “CEP management” and no READ permission, you are able to load, edit and deploy EPL apps and analytic models, but you are not able to see or access the diagnostics information.

Log files of the Apama-ctrl microservice

There are two ways to get the logs of the Apama-ctrl microservice:

• You can download diagnostics information from the Streaming Analytics application as described in Downloading diagnostics and logs.

• In some cases, it is useful to view the log file of the Apama-ctrl microservice directly in Cumulocity. The log file is accessible via the Administration application. You can find it on the Logs tab of the Apama-ctrl microservice. You must subscribe to the microservice so that you can see the logs. For more information on microservices and log files, see Managing microservices and Monitoring microservices.

  Info

  In case of Cumulocity Edge Applicance VM, since Apama-ctrl is not deployed as a microservice in the VM-based distribution, the log file can be retrieved using the diagnostic utility. For more details, see Apama log file locations and Diagnostic utility.

The correlator log is embedded in the log file of the Apama-ctrl microservice. See also Descriptions of correlator status log fields in the Apama documentation.

Contact product support if needed.

Audit logs for Streaming Analytics

Activation and deactivation of analytic models and EPL apps is shown in the audit logs. The audit logs are accessible via the Administration application and the audit API. See Audit logs and Audit API in the Cumulocity OpenAPI Specification for details of accessing the audit logs.

Audit log entries include the current action and the name of the user performing that action. For example:

Diagnostics REST endpoints

The following diagnostics endpoints are available for REST requests. These require authentication as a user with READ permission for “CEP management”:

• /service/cep/diagnostics/metrics
  GET only. Plain text format.
  Prometheus metrics from the correlator. For details, see Monitoring with Prometheus in the Apama documentation.
• /service/cep/diagnostics/overview
  GET only. ZIP file download.
  Obtains the diagnostic-overview<timestamp>.zip file as described above.
• /service/cep/diagnostics/enhanced
  GET only. ZIP file download.
  Obtains the diagnostic-enhanced<timestamp>.zip file as described above.
• /service/cep/diagnostics/request
  PUT only. JSON.
  Gives access to generic management requests against the correlator. For details, see Shutting down and managing components in the Apama documentation.
• /service/cep/diagnostics/correlator/info
  GET only. JSON.
  Obtains the engine_inspect information.

Monitoring REST endpoints

The following monitoring endpoints are available for REST requests. These require authentication as a valid user, but do not require any special roles.

• /service/cep/health
  GET only. JSON.
  Obtains information on whether the Apama-ctrl microservice is up or not.

  Info

  For Apama-ctrl-smartrulesmt (that is, the multi-tenant variant of the Apama-ctrl-smartrules microservice), only basic microservice status values are provided.

• /service/cep/prometheus
  GET only. Plain text format.
  Prometheus metrics from the correlator and microservice. For details, see Monitoring with Prometheus in the Apama documentation.

Alarms generated by the Apama-ctrl microservice

Alarms are created by user applications in the Cumulocity tenant (for example, by an analytic model, an activated EPL file, or a smart rule). To learn about alarms in general, refer to Working with alarms. The Apama-ctrl microservice also generates alarms because it has encountered some problem, so that the user is notified about the situation. The information below is about alarms that are generated by the Apama-ctrl microservice, their causes, consequences and possible ways to resolve them.

You can view alarms in the following ways:

1. In the Cockpit application. See Cockpit for detailed information.
2. In the Administration application, under Ecosystem > Microservices. Click the Apama-ctrl microservice and then click Status. See Monitoring microservices for detailed information.
3. From the Streaming Analytics application. Click the User button to display the right drawer and then click one of the links in the Diagnostics section to download a ZIP file that contains alarms information under /alarm/alarms_apama-ctrl-object.json. See Downloading diagnostics and logs for detailed information.

Alarm severities

Alarms created by the Apama-ctrl microservice

Apama-ctrl can create alarms to notify users in scenarios such as the correlator running out of memory, uncaught exceptions in activated EPL files, and so on. Once you see an alarm in the Cumulocity tenant, you should diagnose it and resolve it depending on the severity level of the raised alarm. Each alarm has details such as title, text, type, date, and count (represents the number of times the alarm has been raised).

The following is a list of the alarms. The information further down below explains when these alarms will occur, their consequences, and how to resolve them.

• Change in tenant options and restart of Apama-ctrl
• Safe mode on startup
• Deactivating models in the Apama-ctrl-starter microservice
• High memory usage
• Warning or higher level logging from an EPL file
• An EPL file throws an uncaught exception
• An EPL app is running in an infinite or long-running loop
• EPL app restore timeout on restart of Apama-ctrl
• Multiple extensions with the same name
• Smart rule configuration failed
• Smart rule restore failed
• Connection to correlator lost
• Performance alarms
• Parent tenant not subscribed
• Analytics Builder dropped events

Once the cause of an alarm is resolved, you must acknowledge and clear the alarm in the Cumulocity tenant. Otherwise, you will continue to see the alarm until a further restart of the Apama-ctrl microservice.

Change in tenant options and restart of Apama-ctrl

This alarm is raised when a tenant option changes in the analytics.builder or streaminganalytics category. For details on the tenant options, refer to the Tenant API in the Cumulocity OpenAPI Specification for more details.

• Alarm type: tenant_option_change
• Alarm text: Detected changes in tenant option. Apama-ctrl will restart in order to use it.
• Alarm severity: MAJOR

Analytics Builder allows you to configure its settings by changing the tenant options, using key names such as numWorkerThreads or status_device_name. For example, if you want to process things in parallel, you can set numWorkerThreads to 3 by sending a REST request to Cumulocity, which will update the tenant option. Such a change automatically restarts the Apama-ctrl microservice. To notify the users about the restart, Apama-ctrl raises an alarm, saying that changes have been detected in a tenant option and that Apama-ctrl will restart in order to use it.

Once you see this alarm, you can be sure that your change is effective.

Safe mode on startup

This alarm is raised whenever the Apama-ctrl microservice switches to safe mode.

• Alarm type: apama_safe_mode
• Alarm text: Apama-ctrl appears to be repeatedly restarting. As a precaution, user-provided EPL, analytic models and extensions that might have caused this have been disabled. Refer to the audit log for more details. Please check any recent alarms, or contact support or your administrator.
• Alarm severity: CRITICAL

Apama-ctrl detects if it has been repeatedly restarting and if user assets (EPL apps, analytic models, extensions) have been modified recently. Apama-ctrl disables all user assets as a precaution. Potential causes are, for example, an EPL app that consumes more memory than is available or an extension containing bugs.

You can check the mode of the microservice (either normal or safe mode) by making a REST request to service/cep/diagnostics/apamaCtrlStatus, which contains a safe_mode flag in its response.

To diagnose the cause of an unexpected restart, you can try the following:

• Check the EPL apps memory profiler by making a REST request to /service/cep/diagnostics/eplMemoryProfiler for any memory leaks. Note that you must re-activate the EPL apps that were active before as the Apama-ctrl microservice loses information about the previous microservice instance when it restarts due to safe mode. To replicate the previous scenario, run the EPL apps and process some events to trigger a leak and then use the memory profiler to check for any memory leaks.

• Check the microservice logs for any exceptions by downloading the basic diagnostics ZIP file as described in Downloading diagnostics and logs. In the downloaded ZIP file, you can find the logs under /diagnostics/. As mentioned in the above point, re-activate the EPL apps and analytic models that were active before and then check the logs.

• Check the audit logs. The audit logs are accessible via the Administration application and the audit API. See Audit logs and Audit API in the Cumulocity OpenAPI Specification for details of accessing the audit logs.

In safe mode, all previously active analytic models and EPL apps are deactivated and must be manually re-activated.

Deactivating models in the Apama-ctrl-starter microservice

This alarm is raised when Apama-ctrl switches from the fully capable microservice to the Apama-ctrl-starter microservice with more than 3 active models.

• Alarm type: apama_ctrl_starter
• Alarm text: The following models were de-activated as Analytics Builder is restricted to <activate limit> active models: (<models>).
• Alarm severity: MINOR

With the Apama-ctrl-starter microservice, a user can have a maximum of 3 active models. For example, a user is working with the fully capable Apama-ctrl microservice and has 5 active models, and then switches to Apama-ctrl-starter. Since Apama-ctrl-starter does not allow more than 3 active models, it deactivates all the active models (5) and raises an alarm to notify the user.

High memory usage

This alarm is raised whenever the Apama-ctrl microservice consumes 90% of the maximum memory permitted for the microservice container. During this time, the Apama-ctrl microservice automatically generates the basic diagnostics ZIP file which contains diagnostics information used for identifying the most likely cause for memory consumption.

There are 3 variants of this alarm, depending on the time and count restrictions of the generated basic diagnostics ZIP file.

First variant:

• Alarm type: apama_highmemoryusage
• Alarm text: Streaming Analytics is using 90% of available memory (<totalMemory>). Your apps will be in danger of crashing. Diagnostics file is located at <URL-to-ZIP-file> You can also download the file by navigating to Administration > Management > Files Repository
• Alarm severity: WARNING

Second variant:

• Alarm type: apama_highmemoryusage
• Alarm text: Streaming Analytics is using 90% of available memory (<totalMemory>). Your apps will be in danger of crashing. Have recently created diagnostics snapshot (within last hour).
• Alarm severity: WARNING

Third variant:

• Alarm type: apama_highmemoryusage
• Alarm text: Streaming Analytics is using 90% of available memory (<totalMemory>). Your apps will be in danger of crashing. Have created 5 diagnostics snapshots, not creating any more, refer to past alarms.
• Alarm severity: WARNING

Running EPL apps (and to a lesser extent, smart rules and analytic models) consumes memory, the amount will depend a lot on the nature of the app running. The memory usage should be approximately constant for a given set of apps, but it is possible to create a “memory leak”, particularly in an EPL file or a custom block. The Apama-ctrl microservice monitors memory and raises an alarm with WARNING severity if the 90% memory limit is reached along with the basic diagnostics ZIP file and saves it to the files repository (as mentioned in the alarm text).

Apama-ctrl generates the basic diagnostics ZIP files with the following conditions:

• Only if it has not been generated in the last 1 hour.
• A maximum of 5 basic diagnostics ZIP files from its start time until it stops.
• Overall, it can generate a maximum of 20 ZIP files per Cumulocity tenant, beyond which it keeps deleting the oldest ZIP files during its startup process.

To diagnose high-memory-consuming models and EPL apps, you can try the following (it could be listener leaks, excessive state being stored or spawned monitors leaking, and so on):

• Download the automatically generated basic diagnostics ZIP file (refer to the alarm text for its location) and look at correlator/inspect.json and correlator/status.json for the number of listeners. This number may be large in the case of a listener leak. Note that this ZIP file includes the EPL memory profiler snapshots.

• Download the diagnostics information from the Streaming Analytics application. Click the User button to display the right drawer and then click one of the links in the Diagnostics section (as described in Downloading diagnostics and logs). The EPL memory profiler from the Basic diagnostics (ZIP) link in /diagnostics/eplMemoryProfiler.csv gives the memory consumed by each monitor along with details such as the number of listeners or the number of monitor instances running something like shown in the snippet below. This can help you to understand which monitor is consuming more memory and try to reduce it.

  Monitor	Monitor instances	EPL objects	Listeners	Bytes	Overhead bytes
  mon1	1	5384	4	1073908	383240
  mon2	1	2	2	696	2280
  mon3	1	4	1	840	752

• When using the Enhanced diagnostics (ZIP) link in the Streaming Analytics application, the diagnostics information includes, in addition to the information that you get with the Basic diagnostics (ZIP) link, requests that are more resource-intensive and may significantly slow down the correlator. This includes the contents of the queues. So when diagnosing the cause for the first time, it is recommended to use the ZIP file from the Basic diagnostics (ZIP) link, unless additional information is required.

• Also check for memory usage on all the input and output queues available from the Enhanced diagnostics (ZIP) link in /diagnostics/toStringQueues.txt.

If the memory continues to grow, then when it reaches the limit, the correlator will run out of memory and Apama-ctrl will shut down. To prevent the microservice from going down, you must fix this as a priority.

See also Diagnostic tools for Apama in Cumulocity in Software AG’s Tech Community.

Warning or higher level logging from an EPL file

This alarm is raised whenever messages are logged by EPL files with specific log levels (including CRITICAL, FATAL, ERROR and WARNING).

The Streaming Analytics application allows you to deploy EPL files to the correlator. The Apama-ctrl microservice analyzes logged content in the EPL files and raises an alarm for specific log levels with details such as monitor name, log text and alarm type (either of WARNING or MAJOR), based on the log level.

For example, the following is a simple monitor which prints a sequence and logs some texts at different EPL log levels.

monitor Sample{
   action onload() {
      log "Info"; // default log level is now INFO
      log "Fatal Error" at FATAL; // log level is FATAL
      log "Critical Error" at CRIT; // log level is CRITICAL
      log "Warning" at WARN; // log level is WARNING
   }
}

Apama-ctrl analyzes all the log messages, filters out only certain log messages, and raises an alarm for the identified ones. Thus, Apama-ctrl generates the following three alarms for the above example:

First alarm:

• Alarm type: APAMA_CTRL_FATAL_<HASHCODE>
• Alarm text: <Monitor name>-Fatal Error.
• Alarm severity: MAJOR

Second alarm:

• Alarm type: APAMA_CTRL_CRIT_<HASHCODE>
• Alarm text:<Monitor name>-Critical Error.
• Alarm severity: MAJOR

Third alarm:

• Alarm type: APAMA_CTRL_WARN_<HASHCODE>
• Alarm text: <Monitor name>-Warning.
• Alarm severity: WARNING

An EPL file throws an uncaught exception

You have seen that the Apama-ctrl microservice raises alarms for logged messages. In addition, there can also be uncaught exceptions (during runtime). Apama-ctrl identifies such exceptions and raises alarms so that you can identify and fix the problem.

For example, the following monitor throws IndexOutOfBoundsException during runtime:

monitor Sample{
   sequence<string> values := ["10", "20", "30"];
   action onload() {
      // IndexOutOfBoundsException (runtime error)
      log "Value = " + values[10] at ERROR;
   }
}

Apama-ctrl generates the following alarm for the above example:

• Alarm type: APAMA_CTRL_ERROR_<HASHCODE>
• Alarm text: <Monitor name>-Error on line <x> of monitor : IndexOutOfBoundsException - Out of bounds index passed to sequence [] operator - Sample
• Alarm severity: MAJOR

You can diagnose the issue by the monitor name and line number given in the alarm.

For more details, you can also check the log files of the Apama-ctrl microservice if the tenant has the “microservice hosting” feature enabled. Alarms of this type should be fixed as a priority as these uncaught exceptions will terminate the execution of that monitor instance, which will typically mean that your app is not going to function correctly. This might even lead to a correlator crash if not handled properly. See also Log files of the Apama-ctrl microservice.

An EPL app is running in an infinite or long-running loop

If an EPL app has an infinite or long-running loop, it may block the correlator context for too long, not letting any other apps run in the same context or, even worse, causes excessive memory usage (as the correlator is unable to perform any garbage collection cycles) leading to the app running out of memory. The Apama-ctrl microservice identifies such scenarios (the correlator logs warning messages if an app is blocking a context for too long) and raises alarms, so that the user can identify and fix the problem.

For example, the following monitor blocks the correlator main context:

event MyEvent {
}

monitor Sample{
    action onload() {
        while true {
            // do something
            send MyEvent() to "foo";
        }
    }
}

Apama-ctrl generates the following alarm for the above example:

• Alarm type: APAMA_CTRL_WARN_<HASHCODE>
• Alarm text: <EPLAppName>.<monitorName> - This EPL app is probably in an infinite or long-running loop and impedes the operation of your other apps. It is recommended that you deactivate and diagnose this app.
• Alarm severity: WARNING

You can diagnose the issue by the monitor name given in the alarm.

For more details, you can also check the log files of the Apama-ctrl microservice if the tenant has the “microservice hosting” feature enabled. Alarms of this type should be fixed as a priority as these scenarios may lead to the microservice and correlator running out of memory. See also Log files of the Apama-ctrl microservice.

EPL app restore timeout on restart of Apama-ctrl

If restoring an EPL app on a restart of the Apama-ctrl microservice takes a long time and exceeds the time limit specified by the recovery.timeoutSecs tenant option (in the streaminganalytics category) or a default of 60 seconds, the Apama-ctrl microservice times out and raises an alarm, indicating that it will restart and reattempt to restore the EPL app. The alarm text includes the names of any EPL apps that are considered to be the reason for the timeout.

• Alarm type: eplapp_restore_timeout
• Alarm text: Restoring EPL apps after Apama-ctrl microservice restart has timed out. The EPL app <app name> could not be restored. The following EPL apps may be the cause of this: <comma-separated list of app names>. The Apama-ctrl microservice will restart now, and restoring will be reattempted. If this continues to fail, the Apama-ctrl microservice will enter safe mode, disabling all EPL apps.
• Alarm severity: MAJOR

The following information is only included in the alarm text if the Apama-ctrl microservice detects that the timeout is due to some EPL apps: “The following EPL apps may be the cause of this: <comma-separated list of app names>.”. If no such apps are detected, this information is omitted from the alarm text.

Multiple extensions with the same name

This alarm is raised when the Apama-ctrl microservice tries to activate the deployed extensions during its startup process and there are multiple extensions with the same name.

• Alarm type: extension_error
• Alarm text: Multiple extensions with the same name have been found: <list of all duplicate extension names>
• Alarm severity: CRITICAL

This disables all extensions that were deployed to Apama-ctrl. In order to use the deployed extensions, the user must decide which extensions to keep and then delete the duplicate ones.

Smart rule configuration failed

This alarm is raised if a smart rule contains an invalid configuration.

• Alarm type: smartrule_configuration_error
• Alarm text: <Smart rule identifier>: Smart rule create/edit failed. One or more fields are invalid, please check smart rule configuration.
• Alarm severity: MAJOR

To diagnose the cause, download the basic diagnostics ZIP file as described in Downloading diagnostics and logs. Or, if that fails, log on as an administrator and look at the result of a GET request to /service/smartrule/smartrules?withPrivateRules=true. Review the smart rules JSON and look for invalid smart rule configurations. Such smart rules must be corrected.

The Apama microservice log contains more details on the reason for the smart rule configuration failure. For example, it is invalid to configure an “On measurement threshold create alarm” smart rule with a data point that does not exist.

Smart rule restore failed

This alarm is raised if a corrupt smart rule is present in the inventory and the correlator therefore fails to recover it correctly during startup.

• Alarm type: smartrule_restore_failed
• Alarm text: Smart rule restore failed. Contact support.
• Alarm severity: MAJOR

To diagnose the cause, download the basic diagnostics ZIP file as described in Downloading diagnostics and logs. Or, if that fails, log on as an administrator and look at the result of a GET request to /service/smartrule/smartrules?withPrivateRules=true. Review the smart rules JSON and look for invalid smart rule configurations. Such smart rules may need to be deleted or corrected.

Connection to correlator lost

This alarm is raised in certain cases when the connection between the Apama-ctrl microservice and the correlator is lost. This should not happen, but can be triggered by high load situations.

• Alarm type: lost_correlator_connection
• Alarm text: Unable to ping correlator: <message>, Apama-ctrl will restart.
• Alarm severity: MAJOR

Apama-ctrl will automatically restart. Report this to product support if this is happening frequently.

Performance alarms

Input or output queues that are filling up are a symptom of a serious performance degradation, suggesting that events or requests are being produced by Apama or Cumulocity faster than they can be processed by Apama or Cumulocity.

The performance of the correlator’s input and output queues is periodically monitored. Different types of alarms can be raised, where the alarm text contains a snapshot of the correlator status at the time of raising the alarm.

This alarm is raised for the input queues:

• Alarm type: input_queues_filling
• Alarm text: Correlator input queues are filling. If this alarm is being regularly raised, there is a chance that the correlator cannot process the requests at the rate at which they are arriving. Slowest receiver name: <name>, Slowest receiver queue size: <size>, Slowest context name: <name>, Slowest context queue size: <size>.
• Alarm severity: WARNING

This alarm is raised for the output queues:

• Alarm type: output_queues_filling
• Alarm text: Correlator output queues are filling. If this alarm is being regularly raised, there is a chance that Cumulocity is not able to process the requests at the rate the correlator is sending them. Slowest receiver name: <name>, Slowest receiver queue size: <size>, Slowest context name: <name>, Slowest context queue size: <size>.
• Alarm severity: WARNING

This alarm is raised for both the input and output queues:

• Alarm type: input_output_queues_filling
• Alarm text: Correlator input and output queues are filling. If this alarm is being regularly raised, there is a chance that Cumulocity is not able to process the requests at the rate the correlator is sending them, causing the slowest output queue to fill up. This might have also caused the slowest input queue to fill up. Slowest receiver name: <name>, Slowest receiver queue size: <size>, Slowest context name: <name>, Slowest context queue size: <size>.
• Alarm severity: MAJOR

See also List of correlator status statistics in the Apama documentation.

Check the text from the above alarms to get an indication of which queue is blocking. A problem is likely to trigger these alarms, followed by this alarm:

• Alarm text: Real-time event processing is currently overloaded and may stop processing your events. Please contact support.
• Alarm severity: CRITICAL

This alarm is raised whenever the CEP queue for the respective tenant is full. It is coming from Cumulocity Core, but concerns Apama-ctrl.

Karaf nodes that send events to the CEP engine maintain per-tenant queues for the incoming events. This data gets processed by the CEP engine for the hosted CEP rules. For various reasons, these queues can become full and cannot accommodate newly arriving data. In such cases, an alarm is sent to the platform so that the end users are notified about the situation.

If the CEP queue is full, older events are removed to handle new incoming events. To avoid this, you must diagnose the cause of the queue being full and resolve it as soon as possible.

The CEP queue size is based on the number of CEP events, not raw bytes.

To diagnose the cause, you can try the following. It may be that the Apama-ctrl microservice is running slow because of time-consuming smart rules, analytic models or EPL apps, or the microservice is deprived of resources, or code is not optimized, and so on. Check the correlator input and output queues from the above alarms (or from the microservice logs or from the basic diagnostics ZIP file under /correlator/status.json).

• If both input and output queues are full, this suggests a slow receiver, possibly EPL sending too many requests (or too expensive a request) to Cumulocity.
• Else, if only the input queue is full, EPL is probably running in a tight loop. Try analyzing the cpuProfile.csv output in the basic diagnostics ZIP file, especially the monitor name and CPU time. The data collected in the profiler may also help in identifying other possible bottlenecks. For details, refer to Using the CPU profiler in the Apama documentation.
• Else, the cause may be some issue with connectivity or in Cumulocity Core.

Parent tenant not subscribed

This alarm is raised for a subtenant that was subscribed before the parent tenant was subscribed.

• Alarm type: parent_tenant_not_subscribed
• Alarm text: The microservice cannot function fully until the parent tenant is also subscribed to the microservice. Please contact the administrator.
• Alarm severity: MAJOR

The Apama-ctrl microservice allows you to subscribe to tenants in any order. However, as long as the parent tenant is not subscribed, the microservice functionality will not work on the subtenant.

This alarm is cleared once the parent tenant is subscribed.

Analytics Builder dropped events

This alarm is raised when an Analytics Builder model drops an event because it is delayed beyond the reorder buffer duration.

• Alarm type: analyticsbuilder_dropped_events
• Alarm text: Analytics Builder dropped <number> events because they were delayed beyond the reorder buffer duration. The last dropped event was received at <system time> (<number> seconds old): ‘<last dropped event string>’.
• Alarm severity: WARNING

Analytics Builder models use buffers to reorder incoming events by their source timestamp and process them in order. By default, input blocks reorder events with a delay of up to 1 second. If an event is received after a delay of more than the reorder buffer duration, the event may be dropped without processing. See Input blocks and event timing for detailed information.

To resolve the issue, you can either disable the reordering of input events or increase the duration of the reorder buffer. You can disable the reordering of input events by enabling the Ignore Timestamp parameter of the input blocks. When the Ignore Timestamp parameter is enabled, the input events are processed as soon as possible without reordering. You can increase the duration of the reorder buffer by changing the timedelay_secs tenant option in the analytics.builder category. For more information on timedelay_secs, see Keys for model timeouts.