Managing the dashboard data server and display server

Prerequisites

In order to start a data server or display server with all the necessary parameters to support a given deployment, you may need to obtain the following information from the Dashboard Builder:

• For web-based deployments, the data server or display server host and port that was used when preparing the dashboard for deployment. See Generating a deployment package from the command line.
• The logical name for each correlator as well as the host name and port for each deployment correlator (if any) that was specified by the Dashboard Builder in the Apama tab of the Tools > Options dialog prior to the generation of the deployment package. See Changing correlator definitions for deployment.

The port of the Apama dashboard data server must be accessible to the Apama Dashboard Viewer. If you are on a Windows system and the firewall is enabled, unblock network access for this port. The default value for the port is 3278. For security reasons, never change firewall settings such that this port is exposed to untrusted clients.

Command-line options for the data server and display server

The executable for the data server is dashboard_server and the executable for the display server is display_server. They can be found in the bin directory of the Apama installation.

Synopsis

To start the data server, run the following command:

dashboard_server [ options ]

To start the display server, run the following command:

display_server [ options ]

When you run these commands with the -h option, the usage message for the corresponding command is shown.

Start the display server from the dashboards directory of your Apama work directory. To do so, proceed as follows:

1. Open an Apama Command Prompt as described in Setting up the environment using the Apama Command Prompt.
2. Change to the %APAMA_WORK%\dashboards directory.
3. Invoke the executable from the command line.

You can also start display server with the --dashboardDir folder option to specify a folder for your deployed dashboards. When --dashboard folder is used, display server will be looking for the deployed dashboards from the specified folder.

Description

The dashboard_server and display_server executables can be run without arguments, in which case they start a server on port 3278 (for a data server) or 3279 (for a display server) on the local host. You can specify a different port with the -d or --dataPort option.

The -c or --correlator option allows you to specify the deployment host and port for a given correlator logical name. See Changing correlator definitions for deployment.

You can enable logging with the -f and -v (or --logfile and --loglevel) options or with the log4j properties file.

Options

Both the dashboard_server and display_server executables take the following options:

-c default:localhost:15903:false
–c work1:somehost:19999:false

[retry:ms | fail:n | db:name | noinfo |
nopererr | quote]

--queryIndex Products_Table:prod_id,vend_id

Rotating the log files of the data server and display server

Rotating a log file refers to closing a log file being used by a running data server or display server and opening a new log file to be used instead from that point onwards. This lets you archive log files and avoid log files that are too large to easily view.

Each site should decide on and implement its own log rotation policy. You should consider the following:

• How often to rotate log files.
• How large a log file for a data server or display server can be.
• What log file naming conventions to use to organize log files.

There is a lot of useful header information in the log file being used when the data server or display server starts. If you need to provide log files to Apama technical support, you should be able to provide the log file that was in use when the data server or display server started, as well as any other log files that were in use before and when a problem occurred.

Logging for data servers and display servers is configured using standard Log4j configuration files. You can find them in the etc directory of your Apama installation:

• log4j-dashboard-server.xml is the configuration file for the data server.
• log4j-display-server.xml is the configuration file for the display server.

By default, the above files configure the servers to rotate the log files when they reach a certain file size. If you want to enable time-based rotation instead (for example, to rotate the log files on a monthly basis), see the Log4j 2 documentation at https://logging.apache.org/log4j/2.x/.

There are many external resources which can be found online regarding how to configure Log4j for different purposes. In the case of more advanced configurations, you may consider consulting these.

Controlling the update frequency

The correlator sends update events to the data server, display server, or any clients using the Scenario Service API (see also Scenario Service API) for all scenarios with output variables (for example, DataViews, including MemoryStore tables that expose DataViews using the exposeMemoryView or exposePersistentView schema options). These updates are sent whenever the values of fields or output variables in your scenarios change. If you have scenarios that update frequently, you might need to reduce the frequency of update events sent by the correlator.

You can adjust the settings per scenario definition or globally. The global value is used where a given scenario definition has no specific setting. The per-definition values always take precedence over the global values.

A ConfigureUpdates event consists of the following:

• scenarioId (type string): May be the empty string to modify the global values, or a definition’s scenarioId.
• Configuration (type dictionary (string, string)): Configuration key and values. Key can be one of:
  • sendThrottled (type boolean): Whether to send throttled updates (on the scenarioId.Data channel). The default is true.
  • sendRaw (type boolean): Whether to send every update (on the scenarioId.Data.Raw channel). The default is true.
  • throttlePeriod (type float): Throttle period in seconds. A zero value indicates no throttling. The default is 0.0.
  • routeUpdates (type boolean): Whether to route Update events for the benefit of EPL files running in the correlator. The default is false.
  • sendThrottledUser: (boolean): Whether to send throttled updates on a per-user channel. The default is false.
  • sendRawUser (type boolean): Whether to send raw updates on a per-user channel. The default is false.

Those with a User suffix are suitable for using with only custom clients that use ScenarioServiceConfig.setUsernameFilter() on their service configuration.

For example, consider the following:

com.apama.scenario.ConfigureUpdates("DV_scenario1",
    {"sendRaw":"true"})
com.apama.scenario.ConfigureUpdates("",{"sendRaw":"false",
    "throttlePeriod":"0.1"})
com.apama.scenario.ConfigureUpdates("DV_scenario2",
    {"sendRaw":"true"})
com.apama.scenario.ConfigureUpdates("DV_scenario3",
    {"throttlePeriod":"1.0"})

The above examples configure DV_scenario1 and DV_scenario2 to send raw updates; DV_scenario3 to use a throttle period of 1 second; and all other scenarios to not send raw updates, and to use a throttle period of 0.1 seconds.

Earlier releases used the com.apama.scenario.SetThrottlingPeriod(x) event. Note that the use of the ConfigureUpdates events allows greater flexibility than the SetThrottlingPeriod event (which only controlled sending of throttled updates for all scenarios).

The use of com.apama.scenario.SetThrottlingPeriod(x) should be replaced with:

com.apama.scenario.ConfigureUpdates("", {"throttlePeriod":"x"})

Note that by default, routeUpdates is false, so any EPL that relies on Update (and other scenario control events) to be routed should route a ConfigureUpdates event for the scenarioIds it is interested in to route Updates.

The latest values are always used — thus it is not advisable for a client to send an event requesting (for example) raw updates and then undo this when it disconnects, as that will affect other clients. The recommendation is that the administrator should configure settings at initialization time.

Runtime performance of scenarios can be improved by setting sendRaw and routeUpdates to false and throttlePeriod to a non-zero value. In this case, the cost of an update is reduced (as the Update events are only generated when needed, and if throttling, they are only needed at most once every throttlePeriod).

Configuring Trend-Data Caching

By default, dashboard servers (data servers and display servers) collect trend data for all numeric output variables of DataViews running in their associated correlators. This data is cached in preparation for the possibility that it will be displayed as historical data in a trend chart when a dashboard starts up. Without the cache, trend charts would initially be empty, with new data points displaying as time elapses.

Advanced users can override the default caching behavior on a given server, and control caching in order to reduce memory consumption on that server, or in order to cache variables that are not cached by default, such as non-numeric variables.

Important:

In many cases, Server performance can be improved by overriding the default caching behavior, and suppressing the caching of those output variables for which trend-chart historical data is not required.

Caching trend data for string variables is very costly in terms of memory consumption.

You control caching with a trend configuration file, which allows you to specify the following:

• Individual variables to cache
• Classes of variables to cache
• Default caching rules
• Trend depths (number of data points to maintain) for each DataView

You do not need to provide a trend configuration file. If you provide no trend configuration file, dashboard servers use the default caching behavior described above.

Trend charts can include variables whose trend data is not cached, but they will display no historical (pre-dashboard-startup) data for those variables.

When a data server or display server starts, it uses the trend configuration file specified with the -G option, if supplied. Otherwise it uses the file trend.xml in the dashboards directory of your Apama work directory, if there is one. (Note that Apama provides an example trend configuration file, APAMA_HOME\etc\dashboard_onDemandTrend.xml, that you can copy to APAMA_WORK\dashboards\trend.xml as a basis for a trend configuration file.) Otherwise, it uses the default caching behavior described above.

Here is a sample configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <trend>
    <item type="DATAVIEW" correlator="*" name="*"
      vars="ALL_NUMERIC_OUTPUT" depth="10000"/>
    <item type="DATAVIEW" correlator="*" name="DV_scenario1"
      vars="LIST" depth="5000">
    <var name="A"/>
    <var name="B"/>
    </item>

    <item type="DATAVIEW" correlator="production" name="DV_dataview1"
      vars="LIST" depth="5000">
        <var name="A"/>
        <var name="B"/>
    </item>
  </trend>
 </config>

This file specifies the following:

• For DV_scenario1 in all correlators, cache trend data for variables A and B with a maximum trend depth of 5000.
• For all other queries, cache all numeric output variables with a maximum trend depth of 10,000.
• For DV_dataview1 in correlator production, cache variables A and B with a maximum trend depth of 5000.
• For all other DataViews, cache no trend data.

In general, a trend configuration file is an XML file that includes of one or more item elements with the following attributes:

• type:DATAVIEW
• correlator: Logical name of correlator. Use * for if the item applies to all correlators
• name: DataView ID. Use * if the item applies to all DataViews.
• vars: Class of variables to cache trend data for. Specify one of the following:
  • LIST: Cache the individual variables that are listed in var sub-elements.
  • ALL: Cache all input and output variables.
  • ALL_OUTPUT: Cache all output variables.
  • ALL_NUMERIC_OUTPUT: Cache all numeric output variables.
• depth: Maximum depth of trend data to cache.

If the vars attribute of an item element is LIST, the element has zero or more var sub-elements. Each var element has single attribute, name, which specifies the name of a DataView field.

The item elements are nested in a trend element, which is nested within a config element.

If a particular DataView on a given correlator matches multiple item elements in a server’s trend configuration file, the server chooses the best-matchingitem and caches the variables specified in that item. Following are the ways, in order from best to worst, in which an item can match a DataView on a given correlator:

1. Fully resolved: Exact match for both correlator name and DataView name
2. Wildcard correlator: Wildcard correlator and exact match for DataView name
3. Wildcard DataView: Exact match for correlator name and wildcard DataView
4. Fully wildcarded: Wildcard correlator and wildcard DataView

If there are multiple best matches, the last match is used.

Consider, for example, scenarios named DV_scenario1 and DV_scenario2, correlators named production and development, and the following item elements:

1 <item type="DATAVIEW" correlator="production" name="DV_scenario1"
    vars="LIST" depth="5000">
2 <item type="DATAVIEW" correlator="*" name="DV_scenario1" vars="LIST"
    depth="5000">
3 <item type="DATAVIEW" correlator="production" name="*" vars="LIST"
    depth="5000">
4 <item type="DATAVIEW" correlator="*" name="*" vars="LIST"
    depth="5000">

DV_scenario1 running on production best matches item1.

DV_scenario1 on development best matches item2.

DV_scenario2 on production best matches item3.

DV_scenario2 on development best matches item4.

Below are some additional sample configuration files. The following file caches trend data for all input and output variables:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <trend>
      <item type="DATAVIEW" correlator="*" name="*" vars="ALL"
      depth="10000"/>
  </trend>
 </config>

The following caches trend data for all numeric output variables, the default behavior:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <trend>
    <item type="DATAVIEW" correlator="*" name="*"
      vars="ALL_NUMERIC_OUTPUT" depth="10000"/>
    <item type="DATAVIEW" correlator="*" name="*"
      vars="ALL_NUMERIC_OUTPUT" depth="10000"/>
  </trend>
 </config>

The following caches no data, which results in trend-data collection only on demand:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <trend>
  </trend>
 </config>

The following caches a single variable for a single scenario:

<?xml version="1.0" encoding="UTF-8"?>
<config>
   <trend>
    <item type="DATAVIEW" correlator="*" name="DV_scenario1"
      vars="LIST" depth="5000">
        <var name="PRICE"/>
    </item>
  </trend>
 </config>

Managing Connect and Disconnect Notification

Whenever a dashboard connects to or disconnects from a data server or display server, the server sends a special notification event to all connected correlators that include the Dashboard Support bundle.

The events are defined as follows:

event DashboardClientConnected {
  string userName;
  string sessionId;
  dictionary<string,string> extraParams;
}event DashboardClientDisconnected {
  string userName;
  string sessionId;
  dictionary<string,string> extraParams;
}

userName specifies the user name with which the dashboard was logged in to the server.

sessionID is a unique identifier for the dashboard’s session with the server.

extraParams may be used in a future release.

Note that the circumstances under which a dashboard disconnects from a server include but are not limited to the following:

• End user exits the Dashboard Viewer or web browser in which a dashboard is loaded.
• End user exits a web browser tab in which a dashboard is loaded.
• Network failure causes loss of connectivity to viewer or web browser in which a dashboard is loaded.

Note also that disconnect notification might be sent only after a timeout period rather than immediately upon loss of connection.

Follow these steps to manage connect and disconnect notification:

1. Ensure that the Dashboard Support bundle is loaded into all relevant correlators.

2. Use EPLs to process DashboardClientConnected and DashboardClientDisconnected events. Base processing on the values of the userName and sessionId fields.

Working with multiple data servers

Deployed dashboards have a unique associated default data server or display server. For web-based deployments, this default is specified with the following properties of the dashboard_deploy.xml configuration file (see Generating a deployment package from the command line).

• apama.displayserver.port
• apama.displayserver.host
• apama.displayserver.refresh
• apama.displayserver.hiddenmenuitems

For Viewer deployments, it is specified upon Viewer startup. By default, the data-handling involved in attachments and commands is handled by the default server, but advanced users can associate non-default data servers with specific attachments and commands. This provides additional scalability by allowing loads to be distributed among multiple servers. This is particularly useful for display server deployments. By deploying one or more data servers behind a display server, the labor of display building can be separated from the labor of data handling. The display server can be dedicated to building displays, while the overhead of data handling is offloaded to data servers.

Apama supports the following multiserver configurations:

• Builder with multiple data servers. See Builder with multiple data servers.
• Viewer with multiple data servers. See Viewer with multiple data servers.
• Display server (thin client) deployment with multiple data servers. See Display server deployments with multiple data servers.

The Attach to Apama and Define… Command dialogs (except Define System Command) include a Data Server field that can be set to a data server’s logical name. To associate a logical name with the data server at a given host and port, developers use the Data Server tab in the General tab group of the Application Options dialog (select Tools Options in Builder).

For display server (thin client) deployments, you must use the option --namedServerMode whenever you start named data servers. See Display server deployments with multiple data servers.

The logical data server names specified in the Builder’s Application Options dialog are recorded in the file OPTIONS.ini, and the deployment wizard incorporates this information into deployments. You can override these logical name definitions with the --namedServername:host:port option to the Builder, Viewer, data server or display server executable. Below is an example. This is a sequence of command-line options which should appear on a single line as part of the command to start the executable:

--namedServer Server1:ProductionHost_A:3278 --namedServer Server2:ProductionHost_B:4278 --namedServer Server3:ProductionHost_C:5278

Here Server1, Server2 and Server3 are the server logical names.

Builder with multiple data servers

Builder maintains connections with the data servers named in attachments and commands. Note that it connects directly to the correlator (dotted lines in the figure below) in order to populate dialogs with metadata. Correlator event data is handled by the data servers.

You can override the logical server names specified in the Application Options dialog with the --namedServername:host:port option to the Builder executable. Below is an example. This is a sequence of command-line options which should appear on a single line as part of the command to start the executable:

--namedServer Server1:ProductionHost_A:3278 --namedServer Server2:ProductionHost_B:4278 --namedServer Server3:ProductionHost_C:5278

Here Server1, Server2 and Server3 are the server logical names.

Viewer with multiple data servers

Viewer maintains connections with the data servers named in attachments and commands of opened dashboards.

In the data server Login dialog (which appears upon Viewer startup), end users enter the host and port of the default data server (or accept the default field values). If all attachments and commands use named data servers, end users can check the Only using named data server connections check box and omit specification of a default server.

The logical data server names specified in the Builder’s Application Options dialog are recorded in the file OPTIONS.ini, which is found in the deployed .war file along with dashboard .rtv files. You can override these logical name definitions with the --namedServer name : host : port option to the Viewer executable. Below is an example. This is a sequence of command-line options which should appear on a single line as part of the command to start the executable:

--namedServer Server1:ProductionHost_A:3278 --namedServer Server2:ProductionHost_B:4278 --namedServer Server3:ProductionHost_C:5278

Here Server1, Server2 and Server3 are the server logical names.

Display server deployments with multiple data servers

The display server maintains connections with the data servers named in attachments and commands of its client dashboards.

The logical data server names specified in the Builder’s Application Options dialog are recorded in the file OPTIONS.ini, which is used by the Deployment Wizard to define deployment logical names. You can override these logical name definitions with the --namedServer name:host:port option to the display server executable. Below is an example. This is a sequence of command-line options which should appear on a single line as part of the command to start the executable:

--namedServer Server1:ProductionHost_A:3278 --namedServer Server2:ProductionHost_B:4278 --namedServer Server3:ProductionHost_C:5278

Here Server1, Server2 and Server3 are the server logical names.

Managing and stopping the data server and display server

The dashboard_management tool is used to stop a data server or display server and perform certain data server or display server management operations. The executable for this tool is located in the bin directory of the Apama installation. Running the tool in the Apama Command Prompt or using the apama_env wrapper (see Setting up the environment using the Apama Command Prompt) ensures that the environment variables are set correctly.

Synopsis

To manage and stop the data server or display server, run the following command:

dashboard_management [ options ]

When you run this command with the –h option, the usage message for this command is shown.

Description

You can use this tool to shut down, deep ping, or get the process ID of a data server or display server on a specified host and port. A successful deep ping verifies that the server is responding to requests. You can also use this tool to generate a dashboard deployment package, and to sign.jar files as part of deployment-package generation.

When you invoke this tool, you can specify the host and port of the server you want to manage. For the port, specify the port that was specified with the -p or --port option when the desired server was started. If the -p or --port option was not specified, you do not need to supply this option. It defaults to the default management port (28888).

Options

The dashboard_management tool takes the following options: