These Release Notes describe new features and changes introduced with old releases of Apama. For information about recent changes, see Change Logs.
Note that the versions covered in these release notes may no longer be supported.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.11.3 includes new features, enhancements, and changes as described in the following topics.
This release of Apama ships with a new version of PySys, version 2.1.
Some highlights in this release are:
BADPERF outcome for assertions that detect inadequate performance, and automatic printing of the performance results on the console at the end of a test run, including mean and standard deviation statistics (if multiple cycles were executed), and comparison with one or more baseline performance .csv/.json files (if provided).__pysys_XXX__ descriptor fields in pysystest.py files, including an easier way to create parameterized tests, and a more natural syntax for configuring the list of modes.assertMessage= parameter is handled by the assertXXX methods when an assertion fails. The test failure outcome reasons now contain both the standard outcome reason and the human-friendly user-defined assertMessage if one is provided. For the benefit of anyone working on the test or triaging test results, it is a good idea to add an outcomeReason that describes at a high level what the assertion is checking (and why), to any assertion where it is not obvious.There are also several bug fixes and minor enhancements. See the PySys Change Log (PySys Test Framework > Change Log in the API reference for Python) for more information. There are no breaking changes in this release.
Apama 10.11.3 includes the following dashboard enhancements:
The dashboard data server and display server now produce a heap dump file to help with diagnostics should they run out of memory. If you want to disable this new behavior, remove the -XX:+HeapDumpOnOutOfMemoryError JVM flag in either the dashboard_server.ini and display_server.ini files (Windows) or the dashboard_server and display_server scripts (Linux) in the bin directory of the Apama installation.
Note: These files will be overwritten by any subsequent fixes.
Apama 10.11.3 includes the following miscellaneous enhancements and changes:
CUMULOCITY_INITIAL_DELAY_SECS is now available. It is not provided by default in the .properties configuration file. You can add it if you want to set an initial delay (in seconds) for querying tenant subscriptions. By default, it is set to 0 seconds. See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).The following changes in Apama 10.11.3 affect backwards compatibility to previous Apama versions:
The repository location for the Apama images on Docker Hub has changed. The new URLs are:
docker pull softwareag/apama-correlator:*version*
docker pull softwareag/apama-builder:*version*
docker pull softwareag/apama-cumulocity-jre:*version*
See Deploying Apama applications with Docker for more information.
Apama images for Docker from 10.11.3.0 are now compliant with the DoD Security Technical Implementation Guide for containers.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.11.2 contains several error corrections. New functionality is not provided.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.11.1 includes new features, enhancements, and changes as described in the following topics.
Apama 10.11.1 includes the following miscellaneous enhancements and changes:
pageSize is below 50 and currentPage is not set (default). This is because setting a small pageSize without setting currentPage can result in queries that run very slowly. For example, to request the first 20 items, you have to set pageSize to 20 and currentPage to 1. See also Paging Cumulocity queries.The following changes in Apama 10.11.1 affect backwards compatibility to previous Apama versions:
GeoFenceContainer.createGeoFenceContainer() action no longer throws an exception when provided with any lat or lng values that cannot be parsed as a float. Instead, an empty optional<GeoFencePoint> is now created for that coordinate. See the com.apama.cumulocity package in the API reference for EPL (ApamaDoc) for more information on the GeoFenceContainer event.The following Apama features are now deprecated or have been removed in Apama 10.11.1:
engine_deploy tool when you attempt to set the deployment directory (specified with either the -d or --outputDeployDir action) to be a subdirectory of the project directory. The ability to do this will be removed in a future release. See also Deploying a correlator.Apama 10.11.0 is the successor of Apama 10.7.2. There is no version 10.8, 10.9 or 10.10.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.11.0 includes new features, enhancements, and changes as described in the following topics.
This release of Apama comes with a Java 11 JDK instead of Java 8. All Java code executed within the correlator and IAF processes is now executed with Java 11, so you may need to make changes to your EPL plug-ins, connectivity plug-ins, IAF plug-ins, JMON applications, etc. that run inside the correlator or IAF.
If you build separate client processes that use the Apama APIs from ap-client.jar, a Java 8 runtime can still be used if you wish. However, this is deprecated and it is recommended that you migrate to using a Java 11 runtime when possible.
If you are using the “core components” version of the Apama Community Edition, then you must now provide a Java 11 runtime for use with Apama instead of Java 8.
There are plenty of great resources on the internet for finding out how to upgrade from Java 8 to Java 11, and lists of breaking changes between these versions. In practice, the main effort is likely to be upgrading any third-party libraries to Java 11-supporting versions, and adding additional libraries to replace some of the packages removed from the runtime itself (for example, JAXB). Java now uses a different default garbage collector (G1GC instead of the parallel garbage collector) which may result in different memory behavior.
The default locale data provider was changed (to the Unicode Consortium’s CLDR), which may result in dates, times and numbers that used to parse successfully now being rejected. See the Java documentation for details of how to configure the Java locale data provider.
Apama 10.11.0 has upgraded the compilers that it uses:
Consequently, you must rebuild the following components with the new compiler:
Note: Compiling connectivity or EPL plug-ins with Red Hat Enterprise Linux (RHEL) 8 no longer needs -D_GLIBCXX_USE_CXX11_ABI=0 as described in Support for Red Hat Enterprise Linux (RHEL) 8.0 in 10.5.
In addition, when planning migration, consider that new compiler versions are more modern than the previous versions and less tolerant of ambiguous and potentially unsafe code patterns. It is likely that some time may be required to fix compiler errors after moving to the new versions.
Note: Our recommendation is to enable “warnings as errors” in your projects where possible. This encourages good coding patterns and catches errors earlier in the development process. In particular, it helps you catch errors that might cause subtle, difficult to diagnose errors.
This release of Apama comes with Python 3.9.5, replacing Python 3.7.8. For a detailed overview of changes, including new features, deprecated and breaking changes, see https://docs.python.org/3/whatsnew/3.9.html and https://docs.python.org/3.8/whatsnew/3.8.html.
This change affects:
If you are not using the version of Python distributed with the full Apama installation, you will have to update your Python installation to match.
This release of Apama ships with a new version of PySys, version 2.0.
The 2.0 release contains many enhancements including:
pysystest.py file, instead of separate run.py and pysystest.xml files. You can mix and match the old and new styles within the same project. For new PySys projects, a simpler directory layout is now recommended in which the self.input directory is configured to be the main testDir/ (which also contains the pysystest.py file) instead of having a separate testDir/Input/ subdirectory for input files. This can make test contents easier to navigate.pysys make, allowing easy configuration of how new tests are created - on a per-directory basis - and also automatic generation of test identifiers for new tests (when using numeric identifiers).pysys.mappers API for more easily transforming text files during copy and grep operations, including support for multi-line exception stack traces.For more information on the above and also details on the many smaller enhancements in this PySys release, see the PySys Change Log (PySys Test Framework > Change Log in the API reference for Python).
There are several improvements to the Apama helper classes for PySys in this release:
The ApamaPlugin class (self.apama) has a new method JoinEPLStackLines which can be used in an assertGrep(..., mappers=[...]) list to convert multi-line EPL stack dumps to a single line. This can make it easier to ignore expected errors while still checking for unanticipated errors. For example:
self.assertGrep('testCorrelator.log', '(ERROR|FATAL|Failed to parse).*', contains=False,
mappers=[self.apama.JoinEPLStackLines(), pysys.mappers.JoinLines.JavaStackTrace()])
See Apama Helper Classes for PySys in the API reference for Python for more information.
The CorrelatorHelper and IAFHelper classes have a new method waitForLogGrep which (much like waitForGrep) can be used to wait for a log message to appear in the log file, but with automatic checking for errors and an early abort with a clear outcome reason if an error is detected. For example:
correlator.waitForLogGrep('INFO.* Test completed')
The ApamaPlugin class (self.apama) has a new field defaultLogIgnores which holds a list of project-level regular expressions that should be ignored when checking for errors, warnings and other messages in Apama component logs. The default value contains a small set of WARN messages which should be ignored in testcases. Additional ignores can be added to this list with a multi-line list of regular expressions in your pysysproject.xml configuration:
<property name="apamaDefaultLogIgnores">
WARN.*My warning to ignore
ERROR.*My error to ignore
</property>
It is recommended to add at least one maker-template to your pysysproject.xml file to provide a good starting point when creating new testcases using pysys make. A default Apama test template is provided with the product which includes some typical PySysTest commands. You should delete any aspects not needed in your testcase. To make this template available in existing projects, just add the following XML to your pysysproject.xml file:
<pysysproject>
<pysysdirconfig>
<maker-template name="apama-default-test"
description="a test for an Apama application that runs in a local correlator"
copy="${pysysTemplatesDir}/apama-default-test/*"
/>
</pysysdirconfig>
</pysysproject>
There are a several changes that could require changes to existing PySys projects. A full list is provided in the PySys Change Log, but the main ones are summarized below:
mappers= are used (for example, during self.copy), it is now an error for a mapper to strip off the trailing \n character at the end of each line, as failure to do so can have unintended consequences on later mappers. This requirement is also more clearly documented.pysystest.xml structure that were previously tolerated will now produce stderr warning messages (for example, incorrectly nesting <modes> inside <groups>) and others will produce a fatal error (for example, multiple occurrences of the same element). To find out if any tests need fixing up, just execute pysys print in your PySys project directory and act on any warning or error messages.supportMultipleModesPerRun=false project property (only used in very old PySys projects) can no longer be used. Please change your tests to use the modern modes approach instead.testDir (and the input, output, and reference directories) no longer start with the \\?\ long path prefix. Instead, this can be added for operations where it is needed using pysys.utils.fileutils.toLongPathSafe (as the standard PySys methods already do, for example self.copy). Where possible, it is recommended to avoid nesting tests and output directories so deeply that long path support is needed.In addition, your tests may be subject to breaking changes from the upgrade to Python 3.9. See also Python upgrade in 10.11.0.
It is strongly recommended to use the new pysys.constants.PREFERRED_ENCODING constant instead of Python’s built-in locale.getpreferredencoding() function, to avoid thread-safety issues in your tests.
If you have a custom pysys.utils.perfreporter.CSVPerformanceReporter subclass, the signatures for pysys.utils.perfreporter.CSVPerformanceReporter.getRunDetails and pysys.utils.perfreporter.CSVPerformanceReporter.getRunHeader have changed to include a testobj parameter.
Although this should not immediately break existing applications, to avoid future breaking changes, you should update the signature of those methods if you override them to accept testobj and also any arbitrary **kwargs that may be added in future.
The pysys.xml module is deprecated. Use pysys.config instead.
The pysys.utils.fileunzip module is deprecated. Use BaseTest.unpackArchive instead. For example, replace unzip(gzfilename, binary=True) with self.unpackArchive(gzfilename, gzfilename[:-3]).
The (undocumented) DEFAULT_DESCRIPTOR constant is now deprecated and should not be used.
The old <mode> elements are deprecated in favor of the new Python lambda syntax (support for these will not be removed any time soon, but are discouraged for new tests).
The pysys.utils.pycompat module is now deprecated.
The ConsoleMakeTestHelper class is now deprecated in favor of pysys.launcher.console_make.DefaultTestMaker.
If your pysysproject.xml contains an APAMA_COMMON_JRE property with its default value set to ${APAMA_HOME}/../jvm/jvm/jre, you should either remove the property or change the default value to ${APAMA_HOME}/../jvm/jvm as the /jre directory has been removed in this release of Apama.
A quick way to check for the removed and deprecated items in your test project is to use the regular expression as shown in the following grep command:
grep -r "\(supportMultipleModesPerRun.*alse\|DescriptorLoader\|pysys.utils.loader\|_stringToUnicode\|pysys[.]xml\|pysys.utils.fileunzip\|[^_@]DEFAULT_DESCRIPTOR\|pysys.utils.pycompat\|PY2\|string_types\|binary_type\|isstring[(]\|quotestring[(]\|openfile[(]\|ConsoleMakeTestHelper\|def getRunDetails\|def getRunHeader\|locale.getpreferredencoding\|addResource\|CommonProcessWrapper\|TEST_TEMPLATE\|DESCRIPTOR_TEMPLATE\|ThreadFilter\|/jvm/jvm/jre\)".
Apama 10.11.0 includes the following enhancements in Software AG Designer:
Apama 10.11.0 includes the following dashboard enhancements:
Apama 10.11.0 includes the following miscellaneous enhancements and changes:
Apama 10.11.0 incorporates the ICU (International Components for Unicode) time-zone data update 2021a, which is the most recent update at the time of release. This will update time-zone data used by the correlator and Time Format EPL plug-in.
A new key for monitoring the peak physical memory usage of the correlator, peakPhysicalMemoryMB, has been added to the /correlator/status REST endpoint. This is useful for checking whether the correlator is running on an adequately provisioned machine. For more information, see List of correlator status statistics and Managing and monitoring over REST.
Apama now provides a --shutdownTimeoutSecs correlator option. This option is used to specify the maximum time allowed for the correlator to shut down after all persistence activities have been completed. When this time has elapsed, the correlator shuts down forcibly, ignoring any transport or plug-in shutdown activities. This option is especially useful to prevent indefinite hangs caused by plug-ins. If this option is not provided, a default value of 90 seconds is used. See Starting the correlator.
A corresponding YAML element with the name shutdownTimeoutSecs is also provided. See YAML configuration file for the correlator.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.11.0. The platforms listed below are no longer supported.
No changes.
No changes.
Apache Ant 1.10.5 (replaced by support for a more recent version).
Java 8 (replaced by support for a more recent version).
Note: In some situations, Java 8 can still be used. See Java upgrade in 10.11.0 for more information.
MathWorks MATLAB 2016b (replaced by support for a more recent version).
Microsoft.NET Framework 4.5.1 (replaced by support for a more recent version).
PySys 1.6.1 (replaced by support for a more recent version).
Python 3.7 (replaced by support for a more recent version).
Software AG Cumulocity IoT 10.7 (replaced by support for a more recent version).
Software AG Designer 10.7 (replaced by support for a more recent version).
The following changes in Apama 10.11.0 affect backwards compatibility to previous Apama versions:
plugins or dropins directory, they may no longer resolve when you upgrade to a new Eclipse Platform version. In each new version of the Eclipse Platform, there are new versions of bundles included in the platform, and often a small number of removed bundles. This may cause your previously dropped-in bundles to no longer resolve and run. It is always recommended that you install software via the Help > Install New Software mechanism so you are made aware of any install-time failure to resolve dependencies.ap-util.jar. Although the presence of Jackson was never documented, it is possible that some customers are depending on it. If you have Java classes (such as connectivity plug-ins) that depend on any com.fasterxml.jackson.* classes, you should explicitly add the Jackson JAR files to your classpath.Note: There are also other features that affect backwards compatibility. See the following topics for more information: Java upgrade in 10.11.0, Compiler upgrade in 10.11.0, Python upgrade in 10.11.0, and PySys changes in 10.11.0.
The following Apama features are now deprecated or have been removed in Apama 10.11.0:
samples\universal_messaging\datagroups subdirectory have been removed from the Apama installation.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.7.2 includes new features, enhancements, and changes as described in the following topics.
Apama 10.7.2 includes the following connectivity plug-ins enhancements:
keepAliveTimeSecs configuration option. See also Configuring the HTTP server transport.SendSMS events now always pick the latest defaults for the sender name and sender address from the tenant options. It is no longer required to restart the correlator. See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).filterOnContentType configuration option for JSON codec now accepts parameters in the metadata.contentType. Therefore, the filter pattern has been updated from "^application/([^/][+])?json$" to "^application/([^/]*[+])?json(;[^=;]+=(\".*\"|[^=;]+))*$". See also The JSON codec connectivity plug-in.Apama 10.7.2 includes the following EPL enhancements:
It is now easier to identify and extract dictionary and sequence types contained within an any. For this purpose, the following new actions are now available for the any type:
isDictionaryisEventisSequencegetGenericDictionarygetDeclaredElementTypegetDeclaredKeyType
See the API reference for EPL (ApamaDoc) for more information.The following changes in Apama 10.7.2 affect backwards compatibility to previous Apama versions:
GenericResponse.body member is now deprecated and will be removed in a future release. It is recommended that you now use the GenericResponse.getBody action instead. See the com.apama.cumulocity package in the API reference for EPL (ApamaDoc) for more information on the GenericResponse event.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.7.1 includes new features, enhancements, and changes as described in the following topics.
Apama 10.7.1 includes the following connectivity plug-ins enhancements:
multipart/form-data or application/x-www-form-urlencoded media types using the new predefined FormRequest event definition. See also Handling HTML form encoding using a predefined generic event definition.false, but you can turn it on by setting concurrentChains to true in the YAML configuration file. See also Configuring the HTTP server transport. This enhancement was also delivered with Apama 10.5.4.numChains is now available for the HTTP server transport. It indicates the number of active connections into the HTTP server instance. See also Monitoring status for the HTTP server. This enhancement was also delivered with Apama 10.5.4.When the HTTP client’s generic JSON bundle has been added to the project configuration, defaults for the sender name and sender address are now retrieved from the tenant options in Cumulocity IoT if they are not specified in the SendSMS event. If the tenant options cannot be retrieved from Cumulocity IoT, you can now configure the defaults by adding CUMULOCITY_SMS_SENDER_NAME and CUMULOCITY_SMS_SENDER_ADDRESS to the .properties configuration file of the Cumulocity IoT transport. See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).
A geofence helper utility has been added to the Utilities for Cumulocity EPL bundle. See also About the Cumulocity transport. In addition, the following new events are now available in the com.apama.cumulocity package: GeoFenceContainer and GeoFencePoint. See the API reference for EPL (ApamaDoc) for more information on these events.
The com.apama.cumulocity events Alarm, Event, ManagedObject and Operation now have isCreate() and isUpdate() actions which can be used in event listeners to distinguish between events from Cumulocity IoT which are a result of a creation or an update. See the API reference for EPL (ApamaDoc) for more information on these events.
The com.apama.cumulocity.Error event now has a reqId field which corresponds to the identifier of the request from which the error event occurred. See the API reference for EPL (ApamaDoc) for more information.
The Cumulocity API now supports a new withChannelResponse action for the predefined types (ManagedObject, Alarm, Event, Measurement, MeasurementFragment and Operation). This allows your application to receive a response on the *type*.SUBSCRIBE_CHANNEL channel when one of these object types is created or updated.
It is recommended that you now use the new withChannelResponse action instead of the withResponse action which is now deprecated. See also Removed and deprecated features in 10.7.1.
See the following topics for more information:
Apama 10.7.1 includes the following EPL code coverage enhancements:
epl_coverage tool now also generates coverage reports in a widely-used XML format (the format that is used and was popularized by Cobertura, which is a code coverage utility for Java; see also https://cobertura.github.io/cobertura/). This allows coverage from EPL to be checked and visualized by third-party tools. No changes are required to get the XML report. An epl_coverage.xml file is generated automatically whenever the epl_coverage tool is executed. See also Generating code coverage information about EPL files.engine_delete --all or when the file has been reinjected. For more details, see Recording code coverage information.Apama 10.7.1 includes the following enhancements:
com.apama.util.AnyExtractor event for extracting data from an any type now has actions getAnyOr, getStringOr, getIntegerOr, getFloatOr, getBooleanOr, getSequenceOr, and getDictionaryOr. These actions return a user-specified default value if the path to the nested value does not exist, or if the path exists but the value is not of the required type. These offer non-throwing alternatives to the existing getter actions in the AnyExtractor event. See the API reference for EPL (ApamaDoc) for more information.AnyExtractor now supports extracting values from some additional types: events, sequence<ANY_TYPE>, dictionary<string,ANY_TYPE>, dictionary<any,ANY_TYPE> and location objects. The “.” and “[]” operators are also interchangeable, so either can be used for extracting values from any of these types.Apama 10.7.1 includes the following miscellaneous enhancements and changes:
maxMsgKB field which indicates the maximum size in kilobytes of JMS messages that have been received so far. See also Logging correlator-integrated messaging for JMS status.The following changes in Apama 10.7.1 affect backwards compatibility to previous Apama versions:
While sending a request, the HTTP client transport no longer sets a content-type header from metadata.contentType if the body is empty. If you want to send the content-type even if the body is empty, you now have to set metadata.http.headers.content-type explicitly in the request. See also Handling HTTP headers.
The YAML file that was provided with the HTTP client’s generic JSON transport for using predefined generic event definitions is now used as a linked resource. Therefore, the bundle instance file *HTTP\_Client\_bundle\_instance*.yaml is no longer created for the “generic” option (see also Adding the HTTP client connectivity plug-in to a project). To work with the newer version of this bundle, you have to call the engine_deploy tool again on your project.
Existing bundle instances for the “generic” option will be discarded. In case you modified the bundle and still want to use it, the chain definition name must be explicitly mentioned when creating a transport:
HttpTransport.getOrCreateWithConfigurations(*host*, *port*, {HttpTransport.CONFIG_DYNAMIC_CHAIN_DEF_NAME: "*modified-chain-definition-name*"});
Also, the chain name has been changed from HTTPClientGenericChain to HTTPClientGenericJSONChain. If you are using this chain, you have to update any EPL where you refer to HTTPClientGenericChain in the *chainDefnName* argument of a createDynamicChain() call to reflect the changed behavior. See also Creating dynamic chains from EPL.
The following Apama features are now deprecated or have been removed in Apama 10.7.1:
withResponse action, which was added to the Cumulocity API in Apama 10.5.2, is now deprecated for all of the predefined types (ManagedObject, Alarm, Event, Measurement, MeasurementFragment and Operation) and will be removed in a future release. It is recommended that you now use the new withChannelResponse action. See also Connectivity plug-ins enhancements in 10.7.1.Apama 10.7.0 is the successor of Apama 10.5.3. There is no version 10.6.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.7.0 includes new features, enhancements, and changes as described in the following topics.
This release of Apama ships with a new version of PySys, version 1.6.1. The 1.6.0 and 1.6.1 releases contain many enhancements including:
Several new writers for recording test results, including GitHub Actions support and a writer that produces .zip archives of test output directories.
You may wish to copy some of the new writers from the APAMA_HOME/samples/pysys/pysysproject.xml file into your own projects to benefit from these, or see the PySys documentation in the API reference for Python for more information.
A new BaseTest.assertThatGrep() method that extracts a value using a grep expression and then allows asserting its value is as expected. For extract-and-assert use cases, this approach gives much clearer messages when the assert fails than using assertGrep.
Numerous minor usability improvements to the pysys.py command line, project configuration file, and to many PySys helper methods.
Other enhancements and changes as described below.
New plug-in API
PySys 1.6.0 introduces a new plug-in API that provides a simpler and more modular way to share application-specific utility methods across multiple tests than the traditional approach of creating custom BaseTest and BaseRunner classes. Apama makes use of this to provide a new ApamaPlugin that provides easier access to Apama functionality. To use it, just add this to your pysysproject.xml:
<test-plugin classname="apama.testplugin.ApamaPlugin" alias="apama"/>
You can then access the plug-in’s methods using self.apama, for example, to start a correlator:
correlator = self.apama.startCorrelator('testCorrelator',...)
New way of configuring EPL code coverage reporting
There is a new and easier way to configure EPL code coverage reporting. Just remove the references to ApamaRunner (now deprecated) from your pysysproject.xml and replace them with:
<writer classname="apama.coverage.EPLCoverageWriter">
<property name="destDir" value="__coverage_epl.${outDirName}"/>
<!-- The following (optional) properties take a comma-separated list,
for example, "**/foo/Bar*.mon, **/baz/*.mon".
See the EPLCoverageWriter API documentation for details on these and
other available configuration parameters.
-->
<property name="srcIncludes" value=""/>
<property name="srcExcludes" value=""/>
</writer>
See the EPLCoverageWriter class in the API reference for Python for more details.
New way of validating Apama events sent by your application
Your PySys tests can now validate the events that are sent out of your Apama application in a new way. The previous approach of validating the output of the engine_receive tool using assertDiff or assertGrep/assertOrderedGrep regular expressions has several disadvantages including uninformative error messages when the match fails, and no easy way to extract the information of interest from complex Apama events. Instead we now introduce a new way to check the events: instead of CorrelatorHelper.receive(evt, channels= [...] ), call CorrelatorHelper.injectTestEventLogger(channels=[...]) to request a message to be written to the correlator log when an event is sent to the specified channels, and then in your validate() method, use ApamaPlugin.extractEventLoggerOutput(logfile) to get a list of events as a standard Python dictionary. For example:
sensor1_temps = [
# Extract only the field value(s) we care about
# (allows us to ignore unimportant information, timestamps, etc):
(evt['temperature'])
for evt in self.apama.extractEventLoggerOutput('testCorrelator.log')
# Filter to include the desired subset of events:
if evt['.eventType']=='apamax.myapp.Alert' and evt['sensorId']=='TempSensor001'
]
self.assertThat('sensor1_temps == expected',
sensor1_temps=sensor1_temps, expected=[
111.0,
120,
145.2,
])
For a concrete example, see Apama_cor_001 in the samples/pysys directory of your Apama installation.
Property settings affecting compatibility with existing projects
In previous versions of the Apama PySys helper classes, the following properties were set on the project object:
APAMA_BIN_DIRAPAMA_COMMON_JREAPAMA_LIBRARY_VERSIONAs of this version, these properties are no longer set on the project object, except if they are set in the pysysproject.xml file. All uses of APAMA_BIN_DIR and APAMA_LIBRARY_VERSION have been removed from the helper classes. APAMA_COMMON_JRE is still read from the pysysproject.xml file, but is not set on the project otherwise.
If the above three properties are not set in the pysysproject.xml file, a relative directory to APAMA_HOME is now used by default. If you have any PySys tests which explicitly use the above properties, for example, self.project.APAMA_BIN_DIR, then we recommend you rewrite them to use a path relative to self.project.APAMA_HOME instead. As a workaround, you can replicate the original behavior by providing this property in your pysysproject.xml file:
<property name="APAMA_BIN_DIR" value="${env.APAMA_HOME}/bin"/>
Other changes affecting compatibility with existing projects
The new PySys release includes some changes in behavior that may affect existing projects. For example:
os.environ during test execution.descriptor.xml/.pysystest.xml (all tests should now be defined with a pysystest.xml file).defaultAbortOnError) to match best practice and avoid the need for new projects to copy a long list of properties just to get the recommended behavior.As a result, some existing PySys projects will need changes to work with the new PySys release. However, in most cases this will just require copying in some properties to pysysproject.xml (to maintain the previous behavior) and perhaps adding default="..." values for any optional properties, but significant changes to test cases are unlikely to be needed. For full details of how to migrate an existing test project to the new release, including the project properties that you may need to copy in to retain existing behavior, see PySys Test Framework > Change Log in the API reference for Python.
Deprecated APIs
See Removed and deprecated features in 10.7.0 for a list of all PySys APIs that are deprecated in this release.
Apama 10.7.0 includes the following connectivity plug-ins enhancements:
The custom HTTP Client connectivity bundles and also the HTTP Client Generic Events EPL bundle now support sending requests via a proxy server, if configured. See also Configuring the HTTP client transport.
The Cumulocity IoT transport and the Cumulocity IoT REST API now support sending requests via a proxy server, if configured. See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).
The names of the Cumulocity IoT bundles that you can add using Software AG Designer no longer include a version number. These are the following bundles:
The Cumulocity IoT transport now allows you to set the following properties in a .properties configuration file:
CUMULOCITY_ALLOW_UNAUTHORIZED_SERVERCUMULOCITY_FORCE_INITIAL_HOST
In addition, you can now also specify the username configuration option in the form of *tenant*/*username*.For more information, see [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).
The Cumulocity IoT transport now enables you to distinguish between “create” and “update” operations for notifications that are received for managed objects, events, alarms and operations. The new constants NOTIFICATION_CREATED and NOTIFICATION_UPDATED are available for this purpose. For more information, see Receiving update notifications. This enhancement was also delivered with Apama 10.5.4.
The Apache Kafka library which is used by the Kafka transport has been updated to version 2.5.0. This includes support for TLS 1.3 and fixes several vulnerability issues. Some configurations may have changed with the new version.
Apama 10.7.0 includes the following miscellaneous enhancements and changes:
The apama_project tool now allows you to add multiple bundles to a project using a single command. Instead of using a separate add bundle *bundlename* command as in previous versions, you can now add multiple bundles using the add bundle *bundlename* *bundlename*... command. See also Creating and managing an Apama project from the command line.
The EPL memory profiler has been enhanced to include the memory used in any streams networks in your application (for example, from a in all A() within 10.0 select mean(a.i)). This includes the memory used by stream data structures such as aggregate functions and retain and within clauses. This is useful for understanding memory consumption in any application that uses streams (see also Working with streams and stream queries).
For information about how to use the EPL memory profiler, see Using the EPL memory profiler.
The jemalloc memory allocator library for Linux platforms has now been upgraded to version 5.2.1. See also Using jemalloc to optimize memory usage.
A small change has been made to the implementation of listeners using the at keyword which changes the behavior of some patterns when times repeat (for example, DST changes). Before, there were some types of pattern which would only execute during the first of repeated sections of time, and others which would execute during both. Now all patterns which match a time in a repeated section of time will execute both in the first and the second repeat of the time. For more details, see Triggering event listeners at specific times.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.7.0. The following platforms are no longer supported:
The following changes in Apama 10.7.0 affect backwards compatibility to previous Apama versions:
Software AG Designer now uses Eclipse 4.15. Eclipse 4.15 workspaces are upward-compatible, and workspaces created with older Eclipse versions are automatically migrated to 4.15 workspaces. Downward-compatibility of workspaces, however, is not supported. This means that a workspace which has been created or opened with Eclipse 4.15 can not be used with an older Eclipse version. Therefore, it is recommended that you back up your workspaces before opening them with Eclipse 4.15.
As of Apama 10.7.0, the Cumulocity IoT bundles are no longer versioned. Therefore, you have to replace the Cumulocity IoT bundles that you are using. Proceed as follows:
CumulocityIoT.yaml and CumulocityIoT.properties files over from the old, versioned bundles into the new bundles.Generic requests that are sent using the Cumulocity IoT transport now return the response body with exactly what is returned from Cumulocity IoT. In previous versions, this was put into a dictionary with an empty key. So if you have existing code such as the following
AnyExtractor dict :=
AnyExtractor(AnyExtractor(response.body).getDictionary("")[""]);
you have to replace this with:
AnyExtractor dict := AnyExtractor(response.body);
To be more compliant with the HTTP RFC and IANA standards, when automatically constructing the content-type header for HTTP requests, the HTTP client transport now only appends the charset parameter from the metadata.charset field if the content type is in the text/* domain. If you need to add a charset for any other content type, you can explicitly add it by setting the metadata.http.headers.content-type field to include the charset modifier. See Handling HTTP headers for more details.
In 10.7, the defaults of the two configuration options cookieJar and followRedirects, which are used by the HTTP client transport, have changed. Since Apama 10.3.1, Software AG Designer projects created with the HTTP Client connectivity bundle have had these set to true in the bundle configuration, and for those projects nothing will change whether you had the default or changed the configuration in your bundle instance (see also the backwards compatibility note in Connectivity plug-ins enhancements in 10.3.1). For older projects or those using the HTTP client transport in some other fashion, the default is now true. To disable these options, set them to false in the transport configuration. See Configuring the HTTP client transport for more details.
The JSON codec connectivity plug-in now safely handles empty payloads and now converts "" to {} instead of throwing an error in the towards-host direction. See also The JSON codec connectivity plug-in.
If the engine_deploy tool is executed with an --include or --exclude option with an argument containing an unsupported pattern (such as ./), the tool will now exit with a non-zero exit code, abort execution and print an error message. With previous versions, a warning was printed in this case. See also Deploying a correlator.
To avoid confusion, creating user status items through either EPL or from plug-ins now strips any leading or trailing whitespace from key names. This avoids having two distinct keys with visually identical key names. This means that two keys which previously differed only in whitespace will now cause a conflict for having the same name when using the createUserStatusItem interface. When using the setUserStatus interface, they will now update the same value. Keys which previously had trailing whitespace that caused them not to appear in the Prometheus statistics will now appear. Status values in the correlator status REST API will now appear without whitespace if they had them before. In addition, empty keys, or keys only consisting of whitespace, are now illegal and will throw an exception. See also Using the Management interface and User-defined status reporting from connectivity plug-ins.
The Apama images that are available on Docker Hub (apama-correlator, apama-builder and apama-cumulocity-jre) have been changed to use Red Hat UBI 8 minimal as their base operating system. This may affect users who are extending those images. In addition to the upgrade from 7 to 8, the switch to a minimal image results in much smaller images, but as a result some packages that a user relies on may be missing. In particular, this includes the yum tool for installing packages. Instead, the microdnf tool must be used. We have taken the opportunity to also ensure that all of the images will work with non-UTC time zones out of the box. However, the only locale which is shipped is the new default C.UTF-8 locale (previously, it was en_US.UTF-8). Any other locales which are required must be added to the image. See also Deploying Apama applications with Docker.
In Apama 10.7.0, the database driver moved. Software AG Designer projects using the old location for the driver classpath will need to update the configuration of their JDBC adapter to point to the new location.
The new PySys release includes some changes in behavior that may affect existing projects. See PySys changes in 10.7.0 for more information.
Note: There may also be removed features that affect backwards compatibility. See Removed and deprecated features in 10.7.0 for more information.
The following Apama features are now deprecated or have been removed in Apama 10.7.0:
With the new PySys release, several PySys APIs are now deprecated. This includes the following:
-Xeplcoverage, use -XcodeCoverage to enable EPL code coverage. If you need to enable EPL coverage without enabling any other types of code coverage in your project, this can be achieved with -XeplCoverage (note the capitalization of “C”).ApamaRunner is deprecated. Remove it from your pysysproject.xml and instead configure EPL code coverage as described in PySys changes in 10.7.0.ApamaBaseTest is not currently deprecated, but its use is no longer encouraged. New tests subclass pysys.basetest.BaseTest rather than ApamaBaseTest.ThreadFilter class is deprecated and will soon be removed. Usually, it is not recommended to suppress log output and better alternatives are available, for example, the quiet=True option for BaseTest.startProcess, and the BaseTest.disableLogging() method.pysys.basetest.BaseTest.addResource is deprecated and will be removed in a future release. Therefore, change your tests to stop using it. Use pysys.basetest.BaseTest.addCleanupFunction instead.
See also PySys changes in 10.7.0.In previous releases, the correlator and IAF status key names numSnapshots and virtualMemorySize were deprecated and replaced with the keys persistenceNumSnapshots and virtualMemoryMB. The deprecated keys have now been removed and will no longer appear in the status output. This applies to the following:
engine_management, component_management and iaf_management tools (see also Shutting down and managing components), andThe deprecated SOAP-based Web Services Client IAF adapter has been removed. It is recommended that you now use a REST-based HTTP server instead and connect to it with the HTTP client transport, if possible (see The HTTP Client Transport Connectivity Plug-in). However, if you really need to communicate with a SOAP-based server, use the webMethods Integration Server and connect to it from Apama using the Digital Event Services transport.
Support for Software AG’s Event-Driven Architecture (EDA) has been removed. This was deprecated in Apama 10.1. Instead of using EDA events, it is recommended that your Apama applications use digital event types, as provided by Software AG Digital Event Services.
Several minor features which have been deprecated over the past few years have been removed. The following table lists all of the removed APIs or options and their recommended replacements.
|
Removed item |
Replacement |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
No replacement. |
|
|
|
|
|
No replacement, all connections are persistent now. |
|
ADBC: |
|
|
ADBC: |
|
|
ADBC: |
|
|
ADBC: |
No replacement. |
|
ADBC: |
No replacement. |
|
ADBC: |
No replacement. |
|
ADBC: |
No replacement. |
|
ADBC: |
|
|
ADBC: |
|
|
C++ and Java connectivity APIs: |
|
|
C++ EPL plug-in: |
|
|
Java EPL plug-in: |
|
|
|
|
|
PySys: |
|
|
|
No replacement. |
|
|
|
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.5.4 includes new features, enhancements, and changes as described in the following topics.
Apama 10.5.4 includes the following connectivity plug-ins enhancements:
false, but you can turn it on by setting concurrentChains to true in the YAML configuration file. See also Configuring the HTTP server transport.numChains is now available for the HTTP server transport. It indicates the number of active connections into the HTTP server instance. See also Monitoring status for the HTTP server.NOTIFICATION_CREATED and NOTIFICATION_UPDATED are available for this purpose. For more information, see Receiving update notifications.The following changes in Apama 10.5.4 affect backwards compatibility to previous Apama versions:
"" to {} instead of throwing an error in the towards-host direction. See also The JSON codec connectivity plug-in.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.5.3 includes new features, enhancements, and changes as described in the following topics.
Apama 10.5.3 now ships a newer version of the PySys test framework, version 1.5.1.
If you are already familiar with PySys, it is strongly recommended to read the full Change Log in detail to familiarize yourself with the new features and new best practices for improving how you write tests. There is more detail on many PySys features in the User Guide. You can find the Change Log and User Guide in the API reference for Python, which you can also access via the doc/pydoc/index.html file of your Apama installation.
Key features for the new PySys 1.5.1 release include:
BaseTest class (/doc/pydoc/pysys-doc/BaseTest.html) as this is the main page that you will want to refer to regularly when writing PySys tests.assertThat, assertGrep and waitForGrep (formerly known as waitForSignal) have lots of new functionality.<project-help> element that allows you to customize the pysys run --help output.A new Apama helper class for PySys called ProjectHelper is now available in the apama.project package. It provides methods for creating, manipulating and deploying Software AG Designer-compatible projects from your PySys tests and thus provides access to the apama_project tool; see Creating and managing an Apama project from the command line.
Apama 10.5.3 includes the following connectivity plug-ins enhancements:
set rule of the Mapper codec, you can now set the value of a field to a specific value, regardless of whether that field already exists in the event. For more information, see The Mapper codec connectivity plug-in.Apama 10.5.3 includes the following miscellaneous enhancements and changes:
--loglevel option of the correlator executable or by using a YAML configuration file on correlator startup. In previous releases, this was only possible for a running correlator via the -r option of the engine_management tool. As a result, these messages are now logged with a DEBUG prefix when enabled, rather than INFO. There is also a small change to the format of the messages logged. See also Viewing garbage collection and application events information.jvmNumThreads (REST API) and sag_apama_correlator_jvm_num_threads (Prometheus). These statistics will exist when the Java Virtual Machine (JVM) which is embedded in the correlator has been enabled. See also List of correlator status statistics.@since tag can now be used in ApamaDoc comments on event members and constants as well as for events and actions. See also Inserting ApamaDoc tags.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.5.2 includes new features, enhancements, and changes as described in the following topics.
Some events have been added to the Cumulocity API to allow easy retrieval of information in some predefined areas.
Authorization header or OAuth cookies of an incoming request to validate the roles of that user.The Cumulocity API now allows the ability to retrieve a response for the creation or update of the predefined types (Alarm, Operation, Event, Measurement, MeasurementFragment and ManagedObject) by using the newly added withResponse action. Headers can now be set on the request, for example, to determine the processing mode within Cumulocity IoT.
For detailed information, see The Cumulocity transport connectivity plug-in.
Apama 10.5.2 includes the following miscellaneous enhancements and changes:
Using the new --configLog *file* option of the correlator executable, you can now specify the path to the configuration log file. The configuration log file contains the correlator’s configuration, that is, the contents of all YAML configuration files and properties files as well as the correlator startup arguments and environment. The configuration log is a good way to capture useful diagnostic information from the correlator’s startup in performance-critical situations where it is not acceptable to enable the input log. This option was already added with 10.5.1. However, the documentation has only been updated with 10.5.2. See also Starting the correlator.
The results of a eplMemoryProfileMonitorInstanceDetail request are now returned using the following sort order:
The startup log stanza is now repeated even when logging is going into stdout/stderr via the --rotateLogs management request while using the engine_management tool (see also Shutting down and managing components) or by using SIGHUP in Linux.
The following changes in Apama 10.5.2 affect backwards compatibility to previous Apama versions:
list_t or map_t container type while modifying it, these iterators have been modified to assert (or throw an exception) if accessed after the parent container has been modified. These will fail with a message saying that the iterator has been invalidated by modification of either a map or a list. Any code which now throws an exception will have been unsafe previously (even if it did not crash). The new check simply exposes the existing bug in the user’s code. See also C++ data types.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.5.1 includes new features, enhancements, and changes as described in the following topics.
The PySys apama.correlator and apama.iaf modules have been updated to accept arguments such as logfile and verbosity via the constructor as an alternative to specifying them in the start() method. Any individual arguments specified in the start(...) method will take precedence over the constructor arguments. See the API reference for Python for more information.
Apama 10.5.1 includes the following connectivity plug-ins enhancements:
.properties configuration file. For more information, see Monitoring status for Cumulocity and [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).Apama 10.5.1 includes the following EPL enhancements:
The ondie action can now have either no argument or the arguments optional<com.apama.exceptions.Exception> and optional<string>. See also About executing ondie() actions.
In a sequence of sequences, in a dictionary of sequences or in cast operations, it is no longer required to add a space between two closing angle brackets (>). A syntax error no longer occurs when the space is missing. For example, instead of
dictionary<decimal, <sometype> >
you can now write
dictionary<decimal, <sometype>>
Both are valid EPL code.
Apama 10.5.1 includes the following miscellaneous enhancements and changes:
/request REST API on the correlator can now be called with a JSON payload as an alternative to the existing XML payloads. Responses can also be received in JSON. For more details, see the API reference for Component Management REST APIs.PASSWORD, PASSWD or CREDENTIALS with the string “<redacted>” for security.apamadoc command-line tool which can be used to generate ApamaDoc. For more details, see Generating ApamaDoc from the command line.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Due to these upgrades, the following versions are no longer supported:
Apama 10.5.0 and 10.5.1 support running on RHEL 8.0. If you build your own C++ EPL or connectivity plug-ins on RHEL 8.0, you will have to add the following flags to your compiler command-line options:
-D_GLIBCXX_USE_CXX11_ABI=0 -Wno-parentheses -Wno-class-memaccess -Wno-aligned-new
The following changes in Apama 10.5.1 affect backwards compatibility to previous Apama versions:
If the correlator or the IAF is configured to log to a file, then the log file being used is now output to stderr. In the case of the correlator, this is the full path and filename, whereas the IAF only logs the filename.
The shutdown order of the connectivity chain components has changed. In previous versions, the codecs were shut down first, followed by the transport and host. As of this version, the host is shut down first (that is, calls into the host component of the chain are unblocked), followed by the codecs and the transport. See also Lifetime of connectivity plug-ins.
When you develop your own connectivity plug-ins, you may have to update your plug-ins to reflect the changed behavior. Make sure that your testing includes shutting down the correlator (for example, using the PySys correlator.shutdown() method), since it is possible that this change in behavior makes shutdown issues in user-defined plug-ins more likely to show up.
Apama 10.5.0 is the successor of Apama 10.3.1. There is no version 10.4.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.5.0 includes new features, enhancements, and changes as described in the following topics.
A new Apama base image called “apama-cumulocity-jre” is available on Docker Hub. It can be used as a smaller image for Cumulocity microservices. It only contains a correlator and the Cumulocity connectivity pieces. There is a JRE but no JDK, and the image does not come with Python support by default. See also Apama image for Cumulocity microservices.
For more details, see the installation instructions for this image on Docker Hub.
Note: This image is also available for Apama version 10.3.1.
The correlator, the IAF, the dashboard servers, and the com.apama.util.Logger class used by the Java client library now use Log4j 2 rather than Log4j 1.
In Software AG Designer, Apama projects with a Java nature now use Log4j 2. If you are still using Log4j 1.2, you must now upgrade your applications to use Log4j 2 or manually add the Log4j-1-to-2 bridge or the Log4j-1-to-SLF4J bridge to your classpath from common/lib/ext/log4j.
In Log4j 2, the configuration file format, the system property that specifies the configuration file location, and the Java package are all completely different to Log4j 1. So if your application directly uses the Log4j 1 API (org.apache.log4j.*) or you have any custom Log4j configuration files (for example, a log4j.properties file), you will need to change them when upgrading to this Apama release. See the Log4j website at https://logging.apache.org/log4j/2.x/ for detailed information on Log4j 1.x migration.
The Log4j 1.2 configuration files are no longer used. Thus, the log4jConfigFile configuration option for controlling the verbosity, which was previously used by the Kafka and Digital Event Services transports, is no longer supported in an Apama project. If you are using the log4jConfigFile configuration option, you should now set the log levels in the YAML configuration file for the correlator instead, using the correlatorLogging entry. See also Setting correlator and plug-in log files and log levels in a YAML configuration file.
If you are using com.apama.util.Logger within the correlator or IAF processes, no changes should be required. However, if you use com.apama.util.Logger in a standalone Java application, perhaps using the Apama Engine Client or Scenario Service APIs, you will need to provide a Log4j 2 configuration file even if previously you did not, as the com.apama.util.Logger class no longer performs automatic configuration of Log4j. Again, see the Log4j website for sample configuration files and detailed information on the configuration file format.
You may also need to take action if you use a third-party library inside the correlator that itself performs logging, for example, a connectivity plug-in, EPL plug-in, or JMS client library. The correlator’s root classloader now contains Log4j 2, the widely-used SLF4J logging facade, and the Log4j 2 SLF4J implementation binding. So any libraries using SLF4J or Log4j 2 will correctly write to the correlator log file automatically, and their log levels can be customized using correlatorLogging in the YAML configuration file (as described in Setting correlator and plug-in log files and log levels in a YAML configuration file). The root classloader also contains the Log4j 1.x bridge (log4j-1.2-api.jar) which redirects calls to most of the Log4j 1 API through to Log4j 2, so provided you do not use internal features such as programmatic configuration, Log4j 1 libraries should work fine. However, we recommend that you transition away from Log4j 1 since it has reached end-of-life status. If some other logging library is used which does not support SLF4J directly, you should add a jar that bridges from that library’s API to SLF4J or Log4j2 if possible.
Previous Apama versions included the jars for Log4j 1 and SLF4J in the lib directory of your Apama installation. In this release, the SLF4J and Log4j logging jars are now located in the Software AG installation directory under common/lib/ext and common/lib/ext/log4j. Any build scripts (for example, Apache Ant build.xml scripts) that need these jars at compilation time should be updated to point to the new location.
The IAF-based Web Services Client adapter now writes some of its logging output to standard output rather than to a hardcoded file called iaf.log as in the previous version, though the messages written to standard output are likely to be useful only for debugging purposes. This behavior can be customized in APAMA_HOME\adapters\config\log4j-iaf.xml if needed. If you have an existing configuration, you should edit the WebServicesClientAdapter.xml file in your project to use the Log4j 2 configuration file and system property instead of the old log4j.configuration property:
<property name="log4j.configurationFile" value="file:///@APAMA_HOME@/adapters/config/log4j-iaf.xml"/>
Apama 10.5.0 now ships a newer version of the PySys system testing framework, version 1.5.0.
PySys 1.4.0 includes improvements to the writer API such as integrated support for running tests using Travis CI. New test projects can be created with the new pysys.py makeproject command. Among other additions, it includes support for creating background threads from test cases, a new skipTest() method, an IS_WINDOWS constant, and a new stdouterr argument for ProcessUser.startProcess() which makes it easier to specify to which files process output is to be written. There are also some important bug fixes in areas such as international (I18N) character support, logging, assertion error handling, cleanliness of the startProcess environment, and process monitoring.
PySys 1.5.0 brings some significant new features, including support for running a test in multiple modes and pysysdirconfig.xml files that allow you to specify defaults that apply to all test cases under a particular directory - such as groups, modes, a prefix to add to the start of each test identifier, and a numeric hint to help define the execution order of your tests. There are also new features in the pysys run and pysys print command lines, and a host of small additions to the API to make test creation easier, for example, assertEval, copy (with filtering of each copied line) and write_text (for easy programmatic creation of files in the output directory).
If you are already familiar with PySys, it is strongly recommended to read the full PySys change log in detail to familiarize yourself with the new features and new best practices for improving how you write tests. You can find the PySys change log (PYSYS_CHANGELOG.html) in the doc/pydoc directory of your Apama installation, or you can access it from the API reference for Python topic. There is more detail on many PySys features in the user guide (PYSYS_USERGUIDE.html) available in the same location.
PySys 1.4.0 and 1.5.0 include some changes that might require changes to existing tests, including:
assert* methods. Incorrect arguments that were silently ignored in previous releases will now result in a test failure to alert you to the fact that they need fixing.startProcess no longer include environment variables of the parent PySys process by default. Note that this does not affect correlator or IAF processes which continue to be started with the same environment variables as PySys itself.pysysproject.xml file in the root directory containing your tests.pysystest.xml XML descriptors will now prevent any tests from running (previously, they were just logged). Since an invalid descriptor prevents the associated test case from reporting a result, the new behavior ensures such mistakes will be spotted and fixed promptly. If you have any non-PySys files under your PySys project root directory with names such as descriptor.xml which PySys would normally recognize as test cases, you can avoid errors by adding a .pysysignore file to prevent PySys looking into that part of the directory tree.ProcessUser.mkdir now returns the absolute path (including the output directory) instead of just the relative path that was passed in. This makes it easier to use in-line while performing operations such as creating a file in the new directory. Code that relied on the old behavior of returning the path that was passed in may need to be updated to avoid having the output directory specified twice. If you are using os.path.join, then no change will be required.BaseRunner subclass, note that the self.output variable in BaseRunner is no longer set to the current directory, but instead to a pysys-runner-*OUTDIR* subdirectory of the test root (or to *OUTDIR*/pysys-runner if *OUTDIR* is an absolute path). This directory is not created (or cleaned) automatically, so if you need to write to self.output, you should add code to do that to your runner.PROJECT variable in the constants module is deprecated. Use self.project instead (which is defined on classes such as BaseTest and BaseRunner).If you have existing PySys tests, we recommend copying some or all of the properties from APAMA_HOME/samples/pysys/pysysproject.xml into your own pysysproject.xml, especially supportMultipleModesPerRun, defaultAbortOnError, defaultIgnoreExitStatus, and the default-file-encodings section. However, this will change the behavior of existing tests. See the PySys documentation and change log for further details.
We also strongly recommend you to update the requires-pysys and requires-python elements in your pysysproject.xml, to ensure that anyone attempting to run them using an older version than they were designed for gets a clear error message rather than just a subtle difference in behavior.
See the Upgrade guide and compatibility section in the above mentioned PySys change log for further details.
The documentation has been updated with information on how to get started with PySys. See Testing Apama applications Using PySys.
Apama 10.5.0 includes the following connectivity plug-ins enhancements:
mqttClientId and cleanSession are now available for this purpose. See also Configuring the connection to MQTT.onApplicationInitialized() method in the correlator. As long as this method is not called, the server returns a “503 Host Not Ready” HTTP response. See also HTTP response codes.responseCompression is available for the chain on the HTTP server. See also Configuring the HTTP server transport.metadata.http.headers.content-encoding can now be added to the client request. See also Mapping events between EPL and HTTP client requests and Content-encoding example.A new version (10.5) of the Cumulocity IoT transport connectivity plug-in is now available. See also Migrating the Cumulocity IoT transports to 10.5.
The Cumulocity IoT transport now forwards delete notifications for managed objects, measurements and events to Apama using the following EPL events:
com.apama.cumulocity.ManagedObjectDeletedcom.apama.cumulocity.MeasurementDeletedcom.apama.cumulocity.EventDeleted
See the API reference for EPL (ApamaDoc) for more information.A new request/response pair has been added to the Cumulocity IoT transport event API. This allows you to make arbitrary requests to the Cumulocity API to support request types which are not provided as built-in events in the API. See also Invoking other parts of the Cumulocity REST API.
A new CumulocityRequestInterface event has been added to the Cumulocity IoT transport which allows you to invoke other microservices within Cumulocity IoT from an EPL application. This can be used from an Apama instance outside of Cumulocity IoT and within Cumulocity IoT, either as an EPL application or a custom microservice. See also Invoking microservices.
The subscribeToOperations configuration option of the Cumulocity IoT transport is set to false by default when it is not explicitly specified in the YAML configuration file. However, for your convenience, when you add a new Cumulocity Client connectivity bundle in Apama Plugin for Eclipse or by using the apama_project tool, this option is now already set to true in the resulting YAML configuration file.
filterOnContentType is set to true. See also The JSON codec connectivity plug-in.Apama 10.5.0 includes the following enhancements in Software AG Designer:
In Software AG Designer, some of the commands, control names and window titles containing the old term “MonitorScript” have been replaced with new names. The following table lists the new names:
| Old name | New name |
|---|---|
| MonitorScript Build Path | Apama Build Path |
| MonitorScript File Selection | Add Apama External Dependency |
| MonitorScript Path Variables | Apama Build Path Variables |
| MonitorScript Search | Apama Search |
| Export as MonitorScript | Scenario EPL |
| Hide non-MonitorScript projects | Hide non-Apama projects |
| Apama MonitorScript Editor | Apama EPL Editor |
| Launch/Inspect MonitorScript Application | Launch/Inspect Apama Application |
| Edit MonitorScript path | Edit EPL Path |
| MonitorScript Export Wizard | Scenario EPL Export Wizard |
Also, the directory samples/monitorscript has been replaced with samples/epl.
Apama 10.5.0 includes the following EPL enhancements:
10.rand() are now valid. Previously, this resulted in a compiler error as the compiler used to treat “10.” as a float. The workaround in previous versions was to either add a space after the integer (for example, 10.rand()) or to enclose the integer in parentheses (for example, (10).rand()). This is no longer required.Apama 10.5.0 includes the following miscellaneous enhancements and changes:
physicalCores as the number of cores available to the correlator, instead of the number that exists on the host machine. A new metric availableMemoryMB has been added to represent the maximum available memory in MB that the correlator has access to. See also Restricting correlator resource usage with control groups.correlatorLogging section, supporting the ability to change the log level and file used for individual categories. The categories in the correlatorLogging section are treated hierarchically. For example, setting the level for foo.bar will also set the level for foo.bar.baz (unless the log level for foo.bar.baz is also specified). This provides the same capabilities as the existing eplLogging section in a YAML configuration file. See Setting correlator and plug-in log files and log levels in a YAML configuration file for more details.SAG_HOME/common/lib/ant instead of APAMA_HOME/third_party/apache_ant. In addition, Apache Ant has been upgraded from version 1.9.7 to version 1.10.5.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.5.0. The following platforms are no longer supported:
The following changes in Apama 10.5.0 affect backwards compatibility to previous Apama versions:
at) has been changed from CRIT to INFO. See also Specifying log statements.WARN level.onApplicationInitialized() in the correlator before sending requests to the HTTP server. Any requests that are sent before calling this method will fail with a “503 Host Not Ready” response code. See Sending and receiving events with connectivity plug-ins for more information on onApplicationInitialized().url configuration option is now mandatory. The configuration of a tenant identifier (tenant configuration option) is now optional. See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).apama-correlator and apama-builder images published on Docker Hub and other locations now use Red Hat’s Universal Base Images for RHEL 7 as their base images, rather than CentOS which was used in previous releases. The apama-cumulocity-jre image still uses Ubuntu 18.04 (Bionic Beaver).apama-correlator image published on Docker Hub now no longer contains a full JDK. Instead it only contains a JRE suitable for running Java applications in the correlator. If you need to build Java code at runtime, we recommend using a multi-stage build as documented in Building Apama projects during the Docker build. If you must have the full JDK at runtime, then you should use the apama-builder image as your base.Note: There may also be removed features that affect backwards compatibility. See Removed and deprecated features in 10.5.0 for more information.
The following Apama features are now deprecated or have been removed in Apama 10.5.0:
Apama’s Event Modeler in Software AG Designer is no longer supported and has been removed. This includes any associated scenario files (.sdf), scenario block files (.bdf), scenario function files (.fdf), PySys test support (CorrelatorHelper.injectScenario() was removed), Ant macro definitions in apama-macros.xml (see also the list below), and engine_deploy support (the command-line option --generateEventModelerConfig was removed). See also the announcement in Removed and deprecated features in 9.12. The Scenario Browser view and the Scenario Service API are still fully supported, but they are now used only for DataViews and queries.
Important:
You have to re-export or edit all Ant scripts exported from Software AG Designer - even if they do not use scenarios - as several scenario-related macros and macro attributes that were present in the default build.xml file have now been removed.
To manually edit your build.xml file:
<generate-event-modeler-config.../>, and<inject-project-init-list.../> to remove the following attributes:
scenario-config=temp-location=
If your project uses scenarios and/or you have customized your build.xml file, you have to make additional changes.If you wish to clean up your build.xml file to remove redundant scenario-related items, the full list of removed items is:
The following properties have been removed:
catalogs.dirscenarios.direvent_modeler.configevent_modeler.xmlThe following attributes have been removed from inject-project-init-list:
scenario-config=scenario-debug=
In addition, temp-location= is no longer needed. So it is recommended that you remove this.The following macros have been removed:
generate-event-modeler-configstart-event-modelerrun-event-modelergenerate-blockgenerate-scenariogenerate-scenariosinject-scenarioinject-scenariosdelete-scenarioPython 2.7 is no longer supported. You should therefore port any PySys testcases written for Python 2.7 to Python 3.x when possible. Python 2.7 is no longer being tested by Software AG but may continue to work with PySys. You can choose to continue to use your own installation of Python 2.7 at your own risk. See also the announcement in Removed and deprecated features in 10.3.0.
The Cumulocity IoT transports for 9.0 and 9.8 have been removed. A new version (10.5) of the Cumulocity IoT transport connectivity plug-in is now available. See Migrating the Cumulocity IoT transports to 10.5 for details.
The requestAllDevices configuration option for configuring the Cumulocity IoT transport is now deprecated. You should now explicitly request for all available devices on startup using the com.apama.cumulocity.FindManagedObject API.
The pageSize and useBatching configuration options for configuring the Cumulocity IoT transport are now also deprecated and have been removed from the documentation. This is now handled automatically. Specifying values for either of these configuration options will be ignored.
See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport).
The documentation for the old C and C++ EPL plug-in APIs that were supported until version 9.12 has been removed. These APIs are deprecated since 10.0 and will be removed in a future release (see also New API for writing EPL plug-ins in C++ in 10.0). For details on the current C++ EPL plug-in API, see Writing EPL plug-ins in C++. For your convenience, the information on how to migrate the old plug-ins has been added to the Release Notes for 10.0; see Migrating the C and C++ EPL plug-ins written in 9.12 or earlier.
com::apama::setLogFD has been removed from the C++ API. If you are using this, it is recommended that you use setLogFile instead. See also the API reference for C++ (Doxygen).
For the Java and C++ connectivity plug-ins, the LOGGER field and use of the legacy constructor signatures that do not use TransportConstructorParameters or CodecConstructorParameters have been removed.
For C++, the following macros have been removed:
SAG_DECLARE_CONNECTIVITY_CODECSAG_DECLARE_CONNECTIVITY_TRANSPORT
See also the announcement in Removed and deprecated features in 10.0.In addition, the previously deprecated name field was removed from the C++ connectivity plug-ins. You should now use pluginName instead.
Calling a non-static action on a type is now deprecated and will be removed in a future release. Instead, you should now call the method directly on an instance.
For example, you should no longer call a non-static action on a type as follows:
integer range := 100;
integer.rand(range);
Use the following instead:
integer range := 100;
range.rand();
Another example - you should no longer use the following:
event A {
action nonStaticAction() { log "non static" at INFO; }
static action staticAction() { log "static" at INFO; }
}
A.nonStaticAction(new A); // Results in warning
Use the following instead:
A a = new A;
a.nonStatic(); // This is preferred way
The enqueue statement (without a to) is now deprecated and will be removed in a future release. It is recommended that you use the send...to statement instead. For applications which require enqueuing a string rather than an event object, you can either use the any type (where it is an event but not statically known at compile time) or you can switch to a second correlator using the engine_connect tool and the emit statement, which still supports bare strings (for things like &TIME events).
Use of Universal Messaging from the IAF is now deprecated and will be removed in a future release. It is recommended that you now change any IAF-based adapter configurations using Universal Messaging with a <universal-messaging> element in the configuration file to use an <apama> element to talk directly to the correlator. See Apama correlator configuration.
The SOAP-based Web Services Client IAF adapter is now deprecated and will be removed in a future release. It is recommended that you now use a REST-based HTTP server and connect to it with the HTTP client transport, if possible (see The HTTP Client Transport Connectivity Plug-in). However, if you really need to communicate with a SOAP-based server, use the webMethods Integration Server and connect to it from Apama using the Digital Event Services transport.
When using FindMeasurement from the Cumulocity API, use of fragmentType as a filter is now deprecated. You should instead use both of valueFragmentType and valueFragmentSeries together to filter measurements. Use of just one will be significantly slower. See also Querying for measurements.
In order to migrate an existing Apama project to 10.5 which is using the removed 9.0+ and/or 9.8+ Cumulocity IoT connectivity bundles, you must proceed as described below. If you are using Cumulocity IoT with the built-in Apama EPL editor, then no migration steps should be needed.
To migrate the Cumulocity IoT transports to 10.5:
Back up the CumulocityIoT.properties files for 9.0+ and/or 9.8+. These are located in the config/connectivity folder of your project.
Remove the Cumulocity Client 9.0+ and/or Cumulocity Client 9.8+ connectivity bundles from your project.
If present, remove the Cumulocity IoT > Event Definitions for Cumulocity 9.0+ and/or Cumulocity IoT > Event Definitions for Cumulocity 9.8+ EPL bundles from your project.
If present, remove the Cumulocity IoT > Utilities for Cumulocity 9.0+ and/or Cumulocity IoT > Utilities for Cumulocity 9.8+ EPL bundles from your project.
Add the Cumulocity IoT > Cumulocity Client 10.5+ connectivity bundle.
Add the Cumulocity IoT > Event Definitions for Cumulocity 10.5+ EPL bundle.
Add the Cumulocity IoT > Utilities for Cumulocity 10.5+ EPL bundle.
Populate the CumulocityIoT.properties file for 10.5 with the data from your backup.
Use the com.apama.cumulocity.Util.generateReqId() action to generate all request identifiers (reqId).
Important:
There should be no use of integer.getUnique() or integer.incrementCounter() to generate request identifiers.
For detailed information on this connectivity plug-in, see The Cumulocity transport connectivity plug-in.
You can install Apama 10.3.1 by selecting “2019 April Innovation Release” in the Software AG Installer or by using the Software AG Update Manager or Command Central to upgrade an existing Apama 10.3.0 installation.
Important: Installing Apama 10.3.1 as an “Innovation Release” from the Software AG Installer will limit its support to the end of support for the “2019 April Innovation Release”. Only installation via an upgrade from 10.3.0 will result in support until the end of support for 10.3.0.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.3.1 includes new features, enhancements, and changes as described in the following topics.
A new command-line tool, the apama_project tool, is now available. It can be used to:
Each Apama project generated by this tool is compatible with Apama Plugin for Eclipse and can seamlessly be imported into Apama Plugin for Eclipse as an Apama project.
For detailed information on the new tool, see Creating and managing an Apama project from the command line.
A new codec, the Base64 codec connectivity plug-in, is now available. The Base64 codec performs bidirectional translation between binary payloads and a string that is the Base64-encoding of that payload. The codec is able to operate on arbitrary fields of the message, or the entire payload. For detailed information, see The Base64 codec connectivity plug-in.
A new codec, the Batch Accumulator codec connectivity plug-in, is now available. The Batch Accumulator codec can be used to automatically batch messages from an unbatched transport such as the HTTP server transport when sending messages towards the correlator. This can be used to increase the maximum throughput of the system. For detailed information, see The Batch Accumulator codec connectivity plug-in.
A new codec, the Message List codec connectivity plug-in, is now available. The Message List codec can be used to automatically combine a batch of messages produced by the correlator into a single message containing the original batch as a list. This can be used with a service which supports a batch as a list to avoid the system being limited by request round-trip times. For detailed information, see The Message List codec connectivity plug-in.
Apama 10.3.1 includes the following connectivity plug-ins enhancements:
It is now possible for the response to an HTTP request to the HTTP server transport to be handled and created by an EPL application, instead of automatically returning an empty Accepted response. This allows the HTTP server to implement a request/response REST protocol and not just an event submission protocol. For detailed information, see Handling responses in EPL.
The following new configuration options are available for handling responses in EPL: automaticResponses and responseTimeoutMs. For more information, see Configuring the HTTP server transport.
A new metadata.requestId field is available for requests from the transport to EPL, and several new fields are available for EPL-supplied responses. For more information, see Mapping events between EPL and HTTP server requests.
For an example, see EPL-controlled responses.
The HTTP server transport now supports gzip and deflate decoding. For more information, see About the HTTP server transport.
The HTTP server transport now supports HTML form decoding and can decode multipart/form-data or application/x-www-form-urlencoded media types to a dictionary payload. For more information, see Handling HTTP form decoding.
Accept-Encoding header appropriately. For more information, see About the HTTP client transport.multipart/form-data or application/x-www-form-urlencoded media types. For more information, see Handling HTML form encoding.followRedirects configuration option to true. For more information, see Configuring the HTTP client transport.cookieJar configuration option to true. The HTTP client then stores in memory all cookies received from the server and adds them to subsequent outgoing requests. For more information, see Configuring the HTTP client transport and Dealing with cookies.Note: For backwards compatibility, the new followRedirects and cookieJar configuration options are set to false by default when they are not explicitly specified in the YAML configuration file. However, for your convenience, when you add a new HTTP Client connectivity bundle in Software AG Designer, these options are already set to true in the resulting YAML configuration file.
pageSize parameter while querying for resources. This applies to all versions of the Cumulocity bundles (9.0 and 9.8+). See also the following topics:
fromDate and toDate parameters while querying for operations. See also Querying for operations.requestAllDevices configuration option is set to false. See also [Configuring the Cumulocity transport(/standard-connectivity-plugins/the-cumulocity-iot-transport-connectivity-plug-in/#configuring-the-cumulocity-iot-transport). You should now explicitly request for all available devices on startup using the com.apama.cumulocity.FindManagedObject API. For more information, see Sample EPL.eventTypes, fields, and encoding. For more information, see The String codec connectivity plug-in.Apama 10.3.1 includes the following enhancements in Software AG Designer:
com.apama.cumulocity package in the API reference for EPL (ApamaDoc).com.apama.cumulocity.Util event in the API reference for EPL (ApamaDoc).mqtt: in Apama, rather than consume all channels in the whole system. This corresponds to the default setting of the channelPrefix configuration option as described in Configuring the connection to MQTT.Apama 10.3.1 includes the following EPL enhancements:
getActionParameterTypes() has been added for the any type. It returns the required types as a sequence of strings, allowing reflection on action parameters, built-in methods, closures, and plug-in methods. See the API reference for EPL (ApamaDoc) for more information.nextafter method on float and decimal with nextAfter. The previous method name will still work for compatibility, but we recommend you to begin using nextAfter instead.getConfigProperties method on the com.apama.correlator.Component event. See also Using the Management interface and the API reference for EPL (ApamaDoc).Component.incrementUserStatus allows you to atomically increment or decrement a status value from multiple contexts. For more details, see the API reference for EPL (ApamaDoc).Apama 10.3.1 includes the following miscellaneous enhancements and changes:
-XX:+HeapDumpOnOutOfMemoryError argument has been added to the default JVM flags for the Java process within the correlator. This means that whenever a Java OutOfMemoryError is thrown, a java_pid*number*.hprof file is created containing the Java heap. If you want to disable this new behavior, start the correlator with -J-XX:-HeapDumpOnOutOfMemoryError (with a minus sign instead of a plus sign).apama_env script to set up your console environment to run Apama commands, you can also use it as a one-shot wrapper to run a single Apama command without changing your environment. For more details, see Setting up the environment using the Apama Command Prompt.HEALTHCHECK instruction to monitor the status of the correlator. This will be monitored by Docker and other tooling automatically.sag_apama_correlator_slowest_input_queue_size_total and sag_apama_correlator_slowest_output_queue_size_total metrics. See also List of correlator status statistics. In addition, the following Prometheus metrics have been added where the labels give information about the status of the current license or about the configuration: sag_apama_correlator_licensedata and sag_apama_correlator_metadata. See also Monitoring with Prometheus.engine_deploy tool now always generates the connectivity.yaml file for a project which contains connectivity bundles. With previous versions, this file was generated only when a deployment file was provided to the engine_deploy tool. See also Deploying a correlator.client_status_names.hpp has been added to the public includes to contain constants for all standard status values within the correlator. See also List of correlator status statistics.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.3.1.
Due to these upgrades, the following versions are no longer supported:
The following changes in Apama 10.3.1 affect backwards compatibility to previous Apama versions:
*type*.*method*(...) to call a method, you will be passing an instance of *type* as the first parameter name. We have corrected an issue with the any.getActionParameterNames() method so that it now correctly returns self as the first parameter where there is no closure. This matches the behavior of the new any.getActionParameterTypes() method (see also EPL enhancements in 10.3.1).--pidfile option now take an exclusive lock on the pidfile. This means that multiple components using the same pidfile will be prevented from starting up. See also Starting the correlator and IAF command-line options.Location events correctly. The issue occurred only with hosts set up to use a character other than the period (.) as the floating point separator. For example, if the local host was set up to use a comma as the floating point separator, then a Location([4.4,3.3,2.2,1.1]) was interpreted as Location([44,33,22,11]). Now this is correctly interpreted as Location([4.4,3.3,2.2,1.1]).The following Apama features are now deprecated or have been removed in Apama 10.3.1:
The MapHelper class (com.softwareag.connectivity.MapHelper in the Java API and com::softwareag::connectivity::MapHelper in the C++ API) is now deprecated and will be removed in a future release. Use the MapExtractor class instead. This class provides an easy and type-safe way to extract values from maps such as those used for plug-in configuration. It has automatic support for generating user-friendly error messages if configuration items are missing or of the wrong type.
In the Java API, the com.softwareag.connectivity.util.MapExtractor class is available since version 10.0. See the API reference for Java (Javadoc) for further information.
In the C++ API, the com::softwareag::connectivity::MapExtractor class is available since version 10.1. See the API reference for C++ (Doxygen) for further information.
The ADBC reconnection policy constants RECONNECT and RECONNECT_AND_CLEAR_REQUEST_QUEUE, which can be defined in the ADBCHelper API, are now deprecated and will be removed in a future release. It is recommended that you use any of the other policy constants as appropriate (that is, either DO_NOT_RECONNECT or RECONNECT_AND_RETRY_LAST_REQUEST). See also Reconnection settings.
Log4j 1.2 is now deprecated and support for it will be removed in a future release. For now, the Apama Java client libraries in ap-util.jar and ap-client.jar continue to support Log4j 1.2, but can now alternatively write to Log4j 2 if the Log4j-1-to-2 bridge log4j-1.2-api-2.x.jar is added to the classpath before those jars.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.3.0 includes new features, enhancements, and changes as described in the following topics.
A new connectivity plug-in, the Cumulocity IoT transport, is now available. The Cumulocity IoT transport connectivity plug-in allows you to receive information from devices, measurements, events and alarms from Cumulocity IoT and send operations to Cumulocity IoT (and thus to the devices). For detailed information, see The Cumulocity transport connectivity plug-in.
New EPL and connectivity bundles are also available which you can add to your projects using Software AG Designer. See also Apama enhancements in Software AG Designer 10.3.0.
The Cumulocity IoT transport connectivity plug-in and the new bundles were also delivered with 10.2.0.1.
As of 10.3.0, and also with 10.2.0.3, the following enhancements are available:
subscribeToOperations for subscribing to all device operations.pageSize for retrieving resources from Cumulocity IoT.useBatching for creating bulk measurements.A new EPL plug-in, the JSON plug-in, is now available. You can use the JSON plug-in to convert EPL objects to JSON strings, and vice versa. The JSON plug-in requires the new JSON Support bundle for use with Software AG Designer. See also Apama enhancements in Software AG Designer 10.3.0.
Equivalent functionality is available with the JSON codec connectivity plug-in. Use the JSON codec if the JSON is coming in as an entire message of a connectivity chain. Use the JSON EPL plug-in if only one field of an event is JSON, or if the events are going to or are coming from a connection that is not using a connectivity chain.
See Using the JSON plug-in for detailed information.
The correlator now supports the Prometheus monitoring system for observing internal correlator metrics. See Monitoring with Prometheus for detailed information.
Apama supports a new framework for writing EPL plug-ins in Python. See Writing EPL plug-ins in Python for detailed information.
EPL plug-ins written in Python support Python 3, which is shipped with Apama. They do not support Python 2.
Using the new --python option of the correlator executable, you can enable and disable Python support. See Starting the correlator.
You can also enable and disable Python support using a YAML configuration file. The new pythonSupport element is available for this purpose. Using the new eplPlugins element in a YAML configuration file, you can configure your EPL plug-ins written in Python. See Using Python plug-ins.
Sample EPL plug-ins written in Python can be found in the samples\correlator_plugin\python directory of your Apama installation.
Apama now ships PySys 1.3 instead of PySys 1.2. Release notes for PySys 1.3 can be found in Apama\third_party\python\Lib\site-packages\pysys-release.txt on Windows and in Apama/third_party/python/lib/python3.6/site-packages/pysys-release.txt on Linux.
The sample pysysproject.xml file has been updated to support the new features, details of which are explored further down below.
A new Apama PySys base test class for use in new PySys test cases has been added. Using this class means that PySys methods that read or write files (such as waitForSignal or assertGrep) will use character strings rather than byte strings and UTF-8 encoding for certain file types: .xml, .yaml, .json, .log.
Key features for the new PySys 1.3 release include:
PySys tests can now be written and executed using Python 3.
Support was added for coloring console output to highlight passes, fails, warnings, and more. This is enabled by default in the example pysysproject.xml configuration file shipped with Apama. Coloring can also be enabled or disabled for a particular user and/or machine by setting the PYSYS_COLOR environment variable to true or false. Coloring works on any terminal that supports ANSI escape sequences (for example, most UNIX terminals). On Windows, coloring works using the colorama library that is installed into the Python included with Apama.
A framework was added for recording performance results (such as throughput or latency) that are measured by PySys tests. This includes storage of results in a human-readable and/or machine-parseable CSV file, together with run-specific information such as the host where the test was executed. The CSV files can be aggregated across multiple test runs and/or cycles. See BaseTest.reportPerformanceResult() for more details.
A --progress command line option was added (can also be switched on by setting the PYSYS_PROGRESS environment variable to true), which logs a summary of how many tests have executed, the outcomes, a list of most recent failure reasons, and a list of what other tests are currently executing. This provides helpful feedback to you while executing a long test run.
A defaultIgnoreExitStatus project property was added which controls whether non-zero return codes from startProcess() result in test failures when the ignoreExitStatus flag is not explicitly specified. This property defaults to True if not explicitly configured, which retains behavior for existing tests.
However to promote best practice for new PySys projects, the example PySys project configuration file included with Apama sets defaultIgnoreExitStatus to False, which ensures that processes which return failure codes are not ignored unless explicitly intended by the author of the test case. We recommend changing existing projects to set defaultIgnoreExitStatus to False and also to set abortOnError to True in the project file so that unintentional process failures do not go unnoticed.
With PySys 1.3, the pysys.py run command now returns a non-zero exit code if any tests failed (previously, this command returned 0 on any failed tests).
API documentation for PySys and the Apama PySys extensions can be found in the API reference for Python.
The use of PySys 1.3 has introduced some backwards compatibility changes. See Miscellaneous changes in 10.3.0 affecting backwards compatibility for detailed information.
Apama 10.3.0 includes the following connectivity plug-ins enhancements:
The MQTT transport connectivity plug-in now supports client-side TLS. For this purpose, the following new authentication options can be specified in the YAML configuration file: certificateFile, certificatePassword and privateKeyFile. For more information, see Configuring the connection to MQTT.
The Kafka transport connectivity plug-in now uses the Kafka default value for max.block.ms instead of the custom value (2147483647) that was used in previous versions. See also Configuring the connection to Kafka (dynamicChainManagers).
The Classifier codec now supports regular expressions. For more information, see The Classifier codec connectivity plug-in.
In addition, the Classifier codec is no longer provided as a sample.
Apama 10.3.0 includes the following enhancements in Software AG Designer:
The following connectivity bundle group has been added to Software AG Designer:
The following EPL bundles have been added to Software AG Designer:
com.apama.cumulocity package in the API reference for EPL (ApamaDoc).com.apama.cumulocity.Util event in the API reference for EPL (ApamaDoc).onApplicationInitialized is called as soon as the entire application has been injected into the correlator. This is not appropriate in all cases: for more advanced applications where you need to wait for a MemoryStore, database or another resource to be prepared before your application is able to begin to process incoming messages, you should continue to write EPL code to call onApplicationInitialized explicitly instead of using the bundle. See also Sending and receiving events with connectivity plug-ins and Using EPL to send and receive JMS messages.The Cumulocity IoT Event Definitions, Cumulocity Utilities and Automatic onApplicationInitialized bundles were delivered with 10.2.0.1, but in that version they were placed in the Connectivity bundles node. In 10.3.0, they have been placed in the Standard bundles node. In addition in 10.3.0, the Cumulocity IoT Event Definitions bundle has been renamed to Event Definitions for Cumulocity 9.0+, and the Cumulocity Utilities bundle has been renamed to Utilities for Cumulocity 9.0+.
The Connectivity and Adapters node and the EPL Bundles node are now created by default when you create an Apama project. You can use these nodes to add connectivity bundles, adapter bundles, and EPL bundles. In previous releases, these nodes were created only after adding a bundle. For more information, see Adding connectivity and adapter bundles to projects and Adding EPL bundles to projects. As part of this change, the following commands have been removed from the context menu which is shown when you right-click an Apama project:
You can now add Docker support to your Apama project. The following new command is available for this purpose:
Apama 10.3.0 includes the following EPL enhancements:
empty() is now available for the any type. It returns true if the any is empty, false if it wraps an object. See the API reference for EPL (ApamaDoc) for more information on the any type.hash() is now available on all built-in types which can be hashed. It returns an integer hash value in all cases. See the API reference for EPL (ApamaDoc) for more information.Apama 10.3.0 includes the following correlator utility enhancements:
engine_send macro in the apama-macros.xml Ant script now has an additional channel attribute which allows you send events directly to a specified delivery channel. See also About Deploying Apama Applications with an Ant script.apama-macros.xml Ant script, a pidfile (that is, a file containing the process identifier) is now created by default. The default location and name of this pidfile is ${log.dir}/${correlator_name}.pidfile. You can override this location by modifying the corresponding entry in the build.xml file.correlator.properties file, which is generated by the engine_deploy tool, now uses ${correlator.logsDir}/${correlator.name}.pidfile as the default location and name of the pidfile (instead of ${correlator.dataDir}/correlator.pid).--outputDeployDir action with the engine_deploy tool, you can now specify the shorter -d option. In addition, when the output directory or correlator deployment package already exists, it is now overwritten (rather than causing an error). See also Deploying a correlator.-s (or --source) option of the epl_coverage tool can now be specified more than once to allow multiple source directories to be searched. In addition, the performance of the epl_coverage tool has been improved when using source search directories to locate the source for .mon files that are not present in the location used during the inject, or are contained within .cdp files. See also Creating code coverage reports.INFO level, a new jvm=*n* field is now shown in the Correlator Status: line of the main correlator log file. Its value is the sum of all memory used by the JVM in megabytes (that is, the used heap memory, the used non-heap memory, and the used buffer pool memory). See also Descriptions of correlator status log fields.--python option of the correlator executable, you can enable and disable Python support. See also New Python EPL plug-in support in 10.3.0.Apama 10.3.0 includes the following changes for its integration with Command Central:
Apama/samples/command_central/reference directory. This template documents all the configuration options for the correlator, including how to deploy an Apama application in Command Central. It exists for informational purposes only; it is not a well-rounded sample application that you should build upon.Apama 10.3.0 includes the following dashboard enhancements:
Apama 10.3.0 includes the following miscellaneous enhancements and changes:
Apama 10.3.0 incorporates the ICU (International Components for Unicode) time-zone data update 2018e, which is the most recent update at the time of release. This will update time-zone data used by the correlator and Time Format EPL plug-in.
Apama now supports the jemalloc memory allocator as an optional alternative to its standard memory allocator on Linux platforms. For more information, see Using jemalloc to optimize memory usage.
The TCStore driver now supports TLS-secured connections to a secure Terracotta Server Array. See the Terracotta DB documentation for more information on these concepts. The new TCStore driver property securityRootDirectory is available for this purpose. For more information, see TCStore (Terracotta) driver details.
The REST API now supports the following keys which provide statistics information when the Java virtual machine (JVM) which is embedded in the correlator has been enabled:
jvmMemoryHeapUsedMB
jvmMemoryHeapFreeMB
jvmMemoryNonHeapUsedMB
jvmMemoryBufferPoolUsedMB
jvmMemoryAllUsedMB
In addition, the REST API now also supports the following keys:
virtualMemoryMB (new name for the deprecated virtualMemorySize)
physicalMemoryMB
swapPagesRead
swapPagesWrite
For more information, see List of correlator status statistics.
The image for the Apama correlator in Docker Store now supports the Cumulocity IoT transport. See also New Cumulocity IoT transport connectivity plug-in in 10.3.0.
A builder image which supports the engine_deploy tool is now available in Docker Store. If you are using the official Docker Store image, you can use this new builder image for project build steps, as part of a Docker multi-stage build. See also Building Apama projects during the Docker build.
Docker and Kubernetes offer methods of passing secrets to the correlator. This can be used to securely provide credentials to the correlator. New Docker and Kubernetes samples are now available which demonstrate how to use secrets to set variables in correlator configuration files. See also Apama samples for Docker and Apama samples for Kubernetes.
The documentation has been updated with information to help identifying how personal data can be protected when building applications on the Apama platform. See Protecting personal data in Apama applications for detailed information.
The EPL syntax highlighting files for Emacs and Vim have been updated with this release. You can find them in the utilities directory of your Apama installation.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.3.0. Due to these upgrades, the following versions are no longer supported:
The following changes in Apama 10.3.0 affect backwards compatibility to previous Apama versions:
The Row.set() action of the MemoryStore can be used to set non-schema fields in distributed stores which support them. With previous versions, setting a field with an empty any has set a field with the value “empty”. As of this version, setting a field with an empty any removes the non-schema field with the given name.
The engine_deploy tool now fails to generate output when the input project contains errors. This changes the way in which the output is shown when Ant is run for an Ant-exported project. With previous versions, the Ant build was successful and the errors were shown afterwards while injecting the files. As of this version, the errors are already shown in the console when the Ant build fails.
With previous versions, the correlator incorrectly converted a float of the form -0.0 or a decimal of the form -0.0d to a "0" string (that is, the minus sign was missing). As of this version, this is correctly converted to a "-0" string. This change may lead to a backward compatibility issue for any software using Apama that is checking exclusively for "0" strings (without minus signs) and cannot handle the case of a string of the from "-0".
The location of the following files in your Apama installation has changed from Apama/monitors/ to Apama/monitors/cumulocity/9.0/:
Cumulocity_EventDefinitions.monCumulocity_ServiceMonitor.monCumulocity_Utils.mon
These files were also delivered with 10.2.0.1. If you are explicitly using these files in a script or manual injection ordering, update the location of these files.The Docker images are now configured to run processes as a different user within containers. For more information, see Quick start to using an Apama image.
Previous releases of Apama in Docker Store declared the ${APAMA_WORK} directory to be a volume. This made it hard to use the work directory when creating your own derived images because of limitations on how it is handled. Therefore, the VOLUME declaration has been removed. If you need to restore this functionality, you can add a VOLUME statement in your own derivative Dockerfile after populating the work directory.
The MQTT transport connectivity plug-in now uses TLS version 1.2. This may affect existing deployments. See also Configuring the connection to MQTT.
Apama now ships Python 3.6.6 instead of Python 2.7. You may have existing PySys tests written in Python 2 that you may wish to migrate to Python 3. It is recommended to refer to the internet for the details of what this change from Python 2 to Python 3 entails, as well as recipes for upgrading test cases. It is likely that updates will be required, particularly around the handling of strings, file I/O and print statements. There are many tools, such as 2to3.py, available to aid in this. Support for Python 2.7 is now deprecated; see also Removed and deprecated features in 10.3.0.
Although Python 2.7 is no longer shipped, you can continue to use existing PySys tests written for Python 2 with Apama 10.3.0. However, you have to manually copy the pysys and apama directories from the Apama\third_party\python\Lib\site-packages directory on Windows or the Apama/third_party/python/lib/python2.7/site-packages directory on Linux into your own Python 2.7.13 installation. On Windows, you must also ensure that the pywin32 library is installed into your Python 2 installation (from https://pypi.org/).
With PySys 1.3, the pysys.py run command now returns a non-zero exit code if any tests failed (previously, this command returned 0 on any failed tests). See also PySys changes in 10.3.0.
The docker module contains classes to help create and orchestrate Docker and Kubernetes images from within PySys. This module has been renamed to apama.docker. In this release, the old name docker can still be used, but is deprecated. If you run Python with deprecation warnings enabled, you will see a warning if you are using the old name. It is recommended that you update your tests to use apama.docker because the docker module name will be removed in a future release.
The defaultValue rule of the Mapper codec will no longer always be applied to metadata fields, and instead will only be applied in the case of an empty/null metadata field. See also The Mapper codec connectivity plug-in.
The following Apama features are now deprecated or have been removed in Apama 10.3.0:
dashboard_deploy.xml files that specify this deployment type will automatically have this type removed from them when opened. It is then up to you to specify a different deployment type.CLASSPATH environment variable to configure the correlator’s system classpath. See also the announcement in Removed and deprecated features in 10.1.virtualMemorySize is now deprecated and will be removed in a future release. Use the new key virtualMemoryMB instead. See also Miscellaneous enhancements and changes in 10.3.0.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.2 includes new features, enhancements, and changes as described in the following topics.
For Apama queries and applications that do not involve caching (such as store and “system of record” use cases), the TCStore driver is now the recommended driver for the distributed MemoryStore. See TCStore (Terracotta) driver details. See also Configuring the TCStore driver.
Apama continues to support the BigMemory Max driver. This is still the recommended driver for caching use cases, but no longer for queries. See BigMemory Max driver details.
Support for using BigMemory Max for Apama queries is deprecated and will be removed in a future release. It is recommended that you now use Terracotta DB’s TCStore for queries.
In the Distributed MemoryStore Configuration Wizard of Software AG Designer, when you select the third-party cache provider from the Store provider drop-down list box, you can no longer select the option Apama Queries (using BigMemory). This option has been replaced by the Apama Queries (using TCStore) option. See also Adding a distributed store to a project.
If you still want to use BigMemory Max for queries in new projects, you must now select BigMemory as the store provider and specify “ApamaQueriesStore” as the store name. See also Configuring the BigMemory Max driver. Existing deployments using BigMemory Max for queries are unaffected; this only covers developing new projects in Software AG Designer.
For detailed information on how to change a queries project that uses BigMemory Max to use TCStore instead, see Migrating from BigMemory Max to TCStore.
See also Deploying and Managing Queries, which provides detailed information on how to use queries with the distributed MemoryStore.
Apama now supports Kubernetes. Kubernetes can be used for container orchestration, that is, for automating the deployment of your Apama applications. See Deploying Apama applications with Kubernetes for detailed information.
Apama 10.2 includes the following connectivity plug-ins enhancements:
A new EPL sample application is now available in the samples/connectivity_plugin/application/genericsendreceive directory of your Apama installation. This simple sample application provides an easy way to get started with sending and receiving messages to or from any connectivity plug-in. For more information, see the README.txt file in the above directory and Sending and receiving events with connectivity plug-ins.
The Kafka transport is now available as a pre-built connectivity plug-in. See also The Kafka Transport Connectivity Plug-in.
Protection against brute force attacks on users and passwords is now available for the HTTP server. Using the new configuration options coolDownSecs and maxAttempts, a user-configurable cool-down and maximum number of login attempts now allows failed requests to be throttled. In addition, handling of unknown users now mirrors allowed users to avoid user information leakage. See also Configuring the HTTP server transport and Authentication.
The Unit Test Harness codec connectivity plug-in now orders all maps and not just top level maps. The ordering of nested maps in the output file may be different to previous versions. See also The Unit Test Harness codec connectivity plug-in.
The C++ connectivity plug-in API now also supports creating user-defined dynamic chain managers for transports, which was previously only possible in Java (see also Connectivity plug-ins enhancements in 10.0). The C++ com::softwareag::connectivity::chainmanagers namespace has been added for this purpose; this is equivalent to the Java com.softwareag.connectivity.chainmanagers package. See also Requirements of a transport chain manager plug-in class and the API reference for C++ (Doxygen).
In addition, a new macro is available for C++: SAG_DECLARE_CONNECTIVITY_TRANSPORT_CHAIN_MANAGER_CLASS(*class-name*), which is used for exporting a chain manager class. See also Exporting the required symbols from C++ plug-ins.
A skeleton sample for C++ is provided in the samples/connectivity_plugin/cpp/skeleton_chainmanager directory of your Apama installation. You can use this sample as a starting point to write your own C++ chain manager and transport.
A new to_string(...) function is now available in the com::softwareag::connectivity namespace of the C++ API. This function operates on data_t, list_t and map_t classes and returns a string representation of the object. It is intended for logging and similar tasks. It is not tuned for performance. See also the API reference for C++ (Doxygen).
Apama 10.2 includes the following enhancements in Software AG Designer:
You can now add the Kafka connectivity bundle to your projects. For further information, see Adding the Kafka connectivity plug-in to a project.
Software AG Designer now offers several different options to choose from when adding the Universal Messaging connectivity plug-in to your project, including a simple configuration for getting started quickly using JSON messages and an advanced configuration showing a wide range of possible options for configuring the Universal Messaging transport and for customizing the way Universal Messaging messages are mapped to Apama events. See also Adding the Universal Messaging connectivity plug-in to a project.
Software AG Designer now offers several different options to choose from when adding the HTTP client connectivity plug-in to your project: an option that uses application-specific event definitions, transports and mapping configurations, and a “generic” option that uses generic Apama event definitions and a chain configuration to invoke HTTP services with JSON payloads. For the “generic” option, the HTTP client adds a reference to com.apama.util.AnyExtractor which is useful for easily extracting data from an any type. See also Adding the HTTP client connectivity plug-in to a project and Using predefined generic event definitions to invoke HTTP services with JSON and string payloads.
Apama 10.2 includes the following query enhancements:
Apama 10.2 includes the following EPL enhancements:
The reflection capabilities have been improved. For example, it is now possible for an EPL application to query what actions a type has. For this purpose, the following new methods are now available for the any type:
getActionNamesgetActiongetConstantgetConstantNamesgetActionParameterNamesgetActionParametersgetActionReturnTypeNamegetGenericAction
See also Reflection on types.When listening to events of all types, you can now provide the following optional values:
typeName to specify which routable type you are listening for.values to name indexable fields in the event.
See also Listening for events of all types.ApamaDoc is now available for all of the built-in types and their methods. This includes the primitive types, the string type, the reference types, and the monitor pseudo-type. See the API reference for EPL (ApamaDoc) for detailed descriptions of all of these types. You can find them under All Types, and under the headings Built-in types and Aggregates. Alternatively, you can find them in the following packages:
<Default Package> (most of the built-in types can be found here)com.apama (includes the Channel type)com.apama.aggregates (includes all of the built-in aggregate functions such as avg)com.apama.exceptions (includes the Exception and StackTraceElement types)
As the type descriptions are now part of the ApamaDoc, the corresponding topics under Types have been removed from the EPL Reference.The new @throws tag can now be used with ApamaDoc. It documents exceptions which may be thrown from an action and the conditions which would cause the action to throw an exception. See also Inserting ApamaDoc tags.
A new getInfo() action is now available from the com.apama.correlator.Component event of the Management interface, which provides a way for an EPL application to get access to a variety of correlator statistics and operational information. See also Using the Management interface.
Apama 10.2 includes the following correlator utility enhancements:
Using the --outputDeployDir action of the engine_deploy tool, you can now alternatively specify the name of a Zip file that is to contain the contents of the correlator deployment directory. See also Deploying a correlator.
With previous versions, when a directory containing a Software AG Designer project was specified with the engine_deploy tool, the resulting correlator deployment directory would not configure any connectivity plug-ins or any other kind of configuration. This is no longer the case. The resulting correlator deployment directory now accurately reflects the entire launch configuration, unless more than one launch configuration exists in the project in which case you have to choose a launch configuration. See also Deploying a correlator.
The YAML configuration file for the correlator now supports many new elements which correspond to command-line options of the engine_connect tool. For more information, see Setting up connections between correlators in a YAML configuration file.
To mitigate potential injection attacks against the correlator web interface, the address used to access the web interface is now, for certain URLs, validated against a list of permitted host names. By default, this corresponds to the configured IP addresses and host names of the machine on which the correlator is running and the configured correlator port. This means that for most applications this change will be transparent. If your correlator web interface is deployed behind another component or has otherwise non-standard configuration, you may need to set the new configuration option server/permittedRootURIs to include the addresses from which the correlator will be accessed. For more information, see Using the correlator web interface with non-default addresses.
Note: This only applies to the correlator web interface that is accessed via the correlator port (the default is 15903). It does not apply to the HTTP server transport connectivity plug-in.
If persistence is enabled, a new Persistence Status: log line is now shown in the main correlator log file when logging at INFO level. See Persistence status for detailed information on the log fields that are shown in this line.
REST API keys with similar names are also available now. See Miscellaneous enhancements and changes in 10.2.
Apama 10.2 includes the following changes for its integration with Command Central:
APAMA-CORRELATOR-INITIALIZE configuration type and the connectivityYamlPath property of APAMA-ARGS. This was the mechanism by which you configured the EPL and connectivity configuration for the correlator to run, replaced by a new mechanism.APAMA-PROP-OVERRIDES configuration type. In practice, you would use the APAMA-PROP-OVERRIDES configuration type for configuration that is different between your development environment (in your Designer project) and your production environment (as managed by Command Central), like the hostname of a messaging system used by a connectivity plug-in.Apama 10.2 includes the following miscellaneous enhancements and changes:
The REST API now supports the enqueueQueueSize key which indicates the number of enqueued events. The status log field eq with the same meaning, however, is no longer shown in the main correlator log file. See also List of correlator status statistics.
The REST API now supports the following keys which provide statistics information when the correlator is running in persistent mode:
persistenceNumSnapshotspersistenceLastSnapshotTimepersistenceSnapshotWaitTimeEwmaMillispersistenceCommitTimeEwmaMillispersistenceLastSnapshotRowsChangedEwma
For more information, see List of correlator status statistics.Status log fields with similar names are also available now. See Correlator utility enhancements in 10.2.
Generated documentation for the REST API is now available in the doc/rest directory of your Apama installation. See the API reference for Component Management REST APIs. The corresponding information has therefore been removed from Managing and monitoring over REST.
The Docker samples have been updated to support a newer Docker version. This includes making the samples usable with both an image built from an installation using the sample and the published image in the Docker Store. The Docker samples now also come with Kubernetes support. See also Deploying Apama applications with Kubernetes.
In addition, documentation on how to use Docker with Apama is now available. See Deploying Apama applications with Docker.
The correlator no longer refuses to start if a distributed MemoryStore is enabled and a license file cannot be found. See also Using the distributed MemoryStore.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.2. Due to these upgrades, the following versions are no longer supported:
The following changes in Apama 10.2 affect backwards compatibility to previous Apama versions:
Apama 10.2 incorporates the ICU (International Components for Unicode) time-zone data update 2017c, which is the most recent update at the time of release. This will update time-zone data used by the correlator and Time Format EPL plug-in.
In addition, Apama 10.2 contains an upgrade from ICU 58.1 to ICU 60.1 which contains a small behavioral difference which may affect your applications: When formatting time in UTC, the output for a “GMT+offset” time zone is now “UTC” rather than “GMT” (non-UTC time zones are unaffected).
It is no longer legal to define two event types with the same name and package in a Software AG Designer project. In previous versions, two similar event declarations in a project were shown as a warning in Software AG Designer. As of this version, this is shown as an error.
An IllegalArgumentException is now thrown at runtime if an EPL dictionary is initialized with duplicate keys. With previous versions, no exception was thrown; the dictionary contained the key only once and the value was undefined.
The ADBC Event API has been changed such that a single com.apama.database.Query event can no longer be used to run multiple queries concurrently. A single query event was never intended to run concurrent queries. It is recommended to instantiate multiple com.apama.database.Query events if you want to run queries concurrently. To reuse the same com.apama.database.Query event, you must now set up appropriate listeners on it again, by calling setBatchDoneCallback and setSchemaCallback before calling start. See also The ADBC Event application programming interface and the description of the Query event in the API reference for EPL (ApamaDoc).
The following Apama features are now deprecated or have been removed in Apama 10.2:
subscribe(), unsubscribe() and sendToDES() actions on the auto-generated EPL representations of Digital Event Services types have been removed. Instead, you should now use the static CHANNEL constant on the EPL type to access the channel name. With CHANNEL, you can send, subscribe and unsubscribe in the same way as any other Apama channel. See also the announcement in Removed and deprecated features in 10.0.readVisibility and writeVisibility properties in the TCStore driver configuration for distributed memory stores is deprecated and will be removed in a future release. It is recommended that you remove these properties from your Spring configuration files. Setting these properties does not have any effect in version 10.2, but will result in an error in future versions of Apama. See also TCStore (Terracotta) driver details.APAMA-CORRELATOR-INITIALIZE configuration type and the connectivityYamlPath property of APAMA-ARGS. This was the mechanism by which you configured the EPL and connectivity configuration for the correlator to run, replaced by a new mechanism.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers and other products.
Apama 10.1 includes new features, enhancements, and changes as described in the following topics.
EPL now supports any types. Values of the any type can hold a value of a concrete EPL type (that is, a type other than the any type). Different values of any variables may have different types. This is useful when mapping to generic data formats such as JSON, or where the data in an event or being processed by an action may be one of a number of different types. An any value may also be empty and not contain a value (similar to the concept of null in other languages).
The new switch statement operates on an expression of the any type. See Handling any values of different types with the switch statement for more information.
EPL supports casting of the any type to a concrete target type and vice versa. See Handling the any type for more information.
The MemoryStore now supports the any type. A valid schema can be created from an event type using schemaFromAny(*event*). New actions to get and set fields and rows from the any type and from dictionary<string,any> have been added. See Description of row structures for more information.
The connectivity plug-ins can now handle values of the any type. This means that an empty data_t or a null value in Java will translate to/from an empty type. See Map contents used by the apama.eventMap host plug-in.
The EPL plug-ins written in C++ support the any type, converting it to a data_t. See Method signatures.
Note: any type support is currently not available for the EPL plug-ins wr
itten in Java.
The Apama event parser libraries for Java and.NET have been updated to support events with any fields. A new class AnyFieldType has been added to represent the any field type. To access it, you have to create new AnyFieldType(). For complete information about this class, refer to the description of
com.apama.event.parser in the API reference for Java (Javadoc)Apama.Event.Parser Namespace in the API reference for.NETThe Apama event parser library for C has been updated to support events with any fields.
A new connectivity plug-in, the HTTP server transport, is now available. It can handle HTTP requests and deliver the contained events to the correlator, which will return acknowledgment of the requests. In addition to this, it can serve static files. It also provides support for TLS (Transport Layer Security) alongside HTTP basic authentication. For detailed information, see The HTTP Server Transport Connectivity Plug-in.
Using Software AG Designer, you can add the new HTTP server connectivity bundle to your projects. For further information, see Adding the HTTP server connectivity plug-in to a project.
A new EPL plug-in, the R plug-in, is now available. It enables an EPL application to use an external R server using Rserve. The R plug-in requires the new R Support bundle for use with Apama Plugin for Eclipse. See Using the R plug-in for detailed information.
Apama 10.1 adds support for Terracotta DB. This is provided as a driver that can be used by any Apama application using the existing Apama MemoryStore API.
The new driver uses the TCStore API to provide access to Terracotta DB’s storage capabilities. This allows Apama to read and write records in TCStore datasets, which may also be read and written by other Apama correlators and non-Apama components such as other Software AG products or custom clients written against the TCStore API.
The new TCStore driver can be added to projects and easily configured using Apama’s Distributed MemoryStore editor in Software AG Designer (see Configuring a distributed store).
Apama’s TCStore driver only supports clustered mode. Currently, there is no support for embedded TCStore, for performing searches from Apama, or for the Ehcache API.
Apama continues to support the MemoryStore driver for Terracotta BigMemory Max (see BigMemory Max driver details), which remains the recommended driver for both Apama queries and for caching use cases. For applications that do not involve queries or caching, TCStore is now the preferred MemoryStore driver. See TCStore (Terracotta) driver details for detailed information.
Apama 10.1 includes the following enhancements for the MemoryStore:
any type to simplify operations. For example, a schema can be created from an event type using schemaFromAny(*event*) and there are actions that support getting and setting the contents of a Row using an Apama event type with fields matching the fields of the row. There are also new actions to get and set fields and rows making use of the any type. See Description of row structures for more information.Row.toDictionary() action that returns a dictionary<string,any> containing all fields in the row. This is useful for diagnostic purposes such as logging and testing, as well as for application development.Row.get*XXX* and Row.set*XXX* actions in the EPL API can now be used to get and set such extra fields for stores that support this feature, such as TCStore. The new Row.getKeys() and Row.toDictionary() actions provide a way to find out what fields are present in a given row, including both fields from the schema and extra fields. See Description of row structures for more information about using this feature from EPL, and see Creating a distributed MemoryStore driver for more information about writing a driver that supports it.forceCommit() action is now available for the com.apama.memorystore.Row event. This new action allows you to commit the local Row back to the table even if the row in the table has been modified after the Row event was obtained. See also Using transactions to manipulate rows.Apama 10.1 includes the following connectivity plug-ins enhancements:
MessageId annotation. This is used for reliable receiving. The new MessageId annotation names a field that should contain the unique identifier of the connectivity plug-in message that the event came from. It is thus no longer necessary to make use of copyFrom to get the message identifier into a field. For more information, see Adding predefined annotations.ConnectivityPlugins.createDynamicChain is now available in EPL. It replaces the deprecated ConnectivityPlugins.createChain method. The new method allows you to specify an additional argument: *defaultChannelTowardsHost*. For more information, see Creating dynamic chains from EPL.A new helper class called com::softwareag::connectivity::MapExtractor is available in the C++ API. This class provides an easy and type-safe way to extract values from map_t objects such as those used for plug-in configuration. It has automatic support for generating user-friendly error messages if configuration items are missing or of the wrong type. See the API reference for C++ (Doxygen) for further information.
Note: For Java, this functionality has already been added in 10.0. See also Connectivity plug-ins enhancements in 10.0.
acceptUnrecognizedCertificates and certificateAuthorityFile. For more information, see Configuring the connection to MQTT.https URLs. See also About the HTTP client transport.basePath which defines an optional path to be prefixed to the metadata.http.path for all messages sent to the transport. See also Configuring the HTTP client transport.forEach to define mapping rules for elements nested in a list. See also The Mapper codec connectivity plug-in.Apama 10.1 includes the following enhancements in Software AG Designer:
You can now export Apama event definitions as digital event types using the Digital Event Types editor.
The following commands from the File > New menu (and from the New context menu) have been renamed:
For both commands, validations for the name have been implemented such that wizard now shows an error message if name contains special characters, except for the dollar ($) and underscore (_) characters.
The EPL editor now provides a template with the specified monitor name or event definition name.
See the following topics for more information:
Apama 10.1 includes the following query enhancements:
route statement is now allowed in queries, and is now the recommended mechanism for sending events between queries. See also Communication between monitors and queries.Apama 10.1 includes the following EPL enhancements:
integer.incrementCounter(*string*) is now available. It can be used to generate a monotonically increasing integer for the corresponding string key passed as a function argument in the scope of the correlator. This is a type method as well as an instance method.context.iqSize() is now available. It can be used to return an integer that is the current number of elements in the input queue for the context.Apama 10.1 includes the following correlator utility enhancements:
Using the new -v *category*=*level* (or --loglevel *category*=*level*) option of the correlator executable, it is now possible to configure the log level for messages logged by the correlator-internal systems. You can do this either for the whole correlator or for individual categories. This includes setting the log level for individual connectivity plug-ins and EPL plug-ins. For detailed information, see Starting the correlator and Setting correlator and plug-in log files and log levels in a YAML configuration file.
With the correlator executable, it is now possible to set distinct and individual log levels for the correlator and the application. The following options are now available: --applicationLogLevel and --applicationLogFile. If these two options are not set, then the current settings of the -v (or --loglevel) and -f (or --logfile) options are used as the default. For detailed information, see Starting the correlator.
Note that less log levels are now allowed for the correlator log messages, that is, for the -v (or --loglevel) option. OFF, CRIT and FATAL are no longer supported. If specified, they are reset to ERROR and a warning message is printed at the top of the correlator log file. In addition, the use of ERROR and WARN is no longer recommended. If specified, a warning message is also printed at the top of the correlator log file.
The YAML configuration file for the correlator now supports many new elements which correspond to command-line options of the correlator executable. For more information, see YAML configuration file for the correlator.
The engine_deploy tool now converts all of the options in a Software AG Designer launch configuration to the equivalent YAML configuration options, now that they exist. Previously, only a small number were honored. The new options are:
javaApplicationSupportjmsConfigDirdistMemStoreConfigDirinputLogFilelogFilelogLevelnamejvmOptions)
This may be a backwards-incompatible change if you had settings in your launch configuration that were previously being ignored. The generated files also contain support for some options not present in Software AG Designer, but which are often useful at deployment time: pidFile, runtime and runtimeCacheDir.The generated YAML file also contains configuration to set an environment variable APAMA_PROJECT_DIR, which can be used to specify classpath dependencies for EPL plug-ins written in Java in the deployment descriptor file (see also Specifying the classpath in deployment descriptor files). For example:
<classpath>${env:APAMA_PROJECT_DIR}/lib/foo.jar</classpath>
The properties file now contains the properties logsDir and dataDir to allow easily changing the location where data (for example, runtime cache directory, pid file) and log files are written without needing to modify the properties for each file path individually.
The properties file also contains a property startTimeoutSecs. This value is not currently used by any Apama tools, but can be used to provide a hint to customer-developed deployment or testing tools about how long to wait for the component to start up.
Note that the extraArgs settings are copied into the properties file, but cannot be used by the YAML configuration file for the correlator. Therefore, they must be manually passed on the command line by whatever tool is responsible for starting the correlator.
For more information on the engine_deploy tool, see Deploying a correlator.
Instead of using the engine_deploy tool to generate a correlator deployment package (CDP), you can now also use Ant to do this. The apama-macros.xml file includes the new project-to-cdp macro for this purpose. See also About Deploying Apama Applications with an Ant script.
In previous versions, the correlator automatically and silently substituted a dot (.) for ${APAMA_HOME} or ${APAMA_WORK} if they were used within a YAML configuration file without being defined as an environment variable or as a property within the YAML configuration itself. This was hard to diagnose and incorrect. The correlator will now no longer do this automatic substitution and will fail to start up, as with any other undefined YAML configuration variable. Note that this does not affect correlators that are started through Apama Plugin for Eclipse or using the Apama Command Prompt, both of which correctly set these variables.
Apama 10.1 includes the following dashboard enhancements:
Apama 10.1 includes the following miscellaneous enhancements and changes:
samples/monitorscript/hot-redeploy/StoreState directory of the Apama installation which shows how to transfer monitor state using the MemoryStore. For more information, see Versioning and upgrading monitors.optional fields.Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.1. Due to these upgrades, the following versions are no longer supported:
Red Hat Enterprise Linux 6 (x86) 64-bit.
Red Hat Enterprise Linux 7.0 (x86) 64-bit (replaced by support for a more recent version).
CentOS Linux 6 (x86) 64-bit.
CentOS Linux 7.0 (x86) 64-bit (replaced by support for a more recent version).
Oracle Linux 6 (x86) 64-bit.
Oracle Linux 7.0 (x86) 64-bit (replaced by support for a more recent version).
Software AG Common Runtime 10.0 (replaced by support for a more recent version).
Apache Tomcat 8.5.5 (replaced by support for a more recent version).
Software AG Universal Messaging 10.0 as a JMS provider (replaced by support for a more recent version).
IBM WebSphere Application Server 8.5.5.0 as a JMS provider (replaced by support for a more recent version which supports Java 8).
Note: Support for IBM WebSphere Application Server 8.5.5.8 (Liberty) is now deprecated and will be removed in a future release.
Software AG Universal Messaging Client 10.0 (replaced by support for a more recent version).
Oracle Java 8 update 121 (replaced by support for a more recent version).
Software AG Designer 10.0 (replaced by support for a more recent version).
Google Chrome 48 (replaced by support for a more recent version).
The following changes in Apama 10.1 affect backwards compatibility to previous Apama versions:
The evaluation order of some of the operations in the interpreted EPL runtime has been changed and is now in line with the compiled EPL runtime. Anyone who is reliant on the old evaluation order may see a change of behavior. More specifically, the evaluation order for assignment operations and the lookup/store operations for the dictionary and sequence types have been changed. For assignment operations, the left-hand side is evaluated before the right-hand side. The expression to get the dictionary or sequence is evaluated before the expression to get the key or index.
For the sake of consistent naming, several SSL configuration options have been renamed in the standard connectivity plug-ins.
Renamed options in the Universal Messaging transport connectivity plug-in (see also Configuring the connection to Universal Messaging (dynamicChainManagers)):
authentication/certificateAuthority is now certificateAuthorityFile and no longer under authentication.
authentication/certificate is now authentication/certificateFile.
Renamed options in the HTTP client transport connectivity plug-in (see also Configuring the HTTP client transport):
acceptUnrecognizedCertificates is now named tlsAcceptUnrecognizedCertificates.
certificateAuthorityPath is now named tlsCertificateAuthorityFile.
A new subsection authentication has been added which now contains the following options: authenticationType, username (which has been renamed from user), and password.
If you are using the above SSL configuration options, it is recommended that you change your YAML configuration files to use the new names.
For the sake of consistent naming, the following elements in the YAML configuration file for the correlator have been renamed:
In the correlator section: runtime-cache has been renamed to runtimeCacheDir.
In the server section: pidfile has been renamed to pidFile.
In the correlator section, under the persistence entry: snapshotInterval has been renamed to snapshotIntervalMillis.
Note: The -PsnapshotInterval option of the correlator executable has also been renamed to -PsnapshotIntervalMillis. You can still use the old option name, but it is deprecated, and it is recommended that you now use the new name. See also Starting the correlator.
In addition, the elements that were previously located in the messaging section of the YAML configuration file have been moved to the server section. These are maxOutstandingSecs and maxOutstandingKb. Note that the description for maxOutstandingKb has been corrected (it was wrongly called maxOutstandingBytes in the documentation for the previous version). It takes the same units as the -M command-line option.
If you are using the above elements, it is recommended that you change your YAML configuration files to use the new names.
The engine_deploy tool now converts all of the options in a Software AG Designer launch configuration to the equivalent YAML configuration options. This may be a backwards-incompatible change if you had settings in your launch configuration that were previously being ignored. See also the information on engine_deploy in Correlator utility enhancements in 10.1.
Software AG Designer no longer sets the CLASSPATH environment variable when it launches the correlator. This brings Software AG Designer behavior in line with deployment behavior (for example, Ant macros) which also does not set any classpath environment at the correlator level. EPL plug-ins which require classpaths to be set should do this via the apama_java.xml configuration in Software AG Designer (as described in Specifying the classpath in deployment descriptor files) or by starting the correlator with -J-Djava.class.path=*path*.
This also applies to the IAF adapters, which have the classpath defined in their configurations. Software AG Designer no longer sets the CLASSPATH environment variable when it launches the IAF. Therefore, any projects that relied upon that behavior should be changed to set the classpath in the IAF configuration files instead (as described in The IAF configuration file).
Any Software AG Designer project containing queries now automatically requires the Correlator-Integrated JMS Application Support bundle. This enhancement will only require an action on your part if your launch configuration uses the Manual Ordering option for injections, in which case you will need to take care that the new JMS files are injected before any query (.qry) files. See also Injections.
A Universal Messaging cluster which has dropped below quorum of operating nodes is now considered completely down and is therefore disconnected, instead of remaining connected to a cluster which is non-functioning. In particular, this now means that it is not possible to start up a correlator if the cluster to which it is connecting is below quorum. If a cluster drops below quorum while the correlator is operating, the normal reconnect logic applies as it does in a single node or a cluster that has been entirely disconnected.
The query statistics are now updated every 10 seconds by default. In previous versions, they were updated every second. By sending a SetQueryStatisticsPeriod event, you can now control the frequency of the statistics gathering or disable query statistics entirely. See also Monitoring running queries.
The following Apama features are now deprecated or have been removed in Apama 10.1:
Software AG’s Event-Driven Architecture (EDA) is deprecated. Support for EDA events will be removed in a future release. It is recommended that your Apama applications use digital event types instead, as provided by Software AG Digital Event Services.
Use of the CLASSPATH environment variable to configure the correlator’s system classpath is deprecated and will be removed in a future release. It is recommended that you set the classpath on a per-plug-in basis where possible, for example, in the descriptor file for an EPL or JMon plug-in (see Specifying the classpath in deployment descriptor files) or in the configuration file for a connectivity plug-in, JMS or distributed MemoryStore library. For cases where you really need to set the global system classpath, you can use the -J (or --javaopt) option of the correlator executable, along with -Djava.class.path=*path* (as described in Starting the correlator).
The ConnectivityPlugins.createChain method in EPL has been deprecated. Use ConnectivityPlugins.createDynamicChain instead. See also Connectivity plug-ins enhancements in 10.1.
The --remoteDeploy and --remoteUndeploy options of the dashboard_management tool (see Managing and stopping the data server and display server) have been removed in 10.1. These options relied on third-party technology that has not been officially supported since 5.1.
As of this version, the deprecated EngineManagement API for C can no longer be used. It is recommended that you use the EngineManagement API for C++ instead. See Engine Management API. Any clients written using engine_client_c.h will need to be rewritten to use engine_client_cpp.hpp instead. This includes clients written in pure C and clients written using the C++ shims defined in the engine_client_c.h file. See also the announcement in Removed and deprecated features in 9.12.
As of this version, the deprecated native Universal Messaging integration can no longer be used. It is recommended that you use the Universal Messaging connectivity plug-in instead. See also Apama integration with Universal Messaging in 9.12.
When using the connectivity plug-in to mimic the removed native integration, you should use a single dynamicChains definition:
dynamicChains:
umString:
- apama.eventString:
suppressLoopback: true
description: "@{um.rnames}"
remoteAddress: "@{um.rnames}"
- stringCodec
nullTerminated: true
- umTransport:
channelPattern: ".*"
The configuration of the chain manager depends on your previous Universal Messaging configuration:
|
Removed configuration |
New configuration |
|---|---|
|
|
|
|
|
|
|
|
In the removed configuration, it was possible to configure some channels (without a prefix) to be looked up on Universal Messaging but not created, while also specifying a prefix under which channels would be created. This specific use case is not possible to directly port to the new-style configuration.
Note: The above configuration style which has been removed for the correlator is still used for the IAF (see Defining Universal Messaging properties for the IAF). For communication with the correlator, however, the Universal Messaging transport connectivity plug-in is required, and this connectivity plug-in must be configured to be equivalent to the Universal Messaging properties you have configured for the IAF (see also Communicating with the correlator over Universal Messaging).
Apama 10.0 is the successor of Apama 9.12.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers, BigMemory, and Universal Messaging.
Apama 10.0 includes new features, enhancements, and changes as described in the following topics.
A new connectivity plug-in, the HTTP client transport, is now available. It can be used to connect to external services over HTTP/REST, perform requests on them and return the response as an event. For detailed information, see The HTTP Client Transport Connectivity Plug-in.
Using Software AG Designer, you can add the new HTTP client connectivity bundle to your projects. For further information, see Adding the HTTP client connectivity plug-in to a project.
This HTTP client transport connectivity plug-in is also delivered with version 9.12.0.5. In addition to the functionality provided with that fix, version 10.0 provides the following enhancements:
metadata.http is now a map containing statusCode, statusReason, path, and so on. Applications using the Mapper codec to set or read these will continue to work without change. Custom codecs may need changes to support this.A new standard connectivity plug-in, the MQTT transport, is now available. It can be used to communicate between the correlator and an MQTT broker. This implementation of the MQTT transport is based on the Eclipse Paho C library which supports the MQTT v3 protocol. For detailed information, see The MQTT Transport Connectivity Plug-in.
Using Software AG Designer, you can add the new MQTT connectivity bundle to your projects. For further information, see Adding the MQTT connectivity plug-in to a project.
A new standard connectivity plug-in, the Kafka transport, is now available. It can be used to communicate with a Kafka distributed streaming platform. In this release, it is provided as a sample in the samples/connectivity_plugin/java/KafkaTransport directory of your Apama installation. See also The Kafka Transport Connectivity Plug-in.
Apama 10.0 includes the following connectivity plug-ins enhancements.
Connectivity plug-ins can now perform reliable messaging. If you have done APP_CONTROLLED reliable messaging with the Java Message Service (JMS) before, then doing the same with connectivity plug-ins will look very similar. Your old way of doing acknowledgments and flushes in EPL will be quite easy to translate across to the new way of doing this with connectivity plug-ins. For more information, see Using reliable transports.
If you want to write your own transport that supports reliable messaging, see Developing reliable transports.
The Digital Event Services transport connectivity plug-in now supports reliable sending and receiving of events.
A new metadata value, sag.messageId, is now available. This is used for reliable receiving. For more information, see Metadata values.
Connectivity plug-ins can now add any number of user-defined status values which are reported as part of the correlator’s status information from the REST API, the engine_watch tool, the EngineClient API, and from the EPL Management interface. For more information about how to report status, see User-defined status reporting from connectivity plug-ins. See also the samples\connectivity_plugin\cpp\httpclient and samples\connectivity_plugin\java\HTTPServer directories of your Apama installation for examples of how to report status information from a connectivity plug-in.
The message metadata can now also store non-string values. With previous versions, it was allowed to only store string keys and values.
In both C++ and Java Message implementations, the getMetadataMap() function has been added which returns a view on the metadata which can contain non-string values (map_t for C++ and Map<String,Object> for Java). The getMetadata() function still exists, but is deprecated.
While the changes for Java maintain backwards compatibility for users of the getMetadata() function, the changes for C++ do not. For C++, the metadata_t iterator’s key() and value() functions have been changed to return std::string instead of const char*. Only stringifiable values (bool, integer, double, decimal) can be obtained by calling value(), otherwise an exception is thrown. Moreover, for C++, the metadata operator[] has been changed such that it returns std::string instead of const char*. To access the full set of types in C++ metadata, use the new getMetadataMap() function instead.
For the existing connectivity plug-ins (such as the Mapper codec and Classifier codec), the assumption has been removed that only string keys and values can be stored into the metadata.
For more information, see Metadata values.
Connectivity chains can be created in three different ways:
startChains section of a YAML file.ConnectivityPlugins.createChain.In the previous release, it was not possible for user-defined connectivity plug-ins to dynamically manage chain creation, although some standard plug-ins shipped with the product (such as Universal Messaging and Digital Event Services) made use of this capability. In 10.0, there is a new “chain manager” API providing the ability for user-defined plug-ins written in Java to manage chains. In this release, it is not yet possible to do this from C++ plug-ins (it’s only Java right now).
A transport plug-in that manages its own chains must provide a subclass of com.softwareag.connectivity.chainmanagers.AbstractChainManager which can create and destroy chains whenever it wishes, by instantiating any dynamic chain definition in the configuration files that contain the transport plug-in managed. A chain manager can register a listener that is notified whenever a new channel is used for the first time by the Apama EPL application, that is, when the application subscribes or sends to a channel, which it can use to create a new chain to send/receive using that channel as needed.
For more information about configuring transport connectivity plug-ins that have a dynamic chain manager for use in your project, see Configuration file for connectivity plug-ins.
For more information about how to create a dynamic chain manager transport plug-in, see Static and dynamic connectivity chains.
com.softwareag.connectivity.util.MapExtractor is available in the Java API. This class provides an easy and type-safe way to extract values from maps such as those used for plug-in configuration. It has automatic support for generating user-friendly error messages if configuration items are missing or of the wrong type. See the API reference for Java (Javadoc) for further information.A new standard connectivity plug-in, the String codec, is now available. It can be used to translate string payloads to binary payloads, and vice versa. For detailed information, see The String codec connectivity plug-in.
A C++ implementation of the JSON codec is now available. It behaves in the same way as the already existing Java implementation of that codec. For more information, see The JSON codec connectivity plug-in.
The JSON codec now has an additional configuration parameter filterOnContentType which allows it to ignore messages which are not encoded with the “application/json” content type. For more information, see The JSON codec connectivity plug-in.
The Mapper codec has been enhanced as follows:
defaultValue action, an empty or null value is now treated in the same way as an unset value.copyFrom action is available. This new action is exactly the same as the mapFrom action except that the source field is not removed after the copy.
For more information, see The Mapper codec connectivity plug-in.The Mapper and Classifier codecs can now contain a period (.) for both payload and metadata rules. These correspond to nested maps. Mapper rules for built-in codecs and transports will continue to work without changes, but a mapper which is setting a metadata field containing a period will now produce a nested map structure. Custom codecs and transports may need changes to support this. See also The Mapper codec connectivity plug-in and The Classifier codec connectivity plug-in.
When mapping nEventProperties to metadata, the Universal Messaging transport connectivity plug-in now uses the prefix um.properties. You can use the Mapper codec to enable the original behavior. In addition, the ability to set nEventProperties while also sending a binary payload has been added. To do this, set um.properties.*Key*=*Value* in the metadata. See also Supported payloads.
With previous versions, only the host that created a Universal Messaging channel was able to publish to the channel created by Apama. As of this version, it is possible to specify ACL permissions for channels that are automatically created by Apama. This can be done by specifying the new createChannelPermissions option in the YAML configuration file for the Universal Messaging connectivity plug-in. See Configuring the connection to Universal Messaging (dynamicChainManagers). This behavior has also been changed in the deprecated native Universal Messaging integration.
In 10.0, the API for writing EPL plug-ins in C++ has been replaced. Support for the old C and C++ APIs defined in correlator_plugin.hpp and correlator_plugin.h is still retained in 10.0 for backwards compatibility but are deprecated and will be removed in a future release. Use of them will produce warnings both at compile time and when injecting monitors into the correlator which use a deprecated plug-in.
EPL plug-ins should now be written using the API in epl_plugin.hpp instead. See Writing EPL plug-ins in C++ for information on the new API.
For information on migrating plug-ins from the old APIs to the new one, see Migrating the C and C++ EPL plug-ins written in 9.12 or earlier.
The new EPL plug-in API does not include a pure-C API. Instead, the C++ API has been written with a compiler-agnostic ABI. This removes the requirement to use the platform compiler version when creating your plug-ins and therefore the need for a separate C API. All non-Java plug-ins should now be written against the new C++ API.
Apama 10.0 includes the following enhancements in Software AG Designer:
The color in which annotations are shown in the EPL editor can now be configured in the Apama preferences. See also Editor Colors.
The new field Flush between events has been added to the Data Player query page. It controls how many times the correlator’s queues are flushed between every event from the database. This gives correct behavior in multi-context applications at the cost of speed. The default value for new queries is 2, which flushes the correlator’s queues twice. Existing queries are not affected by this enhancement. A blank value is used in this case, which means that the correlator queues are not flushed. If you want to change this, you have to set the required value on the Data Player query page. See also Specifying Data Player playback queries.
If a query pattern uses the or operator (see also Query enhancements in 10.0), then the New Query Send Event Action dialog includes the new Required events section. It sets which events must be matched by the pattern in order for the send event action to send the event. See also Adding query send event actions.
The Open Type dialog functionality has been added for all Apama perspectives. See Opening EPL elements and Java classes using the Open Type dialog.
The Correlator Deployment Package wizard now accepts a launch configuration instead of a project. The ability to select the files to be included has been removed from the wizard. Instead, if you want to create a correlator deployment package (CDP) containing a subset of project’s files, you must now create a dedicated launch configuration to represent the required files before opening the Correlator Deployment Package wizard. See Exporting correlator deployment packages.
When you add a connectivity bundle, the standard-codecs.yaml file is now created in the Linked Resources node. This file contains standard codecs defined by Apama. See also Connectivity bundles.
Apama 10.0 includes the following query enhancements:
find statement, you can now specify the or operator in the event pattern. The events on one side or the other of the or operator must be matched for the query to fire. For detailed information, see Query or operator.engine_delete -F followed by a fully qualified query name. In previous releases, it was possible to delete a query by just using engine_delete if nothing else depended on it. Note that engine_delete -F deletes everything that depends on the query or the query output event. Therefore, caution is advised when deleting a query. See also Deleting code from a correlator.Apama 10.0 includes the following EPL enhancements:
throw statement is now available. It causes an exception to be thrown. See also The throw statement.com.apama.oob package, componentName and address have been added to ReceiverDisconnected and SenderDisconnected so that the behavior is now in line with ReceiverConnected and SenderConnected. A new dictionary<string, string> field called extraParams has been added to all the out of band events. Additionally in the logging, componentName has been changed to be "*name*", unless no name is provided in which case the old format "*name* (on port *port\_number*)" is used. See also Out of band connection notifications.Apama 10.0 includes the following correlator utility enhancements:
Instead of specifying a list of configuration files on the command line with the --config option, it is now also possible to specify these files in the includes section of another configuration file. For detailed information, see Including YAML configuration files inside another YAML configuration file.
It is now possible to specify the correlator port number in a YAML configuration file. For detailed information, see Specifying the correlator port number.
Using the new engine_deploy tool, it is now possible to generate an initialization file list, a correlator deployment package (CDP), or a correlator deployment directory from an Apama project that has been created with Software AG Designer. You can also use this tool to perform the initialization in a running correlator. If you are not using Software AG Designer, you can also use this tool with a directory of Apama files. For detailed information, see Deploying a correlator.
In addition, you can now also use Ant to generate a correlator deployment directory. The apama-macros.xml file includes the new generate-correlator-deploy-dir macro for this purpose. See also About Deploying Apama Applications with an Ant script.
A new management request has been added: startInternalClock. This is available on correlators via the --doRequest option of the engine_management tool (see Shutting down and managing components) and via the apama-macros.xml Ant script (see also About Deploying Apama Applications with an Ant script). This request takes a previously externally clocked correlator and starts to clock it internally.
Apama 10.0 includes the following dashboard enhancements:
PUT /logLevel request for dashboard data servers and display servers. See also Managing and monitoring over REST.Apama 10.0 includes the following miscellaneous enhancements and changes:
When using the IAF-based CSV codec plug-in (JCSVCodec), you can now enable Excel compatibility mode by setting the new excelCompatible property to true in the IAF configuration file. Double quotes are then used to match the behavior of Excel. By default, this new property is set to false. See also The CSV codec IAF plug-in.
In addition, you can also enable Excel compatibility mode when sending a configuration event from the transport that is communicating with the CSV codec using the method described in Multiple configurations and the CSV codec.
The REST API now also supports the /info/stats URI for the correlator and the IAF. It provides access to the same information as the /correlator/status and /iaf/status URIs. Previously, the /info/stats URI was only available for dashboard data servers and display servers. See also Managing and monitoring over REST.
The new method GenericComponentInfo.getDataValue() has been added to the Apama API for Java. Using this method, you can easily look up a value from a GenericComponentManagement information category that has unique keys (that is, dictionary content) without having to iterate through every item. See the API reference for Java (Javadoc) for further information.
For applications that report status to Command Central, the recommended values for the ".status" key are now STARTING, ONLINE or FAILED (instead of ONLINE and OFFLINE, although OFFLINE is still permitted for legacy applications).
A title can now be provided when generating ApamaDoc from an Ant script or from Software AG Designer. See also Generating ApamaDoc from an Ant script and Exporting ApamaDoc.
The ATM Fraud queries sample has been moved to the demos folder of the Apama installation, and it is now available as a demo from the Welcome page. With previous versions, it was located in the samples\queries\ATM_Fraud_Sample folder.
The demos and samples have been updated to use EPL instead of scenarios. Keep in mind that the Event Modeler is deprecated and that support for creating and deploying scenarios will be removed in a future release (see also Removed and deprecated features in 9.12).
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 10.0. Due to these upgrades, the following versions are no longer supported:
Apama has upgraded from Microsoft Visual Studio 2013 (Visual C++ v12) to Visual Studio 2015 Update 3 (Visual C++ v14). Consequently, you must rebuild the following components with the new compiler:
In addition, when planning migration, consider that the C++ compiler in Visual Studio 2015 is more modern than the previous version and less tolerant of ambiguous and potentially unsafe code patterns. It is likely that some time may be required to fix compiler errors after moving to the new version.
For example, one of our samples previously included this macro definition:
#define snprintf _snprintf
This produces a compilation error with the new compiler. So if your code includes this line, you should simply remove it.
Note: Our recommendation is to enable “warnings as errors” in your projects where possible. This encourages good coding patterns and catches errors earlier in the development process. In particular, it helps you catch errors that might cause subtle, difficult to diagnose errors.
The following changes in Apama 10.0 affect backwards compatibility to previous Apama versions:
Apama 10.0 incorporates the ICU (International Components for Unicode) time-zone data update 2016j, which is the most recent update at the time of release. This will update time-zone data used by the correlator and Time Format EPL plug-in.
In addition, Apama 10.0 contains an upgrade from ICU 3.8.1 to ICU 58.1 which contains some small behavioral differences which may affect your applications:
z no longer uses time-zone short codes such as “EST” due to ambiguity. It now uses GMT+offset formats. The behavior of the V, VV and VVV format/parse symbols has also changed. If you wish to specify time zones by name, you should always use the location-based names, such as “America/New_York” using the format/parse symbol VV. For more information, see Format specification for the TimeFormat functions.w and W).The *chainInstanceId* argument of the createChain method must now specify a unique identifier. Thus, an error is now thrown if a chain with a duplicate ID is to be created. See also Creating dynamic chains from EPL.
There is a change in functionality in the case of YAML configuration files. With previous versions, it was possible to specify, for example, ${PARENT_DIR}foo on Windows. This was a valid specification as PARENT_DIR was expanded with a trailing slash. As of Apama 10.0, this is no longer valid. It is now required that you also specify the slash:
${PARENT_DIR}/foo
This behavior is now in line with PARENT_DIR on Linux, and with APAMA_HOME and APAMA_WORK on Windows and Linux.
The message metadata in connectivity plug-ins can now also store non-string values. For connectivity plug-ins in C++, these changes are not backwards compatible. See Connectivity plug-ins enhancements in 10.0 for more information.
The nullTerminated configuration option of the String codec now defaults to false. This means that the String codec no longer adds a null-terminator in messages towards the transport when converting to a byte[] (Java) or buffer_t (C++) type. This change has been made to be more compatible with the common use case. To enable the old behavior (such as when using the Universal Messaging transport connectivity plug-in to connect to a legacy Apama application), set nullTerminated to true in the String codec configuration. See also The String codec connectivity plug-in.
When mapping nEventProperties to metadata, the Universal Messaging transport connectivity plug-in now uses the prefix um.properties. You can use the Mapper codec to enable the original behavior. See also Connectivity plug-ins enhancements in 10.0#connectivity-plug-ins-enhancements-in-100.
The Mapper and Classifier codecs can now contain a period (.) for both payload and metadata rules. Custom codecs and transports may need changes to support this. See also Connectivity plug-ins enhancements in 10.0.
The ConnectivityPlugins.mon file now also requires the new ConnectivityPluginsControl.mon file from the same directory. This is important if you inject files manually. If you are using Software AG Designer to develop applications that use any connectivity plug-in bundles, this does not matter to you as Software AG Designer automatically upgrades the set of .mon files that you are using.
As of 10.0, ifpresent and any are reserved words in EPL. Projects that are using either of these as an identifier will fail in 10.0.
The following Apama features are now deprecated or have been removed in Apama 10.0:
The subscribe(), unsubscribe() and sendToDES() actions on the auto-generated EPL representations of Digital Event Services types are now deprecated. The channel name is now accessed via the static CHANNEL constant on the EPL type.
For the Java and C++ connectivity plug-ins, the LOGGER field and use of the legacy constructor signatures that do not use TransportConstructorParameters or CodecConstructorParameters are now deprecated.
For C++, the following macros are now deprecated:
SAG_DECLARE_CONNECTIVITY_CODECSAG_DECLARE_CONNECTIVITY_TRANSPORT
You should now use the new signature for the constructor, the new logger field, and the new macros as announced in Connectivity API changes in 9.12.0.1. For more information, see Requirements of a plug-in class.The UMStringCodec is now deprecated. Use the new generic String codec instead. For detailed information, see The String codec connectivity plug-in.
For connectivity plug-ins, the getMetadata() function is now deprecated. Use the new getMetadataMap() function instead. See Connectivity plug-ins enhancements in 10.0 for more information.
Support for the old APIs for writing EPL plug-ins in C and C++ is now deprecated and will be removed in a future release. EPL plug-ins should now be written using the new C++ API. See also New API for writing EPL plug-ins in C++ in 10.0.
Dashboard support for applet deployment is no longer supported. The available deployment options are therefore Web Start, Local, or display server.
Compiling with Visual Studio 2013 is no longer supported. See Windows compiler update in 10.0 for more information.
The following information pertains to the Log File Manager plug-in (LoggingManager) which has been removed in 9.12, except for the associated bundle which was only removed in 10.0. If you still have the deprecated Logging Manager bundle in an Apama project, this results in an error indicating that the LoggingManager.bnd bundle cannot be found. To remove the Logging Manager bundle from an Apama project in Software AG Designer, proceed as follows:
To migrate an existing EPL plug-in that has been written in C or C++ and has been created with version 9.12 or earlier, you will need to follow the steps below. The two APIs are quite different, but you will find the implementation overall simpler and easier to manage.
Include epl_plugin.hpp instead of correlator_plugin.hpp.
#include <epl_plugin.hpp>
Import the com::apama::epl namespace.
using namespace com::apama::epl;
Define a class which inherits from EPLPlugin. This will hold all the methods which are exported to EPL. Previously, these could have been free functions or static member functions. Now they are non-static member functions.
class MyPlugin: public EPLPlugin<MyPlugin>
Add a zero-argument constructor.
MyPlugin(): base_plugin_t("MyPlugin") {}
Change all your exported methods to be members of your class and change the void(const AP_Contex &, const AP_TypeList &, AP_Type &, AP_TypeDiscriminator) signature to be the actual C++ types that your method will take and return. You will no longer need to unpack the typelist argument or explicitly set the return value, you can just read the arguments and return directly.
void myMethod(int64_t arg1)
{
data.push_back(arg1);
}
bool anotherMethod(double arg1, double arg2)
{
return arg1 < arg2;
}
Change use of functions on the AP_Context argument to functions to use the getCorrelator() static method instead.
getCorrelator().sendEvent("MyEvent(42)");
Replace the AP_Function array, type signatures and version declaration from the AP_INIT_FUNCTION_NAME free function with an initialize() static function on your class.
static void initialize(base_plugin_t::method_data_t &md)
{
md.registerMethod<decltype(&MyPlugin::myMethod),
&MyPlugin::myMethod>("myMethod");
md.registerMethod<decltype(&MyPlugin::anotherMethod),
&MyPlugin::anotherMethod>("anotherMethod");
}
Other code from AP_INIT_FUNCTION_NAME goes in your constructor.
Code from AP_SHUTDOWN_FUNCTION_NAME goes in your destructor.
~MyPlugin()
{
...
}
Remove AP_LIBRARY_VERSION_FUNCTION_NAME.
Add a call to APAMA_DECLARE_EPL_PLUGIN for your class.
APAMA_DECLARE_EPL_PLUGIN(MyPlugin)
If you use chunks, you no longer need to inherit from a particular class, just declare methods which return custom_t<T>, where T is the type you want to store in EPL.
struct MyChunkType
{
int64_t datum;
};
custom_t<MyChunkType> createChunk(int64_t datum)
{
return custom_t<MyChunkType>(new MyChunkType{datum});
}
int64_t getChunkValue(const custom_t<MyChunkType> &chunk)
{
return chunk->datum;
}
Apama 9.12 is the successor of Apama 9.10. There is no version 9.11.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers, BigMemory, and Universal Messaging.
Apama 9.12 includes new features, enhancements, and changes as described in the following topics.
Apama now supports Software AG Digital Event Services, which is a messaging system for communicating between different Software AG products using events. Digital Event Services itself has event definitions of its own, called digital event types, that all participating products can have in common.
Using Software AG Designer, you can now add a new connectivity bundle named Digital Event Services to your projects. You use it to automatically generate the Apama event types from the Software AG Digital Event Services type definitions.
The new connectivity bundle includes the Digital Event Services connectivity plug-in. You use this plug-in to send and receive the generated Apama events. All of the configuration for this plug-in is driven by Software AG Designer, and you can edit the configuration by just changing a few properties.
The Apama support for Universal Messaging has changed. A new connectivity plug-in is now available for connecting to Universal Messaging realms and channels. This provides for new features in mapping of message types on Universal Messaging.
Using Software AG Designer, you can add a new connectivity bundle named Universal Messaging to your projects. See Adding the Universal Messaging connectivity plug-in to a project. This bundle includes the Universal Messaging connectivity plug-in. All of the configuration for this plug-in is driven by Software AG Designer, and you can edit the configuration by just changing a few properties. For more information, see The Universal Messaging Transport Connectivity Plug-in and especially the topic Configuring the Universal Messaging connectivity plug-in.
The new Universal Messaging connectivity plug-in replaces the old native Universal Messaging integration, which will be removed in a future release. It is recommended that you use the new connectivity plug-in instead.
EPL now supports optional types. An optional is a value that may be a value of some other specified EPL type, or empty and thus have no value. This is useful for mapping to null values in other languages such as Java, or for results or fields which may be missing a value.
optional values can be extracted using the new ifpresent statement. This statement is used to check if one or more values are empty. It unpacks the values into new local variables and conditionally executes a block of code. See Defining conditional logic with the ifpresent statement and The ifpresent statement for more information.
Note: The ifpresent keyword can no longer be used as a valid identifier.
The connectivity plug-ins can now handle values of optional type. This means that an empty data_t or a null value in Java will translate to/from an empty optional. See also Map contents used by the apama.eventMap host plug-in.
The Apama event parser libraries for Java and.NET have been improved to support events with optional fields. A new class OptionalFieldType (accessible via FieldTypes.Optional) has been added to represent the optional field type. For complete information about this class, refer to the description of
com.apama.event.parser in the API reference for Java (Javadoc)Apama.Event.Parser Namespace in the API reference for.NETNote: optional event field support will not be available in the C parser in AP_EventParser and AP_EventWriter.
Apama now supports the Microsoft Edge web browser.
Microsoft Edge does not support Java plug-ins and therefore does not support deployment of Apama dashboards as Java applets. If you need to deploy Apama dashboards as Java applets, use Microsoft Internet Explorer or Mozilla Firefox instead.
The currently supported versions of web browsers are listed in the Supported Platforms document.
Apama 9.12 includes the following connectivity plug-ins enhancements:
The plugins stanza that is used in the configuration file for the connectivity plug-ins has been renamed to connectivityPlugins. It is recommended that you start using the new name in your configuration files. See also Overview of using connectivity plug-ins.
The --connectivityConfig option which is available for the correlator executable has been renamed to --config. It is recommended that you start using the new name. In addition, it is now possible to specify multiple .yaml files containing connectivity plug-in configuration, which will be merged together. It is also possible now to specify a directory containing .yaml and .properties files. It is no longer possible to specify a YAML file that does not have the .yaml file extension; this will now result in an error. See also Starting the correlator.
A new -D option is available for the correlator executable, which specifies a value for a substitution property to be used by configuration files specified with the --config option of the correlator executable. See Starting the correlator. Using this option is similar to specifying the substitution using a .properties file.
Properties file substitution is now supported, allowing any user-supplied ${*varname*} substitution variables to be replaced inside the configuration files for the connectivity plug-ins (.yaml files). See Using properties files.
Chains can now be defined under both the startChains and dynamicChains stanzas in the configuration file. Chains defined under dynamicChains are not instantiated at startup, but instead can be created dynamically from EPL. See also Configuration file for connectivity plug-ins and Creating dynamic chains from EPL.
Dynamic chains are also used by the Digital Event Services transport and the Universal Messaging transport.
A new predefined Unit Test Harness codec is provided. The Unit Test Harness codec and the new Null Transport transport make it easy to send test messages into a connectivity chain and/or to write out received messages to a text file, without the need to write any EPL in the correlator, which is very useful for writing unit tests for connectivity plug-ins, and especially codecs. For more information, see The Unit Test Harness codec connectivity plug-in.
A new predefined Diagnostic codec is provided, which can be used to diagnose issues with connectivity plug-ins. It logs the events that go through a connectivity chain in either direction. For more information, see The Diagnostic codec connectivity plug-in.
The behavior of the Mapper codec has changed as follows: A mapFrom rule where the source field does not exist now uses the default value if the defaultValue exists or if a subsequent mapFrom rule exists for the same destination field. An error is no longer raised in this case.
For example, let us assume that there is no field b in the metadata and the configuration looks as follows:
mapperCodec:
SomeType:
towardsHost:
mapFrom:
- payload.a: metadata.b
defaultValue:
- payload.a: "someDefaultValue"
In the previous version, an error was raised with the message that b does not exist. As of 9.12, the default value for the payload field a (that is, someDefaultValue) is used rather than raising an error. However, if the destination field (which is payload.a in the above example) is also not defined with defaultValue, then an error is raised.
In addition, a new boolean allowMissing is available. By default, this is set to false. When set to true, an error is not raised when a defaultValue has not been set and a source field is missing. allowMissing needs to be defined at the same level as the event types. For more information, see The Mapper codec connectivity plug-in.
The behavior of the allowMissing configuration property in the apama.eventMap host plug-in has changed. Previously only missing fields were set to their default values. As of this version, fields with empty values (null in Java) are also set to their default values. Similarly, empty values in nested events, elements in sequences and key/value pairs in dictionaries are also set to their default values. See also Translating EPL events using the apama.eventMap host plug-in.
New configuration properties suppressLoopback, description and remoteAddress are now available for the apama.eventMap and apama.eventString host plug-ins. For more information, see Host plug-ins and configuration.
You can now use the jvmOptions key in the configuration file to specify JVM options which the correlator is to pass to the embedded JVM. See also Specifying JVM options.
If you are building C++ plug-ins, you now need to link against the apclient library (instead of apconnectivity). See also Building C++ plug-ins.
Version 9.12.0.1 introduces a new signature for the constructor of a C++ or Java connectivity plug-in, which passes in a TransportConstructorParameters or CodecConstructorParameters object instead of a list of individual parameters.
For Java, there is a new logger field that uses the Simple Logging Facade for Java (SLF4J) instead of the Log4J LOGGER available in earlier versions.
For C++, there is a new logger field to match the Java field, but the behavior is identical to LOGGER although we recommend to use the new name where possible for consistency.
In C++ only, there are the following new macros which must be used for classes with the new 9.12.0.1 constructor:
SAG_DECLARE_CONNECTIVITY_CODEC_CLASS (instead of SAG_DECLARE_CONNECTIVITY_CODEC)SAG_DECLARE_CONNECTIVITY_TRANSPORT_CLASS (instead of SAG_DECLARE_CONNECTIVITY_TRANSPORT)The previous legacy constructors, macros and logger are still available but will be deprecated in a future release. Therefore, we recommend migrating to the new ones when possible, especially for any newly created plug-ins.
To migrate from the old C++ transport constructor and macro to the new one, change the following:
class MyPlugin: public AbstractSimpleTransport
{
public:
MyPlugin(const std::string &name, const std::string &chainId,
const map_t &config)
: AbstractSimpleTransport(name, chainId, config)
{
// do stuff with config here
// use "LOGGER" for logging
}
...
}
SAG_DECLARE_CONNECTIVITY_TRANSPORT(MyPlugin)
to:
class MyPlugin: public AbstractSimpleTransport
{
public:
MyPlugin(const TransportConstructorParameters ¶ms)
: AbstractSimpleTransport(params)
{
// do stuff with params.getConfig() (or the config class member) here
// recommended to use "logger" for logging instead of "LOGGER"
}
...
}
// use new 9.12.0.1 macro for declaring the class
SAG_DECLARE_CONNECTIVITY_TRANSPORT_CLASS(MyPlugin)
The same applies to the C++ codecs, just replace “transport” with “codec”.
To migrate from the old Java transport constructor and logger to the new one, change the following:
public MyTransport(Map<String, Object> config, String chainId,
org.apache.log4j.Logger log4jLogger) throws Exception {
super(config, chainId, log4jLogger);
// do stuff with params.getConfig() (or the config class member) here
LOGGER.info("Transport was created correctly with config: "+config);
...
to:
public MyTransport(org.slf4j.Logger logger,
TransportConstructorParameters params) throws Exception {
super(logger, params);
logger.info("Transport was created correctly with config: "
+params.getConfig());
...
The same applies to the Java codecs, just replace “transport” with “codec”.
Be sure to change all uses of the legacy “LOGGER” to the new “logger” in your Java plug-ins. Most of the logger methods are identical (except that the new logger does not permit logging an exception without an accompanying string message).
See Requirements of a plug-in class for more information about plug-in constructors.
Apama 9.12 includes the following enhancements in Software AG Designer:
When you create an adapter, the instance name that Software AG Designer offers by default now
It is recommended that you specify your own instance names instead of just accepting the default names. An instance name should clearly identify the purpose of the adapter. For example, if you use the correlator-integrated adapter for JMS to connect to a customer tracking system, you should use a name such as “CustomerTracking” (and not just “JMS”).
When you specify your own instance name, it is no longer possible to specify spaces within the name.
Software AG Designer now supports adding multiple User Connectivity configuration instances (each represented by a directory of configuration files) to your project. The instances can contain multiple .yaml and .properties files. You can selectively enable or disable individual User Connectivity instances from the correlator launch configuration dialog. See Creating Apama projects for more information about adding User Connectivity support to a project.
The Connectivity Plug-ins Support option has been removed from under Startup options in the Correlator Configurations dialog, and the Connectivity tab has been added.
In the dialog for configuring correlators in a launch configuration, the Xconfig text box has been removed and replaced with a text box called Configuration. The Configuration text box accepts the path of a configuration file in either YAML (preferred) or the deprecated TXT configuration file format. This configuration can be used for a variety of purposes such as configuring per-package EPL logging settings. See Configuring the correlator.
Apama 9.12 includes the following query enhancements:
A query may now also use the result of an action call on the event as the key for a query. For further information, see Defining actions as query keys.
Coassignment in query files can now use the as operator. For example:
using com.apama.aggregates.last;
query Example {
find Event as e select last(e.value) as lastValue without Stop as s {
}
}
Software AG Designer has been updated to use the as operator when constructing queries graphically. The current : assignment operator is still valid for backwards compatibility. For further information, see Defining event patterns.
Apama 9.12 includes the following EPL enhancements:
The following new string methods are available to search and replace strings using regular expressions:
matches(string *regex*)search(string *regex*)replace(string *regex*, string *replacement*)
Apama currently uses ICU 3.8.1 to implement the regular expressions.Event listeners (on statements) and stream listeners (from statements) can now coassign matching events to an implicitly declared variable using the as operator. This variable is only in scope in the processing block of the listener and has no effect on clashing local or global variables. For example:
// Variable newTick is not yet defined
on StockTick(*,*) as newTick {
// The variable newTick is only in scope in this block
}
The current : assignment operator is still available for backwards compatibility. See Specifying the on statement and Using output from streams for further information.
Data views now support the decimal field type. Decimals can thus be displayed in the Scenario Browser and used in dashboards. An event with a decimal field type can now be sent from a dashboard.
Decimals are now also supported in C, C++ and Java plug-ins, in the MemoryStore and in the distributed MemoryStore.
In C and C++, the decimal value is passed as an AP_decimal which is really a uint64 in the IEEE 754-2008 decimal format. We suggest using the Intel Decimal Floating Point library v2.0U1 (see https://software.intel.com/en-us/articles/intel-decimal-floating-point-math-library) if the actual decimal value needs to be used.
In Java, both BigDecimal (for backwards compatibility) and Number are now supported types for decimal in a plug-in API. For Number, it is either a BigDecimal type for a value, or a Double type for NaN/infinity. NaN/infinity are still not supported when using the BigDecimal type, and will throw an exception that kills the monitor instance if not caught.
It is now possible to generate ApamaDoc from an Ant script on UNIX. For further information, see Generating ApamaDoc from an Ant script.
A new OutOfBandConnections event has been added to the com.apama.oob package, which provides a mechanism to synchronously get the currently connected senders and receivers. For further information, see Out of band connection notifications.
It is now possible to declare static actions inside an event type. For further information, see Defining static actions.
The if statement no longer requires the then keyword. It can still be specified optionally (for backwards compatibility), but you can now write a shorter form. For example:
if seq.size() > 0 { print "First element is " + seq[0]; }
See also Defining conditional logic with the if statement and The if statement.
Apama 9.12 includes the following correlator utility enhancements:
--config option when starting the correlator. For detailed information, see Configuring persistence in a YAML configuration file.--pidfile *file* is available for the correlator, iaf, dashboard_server and display_server tools. It specifies the name of the file that contains the process ID. For more information, see:
toStringQueues, which can be issued with the -r (or --dorequest) option, is available for the engine_management tool. It outputs the current contents of all input and output queues within the running correlator. This can be helpful for identifying slow senders and receivers, and potential causes (such as very large events or excessive flow). See also Shutting down and managing components.getEventTypes, which can be issued with the -r (or --dorequest) option, is available for the iaf_management tool. It returns a string representation of the event types known to the running IAF. The output is equivalent to that of the iaf -e *config.xml* command. The getEventTypes request type, however, does not require a configuration file as input. See also Shutting down and managing components.Apama 9.12 includes the following dashboard enhancements:
GET request, both for XML and JSON: /info/stats. This displays the statistics of instance/trend table queries to dashboard data servers and display servers. In addition, /connections now also shows the connected correlators as well as any configured named dashboard data servers and display servers. See also Managing and monitoring over REST.Apama 9.12 includes the following PySys enhancements:
Correlator class has a new method called initialize() which can be used to initialize the correlator with all the files making up an Apama application (such as .mon, .qry and .evt) by specifying just the path of the Software AG Designer project’s .deploy file, or a .txt file or directory, without the need to list all the files making up the application individually. See the Apama PySys samples and Pydoc for more information and an example of how to use this.Apama 9.12 now supports the following in Command Central:
Apama 9.12 includes the following miscellaneous enhancements and changes:
Apama 9.12 incorporates the ICU (International Components for Unicode) time-zone data update 2016f, which is the most recent update at the time of release. This will update time-zone data used by the correlator and Time Format EPL plug-in.
The BigMemory Max driver now supports supplying converters per table to handle non-Apama managed data. The following new properties, which can be set in the spring.xml configuration file, are available for this purpose:
converterConfig.defaultconverterConfig.byTable
For more information, see BigMemory Max driver details.In Apama versions 9.10 and prior, Universal Messaging channel names were escaped so that only a small subset of the ASCII characters were used in channel names in Universal Messaging. As of Apama 9.12, the new boolean configuration property um.channels.escaped is available which specifies whether channel names are escaped or not. To maintain backwards compatibility, this property is currently set to true, but this may change in future releases. If you set um.channels.escaped to false, Apama passes channel names directly to Universal Messaging without escaping.
When the slash (/) and backslash (\) characters are not escaped, they can now be used to create nested channels. However, you must take care not to use both slash and backslash characters as path separators within the same application as this will result in undefined behavior. See also Considerations for using Universal Messaging channels.
The Docker samples have been updated to support the currently supported Docker and Docker Compose versions.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 9.12. Due to these upgrades, the following versions are no longer supported:
The following changes in Apama 9.12 affect backwards compatibility to previous Apama versions:
Due to tightening of the canParse and parse methods on the EPL types of integer, float and decimal, some previously allowed behavior is now invalid. These EPL types now only allow trailing and leading whitespace, excluding special characters on the float and decimal types such as “d”.
As of this version, integer parsing will only accept integers and whitespace. Any other characters are invalid, which also means that whole numbers such as "1.0" and "2.0" will not parse. The following examples illustrate the new behavior for the canParse and parse methods:
| Now false | Was previously parsed to |
|---|---|
| " " | 0 |
| “2.2” | 2 |
| “2e3” | 2 |
| “2 2” | 2 |
| “2garbage” | 2 |
float and decimal parsing remains unchanged, except that hexadecimal integers will no longer parse. The following example illustrates the new behavior for the canParse and parse methods:
| Now false | Was previously parsed to |
|---|---|
| 0x1d | 0 |
In previous releases, any invalid keyword arguments passed to methods in the Apama PySys extension classes were silently ignored with no error. For example, the following did not produce an error despite being incorrect:
correlator.start(foobar='x')
As of this release, an exception will be thrown if an invalid argument is specified. Thus, test cases that have errors in them are now easy to notice.
The following Apama features are now deprecated or have been removed in Apama 9.12:
Event Modeler is now deprecated. Support for creating and deploying scenarios will be removed in a future release. It is recommended that you use EPL or queries to build new Apama applications.
The EngineManagement API for C is now deprecated. It is recommended that you use the EngineManagement API for C++ instead.
The old native Universal Messaging integration is now deprecated and will be removed in a future release. It is recommended that you use the new Universal Messaging connectivity plug-in instead. See also Apama integration with Universal Messaging in 9.12.
The -r (or --rnames) and -UMconfig ( or --umConfigFile) options of the correlator executable, which pertain to the old native Universal Messaging integration, are now deprecated. Use the --config option together with one or more .yaml files and (typically) also .properties files instead.
The um.install.dir property, which is used by the old native Universal Messaging integration to set the location of the Universal Messaging libraries in the UM-config.properties file, is now deprecated. If Universal Messaging has been installed in the same Software AG installation directory as Apama, Apama will now locate the Universal Messaging installation automatically at ${APAMA_HOME}/../UniversalMessaging and will load the Universal Messaging libraries from the appropriate subdirectory. See also Setting up Universal Messaging for use by Apama.
The -Xconfig option of the correlator executable, which specifies a special configuration file (see Using the Apama component extended configuration file), is now deprecated in favor of YAML configuration via --config. For detailed information on configuration using YAML, see Configuring the correlator and the description of the --config option in Starting the correlator.
As of this version, the deprecated profiling request can no longer be used with the -r option of the engine_management tool. Instead, you now have to use the cpuProfile request, which provides the same functionality. For further information, see Using the CPU profiler. See also the announcement in Removed and deprecated features in 9.10.
When using the engine_management tool with the -r option, it is no longer possible to specify a single string, enclosed in quotation marks, which delimits multiple arguments by spaces. As of this version, it is only possible to specify the arguments as shown in the following example:
engine_management -r cpuProfile frequency
Make sure that multiple arguments are no longer enclosed in quotation marks. Quotation marks are now only used for arguments that contain spaces, such as a file name. See also the announcement in Correlator utility enhancements in 9.9.
As of this version, the deprecated Log File Manager plug-in (LoggingManager) can no longer be used. Instead, you now have to use the built-in logging functionality. For further information, see Logging and printing. See also the announcement in Removed and deprecated features in 9.9.
As of this version, the deprecated Enterprise Management and Monitoring (EMM) and sentinel agent features can no longer be used. It is recommended that you now use Command Central for all deployment tasks. See also the announcement in Removed and deprecated features in 9.10.
The apamadoc.xml and apamadoc.bat files have been removed from the utilities directory of the Apama installation. To generate ApamaDoc without a user interface on both Windows and UNIX, you now have to use the generate-apamadoc macro from the apama-macros.xml Ant script, which is located in the etc directory of your Apama installation. For further information, see Generating ApamaDoc from an Ant script.
As of this version, the Apama EPL Visualization tool for Software AG Designer is no longer installed with Apama. The tool is now available for download from the Apama area of the Software AG Tech Community website at https://techcommunity.softwareag.com.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers, BigMemory, and Universal Messaging.
Apama 9.10 includes new features, enhancements, and changes as described in the following topics.
Apama now provides a new way for developing adapters called connectivity plug-ins.
Connectivity plug-ins can be written in Java or C++, and run inside the correlator process to allow messages to be sent and received to/from external systems. Individual plug-ins are combined together to form chains that define the path of a message, with the correlator host process at one end and an external system or library at the other, and with an optional sequence of message mapping transformations between them. A configuration file describes both the chains and the plug-ins making up each chain. The configuration file is written using the YAML markup language, and can express structured configuration (maps, lists and simple values) for plug-ins. For detailed information, see Using connectivity plug-ins.
A new option, --connectivityConfig, is available for the correlator executable, which specifies the path to the above mentioned configuration file. See Starting the correlator.
A new predefined annotation, com.softwareag.connectivity.ExtraFieldsDict, is available, which can be used to place map keys that do not name fields into a dictionary. See Adding predefined annotations.
You can now record information about a query in the metadata section. This can be, for example, the recording author, the version number, or the last modified date of a query. See Defining metadata in a query for more detailed information.
Metadata properties can be specified for a DataView by adding keys with the prefix DataViewDefinition.EXTRA_PARAMS_METADATA_PREFIX to the extraParams dictionary of DataViewAddDefinition when adding the new DataView definition.
A new constant, EXTRA_PARAMS_METADATA_PREFIX, has been added to the ScenarioService API. Any metadata properties that were set as part of a query or DataView definition will appear in the extraParams with this prefix. See the IScenarioDefinition interface in the API reference for Java (Javadoc) or API reference for.NET for more information.
Once defined, metadata information about a query can be viewed in the Scenario Browser. The Scenario Browser shows metadata properties for queries and DataViews. Metadata properties are shown on the main panel when the query node is selected.
See also Miscellaneous enhancements and changes in 9.10 for information on irrelevant sections in the Scenario Browser that are now hidden from the main panel.
See also Dashboard enhancements in 9.10 for information on the new “Definition extra params table” which can be used to display metadata (and other extraParams information) in a dashboard.
An EPL memory profiler is now provided that you can use to display information on monitors and monitor instances. This is helpful for checking the performance and for finding memory leaks.
To make use of the EPL memory profiler, you use the -r (or --dorequest) option of the engine_management tool with one of the following request types:
eplMemoryProfileOvervieweplMemoryProfileMonitorInstanceDetaileplMemoryProfileMonitorDetailThe information that is returned for a request can be written to a comma-separated values (CSV) file which can easily be viewed in tabular form using a tool such as Microsoft Excel. For detailed information, see Using the EPL memory profiler.
The correlator can now generate “code coverage” information about EPL files, indicating which lines have been executed. This is useful for measuring the quality of test cases, discovering lines of EPL code which are not being exercised by any tests, as well as for helping diagnose bugs or understand complex interactions in the EPL.
The recording of code coverage information can be enabled and written to disk using management requests, or using an environment variable that automatically writes out a coverage file when the correlator is shut down or when code is deleted from the correlator.
The new epl_coverage tool can then be used to merge together the coverage files that have been produced by the correlator and produce summary statistics about how much of each source file is covered, as well as an HTML report where each source line is shown annotated with different colors to indicate which lines are not being covered.
The Apama extensions to the PySys test framework can now enable code coverage recording, and automatically run a coverage report at the end of test execution, which will help users to create better test cases and to find code paths in their EPL applications that do not have adequate test coverage.
For detailed information, see Generating code coverage information about EPL files.
Apama 9.10 ships a new release of PySys, version 1.1.0. This new release includes many useful fixes and enhancements. For full details, see the PySys release notes.
One significant addition in PySys 1.1 is an optional abortOnError capability that produces clearer and more timely “outcome reason” messages to explain why a testcase has failed when something goes wrong, such as an injection failure or an expected signal not being found in a file. The abortOnError functionality can be enabled by setting the following property in your project file:
<property name="defaultAbortOnError" value="true"/>
See the pysysproject.xml file in the samples/pysys directory of your Apama installation for examples of all the settings we recommend for all new test projects. Existing projects may wish to continue with the older settings to avoid the need to fix up any failing tests.
Support has been added for generating EPL code coverage reports from PySys test executions. See Using EPL code coverage with PySys tests for more details.
Correlator and IAF components are now shut down cleanly by default at the end of each test execution, which is a prerequisite for getting EPL code coverage information. See also Miscellaneous changes in 9.10 affecting backwards compatibility for information on how to disable this new behavior.
A new antBuild method has been added which can be used to invoke Apache Ant builds with the Apama environment variables defined. This can be used, for example, to execute an exported build.xml file which references the apama-macros.xml file.
Several new methods have been added to the Apama PySys correlator object:
inspectsendEventStringsinputLog argument to startCorrelatorOn Linux, PySys Docker extensions are now available in the third_party/python/lib/python2.7/site-packages/docker directory of your Apama installation.
You can now monitor Apama dashboard data server and display server components with the same Apama Representational State Transfer (REST) HTTP protocol that is already supported by the correlator and IAF. This provides monitoring capabilities to third-party managing and monitoring tools or to any application that supports sending and receiving XML or JSON documents over the HTTP protocol.
The HTTP protocol uses the management port of the dashboard servers. As a result, in addition to the existing dashboard_management tool, it is now possible to use the component_management tool, the Java GenericComponentManagement API or any HTTP REST client supporting XML or JSON to perform tasks such as pinging the server, listing the version number, memory usage, environment variables and start-up arguments, shutting down the server, and sending other dashboard-specific requests. See Managing and monitoring over REST for more information about using REST with Apama.
Support has been added for the JSON equivalent to most XML URI GET requests as per those listed below:
/correlator/info/correlator/status/correlator/appLogging/correlator/types/iaf/status/info/logLevel/connectionsFor detailed information, see Managing and monitoring over REST.
The correlator types are now also accessible from the web browser using http://localhost:15903/correlator/types. Note that there is no security around these requests. Anyone who has access to the host and port will be able to make these requests. See Security Requirements for Apama for more information.
A Platform Manager Plug-in node is now available in the product selection tree of the Software AG Installer. You can see it when you expand the Apama node. This plug-in allows you to manage the server components of Apama using Command Central.
Apama integration with Command Central supports the following features:
Apama 9.10 includes the following enhancements in Software AG Designer:
Apama 9.10 includes the following EPL enhancements:
A new method add() has been added to the Table class of the MemoryStore. add() does the same as get(), except that it does not check if the row that is to be added already exists in the table until commit() is called and it therefore never throws an exception. If you are sure that the row does not yet exist, you can use add() as this is faster than get(). See also Creating and removing rows.
All comparable types can now be used with the comparison operators (<, >, <=, >=, = and !=). This includes sequences and dictionaries where all members are of a comparable type. See also Comparable types.
The following new built-in aggregate functions are now available:
countUnique(*value*)percentile(*value*,*rank*)
The built-in positional aggregate functions first(*value*), last(*value*) and nth(*value*,*index*) now also support integer, string, boolean and location types. In addition, nth(*value*,*index*) now also supports negative indices to get items from the end of the window (-1 means the last item, -2 means the second last item, and so on).The built-in aggregate functions min(*value*) and max(*value*) now also support integer types.
Apama 9.10 includes the following correlator utility enhancements:
Using the compiled runtime now requires that the binutils package is installed on the Linux system. For the Linux platforms supported by Apama, this is part of the default installation.
Injection times of applications using the compiled runtime have been improved in this release. There is a performance improvement in initial injections for codebases with a large number of actions.
The correlator can also be configured to use a cache of the compiled artifacts to speed up subsequent re-injection of the same source files. For this purpose, the following new option is available for the correlator executable:
--runtime-cache *dir*
For more information, see Starting the correlator and Injection time of compiled runtime.Using the new optional ${PID} tag, you can now add the process ID to the name of a log file. This is helpful if you want to run multiple correlators with the same arguments but with separate log files. For more information, see Specifying log filenames.
New options for log handling which were previously only available as request types of the -r option are now available with the engine_management, component_management and iaf_management tools.
The following options are available for all of the above-mentioned tools:
--setLogFile *path*
--reopenLog
The following options are only available for the engine_management tool:
--rotateLogs
--setApplicationLogLevel [package=]*level*
--setApplicationLogFile [package=]*path*
--unsetApplicationLogLevel *package*
--unsetApplicationLogFile *package*
--unsetRootApplicationLogLevel
--unsetRootApplicationLogFile
--getApplicationLogLevel *package*
--getApplicationLogFile *package*
--getRootApplicationLogLevel
--getRootApplicationLogFile
For more information on these options, see Shutting down and managing components.
The following new options are now available for the engine_management, component_management and iaf_management tools, which can be used to get the uptime, and the virtual and physical memory usage of the component:
-M | --getuptime-Vm | --getvmemory-Pm | --getpmemory
For more information on these options, see Shutting down and managing components.Corresponding methods are also available in the Java EngineClient API.
Apama 9.10 includes the following dashboard enhancements:
extraParams member of a definition (scenario, DataView or query). An entry in extraParams is considered metadata if its key name starts with the Metadata: prefix. This prefix will be automatically added for the query properties.--dashboardDir *folder* is now available. The display server can be run from any directory by using this option to specify an arbitrary *folder* where deployed dashboards can be found. If this option is not used, then APAMA_WORK/dashboards is assumed.--dashboardExtraJars *jarFiles* is now available. Custom function, custom command and/or any extra jar files referenced by the dashboard can now be specified by using this --dashboardExtraJars option. *jarFiles* contains a list of semicolon-separated JAR files to be added to the classloader at runtime. If this option is not used, the APAMA_DASHBOARD_CLASSPATH environment variable must be set accordingly for the JAR files to be accessible by dashboard processes.It is now possible to pass the username and password on the JDBC connection URL, rather than having to set it in EPL. To do so, you must set the special value ADBC_NULL on both the username and password parameter on the EPL side, for example, via the Connection.openDatabaseFull() action in ADBCEvents.mon.
The new IAFStatusDataViewService provides support for publishing the status of an IAF adapter as a DataView item. It can be used to easily monitor the adapter status using Apama’s Scenario Browser or to visualize the adapter status using an Apama dashboard. For more information, see DataView support.
ApamaDoc documentation is now provided for most of Apama’s public event APIs.
By default, the ApamaDoc export wizard does not generate documentation for inner fields of a monitor. If you want to include inner fields of a monitor, run ApamaDoc in headless mode using the --includeMonitorMembers option.
ApamaDoc now supports <code/> tags inside ApamaDoc comments.
The correlator now logs INFO log lines if an external sender sending events in to the correlator is blocking, giving a reason as to why the queue is full. For example:
Blocked adding 23 events to context main (id=1)'s queue; context is blocked
waiting on correlator (on port 62156) which is flagged as a slow receiver
Blocked adding 2 events to context main (id=1)'s queue; context is blocked
waiting on ctx1 (id=2) which is running a plug-in
The Java APIs now implement the AutoCloseable interface for simple cleanup using the Java try-with-resources statement, where applicable. See the API reference for Java (Javadoc) for more information.
The Apama work directory (APAMA_WORK) now has a lib directory which is on the PATH/LD_LIBRARY_PATH set up by the Apama Command Prompt (apama_env script). This lib directory can be used for user-supplied C++ correlator and IAF plug-ins.
Client applications can now add user-defined status values which can be appended to the standard status messages of the correlator:
get, set and delete actions for the user-defined status values are now available from the EPL Management interface. See the corresponding section in Using the Management interface.-a (or --all) option is specified for the engine_watch tool, any user-defined status values are now appended to the standard status messages. See Watching correlator runtime status.The getMostBackedUpInput and getMostBackedUpQueueSize methods of the EngineStatus interface have changed meaning. They now give the most backed up, or slowest, context name and queue size across all contexts, not just public contexts, as non-public channels can subscribe to specific channels and thus receive external events.
The Scenario Browser now hides the irrelevant sections from the main panel.
Scenario, the metadata section is hidden.Dataview, the input section is hidden.Query, the output section is hidden.On Linux, a Universal Messaging Docker sample is now available in the samples/docker/applications/UniversalMessaging directory of your Apama installation.
With the previous Apama version, the correlator automatically shut down after 30 minutes when a license file could not be found. As of this version, when a license file cannot not found, Apama can be used without a time limit, but with reduced capabilities. See Running Apama without a license file.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Support for the following has been removed:
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 9.10. Due to these upgrades, the following versions are no longer supported:
Due to changes to the supported JDK version (see Supported JDK version in 9.10), support for the following application servers has been removed in order to support application server versions that would run using a supported JRE version:
Apama is now built against Oracle JDK 8, and installs with Oracle JRE 8. JDK 7 is no longer supported.
To develop, build, and test an Apama 9.10 application, you must use Oracle JDK 8, which is the version that Software AG Designer installs. Use of any JRE other than the one that the product ships with is discouraged.
However, for applet and/or Webstart dashboards to run properly in 32-bit browsers, the latest 32-bit JREs are recommended on the client machines.
With previous versions, the EPL coassignment mechanism allowed coassignments from different triggerings of a listener instance to leak into each other. For example, consider the following code:
A a := A(0);
B b := B(0);
on all (A():a or B():b) {
print a.toString() + " " + b.toString();
}
Previously, sending A(1) and B(1) printed the following, where the coassignment of A(1) carried over into the triggering of the listener caused by B(1):
A(1) B(0)
A(1) B(1)
This has been corrected in this version so that the above code will now print the following:
A(1) B(0)
A(0) B(1)
This change may also cause a slight reduction in memory usage and an increase in performance of persistent applications.
The isExternal() method on events in EPL has changed behavior. While in previous releases, the route, send..to or enqueue..to statements would change an event to always be internal, in 9.10, these do not modify the isExternal() property. For example, an external event that is forwarded to a second context will still appear to be external. The clone() method always returns an internal event, so if an internal event is required, then enqueue evt.clone() to target will give the same behavior as previous releases.
Apama now ships with the files QueryServices.cdp and SetSingleContext.cdp instead of QueryServices.mon and SetSingleContext.mon. That is, correlator deployment packages are now provided instead of EPL files.
The Apama extensions to the PySys testing framework now perform a clean shutdown of correlators and IAFs before the test ends, rather than terminating the process without warning. This makes it easier to detect problems associated with the shutdown, and it also allows code coverage information to be retrieved. If you want to disable this new behavior, add the boolean property shutdownApamaComponentsAfterTest to the project file.
When a launch configuration is created for a project in Software AG Designer, the name “defaultCorrelator” is now used. Backwards compatibility is provided for the Apama projects that you have created with previous versions: the name “Default Correlator” is automatically considered as “defaultCorrelator”.
However, if you still have Apama projects that have been created before version 5.2 and if these projects have adapter configurations that use the ${Default Correlator.port} and ${Default Correlator.hostname} properties and do not have any launch configurations, the Ant export of these projects will not work seamlessly. As a workaround, you have to apply any of the following changes to these projects:
${Default Correlator.port} and ${Default Correlator.hostname} properties to ${defaultCorrelator.port} and ${defaultCorrelator.hostname}.environment.properties file and add the value Default Correlator.port=*portValue*.The random number generator in the Apama correlator has been updated to provide higher resolution and improved performance. This change will result in different sequences of random numbers being produced and as such any code or tests which rely on specific sequences of random numbers via seed values (provided, for example, via the -XsetRandomSeed option of the correlator executable) or which use the “replay log” feature will need to be updated.
Performance improvements in this version have eliminated the need for different correlator scheduler types. The --scheduler option has therefore been removed from the correlator executable. Note that the correlator will fail to start up if you continue to use this option.
The location of Apache Ant which is shipped with Apama has been changed and it can now be found in the third_party\apache_ant directory of your Apama installation. If you have customized any scripts to rely on the previous location, we recommend that you test your scripts carefully to ensure that they work as expected.
The correlator now uses separate Java child classloaders for loading the libraries required for distributed MemoryStore and JMS support, which substantially reduces the number of libraries present in the top-level system classloader that all correlator, connectivity, JMS and distributed MemoryStore plug-ins share, and therefore reduces the potential for library conflicts between them and customer plug-ins. In previous versions it was possible (though not recommended) to start the correlator with distributed MemoryStore and JMS configuration files in the same directory; this is no longer possible and will result in “Failed to initialize correlator: [ADAPTER_NAME] configuration is invalid: Line XX in XML document from URL…” errors.
To avoid “class not found” problems, ensure you are specifying classpaths using the settings in the relevant XML configuration files for JMS and distributed MemoryStore rather than using command-line options that change the correlator’s system classpath such as -J-Djava.class.path. The vast majority of applications will be doing this already and will require no modification. In some situations, if your plug-in or a library it depends upon uses the context (thread-specific) classloader to locate classes, it may also be necessary to set that explicitly using the following:
Thread.currentThread().setContextClassLoader(
*YourClassHere*.class.getClassLoader() )
action *action-name* {...} are deprecated as of this version. If you still use such a declaration in your code, make sure to change this to action *action-name*() {...} (with additional parentheses after the action name).count(*value*). Use the alternative predicate aggregate function count(not value.isNaN()) instead. Note that only the aggregate function is deprecated which uses the decimal and float values. The aggregate functions count() (without an argument) and count(*predicate*) (with a boolean argument) are still supported.prior(*value*,*index*). Use the alternative aggregate function nth(*value*,*index*) with a negative index instead.profiling request of the engine_management tool has been deprecated. A new cpuProfile request, which provides the same functionality, is available instead. You can still use the deprecated profiling request, however, it is recommended that you now use the new cpuProfile request.Apama 9.9 is the successor of Apama 5.3.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release. Be sure to consult that document for details about supported versions of operating systems, compilers, BigMemory, and Universal Messaging.
Apama 9.9 includes new features, enhancements, and changes as described in the following topics.
As of version 9.9, Apama is installed using the Software AG Installer, and official fixes are installed using the Software AG Update Manager. See the new Installing Apama guide for more information.
Previous Apama releases (such as 5.3) allowed you to select one of three installation types: Server, User and Developer. In the Software AG Installer, these correspond to selecting the following options from the product selection tree:
Apama > Server. Installs the server components of Apama, including the correlator and associated command line tools.
Note: On UNIX, this is the only installation option. All other options listed here are only available on Windows.
Apama > Dashboard Viewer. Installs the viewer tool that allows you to view and interact with dashboards that are receiving live data from an Apama dashboard server.
Note: Unlike the old User installation type, the Software AG Installer only installs the viewer tool, and does not include any other Java/C++ libraries.
Designer > Apama > Apama Development Environment. Installs Software AG Designer which provides graphical tooling for creating Apama applications.
All Apama documentation is now installed regardless of which installation type you select.
You can quickly select all Apama components (that is, the full functionality of Apama) by clicking Typical Installations which is shown next to the product selection tree and then selecting Apama.
Due to the above-mentioned changes, all Windows start menu shortcuts have changed.
The Eclipse-based development environment previously known as Apama Studio is replaced by integration into Software AG Designer. You can open Software AG Designer, which is also based on Eclipse, by choosing the following from the Windows Start menu: All Programs > Software AG > Tools > Software AG Designer n.n. Note that Software AG is the default group name that you can change during the installation.
On Windows, the Apama work directory (APAMA_WORK) now defaults to the all-users %PUBLIC% directory, for example, C:\Users\Public\SoftwareAG\ApamaWork_*n.n* (with previous versions, it was in C:\Users\*username*).
On UNIX, the default for the Apama work directory is now ~/softwareag/apamawork_*n.n*.
With previous versions, the Eclipse workspace was stored under APAMA_WORK by default. Now it is in the location that you select when you start Software AG Designer for the first time. By default, the workspace is located in the C:\Users\*username* directory.
It is now also possible to install Apama using Command Central. You can then install Apama to any machine on which the Platform Manager is running. See the new Installing Apama guide for more information.
Apama 9.9 includes the following query enhancements:
with unique clause which discards all but the latest event when many events have the same value for a specified field. See Matching only the latest event for a given field.As of version 9.9, Apama’s main tool for implementing Apama applications runs in Software AG Designer, which is based on Eclipse, and is no longer called “Apama Studio”.
New Apama features in Software AG Designer include:
.ste files are picked up has been changed to *software\_ag\_install\_dir*/Designer/extensions.apama-macros.xml file. The build.xml file, which is generated from the ANT export action, now references the apama-macros.xml file in APAMA_HOME.Apama 9.9 includes the following EPL enhancements:
TimeFormat event in the API reference for EPL (ApamaDoc).Apama 9.9 includes the following correlator utility enhancements:
engine_management
When using the -r option, you should now specify distinct arguments: the request type and a list of strings. For example:
engine_management -r profiling frequency
In previous versions, you had to specify a single string that delimited the arguments by spaces:
engine_management -r "profiling frequency"
When specifying your management requests now, make sure that the arguments are no longer enclosed in quotation marks. Quotation marks are now only used for arguments that contain spaces, such as a file name. For example:
engine_management -r setLogFile "my new log file.log"
For more information on engine_management, see Shutting down and managing components.
Equivalent changes have been made to the doRequest methods in the Java client API and the engine-management-request macro in the Ant macros. Applications making use of either must be updated accordingly.
Apama 9.9 includes the following dashboard enhancements:
webChartFlag property is set to true by default.webLabelFlag is now available for all objects on the Labels and General tabs of the object palette. It is also available for the checkbox object on the Controls tab. The webLabelFlag property is set to true. With this property, the display server client will render the objects from the above-mentioned tabs using HTML5 (on supported browsers) rather than in the image generated by display server. This enhancement can improve system performance in some cases and will also allow users to copy text strings from the objects to the clipboard. This property will be ignored in dashboard_builder, dashboard_viewer, Applet and WebStart clients.Replace All, is now available. It replaces all occurrences of a given string which matches the pattern of the regular expression with another string.The Apama installation includes an Oracle Java(TM) SE Development Kit (JDK). To write and compile Java applications, you need a Java compiler such as javac, and the jar utility. Software AG recommends that you use Oracle JDK 8 included in the Apama installation to develop, build, and test your applications. The minimum JDK version you can use is JDK 7.
Apama 9.9 incorporates ICU (International Components for Unicode) time-zone data update 2015e, which is the most recent update at the time of release. This will update time-zone data used by the correlator and TimeFormat correlator plug-in.
The correlator will now search APAMA_WORK/license/ApamaServerLicense.xml to locate a license file if none is specified on the command line. If the license file is not found at that location, the correlator will also search Apama/etc/ApamaServerLicense.xml in the installation directory. If the found license file is expired, the correlator will fail to start up. In order to run the correlator in evaluation mode, any expired license file must be removed.
Note: The default name for the license file is now ApamaServerLicense.xml and no longer license.txt.
If the license cannot be found, remote connections are now allowed. You are no longer restricted to communicating only with processes on the same machine.
The correlator log will now include the type and size of the largest container (sequence, dictionary, listener, stream window) in each MThread when the logging of garbage collection events has been enabled using the verbosegc command. This is to allow easier diagnostics of applications that are using unusual amounts of memory.
The file jms-mapping-spring.xml, which stores the rules for mapping between JMS messages and Apama events, is now generated directly into the bundle_instance_files\Correlator-Integrated_JMS folder of an Apama project. With previous versions, this file was generated into the bundle_instance_files\Correlator-Integrated_JMS\Generated folder. The Generated folder is no longer created and used by Apama.
The connection event now has a new reopen action: reopenWithACK which requires a callback. This is useful to know when a reopen succeeds or fails. We recommend that you use this method over the deprecated reopen().
The shutdownConnectionOnError action will now be based on the reconnect policy and will therefore no longer remove outstanding queries from the queue if, for example, you have a reconnect policy of RECONNECT_AND_RETRY_LAST. In this case, outstanding queries will remain on the queue to be retried on a successful reconnection.
The Docker sample directory contains configuration and samples to help you containerize and run Apama components and applications on the Docker platform, including
For more information about the sample, see README.txt in APAMA_HOME/samples/docker.
Apama runs on the platforms listed in the Supported Platforms document for the latest version. For older versions, see the changelog entries adding and removing platforms for that release.
Support for the following has been removed:
Check the above mentioned Supported Platforms document for details about the newer platform versions supported by Apama 9.9. Due to these upgrades, the following versions are no longer supported:
Several libraries have been renamed. In most cases, the version number has been removed from the file names and an “ap” prefix has been added. Due to this change, you may need to update your build scripts and automated tests. Software AG Designer will automatically upgrade any Apama project with the “Apama Java” nature to have the renamed jar files on its build path.
The following table lists the most important name changes:
| Old name | New name |
|---|---|
| Windows: | |
| apcommon5.3.lib | apcommon.lib |
| engine_client5.3.dll | apclient.dll |
| engine_client5.3.lib | apclient.lib |
| engine_client_dotnet5.3.dll | apclientdotnet.dll |
| iafcore5.3.lib | apiaf.lib |
| Linux: | |
| libengine_client.so | libapclient.so (-lapclient) |
| Jar files: | |
| dashboard_client5.3.jar | ap-dashboard-client.jar |
| correlator_extension_api5.3.jar | ap-correlator-extension-api.jar |
| engine_client5.3.jar | ap-client.jar |
| jplugin_public5.3.jar | ap-iaf-extension-api.jar |
| util5.3.jar | ap-util.jar |
| distmemstore-bigmemory.jar | ap-distmemstore-bigmemory.jar |
| distmemstore5.3.jar | ap-distmemstore.jar |
Note: You should no longer reference dashboard_studio*n*.*n*.jar. All required classes are now available in ap-dashboard-client.jar.
WARN message to make it clear when this has happened. If you have previously been running with an insufficient license, then you may experience a corresponding reduction in performance after upgrading.<start-correlator> (generated by the Ant Export feature) checks whether there is already a correlator running. However, previously that was ignored allowing <start-correlator> to succeed even if a new correlator with the specified parameters was not started. Now it will log a warning and (by default) fail the task if a correlator is already running. An Ant build script exported from a launch configuration with Reuse Correlator selected will therefore now fail. To return to the previous behavior, edit the build.xml to set the failoncorrelatorrunning=false attribute on the <start-correlator> task, or set the global property correlator.failonrunning, for example, by specifying -Dcorrelator.failonrunning=false on the command line when executing Ant.LoggingManager) is being deprecated and should not be used any more, as it will be removed in a later release. In the future, you should just use the EPL log statement to write log messages, and configure logging using the Management interface (see Using the Management interface). New actions have been added to the com.apama.correlator.Logging event, such as setApplicationLogFile, setLogFile and setApplicationLogLevel. These actions are the equivalent of using the engine_management requests to configure logging (see also Shutting down and managing components). In previous releases com.apama.correlator.Logging had only one action on it, namely rotateLogs().findAvailableServers and findAvailableServersFull have been deprecated, and replaced with more appropriately named findAvailableDataSources and findAvailableDataSourcesFull respectively. These are identical in behavior.reopen action on the connection event of ADBCEvents.mon has been deprecated. Please use the action reopenWithAck() instead.