Note: Apama queries are deprecated and will be removed in a future release.
An Apama query is a self-contained processing element that communicates with other queries, and with its environment, by sending and receiving events. Queries are designed to be multithreaded and to scale across machines.
Note:
Queries and monitors are the two main EPL programming units. A query cannot contain a monitor. A monitor cannot contain a query. Each unit offers a different approach to event processing.
You use Apama queries to find patterns within, or perform aggregations over, defined sets of events. For each pattern that is found, an associated block of procedural code is executed. Typically this results in one or more events being transmitted to other parts of the system.
A query is defined in a .qry file. A query finds specified event patterns or aggregates event values.
Apama queries are useful when you want to monitor incoming events that provide information updates about a very large set of real-world entities such as credit cards, bank accounts, or cell phones. Typically, you want to independently examine the set of events associated with each entity, that is, all events related to a particular credit card account, bank account, or cell phone. A query application operates on a huge number of independent sets with a relatively small number of events in each set.
The following topics provide reference information for the parts of a query definition. For user guide type information, see Defining queries.
Query lifetime
You inject queries into a running correlator with the Apama macros for Ant, (*install\_dir*\etc\apama-macros.xml) or with Apama Plugin for Eclipse. You can delete queries from a running correlator by performing a delete operation and specifying a query name. You can use the same tools that you use to delete monitors: engine_delete utility, Apama Plugin for Eclipse, Apama macros for Ant (apama-macros.xml), or deleteName() method on the engine client API.
If you are using a cluster of correlators, it is your responsibility to inject each query into each correlator in the cluster, and to delete a query from each correlator in a cluster. This keeps deployed queries in sync across the cluster. In other words, injecting or deleting a query on one host in a cluster does not automatically inject or delete the query on the other cluster members.
Unlike monitors, the lifetime of query instances is either automatic (for non-parameterized queries) or controlled by the Scenario Service (for parameterized queries, see Scenario Service API). There are no spawn or die equivalents in queries, and you cannot use these EPL statements in queries.
When a non-parameterized query is injected, a single instance of the query is automatically created at injection time and it begins processing events. You cannot use the Scenario Service API to edit or delete this single instance or to create new instances. For parameterized queries, after injection, only the query definition is created automatically. The query does not start processing events specified in its inputs section until at least one parameterization is created by means of the Scenario Service. You can control this by using a dashboard or scenario browser. The Scenario Service has methods to create new query instances, edit instances and delete instances.
When using a cluster of correlators, the parameterizations are kept in sync across all members of the cluster. Creating a query instance while connected to one cluster member will create it on all members. The instance can be edited or deleted by any client connected to any member. There may be short delays in replicating parameterization data on each cluster member because this happens asynchronously. However, the recommendation is to edit or delete a particular parameterization from Scenario Service clients that are all connected to the same correlator. This ensures that edit and delete operations are performed in the same order on every cluster member. If you try to edit or delete the same parameterization from different cluster members the results are unpredictable.
If a query executes code in a where clause, aggregate or other expression that results in an exception due to the current values in the window, the query ignores the exception and continues running. For example, an attempt to divide an integer by zero causes an ArithmeticException. If a query experiences an exception that means it cannot continue (such as repeated exceptions while trying to retrieve or store window data), then the query instance will enter the failed state, which will be reported by the Scenario Service. In this case, the query does not process additional events. The correlator log file should contain information that explains why the query failed. The problem that caused the failed state needs to be corrected. After correcting the problem, if the query is a parameterized query, you should delete the failed parameterization and then re-create it. For a non-parameterized query, you must delete and then re-inject the query.
When a query is deleted with the engine_delete utility or equivalent, all instances of the query are terminated and the Scenario Service will reflect that the query definition has been unloaded. The query can be re-injected, if needed. Remember that deletions and injections must be performed on every member in a cluster.
Lifetime of find statements
As long as a query is active, the find statement in a query is active for each value of the key that is specified in the query’s inputs section. Thus, find A as a in a query is similar to on all A() as a in a monitor. The find statement generates a match set each time the latest event causes a match. If the find statement specifies any aggregates and the every modifier, which can only be used with aggregates, then each new match set causes the find statement to add to the aggregate.
In monitors, listeners can match either the first set of matching events, or specify the all operator to fire for every set of matching events. For example, on A() as a -> B() as b fires on the first A and B events, while on all A() as a -> all B() as b fires for every combination of an A event with a later B event. In a query, find A as a -> B as b fires on every B event after an A event if an A event is still in the window defined in the inputs section. The match set contains the most recent A event and the most recent B event. The following table provides examples. The assumption is that all input events remain in the query’s window.
Input events
Query match sets for: find A as a -> B as b
Query match sets for: find every A as a -> B as b
select… - inputs to aggregates
Monitor match sets for: on A() as a -> B() as b
Monitor match sets for: on all A() as a -> all B() as b
A(1)
B(1)
A(1), B(1)
A(1), B(1)
A(1), B(1)
A(1), B(1)
A(2)
B(2)
A(2), B(2)
A(1), B(1)A(1), B(2)
A(2), B(2)
A(1), B(2)A(2), B(2)
B(3)
A(2), B(3)
A(1), B(1)A(1), B(2)
A(2), B(2)
A(1), B(3)
A(2), B(3)
A(1), B(3)A(2), B(3)
Query definition
A query searches for an event pattern that you specify. You define a query in a file with the extension “.qry”. Each .qry file contains the definition of only one query.
If specified, any package or using statements must be before the query declaration. See Packages and The using declaration.
You must specify an identifier for the query name. See Identifiers. The convention for specifying the name of a query is to use UpperCamelCase, as shown in the example below.
Specification of metadata is optional. See Metadata section. The convention for specifying the key in the key-value pair of the metadata is to use lowerCamelCase as shown in the example below.
An inputs section is required. It specifies at least one event type. These are the event types that the query operates on. See Inputs section.
The find statement is required. It specifies the event pattern of interest and a block that contains procedural code. See Find statement.
Action definitions, in the same form as actions in events, are optional. See Event actions.
Example:
query ImprobableWithdrawalLocations {
metadata {
"author":"Apama",
"version":"1" }
parameters {
float period;
}
inputs {
Withdrawal() key cardNumber within (period);
}
find
Withdrawal as w1 -> Withdrawal as w2
where w2.country != w1.country {
log "Suspicious withdrawal: " + w2.toString() at INFO;
}
}
Metadata section
In a query, the optional metadata section specifies a list of key-value pairs. If there is a metadata section, it must be the first section in the query. See Defining metadata in a query for further information.
Parameters section
In a query, the optional parameters section specifies any parameters used by the query. If there is a parameters section, it must follow the metadata section, if defined, and it must precede the inputs section. Parameter values are available throughout a query. See Implementing parameterized queries for further information.
Inputs section
In a query, the required inputs section specifies the events that the query operates on.
At least one input definition is required. Typically, no more than four input definitions are specified.
If there is a parameters section, then the inputs section follows it. The inputs section must be before the find statement.
Example:
inputs {
A() key k retain 20;
B() key k retain 10;
}
In a query, the required inputs section must contain at least one input definition.
An event type you specify must be parseable. See Type properties summary. Event type names can come from the root namespace, a using declaration, or a local package as specified in a package declaration.
Event filters are optional. Specifying a filter here determines which events are added to a query window. The rules for what you can specify for the event filter are the same as for what you can specify in an event template in EPL. See Event templates.
Specification of a key is optional, but rarely omitted. If there is no key specification, all events are in one partition. The correlator uses the key to partition events. Each partition is identified by a unique key value. Specify one or more fields that are in the input event type. One or two fields in a key is typical. Three fields in a key is unusual and rarely needed. More than three fields is discouraged. If you define more than one input in a query
The number, type, and order of the key fields in each input definition must be the same.
If the names of the key fields are not the same in each input definition, you must insert the as keyword to specify aliases so that the names match. For details, see About keys that have more than one field.
A retain clause or a within clause is required. Alternatively, you can specify both.
A retain clause indicates how many events to hold in the window. Follow the retain keyword with a positive integer. If you specify a negative integer or zero, it is a runtime error that terminates the query.
A within clause indicates the length of time that an event stays in the window. Follow the within keyword with a positive float expression or a time literal. If you specify a negative float value or zero it is a runtime error that terminates the query.
A query find statement tries to find a match for the event pattern that the find statement specifies. When the query finds a match it executes the EPL in the find statement block.
When a find statement specifies a select or having clause, the every modifier is required. Conversely, you cannot specify the every modifier if you do not specify a select or having clause.
When a find statement specifies the every identifier, the identifiers in the select clause are available in the having clause and in the find block, but the coassignments in the pattern are not available.
Pattern coassignments are available in a where clause that applies to the pattern.
When you do not specify the every modifier, all pattern coassignments, except a without clause coassignment, are available in the find block.
In a where clause that is part of a without clause, pattern coassignments as well as the coassignment in the without clause are available.
Example:
find Withdrawal as w1 -> Withdrawal as w2
where w1.country = "UK" and w2.country = "Narnia" {
// Recent card fraud in Narnia against UK customers
send SuspiciousWithdrawal(w2) to "SuspiciousChannel";
}
In a query definition, the find statement specifies the event pattern of interest followed by a procedural block that specifies what you want to happen when a pattern match is found.
A coassignment variable specified in an event pattern is within the scope of the find block and it is a private copy in that block. Changes to the content that the variable points to do not affect any values outside the query. Unlike EPL event expressions, you need not declare this identifier before you coassign a value to it.
In an event pattern in a find statement, each coassignment variable identifier must be unique. You must ensure that an identifier in an event pattern does not conflict with an identifier in the parameters section or inputs section.
If a pattern specifies a wait operator, then it must be at the beginning of a pattern, at the end of a pattern, or both. It cannot be in the middle of a pattern. The followed-by operator (->) must be after or before each instance of the wait operator. For example:
wait(1) as w -> (A as a and B as b) // Allowed
{(A as a and B as b) -> wait(1) as w } // Allowed
wait(1) as w1 -> (A as a and B as b) -> wait(1) as w2 // Allowed
wait(1) as w and A as a // Not allowed
A as a -> wait(1) as w -> B as b // Not allowed
A wait operator must specify a positive float value or a time literal. A float value always indicates a number of seconds.
Optionally, specify and, or or -> and then specify an *event\_type* and coassignment variable. Parentheses are allowed in the pattern specification and you can specify multiple operators, each followed by an *event\_type* and coassignment variable. For example, the following is a valid find statement:
find (A as a1 -> ((A as a2)) -> (A as a3) ->
(A as a4 -> A as a5 -> A as a6) ->
(((A as a7) -> A as a8) -> A as a9) -> A as a10 {
print "query with 10: "+a1.toString()+ " - "+a10.toString();
}
Where condition
A find statement can specify a where clause that filters which events match the specified event pattern.
Note:
You can specify a find where clause that applies to the event pattern, and you can also specify a without where clause that is part of a without clause. Any where clauses that you want to apply to the event pattern must precede any within or without clauses.
Specify the where keyword followed by a Boolean expression that refers to the events you are interested in. The Boolean expression must evaluate to true for the events to match.
The where clause is optional. You can specify zero, one or more where clauses.
Coassignment variables specified in the find or select statements are in scope in a find where clause. Also available in a find where clause are any parameter values and key values.
Example:
find Withdrawal as w1 -> Withdrawal as w2
where w2.country != w1.country {
log "Suspicious withdrawal: " + w2.toString() at INFO;
}
Within condition
In a find statement, a within clause sets the time period during which all events in the match set or some events in the match set must have been added to their windows.
A pattern can specify zero, one, or more within clauses. These must appear after any find where clauses and before any without clauses.
Specify the within keyword followed by a float expression or a time literal, which indicates the time period during which the events in the match set must be received.
Optionally, specify a between clause to indicate that the time constraint applies to only some of the events in the match set. See Between clause.
Example:
find LoggedIn as lc -> OneTimePass as otp
where lc.user = otp.user
within 30.0 {
emit AccessGranted(lc.user);
}
Without condition
In a find statement, a without clause specifies an event type whose presence prevents a match.
Specify the without keyword followed by an event type coassigned to an identifier.
An event type that you specify in a without clause must be specified in the inputs block of the query. A pattern can specify zero, one, or more without clauses.
Optionally, after each without clause, you can specify one where clause, which is referred to as a without where clause to distinguish it from a find where clause. When a where clause is part of a without clause:
The Boolean expression must evaluate to true for the presence of the specified event to prevent a match. In other words, when the Boolean expression evaluates to false, then there can be a match even when the specified event is in the window.
The where clause applies to the event specified in its without clause.
The Boolean expression can refer to parameters, coassignment identifiers in the event pattern, and the coassignment identifier in the without clause.
A without clause cannot use the -> or and pattern operators. However, you can specify multiple without clauses. If there are multiple without clauses each one can refer to only its own coassignment and not coassignments in other without clauses. However, all without clauses can make use of the pattern’s standard coassignments.
If there are multiple without clauses, a matching event for any one of them prevents a pattern match. Multiple without clauses can use the same type and the same coassignment, which is useful only when their where conditions are different.
Typically, a without where clause references the event in its without clause, but this is not a requirement.
Optionally, after each without clause, you can specify a between clause, which lists two or more coassigned events or wait operators. For an event to cause a match, the type specified in the without clause cannot be added to the window between the points specified in the between clause. See Between clause.
Any without clauses must be after any find where clauses and within clauses. If you specify both optional clauses, the without where clause must be before the between clause.
Example:
find OuterDoorOpened as od -> InnerDoorOpened as id
where od.user = id.user
without SecurityCodeEntered as sce where od.user = sce.user {
emit Alert("Intruder "+id.user);
}
Between clause
In a within clause and in a without clause, an optional between clause restricts which part of the pattern the within or without clause applies to.
Specify the between keyword followed by two or more identifiers that are specified in the event pattern. Enclose the identifiers in parentheses.
The identifiers set a period of time that starts when one of the specified events is received and ends when one of the other specified events is received. The range is exclusive. That is, the range applies only after the first event is received and before the last event is received.
A between clause is the only place in which you can specify a coassignment identifier that was assigned in a wait clause. You cannot specify identifiers used in a without clause. Also, the same event cannot match both the coassignment identifier in the without clause and an identifier in a between clause.
The condition that the between clause is part of must occur in the range of identifiers specified in the between clause.
It is illegal to have two within clauses with identical between ranges. This would be redundant, as only the shortest within duration would have any effect. It is, however, legal to have more than one without clause with the same between range. Typically, these would refer to different event types or where conditions.
Example:
find A as a -> B as b -> (C as c and D as d)
within 10.0 between (a b)
within 10.0 between (c d)
A find statement that specifies the every keyword can specify a select clause to calculate an aggregate value in order to find data based on many sets of events.
Specify the select keyword followed by a projection expression coassigned to an identifier. The projection expression contains aggregate function(s) that operate on one or more input events. See Built-in aggregate functions as well as Custom aggregates.
The projection expression can use coassignments from the pattern if the coassignments are within a single aggregate function call. For example, the following pattern computes the average value of the x member of event type A in the query’s input and coassigns that average value to aax.
find every A as a select avg(a.x) as aax
A select clause can use parameter and key values.
In an aggregating find statement, only the projection expression can use the coassignments from the pattern. The procedural block of code can use projection coassignments and any parameters or key values, but it cannot use coassignments from the pattern.
In find statements without the every modifier, only the most recent set of events that match the pattern are used to invoke the procedural code block. With the every modifier, every set of events that matches the pattern is available for use by the aggregate function, provided that the latest event is present in one of the sets of events. Any events or combinations of events that do not match the pattern or do not match the where clause, or are invalidated due to a within or without clause, are ignored; their values are not used in the aggregate calculation.
Examples:
find every ATMWithdrawal as w
select last(w.transactionId) as tid
having last(w.amount) > THRESHOLD * avg(w.amount) {
route SuspiciousTransaction(tid);
}
find every A as a -> B as b
where b.x >= 2
select avg(a.x + b.x) as aabx {
print aabx.toString();
}
A find statement that specifies the every keyword can specify a having clause to restrict when procedural code is invoked.
Specify the having keyword followed by a Boolean projection expression. The Boolean projection expression refers to an aggregate calculation. Procedural code is executed only when the Boolean projection expression evaluates to true.
You can specify zero, one, or more having clauses. When you specify more than one having clause, it is equivalent to specifying the and operator. That is, each Boolean projection expression must evaluate to true for the procedural code to be executed.
A having clause can refer to an aggregate value by using the select coassignment name.
When you want to test for an aggregate condition but you do not want to use the aggregate value, you can specify a having clause without specifying a select clause.
Examples:
find every A as a
select avg(a.x) as aax
having aax > 10.0 {
print aax.toString();
}
find every A as a
having avg(a.x) > 10.0 {
print "Average value is greater than ten!";
}
