Release Highlights

• Many improvements to command line tools' usability.
• Transaction Log Tool Query Improvements: LIKE, REGEX_LIKE, GROUP BY and Aggregation
• Code generation enhancements for reduced latency field access (new UUID data type, timestamp accessors for dates, raw getters for Strings)
• Reduced latency in dynamic topic construction.
• Bug Fixes

New Feature

• [XPLATFORM-391] - Implement Quiescence Activity Tracker
  This release implemented the Loopback bus quiescence activity tracker. Quiescence tracking, enabled via the nv.sma.loopback.activitytracker.enabled configuration parameter, records relevant information each time the quiescence activity counter is incremented or decremented. The collected information can be dumped to the bus' logger on demand via LoopbakBus.dumpActivityTrackingInfo() and each time waitForQuiescence() returns false


• [XPLATFORM-199] - Add ability for Command line tool to import a command script
  The transaction log tool was updated allow it to take a -s [scriptFile] argument from the command to run a script. Command line tools in general have been enhanced to allow the user to enter the command`script ` to run a script from within the tool.


• [XPLATFORM-282] - Support for aggregation functions in Transaction Log Tool
  Support for the aggregate functions, COUNT, MAX, MIN, SUM and AVERAGE were added to the transaction log tools query engine.


• [XPLATFORM-283] - Add support for GROUP BY in Transaction Log Queries
  Support from GROUP BY was added to transaction log tool querying and can be used with the newly introduced aggregate functions. At this time support for HAVING has not yet been implemented.


• [XPLATFORM-327] - Support for regular expressions in transaction log queries
  Support for standard SQL LIKE and more powerful REGEX_LIKE were added to Transaction Log Tool set of query operators. See the transaction log query documentation for more details.


• [XPLATFORM-333] - Support command history in Transaction Log Tool
  Command line tools such as the Transaction Log Tool now support persistent command history. History can be navigated from the tool prompt using the up and down arrows, and a new history command was introduced for displaying and manipulating command history.


• [XPLATFORM-341] - Implement persistent property support for interactive command line tools
  Interactive Tools were updated to allow persistent configuration properties. Properties files are stored in ${user.home}/{tool-name}/config.props. Tools such as the Transaction Log Tool now allow users to set properties via:
  
  set [propName] [value]
  or
  [propName]=[value]
  
  Users may also issue 'set' with no argument to display the current configuration property values


• [XPLATFORM-354] - Ability to provide message key to SMA runtime by binding on message receipt
  Prior to this release, there was no mechanism by which an SMA binding can provide a message key to the SMA runtime to set in a message post deserialization. This story enhances the SMA runtime to overcome this limitation.


• [XPLATFORM-363] - Add timestamp based date setter/getter methods to ADM generated code
  The following additional methods were added for Date fields to JSON, Proto and Xbuf generated messages and entities

  • long getXXXAsTimestamp()
  • void setXXXAsTimestamp(long val)

  
  These methods can be interchangeably used with the incumbent getXXX() and setXXX() methods


• [XPLATFORM-372] - Add raw string setter/getter methods to ADM generated code
  The following additional methods were added for String fields to JSON, Proto and Xbuf generated messages and entities

  • Object getXXXAsRaw()
  • void setXXXAsRaw(Object val)

  These methods can be interchangeably used with the incumbent getXXX() and setXXX() methods


• [XPLATFORM-375] - Add raw fixed length string setter/getter methods to ADM generated code
  The following additional methods were added for Fixed Length String fields to JSON, Proto and Xbuf generated messages and entities

  • ByteBuffer getXXXAsRaw()
  • void setXXXAsRaw(ByteBuffer val)

  
  These methods can be interchangeably used with the incumbent getXXX() and setXXX() methods


• [XPLATFORM-381] - Implement Detached Pool Item Washer


• [XPLATFORM-387] - Add Stuck Alert Event for stuck engine detection
  This release introduces an additional event - AepStuckAlertEvent - which is dispatched when the engine determines that the event multiplexer thread may be "stuck". The "stuck" determination is performed by checking the transaction pipelines size i.e. the pipeline is considered to be stuck if greater than zero and has not changed for a configurable period of time. An application configure the StuckAlertEventThreshold in the engine's desciptor and can register itself as an IAepAsynchronousEventHandler with the engine to receive alerts. As checks for stuck engines are driven off of the engine's statistics collection thread the engine's stat's interval must be configured to perform the checks.

Improvement

• [XPLATFORM-270] - Enhanced Command Line Tool Key Binding Support
  Command line tool key binding support was improved to provide a bash like experience. Tools like the Transaction Log Tool now have better support for backspace, home, end and a host of other familiar bash keystrokes.


• [XPLATFORM-298] - Transaction log tool queries should be able to use logName instead of x_repository_name
  TransactionLogTool was enhanced to support selection on 'logName'.


• [XPLATFORM-304] - Enhance ADM to bundle google protobuf idl
  The ADM Code Generator has been enhanced to bundle the google protobuf IDL (google/protobuf/descriptor.proto). In previous versions it was necessary for users to explicitly set the path where this file could be found, and this is no longer necessary, instead ADM will extract its bundled version to the user's home folder under .nvx//adm/protoc.


• [XPLATFORM-305] - Enhanced ADM to output generated protobuf idl in a fully qualified fashion to the generated source output folder
  The ADM Code generator has been enhanced to automatically output generated protobuf idl (.proto files) to the generated source output folder using a namespace qualified path. The outputed IDL is intended to be bundled with the generated classes so that it can be located by other models that attempt to import the it. Build tools should be configured to copy the .proto files to the classes output directory and the .proto should be bundled in the resulting jar
  
  Consequently the AdmModel.setProtobuIdlOutputDir method has been deprecated. If specified the generated idl will be copied to the specified (in addition to fully qualified output folder)


• [XPLATFORM-311] - Allow configuration of the Native Store Binding's listener port
  It is now possible to specify the listening port in the native store binding by specifying localPort in the provider config of the store. This is useful in cases where there are firewall restrictions or it otherwise desirable to set the listener port. If not specified a free port will be chosen.


• [XPLATFORM-316] - When double values are queried in the log tool they should not be rendered in scientific notation
  Tabular rendering of float and double fields in Transaction Log Tool Queries was improved to avoid scientific notation when their value can be represented in few enough characters to fit in the table. Prior to this fix 25.0 would have been represented something like: 2.50000+1


• [XPLATFORM-325] - Add StorePacketJournaller stats
  Store Binding Stats were enhanced to include disruptor stats for the store's persister.


• [XPLATFORM-329] - New option in Transaction Log Tool to display command execution time
  A new configuration option was added to command line tools that will print out the time of long running commands. By default any command that takes longer than 5 seconds will have its execution time printed to the console. The property can be changed with the command:
  
  set longCommandDisplayThreshold


• [XPLATFORM-330] - Enhance Transaction Log Tool tail command to limit the number of entries displayed
  The Transaction Log Tool's tail command was enhanced only to show the last 10 entries by default and a -n option was added to allow the user to change the number of trailing entries to display.


• [XPLATFORM-335] - Transaction Log Tool should display first 5 results by default
  The Transaction Log Tool was enhanced to allow users to see a result preview after executing a query. To enable this functionality issue the command:
  
  set selectPreviewCount 5
  
  This will cause the tool to show the first 5 results in tabular format. After displaying the results the result set is rewound, and the results can be navigated with the 'next' or 'dump' commands.


• [XPLATFORM-337] - Change persister and ICR sender journalling thread names
  Prior to this release, the ODS store persister and ICR sender journalling threads are given the same name - X-ODS-PacketJournaller-. This release assigns names to the journalling thread specific whether it is a persister or ICR sender thread. The new names are as follows:
  
  Persister: X-ODS-Store-Persister-{persisterName}-{serialNo}
  ICR Sender: X-ODS-Store-ICR-Sender-{storeName}-{serialNo}
  ICR Receiver: X-ODS-Store-ICR-Receiver-{storeName}-{serialNo}


• [XPLATFORM-342] - Enhance Transaction Log Tool list types command to accept a filter
  The transaction log tool now 'list' command now accept a case sensitve filter at the end against which the output line is matched.


• [XPLATFORM-344] - Create engine store and bus managers when engine is created
  Prior to this release, an AEP engine store binding was created and started when the engine was started. Similarly, an engine's bus bindings were both created and opened when the engine was activated i.e. when it becomes primary. The enhancements made for centralized statistics gathering and dissemination in this release require access to bus managers and store before the engine is started. Furthermore, having bus manager lists available right after the engine has been created and immutable thereafter eliminates race conditions and therefore associated locks requires for consistent access to the lists across engine states. In this release, we change the creation and start of store bindings and bus managers to behave as follows:

  • Create the store and bus managers when an engine is created
  • Start the store when an engine is started and start the bus managers when the engine is activated
  • Add a new AepEngine.getBusManagers() method
  • Remove state checks in getStore() and getBusManagers() method to allow for the store binding and bus managers to be fetched at any point during an engine's lifecycle


• [XPLATFORM-365] - A query with only unqualified fields returns all null columns for entries whose class doesn't match any
  Transaction Log Query was updated to perform additional matching on queries that use all type unqualified fields. Prior to this change, all rows were returned even if they didn't have a field path matching the fields in the query. This change performs further matching on rows to ensure that the row matches at least one of the paths in the query.


• [XPLATFORM-369] - Transaction Log Tool should provide an option to echo commands and output from script files
  Interactive Tools now echo output from script files by default. scripts may change echoing with the echo on|off command, and can also control ansi output with ansi on|off (off by default)


• [XPLATFORM-371] - Channel Key Resolution and Message Reflector Performance Improvements
  This addresses to performance issues identified with channel key resolution via message reflectors.
  

  • The message reflector api was enhanced to allow the field path to be passed in as an array of the bean path components. Prior to this change a reflector would have to split the path on every invocation. With this fix the engine and channel can pre-split the bean path and pass it to the MessageReflector.This change requires code regeneration
  • Protobuf message reflector performance was improved significantly by creating dedicated static field accessors for field values rather than relying on reflection.
  • Support for accessing additional field types including enum, double, boolean and float in the message was added to avoid the need for using a key resolution table for such fields.

Bug

• [XPLATFORM-266] - Dropping an index and restarting the log tool can cause the index drop to be rolled back
  Fixed an issue in which index drops were not being committed and could be 'resurrected' on a subsequent invocation of the tool.


• [XPLATFORM-290] - IllegalArgumentException: Unknown encoding type '-1' during message send
  Resolves an issue affected xbuf generated messages where the message encoding does not get set correctly on messages that are reused within a pool.


• [XPLATFORM-292] - The internal aep engine control bus uses busy spin on the default cpu with nv.optimizefor=latency
  This change fixes a processor affinitization issue for AepEngines configured with nv.optimizefor=latency and cpu affinitization in which the engine's internal control bus writer thread was being configured to use BusySpin and the default cpu affinity mask. With this fix in place the control busy uses blocking dispatch alleviating contention with other threads using the default processor affinity.


• [XPLATFORM-296] - Protobuf code generation fails for Entities with only enum array fields


• [XPLATFORM-297] - Transaction log archive classpath bundle should include manifest classpath entries.
  Fixes an issue in the Transaction Log archive feature in which classpath elements referenced by a Class-Path attribute from jar manifest entries were not getting included in the transaction log archive.


• [XPLATFORM-299] - Potential waitForQuiescence issue in LoopbackBus


• [XPLATFORM-312] - IllegalArgumentException: object not in store after an entity field is nulled.


• [XPLATFORM-313] - AepEngine.getEngineTime() NullPointerException when called from non engine thread
  Fixed a race condition that can occur when a non engine thread calls AepEngine.getEngineTime() on an engine using EventSourcing


• [XPLATFORM-324] - Fix Issue with Out-Of-Order Commit Completions with Multiple Bus Bindings
  Out of order commit completion detection logic was updated to avoid generating erroneous alerts. When multiple buses are in play it is possible to get out of sequence commit completions for example:
  
  - T1 (B1, B2)
  - T2 (B2)
  ->B1 is slow, B2 completes T2, in this case T2 can complete first we'll stop at T1, not an error.
  
  In this case an alert would not be generated if T2 completes first. This fix does not at present include a facility for checking if B1 is hung.
  
  This change does introduce slightly higher overhead in out of sequence commit detection. The feature can be disabled with the following environment variable:
  

  • nv.aep.enableSendCommitCompleteSequenceAlerts=false


• [XPLATFORM-336] - Interactive command line tools don't provide a mechanism to interrupt long running commands without killing the tool.
  Interactive Tools can now be interrupted with ctrl-c on linux. If ctrl-c is entered in the middle of a command execution the user is prompted if they want to interrupt the command. On both Windows and Unix ctrl-c is trapped to prevent exit from the tool. Support for windows command interrupt is planned for a future release.
  
  Note interrupting some commands may have adverse side effects if the command is the middle of an I/O operation, it may end up closing the corresponding stream. This feature's primary concern is to prevent a user from getting stuck in a hung state. For users of the Transaction Log Tool, it is recommended to close and reopen the log if a command interrupt appears to have left the tool in an inconsistent state.


• [XPLATFORM-338] - Types other than the ones specified in a Select clause can returned by Transaction Log Queries
  Fixes an issue when a SELECT statement is executed that selects a concrete field type but uses a non concrete field such as SELECT Customer FROM logs WHERE firstName = 'Bob', result rows were being returned for all log entries.


• [XPLATFORM-343] - Error issuing SELECT without space after comma.


• [XPLATFORM-349] - Jline console is garbled when opening a transaction log archive
  Several minor bugs were fixed in the the Transaction Log Tool's archive command, and the tool now prints addition during archive and extraction phase to alert the user of archive progress.


• [XPLATFORM-370] - Transaction Log Tool should automatically exit when finished with a script specified on command line


• [XPLATFORM-377] - Transaction Log Tool queries are slow when indexingPolicy is set to wait for index builds
  An issue was identified in the transaction log tool which would slow down queries when autoindexing=true and an indexingPolicy!=flush_never is set. With this combination the indexing thread was committing index updates too frequently resulting in slow query performance.