Release Date: Monday, February 27, 2017

Release Highlights

  • Adds support for property based DDL Configuration ... to allow external configuration formats to be used more easily.
  • Various Configuration Improvements and Knobs:
    • Ability to configure xvm stats via DDL
    • Ability to disable support related metadata fields from being replicated to reduce overhead for low latency apps
    • Support for setting channel filters in DDL.
  • Performance improvements - reduced buffer copies.
  • Support for running ADM maven plugin goals from the platform maven plugin (useful for users of the maven bom as the nvx-platform-maven-plugin follows the nvx-platform version)
  • Reduced memory footprint for State Replication applications by employing more compact metadata representation and late allocation of fields that are not used.
  • Solace Binding Improvements
    • Option not to attempt to create a queue on behalf of the session.
    • Support for querying the list of subscriptions for a solace session (JSCMP binding only ... CCSMP support to be added in a following release).
    • This release introduced javadoc for the solace binding making it easier to understands solace specific configuration properties. See Solace Binding Javadoc
  • Bug Fixes

Compatibility Notes

ADM Compatibility Level: 3.4.0 (see ADM Compatibility Matrix) The Adm Compatibility level last changed in Core X  3.4.373 release

(warning) The following issues have compatibility implications:

Change Log

New Feature

XPLATFORM-584 - Enhance packet recovery log to use memory mapped IO for flush operations

Prior to this release, the packet recovery log used RandomAccessFile to perform file reads and writs. This story enhances the recovery log to use memory mapped IO instead for flush operations improved write latencies and performance. The recovery log continues to use the RandomAccessFile for read operations

XPLATFORM-1343 - DDL should allow setting a channel key filter for an app's bus.

This release update DDL xml to allow configuring a channel filter for an apps channels:

<apps>
  <app mainClass="com.acme.MyApp" name="MyApp.1" enabled="true">
    <messaging>
      <buses>
        <bus name="MyApp.1" enabled="true">
          <channels>
            <channel name="marketaccess-NewOrderSingle" join="false">
              <filter>Region=EMEA|US</filter>
           </channel>
         </channels>
       <bus>
     </buses>
   </messaging>
<app>
</apps>

XPLATFORM-1382 - Allow Solace Binding not to try to create the configured queue if it doesn't exist

This release introduces the ability to suppress automatic provisioning of a queue for solace message bus bindings when the 'queue_name' property is set for a solace binding by setting the property 'provision_queue=false'.

See: SolaceBindingProperties.PROVISION_QUEUE_PROPNAME

XPLATFORM-1541 - Create nvx-platform-maven-plugin that provides goals of all individual plugins

This release introduces an nvx-platform-maven-plugin that aggregates the goals from nvx-core-maven-plugin, nvx-robin-maven-plugin and nvx-eagle-maven-plugin. This is particularly useful for applications that use the nvx-platform-bom to provide versions of various platform components because maven boms don't cover plugin versions. With the nvx-platform-maven-plugin, the plugin version will remain the same as the platform-bom which reduces the number of versions that need to be update in the pom.

The nvx-platform-maven-plugin uses the convention of prefixing the goals with individual plugins' short name to avoid conflicts in the names of goals. The individidual plugins have been correspondingly updated to support these additional goal names as well:

Plugin

Individual Plugin Goal

Platform Plugin Goal

nvx-core-maven-plugin

generate, adm-generate

adm-generate

nvx-core-maven-plugin

generateTest, adm-generateTest

adm-generateTest

nvx-eagle-maven-plugin

generate, eagle-generate

eagle-generate

nvx-eagle-maven-plugin

generateTest, eagle-generate

eagle-generateTest

nvx-robin-maven-plugin

xar, robin-xar

robin-xar

Usage

<!-- Generates messages from xml based message models -->
<plugin>
    <groupId>com.neeve</groupId>
    <artifactId>nvx-platform-maven-plugin</artifactId>
    <version>${nvx.platform.version}</version>
    <executions>
        <execution>
            <id>Generate-Messages</id>
            <goals>
                <goal>adm-generate</goal>
            </goals>
            <configuration>
                <modelFile>${basedir}/src/models/messages.xml</modelFile>
                <encodingType>Xbuf</encodingType>
            </configuration>
        </execution>
    </executions>
</plugin>

XPLATFORM-1564 - Provide support for property based DDL configuration

This release allow representing a DDL xml document as a set of properties. This change makes it easier for external configuration mechanism to map into platform configuration. For example, it is now trivial to use HOCON to configure as it maps very easily to properties.

Applications that will use external configuration and bypass XML DDL configuration entirely may do so, by calling VMConfigurer.configure(properties)

See Also:

  • DDL Config Reference
  • VMConfigurer JavaDoc
  • DDL schema
    Compatibility Notes:

     

    Note this changes the defaul XML DDL config prefix from 'nv.ddl' to a more succinct prefix of
    'x.'. To continue using 'nv.ddl' as an override prefix '-Dnv.ddl.prefix=nv.ddl' must be specified prior to launching an application.

XPLATFORM-1569 - Enable server heartbeat and stats config in DDL

 
<!--
  The 'enabled' attribute replaces 'nv.server.stats.enabled' env property.

  The 'interval' attribute replaces 'nv.server.stats.interval' env property and
  accepts the value in seconds rather than milliseconds.   
    -->
<heartbeats enabled="true" interval="5">
  <!-- 'collectSeriesStats' element replaces 
         'nv.server.stats.includeSeriesStats' env property
    -->
  <collectSeriesStats>false</collectSeriesStats>

  <!-- 'collectSeriesDatapoints' element replaces 
         'nv.server.stats.includeSeriesDataPoints' env property
     -->
  <collectSeriesDatapoints>true</collectSeriesDatapoints>

  <!-- 'collectSeriesDatapoints' element replaces 
         'nv.server.stats.pool.enable' env property
     -->
  <collectPoolStats>false</collectPoolStats>

  <!-- 'maxTrackableSeriesValue' element replaces 
         'nv.server.stats.pool.maxTrackableSeriesValue' env property
     -->
  <maxTrackableSeriesValue>40</maxTrackableSeriesValue>

  <!-- 'poolDepletionThreshold' element replaces 
         'nv.server.stats.pool.poolDepletionThreshold' env property
     -->
  <poolDepletionThreshold>.5</poolDepletionThreshold>

  <!-- new feature ... allows printing of heartbeat messages
         in same format as stats dump tool. 
     -->
  <tracing enabled = "false">
    <traceAppStats>true</traceAppStats>
    <tracePoolStats>true</tracePoolStats>
    <traceSysStats>true</traceSysStats>
    <traceUserStats>true</traceUserStats>
  </tracing>

  <!-- 'logging ' element moved from 'heartbeatLogging'
         under server
     -->
  <logging enabled="false">
    <flushOnCommit>true</flushOnCommit>
    <autoFlushSize>4096</autoFlushSize>
    <autoRepair>true</autoRepair>
    <storeRoot>temp</storeRoot>
    <initialLogLength>1.0</initialLogLength>
    <zeroOutInitial>true</zeroOutInitial>
    <pageSize>4096</pageSize>
    <detachedWrite enabled="false">
        <queueDepth>512</queueDepth>
        <queueOfferStrategy>SingleThreaded</queueOfferStrategy>
        <queueWaitStrategy>Yielding</queueWaitStrategy>
        <queueDrainerCpuAffinityMask>[8]</queueDrainerCpuAffinityMask>
    </detachedWrite>
  </logging>
</heartbeats>

Compatibility Notes:

 

This change moved the heartbeatLogging element in ddl so that it is now specified under a server's heartbeat element:

<server name = "foo" >
  <heartbeatLogging enabled = "true" />
</server>
<server name = "foo">
  <heartbeats enabled = "true" />
    <logging enabled = "true" />
  </heartbeats>
</server>

The DDL XML parser will still parse the old heartbeatLogging element, but will generate a warning.

This change also introduces new configuration properties for enabling server stats and configuration heartbeat emission so that these properties no longer need to be set in the environment.

Improvement

XPLATFORM-923 - Reduce Memory Footprint of Rog Node metadata.

 

This release significantly reduces the memory footprint of associated with metadata stored in entities held in a state graph. The reduction in memory usage is approximately 800 bytes per entity. This change brings memory footprint associate with state entities to approximately 140 bytes greater than it was prior to the release of the 1.8 version of the platform.

XPLATFORM-1425 - Remove warnings in generated code

This release improve the ADM code generators import statement generations to reduce unused imports which results in fewer warnings in eclipse.

XPLATFORM-1531 - Store Member should display its advertised connector and discovery settings on startup

When config trace is enabled for nv.ods, a store configuration for replication will now trace its discovery provider and connection information when it advertises itself.

XPLATFORM-1533 - Stamp Post Wire Send Timestamp and report Wire Send latency in bus binding stats

This release adds a new message bus stat to capture the time spent in a message bus provider's send call when sending messages.

Prior to this change, the message latency stats stopped at the point just before a message is written to the wire. This was because message latency stats were designed to cover the portion of time spent in the platform and provider specific send time was considered to be external. However, in the case that a provider blocks in flow control or is otherwise slow, the provider send time may have an impact on the sending thread's throughput making the time spent writing to the wire of interest from a platform statistic standpoint. This story implements capturing the post wire send timestamp and reports the delta between preWireTs and this timestamp as wire send time. This new latency stat is reported in engine stats output with other bus stats as 'ws' for wire send times and in hearbeats as WireSendLatencies.

Sample of printed output

...[ ws] [sample=65536, min=7 max=54 mean=10 median=11 75%ile=11 90%ile=13 99%ile=19 99.9%ile=23 99.99%ile=26]
......NewOrder...[ ws] [sample=0, min=-1 max=-1 mean=-1 median=-1 75%ile=-1 90%ile=-1 99%ile=-1 99.9%ile=-1 99.99%ile=-1]
......OrderEvent...[ ws] [sample=65536, min=7 max=24 mean=8 median=8 75%ile=9 90%ile=12 99%ile=15 99.9%ile=19 99.99%ile=22]

These latencies are also emitted in heartbeats.

See Also:

XPLATFORM-1536 - Channel Key, Filter, QoS, Id and join status should be displayed in engine config trace.

This release improves the trace of an engine's config to include channel details when engine config trace is enabled.

<1,24578,perf4> 20161213-01:47:01:495 (cfg)...
Engine 'market' Configuration {
...state_factories=[]
...message_factories=[]
...channels
......xsim-morders@market [join=true]
.........Id     : 1
.........Key    : orders/${Region}
.........Filter : null
.........QoS    : Guaranteed
......xsim-mconfirm@market [join=false]
.........Id     : 2
.........Key    : null
.........Filter : null
.........QoS    : Guaranteed

XPLATFORM-1537 - Emit Inbound and Outbound Message Logger Statistics

This release adds inbound and outbound message logger stats to heartbeats and stats trace:

[Inbound Message Logger]
<No Inbound Message Logger>

[Outbound Message Logger]
Put {0(0 0)} Updates {0(0 0)} Removes {0(0 0)} Sends {979,561(0 11,000)} Commits {534,306(0 6,000)} Forks {0(0 0)}
Disruptor (SingleThreaded, BusySpin) {[0 of 1,024] 0%}
...[o2p] [sample=65536, min=0 max=93 mean=26 median=26 75%ile=45 90%ile=51 99%ile=57 99.9%ile=64 99.99%ile=78]
Writer {qsize=0}

These stats are emitted in server heartbeats as part of engine stats.

See Also:

XPLATFORM-1538 - Enhance StatsDumpTool to allow filtering of admin app stats.

This release changes the StatsDumpTool to filter admin app stats by default when generating stats trace from hearbeat logs. Admin app stats can still be done by passing the '-a' flag to the tool.

XPLATFORM-1539 - Change StatsDumpTool to accept start and end time using query format.

This release enhances the StatsDumpTool to allow accept the start and end time options in the same time formats accepted by transaction log tool queries. For example, one can now launch the tool with a time range such as:

java -cp "libs/*" com.neeve.tools.StatsDumpTool -s "30 minutes ago" -e "20 minutes ago" rdat/server-hearbeats/myapp-heartbeats.log

java -cp "libs/*" com.neeve.tools.StatsDumpTool -s 2:00pm -e 2:05pm rdat/server-hearbeats/myapp-heartbeats.log

It is still possible to specify these arguments in the prior timestamp format such as:

java -cp "libs/*" com.neeve.tools.StatsDumpTool -s "20161216-02:06:00" rdat/server-hearbeats/myapp-heartbeats.log

XPLATFORM-1540 - Emit 'o2p' (OfferToPoll) stats in heartbeart for persister and ICR sender

These release now populates the o2p stats in store persister and icr sender stats emitted in heartbeats on the SrvMonDisruptorStats entity.

XPLATFORM-1545 - Add disk usage stats to server heartbeats.

This release adds disk usage stats collection to server heartbeats which can be used to monitor low disk space issues.

See: ISrvMonSysStats.html.getDiskStats()

XPLATFORM-1547 - Enable direct serialization of object metadata into ODS Subheader buffer

This release removes an intermediate copy that was incurred copying an object's metadata into the Store replication subheaders.

XPLATFORM-1552 - Add SMA bus configuration flag to control whether sequence number related metadata is set by an SMA bus or not on inbound messages

This release adds a new SMA bus configuration flag (set_sno_on_receipt specified on bus descriptor) to control whether sequence number related metadata is set by an SMA bus on inbound messages

XPLATFORM-1553 - Add SMA bus configuration flag to control whether flow related metadata is set by an SMA bus or not on inbound messages

This release adds a new SMA bus configuration flag (set_flow_on_receipt specified on bus descriptor) to control whether flow related metadata is set by an SMA bus on inbound messages

XPLATFORM-1554 - Add AEP engine configuration flag to control whether support related ROG metadata is set by an AEP engine or not

This release adds an AEP engine configuration flag (setSupportMetadata) to control whether support related ROG metadata is set by an AEP engine on inbound/outbound messages

XPLATFORM-1555 - Add AEP engine configuration flag to control whether sequence number related ROG metadata is set by an AEP engine on outbound messages

This release adds an AEP engine configuration flag (setOutboundSequenceNumbers) to control whether sequence number related ROG metadata is set by an AEP engine on outbound messages

XPLATFORM-1562 - Create Goal Aliases for ADM Maven MOJOs

Added aliases for ADM maven plugin goals: "generate" same as "adm-generate"; "generateTest" same as "adm-generateTest"

XPLATFORM-1563 - Support generating code for multiple models from maven plugins

Maven plugin supports new parameter called 'modelFiles' which enables code generation for multiple models per single execution. Either 'modelFile' or 'modelFiles' can be used but not both. When 'modelFiles' is used, 'namespace' parameter is not applicable and will break the build.

XPLATFORM-1567 - Add model validation to prevent duplicate names of model elements

This release improves ADM model validation to fail fast if the same name is used in a model for multiple entities, multiple messages or combination of them. With this release an error will be reported when unless -Dnv.adm.suppressduplicatechecks=true is set.
Compatibility Notes:

 

This release introduces stricter code generation checks for duplicate type names in ADM models.

XPLATFORM-1578 - Javadoc solace binding properties.

 

This release provides javadoc for the solace binding and its properties.

See http://build.neeveresearch.com/solace/javadoc/3.5

XPLATFORM-1579 - Update DDL to allow adding displayName on ddl elements

This release adds the ability to specify a displayName attribute in DDL for buses, apps and servers. The displayName can be used by tools to provide a more human readable name for displaying these elements.

Change

XPLATFORM-1454 - Disallow using same transaction log for inbound and outbound message loggers

This release makes the following changes with respect to inbound and outbound message logging policies.

  • Removed UseInbound and UserSupplied OutboundMessageLogging policies
  • Removed Default and UserSupplied InboundMessageLogging policy
  • Make Off the default policy for both Inbound and Outbound message logging policies
  • Remove constraint that a separate inbound message logger cannot be configured for Event Sourcing

In essence, this release allows for inbound logging, outbound logging and application HA to be configured independently of each other.
Compatibility Notes:

 

With this release Default is no longer a valid policy for Inbound or Outbound message logging. Configuration that uses Default as a value must be updated to specify 'Off' instead before moving to this release.

XPLATFORM-1561 - Change DDL app message bus configuration schema to wrap bus lists in a <buses> element

 

To make is easier to specify application bus configuration via DDL properties this release changes the DDL xml schema to define bus configuration under a buses element instead of directly under the messaging element:

Consequently applications' DDL xml should be updated from

<apps>
  <app name="MyApp">
    <messaging>
      <bus name="bus1">
      <bus name="bus2">
    </messaging>  
  </app
</apps>

to

<apps>
  <app>
    <messaging>
      <buses>
        <bus name="bus1">
        <bus name="bus2">
      </buses>  
    </messaging>  
  </app
</apps>  

This change is a recommended change ... the old schema is still supported, but is now deprecated and may be removed in a future release.
Compatibility Notes:

 

To support properties based DDL in a consistent fashion the DDL schema has been updated such that an applications configured buses are now listed under a buses element. While the old configuration schema is still valid applications are advised to update to the new format.

XPLATFORM-1577 - Upgrade Solace CCSMP to 7.2.4.8 and JCSMP to 7.2.2.100 to allow better support for authentication schemes.

 

This release upgrades the dependent solace binaries as follows:

  • For JNI binding: CCSMP libraries were upgraded from 7.1.2.137 to 7.2.4.8
  • For Java, sol-jcsmp and sol-common from 7.1.2.248 to 7.2.2.100.

The motivation for the upgrade was to allow better support for authentication schemes provided by solace in the newer binaries.

|

Bug

XPLATFORM-1525 - nv.aep.trace, nv.sma.trace and nv.ods.trace does not take effect in the env section of a DDL

Prior to this release, the nv.aep.trace, nv.sma.trace and nv.ods.trace configuration settings to control trace level for AEP, SMA and ODS modules respectively did not take effect when set in the <env> section of a DDL. This release fixes this issue

XPLATFORM-1532 - StatsDumpTool reports the store's commit latencies instead of cqs in stats dump

This release fixes the the StatsDumpTool to correctly display the stores Commit Queue Size (cqs) stats.
Compatibility Notes:

 

This release fixes the the StatsDumpTool to correctly display the store's Commit Queue Size (cqs) stats. Prior to this fix this stat was erroneously displaying the values for a store commit latencies

XPLATFORM-1534 - It is not possible to specify channel key variables in DDL because they get replaced during DDL substitution.

 

This release fixes an issue the DDL xml parser to avoid doing environment based substitution on channel key variables. Prior to this fix it was not possible to use variable keys with with default values.

XPLATFORM-1535 - Increased serialization costs on engine thread for outbound logging.

This release fixes a performance regression that can occur in detached outbound message logging. The regression was caused by the movement of some of the serialization logic for outbound message metadata to be done by the aep engine's thread along with the dispose call for the message being logged. The additional logic in the aep engine's thread can cause reduced throughput for applications that have heavy outbound message loads and are close to be cpu bound on the business logic thread.

XPLATFORM-1559 - preProcessingTs is not persisted or replicated when replicateInParallel is true

This release fixes an issue with parallel cluster replication in which the pre processing timestamp was not being replicated to the backup. This was problematic because the pre processing timestamp is used to return the engine time during recovery and and on the backup to prevent divergence.

XPLATFORM-1568 - Host field not properly set on ESA packets

This release fixes a bug in discovery packets in which the host name was not getting set. The host name is used for informational and support purposes by the platform and tools so this issue was a minor one.

XPLATFORM-1574 - Talon distribution xconfig scripts fail with "X license verification failed [Heap limit exceeded" with free license

This release sets the '-Xmx' variable in a talon distributions configure.sh script allow the distribution's environment to be safely configured when using a free X license which imposes maximum heap sizes.

XPLATFORM-1575 - EXbufFieldCorruptException: deserialization error: corrupt data stream [unsupported field wire type code 4

This release fixes a wire corruption issue in Xbuf encoded entities that can occur with array fields with the following sequence of events:

  1. Create an Xbuf generated Message (or entity).
  2. Add a value to an array field on the message.
  3. Serialize the Message
  4. Add 1024 more values to the message
  5. Serialize the message to bytes
  6. Deserialize the message from bytes.

This fix address a problem in step 4: if enough values are added to the array field that it changes the size of the value needed to encode the length of the array, corruption will occur.

This issue is most likely to impact State Replication applications. For this to impact an application, the application would have to:

  • Modify an array field on a message after it is serialized. Note that it is not legal for an application to modify a message after it is sent.
  • Modify an embedded entity's array fields after it is set on a message that is serialized (also not legal).
  • Modify an entity or embedded entity in a StateReplication app state entity between transactions with the entity size modified significantly enough to trigger this bug.

XPLATFORM-1576 - Deserialization of an Xbuf message with String arrays can require reserialization.

This release fixes an issue with Xbuf repeated string fields that could cause a message to be reserialized if written after being deserialized which could slow down Event Sourcing applications that receive messages with String array fields.

XPLATFORM-1580 - NPE in Solace binding during close after messaging start failure.

Fixes an issue in which a null pointer exception can be thrown during close() on a solace message bus binding when an exception is thrown starting messaging (for example, as a result of a subscription validator throwing an exception).

XPLATFORM-1587 - Solace reconnect attempt doesn't use all properties of original connect attempt

When an attempt is made to open a solace bus binding, properties on the bus descriptor are normalized to remove solace agnostic properties and to rationalize between jcsmp and ccsmp binding format. Because this normalization process modifies the properties supplied in the bus descriptor, a second attempt using the same bus descriptor will have properties missing. This release fixes this issue.

XPLATFORM-1589 - NullPointerException resolving unqualified type import

This release fixes a NullPointerException that can occur during ADM code generation when resolving an imported type from another model that is not fully qualified.

java.lang.NullPointerException
        at com.neeve.adm.AdmModel.lookupType(AdmModel.java:2306)
        at com.neeve.adm.AdmXMLParser.createField(AdmXMLParser.java:1265)
        at com.neeve.adm.AdmXMLParser.getFieldRef(AdmXMLParser.java:1428)

XPLATFORM-1590 - XLinkedHashMap can erroneously return null retrieving an existing entry after another entry is removed

This release fixes a bug in the XLinkedHashMap implementations that could result in null be returned when getting a value from the map after another element was removed.