Modeling Basics

The X Platform's Application Data Modeler (ADM) allows modeling of messages and entities in XML and a code generator to generate java code and protobuf idl that can be used by applications. While this document's focus is on message modeling, it is important to consider state modeling as well as they share many of the same modeling constructs. The model allows definition of 3 types of complex or composite types that are comprised of multiple fields: Entities, Embedded Entities and Messages. These will be discussed in a bit more detail below, but at a high level here is how to think about the three:

• Entity: An entity is used to model application state. An entity serves as a node in the platform's Replicated Object Graph (ROG) framework. Entities can reference other entities, which are replicated independently
• Message: A message can be thought of a special type of entity that is enhanced to allow transport over the platform's Simple Message Abstract (SMA) layer. A message can't reference other entities because it needs to be an independently transportable unit.
• Embedded Entity: An embedded entity models a collection of fields of primitive types or other embedded entities. An embedded entity can be embedded into a Message, Entity or other Embedded entity as a field, and because part of that type's serializable unit (e.g. not another node from the standpoint of ROG, and a transported part of a message from the standpoint of SMA. 

As a convenience there is also:

• Inlined Enity: An embedded entity can be inlined into a message such that the message assumes all of that entities fields, but does not contained the inlined entity as a field. 

Types

Primitive/Built-In Types:

The X Platform supports the following built in types which can be used as fields when building composite entity types:

Collections and Arrays

The following built in collection types are supported (where Entity is an X Entity, Message, Enum, primitive or built in type):

Enumerations

The X Platform supports modeling of enumeration types. Enumeration types can be of type int, char or string and according to their type must be assigned a code value that is used to generate their field tag for for protobuf/xbuf encoding:

• For integer coded enums, the value is the same as the (integer) code, 
• For character coded enums, the value is the ASCII value of the (character) code
• For string coded enums, the value is the hash code of the (string) code.

Entities

Composite data types are defined as entities which can contain fields that are primitive, built in, collection, enumerations, entity or message types. Because Messages are a special case of Entities we start by describing Entities here first.

An application's state is represented as an object graph in which entities represent the nodes. The platform replicates the deltas to this object graph as changes are made to the nodes in the graph. When an entity is set as a field of another object it constitutes an addition to the object graph which is then replicated to other peers. Similarly when a field of an entity is removed (set to null) it constitutes a removal which is then replicated to the peers and possibly persisted.

An entity that serves as the root of an object graph can be marked as root which indicates it will not have any parents in the object tree. A root object should not be declared as the child of another object. Application provided state should not be defined as root objects as the server appends them to an underlying root object internally. 

An entity type can be marked as transactional. The platform tracks field updates to transactional entities and can rollback changes made to them in the event of an application exception or an explicit rollback. By default entities and collections are transactional while messages are not. A message can be declared transactional which is useful in the event that a message is stored as part of an application's state graph. 

Embedded Entities

An entity can be declared as embedded. Embedded entities are serialized as part of the entity/message to which it is declared as a field - as part of the same transported unit. Embedded entities can define fields that are primitive or built in types, other embedded entities, or arrays, but may not use collections or non embedded entities. 

Restrictions on the use of entities as fields in other objects

• Entities declared as embedded cannot be defined such that there are cycles in the resultant graph.
• Entities declared as embedded cannot currently be used as members in collections, but may be used as array members. 

Messages

Messages can be thought of as a special type of entity that additionally implement and exposes a MessageView interface allow them to be transported via the platform's Messaging layer. Because messages must be transportable as a single serialized unit, messages can only declare fields that are primitive, built-in, embedded entities or arrays of the previous; non embedded entities and collections aren't supported.

Messages can be used independently from the Messaging Engine and Message Store.

• Messages can be used used with the the platform's SMA (Simple Message Abstraction) layer using its MessageBus and MessageChannel abstractions. See: http://build.neeveresearch.com/core/javadoc/LATEST/com/neeve/sma/package-summary.html
• Or can be used truly independently via external frameworks by utilizing their byte[] or ByteBuffer serializers and/or their generated factory methods. To assist in determining type/factory information the platform provides a MessageMetadata class that can be transported along with the message for type resolution.

XML Based Modeling

The X Platform generates source code for entities and messages for use with the runtime from an xml model specified by the x-adml.xsd schema included at the root of the nvx-adm-<version>.jar and also bundle into the nvx-all.jar. Be sure to update your editor's schema validator to reference it.

Model Element

The root element of a message model is the model element which is used to define a namespace qualified set of modeling elements. To conform with x-adml.xsd schema and pass validation, model must define target XML namespace xmlns="http://www.neeveresearch.com/schema/x-adml". This is not to be confused with namespace attribute described below.

Naming the Model

ADM model has a name property which is used in certain ways during code generation.

Two main uses of model name are:

• Name of .proto file when IDL is generated
• Name of outer java class that wraps model types  when code is generated with Protobuf encoding.

Model name can be derived from the filename, or by defining it explicitly in XML. Name explicitly defined in XML takes precedence over filename.

For the example above, name property will have value of ModelNameGoesHere (it will be converted to Pascal Notation). If we were not to define name attribute, the model name would be derived from filename: sampleModel.xml -> SampleModel.

Documentation and Javadoc

Most model elements can be documented by supplying a child <documentation> element or doc attributes. The documentation added to the model is used to generate javadoc in the generated source code.

XML

Generated Code

Model Element Deprecation

Types in an ADM Model can be marked as deprecated via a deprecation attribute. Such elements are generated with deprecation annotions and javadoc. Adding a child deprecation element optionally allows providing a 'since' attribute and brief message describing the reason for deprecation. 

XML

Generated Code

All methods related to a deprecated field are marked as deprecated, the snippet below shows some examples of this. 

Imports

The import statement allows you to import another model from a different namespace to use with this model. Imported definitions can then be used within the model, though you must provide fully qualified names such as com.example.other.SomeMessage when referencing imported types.

The code generator needs to be able to resolve imported models to correctly generate source code. This is accomplished using an import statement that specifies the imported model location:

See also Imported Model Resolution

Factories

The factories section defines object factories that are used to instantiate the generated object. Each factory in an application must have a unique factory id. The ID is serialized along with object or transported in MessageMetadata and is used to reconstitute its objects during deserialization. A single model can define multiple factories, allowing Messages and Entities to be grouped together as the application sees fit. A factory can contain a maximum of 32767 types so in practice it is rarely a requirement to use multiple factories within a single model. Each type in the model defines the factory to which it belongs via its factoryid attribute. User application may define factories with ids greater than or equal to 1, ids <= 0 are reserved for platform internal use. 

Enumerations

Enumeration types can be modeled as follows and can be declared as fields of entity or message types. Unlike other model elements enumerations aren't tied to a particular factory despite being scoped to the model's name space. Generated java enumeration don't have dependencies on the rest of the platform and can be used standalone. 

XML

Generated Enumerations can specify a type which is accessible in the generated code. The type can be int, char or String. A const should not be removed from an enumeraion, if the newly generated code is expected to deserialize enums that were persisted with an earlier version, in addition the order of constants is important and should not be changed.

Generated Source

For an entity named "MyEntity" an interface and an implementation are generated using the model's namespace. Entities will extend IRogNode or IRogContainerNode marking that they can be used as nodes with the platform's Replicated Object Graph (ROG) framework. 

Fields and Accessors

The ADM model allows fields either to be defined in place (directly on the message or entity that is using the field) or by reference to a field declared in the model's <fields> section. It is a matter of preference which approach an application developer uses, if many messages contain the same field then it may be more convenient to model the fields in a reusable fashion in the <fields> element, but in most cases it is more convenient to define the fields in place. 

XML

Field Attributes

Attribute	Description	Required
type	The type of the element. If the type is defined in this namespace or is a primitive or collection type only the simple name of the type need be used. If the type belongs to another namespace from an imported file, then the fully qualified name should be used. If the field is an arraytype it should be suffixed with array indeces such as MyEntity[] to denote it as an array.
name	The name of the field, must be unique within the model.
jsonName	Contains the name of the JSON property that will be used for the field when the message is serialized to JSON. Defaults to use the value defined in name
id	The id of the field. The id must be unique with the scope of the containing type. For Xbuf/Protobuf encoding this tag is used as the tag value for the field on the wire. If not set, a unique id will be generated by the source code generator. For better control over compatibility it is reccomended that application set this value manually.
length	If this field refers to a variable length type (such as string), this indicates the maximum length of the field.
doc	A brief, one line string describing the field. If more detailed documentation is desired, a child <documentation> element may be used.

Field Reference Attributes

<fieldRef> element can override most attributes in the referenced field:

Attribute	Description	Required
ref	The name of the referenced field. Name may be with or without the namespace of owner model (i.e. full-qualified name). In both cases, field will be looked up in both current model and its imports. If non-fully-qualified name is declared in current model and in one of imports, the field from current model will take precedence. If multiple imports define the same name, an error will be reported. Best correction in such case is to use fully qualified name.
name	Optionally, a name for the field that overrides the referenced field's name. Otherwise the field name defaults to that of the referenced field.
jsonName	Optionally overrides the referenced field's jsonName
id	Optionally overrides the id specified by the field being referenced.
required	Whether or not the field is a required field for the type. Fields declared as required will cause a check to be added for the value being set in the generated types isValid() method.
pinned	Indicates whether or not the field is pinned. A pinned field always occupies the same space on the wire.
isKey	True if this field should serve as the key when inserting the entity into a LongMap or StringMap collection. In this case insertion of the message in the collection with a given key will update this field with the key value.
doc	The doc to use for the field which overrides that specified by the referenced field.

Generated Field Accessors

Standard Setters/Getters

All field types will produce a regular getXXX and setXXX for the field along with clearXXX and hasXXX methods.

Latency sensitive applications should note that assigning a returned primitive value to an object wrapped primitive type (e.g. Integer int = message.getMyIntField()) will result in an object allocation. 

Zero Garbage Setters/Getters

For UUID and XString types additional zero garbage accessors to the underlying raw type which is backed by the message buffer are additionally generated (though only zero garbage when used in conjunction with Xbuf with pooling enabled). For these types some additional accessors are also generated to allow copying the value into and out of the message in an efficient zero garbage fashion:

Standard Accessors

The standard accessors are not guaranteed to be zero garbage, and are less efficient because the require encoding of the type into its underlying byte representation. 

Copying Accessors

The setXXXFrom / getXXXTo accessors allow zero garbage copying of the field value when Xbuf encoding is used for the message. These copying operations are zero garbage and can be very efficient because the copy operation can be done directly using the preserialized bytes for the type. 

Unsafe Accessor

The unsafe accessor allows the user to get a direct reference to the field with no copy - the returned type directly references the underlying raw bytes in the serialized message. This accessor is useful in cases where the the caller will itself copy the value into an outbound message using a copying setter, or if the caller will use the source type purely within the handler of the message from which it takes the reference. The caller is not permitted to retain the object beyond the source object's lifespan and may not in any way modify the returned type.  

This operation can be dangerous in the presence of object pooling because if the application erroneously retains a reference to the type its value will be invalidated when the message is returned to its pool and its backing buffer will be wiped or repurposed. Applications that use this api should operate with special care and should avoid using this accessor unless performance is of the utmost concern. 

List Accessors for array types

When xbuf encoding is used for an array type and pooling is enabled the underlying implementation doesn't store the field value in an array as pooling varying array sizes is impractical. To achieve zero garbage for fields modeled as arrays, a List interface is more appropriate, and an XIndexedList is returned. An XIndexedList implements java.util.List, but also provides reusable iterators. These accessors are discussed in detail in the knowledge base article at http://docs.neeveresearch.com/display/KB/Zero+Garbage+Array+Accessors

Date field accessors

For zero garbage access on date fields it is possible to use the 'AsTimestamp' accessors to access the field as the number of milliseconds since the epoch. 

Embedded Entity Accessors

To avoid garbage for embedded messages entities when using Xbuf encoding while still provided a mechanism for applications to hold such fields in state beyond the lifespan of the message from whence they came additional take/lend accessors are generated for embedded entity fields. Considerations for using embedded entity fields in the context of message pooling are discussed in detail in Zero Garbage Array Accessors

Unset Fields Getters

When a field has not been set on a type the following table indicates what the getXXX() accessor will return:

Type	Unset Field Return Value	hasXXX	Notes
Java Primitive	Default value for java primitive type		For numeric values 0, for boolean false, for char ''.
String, Date,	null
Enum	null
Entity	null
Array	Empty Array		Depends on code gen configuration property generateArrayGetterEmptyIfNull

For java primitive types the hasXXX() method can be called to distinguish between whether the value was explicitly set or is just returning the default value.

For arrays it is not possible to distinguish between an empty array or the array not being set. This allows ADM code to operate with protobuf encoding under the covers which is not able to distinguish between an empty or unset array. For this reason ADM generated code does not generate a hasXXX method for arrays. Setting the code generation property generateArrayGetterEmptyIfNull controls whether an empty or null array is returned when there are no values encoded in the underlying message or entity. 

Default Getters

The ADM Model supports the ability to generate 'default getters' which accept a default value to return when the field being accessed is not set. This can be enabled by passing the generateDefaultGetters=true directive to the AdmCodeGenerator.Default getter accessors will be created for any field type that has a corresponding hasXXX getter to accept a default value to return when the field is not set. While supported, usage of this setting is not recommended: the preferred mechanism is for applications to call hasXXX() directly instead. 

XML

Generated Code

Semantic Types

The ADM model provides a <types> section that can be used to define types based on the platform's primitive and built in types. Such semantic types can be used in place of their corresponding primitive type in the model and will inherit their documentation. The type used for the field in generated messages and entities will be the base type specified by the named field.

XML

Generated Code

Note that "Price" doesn't result in a new java type being created generated source uses it's base type (float) directly. This is true of all semantic types with the exception of types with a base of String that are declared as poolable: the model above would result in the creation of a poolable SocialSecurityNumber as illustrated below:

Poolable String Types

The X Platform provides an XString implementation that allows zero garbage manipulation of Strings. Low latency applications should use XStrings instead of Java Strings in domain state  This allows for zero garbage copy of fields to and from outbound and inbound messages and zero garbage manipulation of fields by domain logic. Although the use of XStrings will eliminate garbage, it will not eliminate object promotion across heap generations which can be an expensive operation. To avoid promotions, low latency applications also need the ability to pool and pre-allocate such fields. To make working with such types easier, the ADM model provides support for creating sub types of XString that are poolable. For String fields that are declared as poolable, the ADM code generator will generate code implementing a new type for the field that is a subtype of XString and associated classes that allow for the pooling and pre-allocation of fields of this type. All of the methods currently available on XStrings will be available to the subtype. The XString documentation can be found at: http://build.neeveresearch.com/core/javadoc/LATEST/com/neeve/lang/XString.html

Zero garbage applications typical operate with poolable string fields in one of two fashions:

• Creating a corresponding pooled string factory that is configured for preallocation and using that to source strings stored in state at runtime that are intialized by copying a value out of a message. It is important that string fields that are held in application state are copied out of the message because if the message itself is pooled, the value of the string will be reset after the message handler returns. 

• Creating a working variable to temporarily hold a mutable pooled string value. 

Declaring a String field to be poolable

A poolable type will be created for a String field if:

• The field or type element that defines it has a poolable="true" attribute. The String type will be created based on the name of the field. 
• A semantic type is defined with a base type of string and the poolable attribute is set to true. The String type will be based on the name of the type. 
• For applications that would rather separate the notion of pooling from the model itself, The Code Generator can be run with a directive indicating that all String fields should generate a poolable type (see Directives in the sections below).
• In the future different narrower directives may be introduced for finer grained control over what String fields will create poolable types.

Restrictions:

• A poolable string type cannot be created such that it conflicts with a built in or primitive type name (string, integer, Currency, etc), nor can the poolable type conflict with a type name used elsewhere within the model (enumerations, entities, messages or collections). In the case where a field name conflicts with one of these types, a semantic type with a different name should be declared to avoid the naming conflict.

Pooled String Type Name Conflicts

As stated above, a poolable string type's name may not clash with built in platform types, other types declared in the model or types in java.lang. For pooled string types that are generated from a field name (rather than a semantic type definition), the ADM code generator supports two optional directives that can make such collisions less likely: pooledStringFieldTypeNameSuffixPolicy, which specifies the policy for resolving conflicts detected by the code generator, and pooledStringFieldTypeNameSuffix to allow the developer to specify a suffix that should be applied to generated pooled types (defaults to 'String' if not specified). 

• None: (Default) With this policy field names are not suffixed, and it will result in a either a code generation or compilation error if a conflict occurs. This policy is useful where the model developer want to proactively monitor for such conflicts, and perhaps declare the pooled string field as a semantic type in the <types> section to resolve the conflict. 
  
• Always: With this policy, the pooled string type is always suffixed with String which greatly reduces the likely hood of a name conflict. 

• OnConflict: with this policy the code generator will suffix the generated pooled string type name with 'String' when the name conflicts with another type. For example:

  would yield a pooled string type of MyEnumString to avoid conflicting with the MyEnum enumeration type.

XML

Note that for pooled strings if the length parameter is not specified the backing buffer size for preallocated Strings will default to 32 bytes. If length is specified then that size will be used for the default backing buffer size value for preallocating Strings of that type. . 

Generated Code

From the standpoint of a message or entity, a pooled string field is treated no differently than a normal String field. The field's setters accept an XString and the getters return an XString. The field value in the message is not itself pooled, rather the value is pooled along with the message. In other words a pooled string type allows pooling of strings in application code not message code. The code generator will generate a subclass of XString with an inner class factory type that can be used for preallocating XStrings for use by the application:

Notes

• The application can create factories for the generated types to allow application preallocation of the pooled type. 
• The generated factory returns unitialized XStrings with the option of creating immutable (constant) instances. 
• A newly created object will start with an ownership count of 1. The ownership count is used to track an object’s reference count. Users can acquire a pooled object to prevent it from being disposed and cleared. Calling dispose on the type will return it to the pool when the ownership count reaches 0. If the objects returned by the pool will be acquired / disposed across multiple threads the factory should be created as threaded. 

Preallocation of Pooled Strings

Strings from inbound message fields are often stored in domain state. To that end poolable string types need to be preallocated along with the rest of application to avoid creating new instances at run time. There are two approaches to handling preallocation:

1. Preallocate the pooled string type with preallocated application state, and copy the string from the inbound message into domain state. In this approach the message keeps it's reference to the pooled string field and reuses the same reference when the message is recycled through its pool. 
2. Use take/lend semantics in the same fashion as with a Nested Entity as described in Xbuf and Embedded Entities. The domain state needn't preallocate the pooled string type, but rather takes if from the message which then draws a new instance from a globally configured preallocated when the message itself is recycled to its pool.  

Configuring Preallocation and the Pooled Type's Default Factory

Using the static creation methods on a generated pooled type will draw a new instances from the generated type's default factory. The default factory is created in the pooled types static initializer block. So any configuration of the default factory needs to be done before the class is loaded for the values to take effect. Generated Messages draw their values from the default pool. Because messages are themselves pooled, it is not necessarily required that the factory's pool use preallocation to avoid allocating new pooled types at runtime: if the application preallocates the strings and copies values out of the message. 

The following properties may be set for each pooled type:

So, in our example above to enable pooling for a ComplianceId and to preallocate 10,000 instances, the following code should be executed before any code makes a reference to ComplianceId:

Creating New Factory Instances

If the application will handle allocating Strings used in application state up front and copy values from messages into application state, then the application should create a separate Factory that preallocates the expected number of types. This can be done with the Factory's newFactory(...) method:

Depending on the application it is possible that the Factory need not be threaded if only one thread will be creating Strings from this factory and will never dispose the strings it creates. If this is the case, then there is no risk that another thread will return the string to the pool and introduce concurrency issues.  

Creating New Factory Instances Generically

It is also possible to create pooled types in a more generic fashion as illustrated below. This approach can be more convenient in cases where application code dynamically pre-allocates objects at runtime. 

Entities

Entities are defined in the entities element of the model and must have a unique id with respect to other types within the scope of their factory 

XML

Entity Attributes

Attribute	Description	Required
name	The name of the field, must be unique within the model.
id	The id of the entity which must be unique within the scope of all types in the entity's factory
factoryId	The id of the factory which must be unique within the scope of all factories used within an application. When a message is received, the factoryId and entityId uniquely identify the type to be deserialized. factory ids <= 0 are reserved for platform use.
asEmbedded	Defaults to false, Indicates whether or not this entity is generated to be used as an embedded or child field of another entity. Embedded entities are always serialized transported with its parent entity. Entities used as fields in messages must be declared as embedded
transactional	Whether or not this entity supports transaction commit and rollback via the applications ODS store

Entity Field Attributes

Attribute	Description	Required
type	The type of the element. If the type is defined in this namespace, is defined in only one of the imported models, or is a primitive or collection type only the simple name of the type need be used. Non fully qualified name will be looked up in both current model and its imports. If name is declared in current model and in one of imports, the type from current model will take precedence. If multiple imports define the same name, an error will be reported. Best correction in such case is to use fully qualified name.  If the field is an arraytype it should be suffixed with array indeces such as MyEntity[] to denote it as an array.
name	The name of the field, must be unique within the model.
jsonName	Contains the name of the JSON property that will be used for the field when the message is serialized to JSON. Defaults to use the value defined in name
id	The id of the field. The id must be unique with the scope of this entity. For Xbuf/Protobuf encoding this tag is used as the tag value for the field on the wire. If not set, a unique id will be generated by the source code generator. For better control over compatibility it is reccomended that application set this value manually.
length	If this field refers to a variable length type (such as string), this indicates the maximum length of the field.
pinned	A pinned field always occupies the same space on the wire.
isKey	True if this field should serve as the key when inserting the entity into a LongMap or StringMap collection. In this case insertion of the message in the collection with a given key will update this field with the key value.

Generated Source

For an entity named "MyEntity" an interface and an implementation will be generated using the model's namespace. Entities will extend IRogNode or IRogContainerNode marking that they can be used as nodes with the platform's Replicated Object Graph (ROG) framework. 

Messages

Messages are defined in the messages element and must have a unique id with respect to other types within the scope of their factory. Message can use all of modeling capabilities of entities, but cannot use non embedded entities or non array collections as fields.

XML

Message Attributes

Messages support the same attributes as entities (listed above) with the exception that transactional defaults to false for messages. 

Message Field Attributes

Message fields support the same attributes as entities (listed above)

Notes

For Xbuf and Protobuf encoding, the id is used to generate the protobuf field tag, so changing field ids will break wire compatibility. If fields are not explicitly assigned ids in the model, then the ADM generate will assign them automatically in this case fields should not be removed or changed in order to maintain wire compatibility with earlier version of the generated code. 

Messages can only declare primitive types, built in types, embedded types as fields (or arrays of those types). Collections, Messages and non embedded Entities can't be used.

Generated Source

For an entity named "MyMessage" an interface and an implementation will be generated using the model's namespace. Entities will extend IRogNode or IRogMessage marking that they can be used as nodes with the platform's Replicated Object Graph (ROG) framework and the platforms Simple Message Abstraction (SMA) Layer.

Collections

Collections are defined in the collections element and must also have a unique id with respect to other types within the scope of their factory. Collections may not be declared as embedded at this time. 

XML

Generated Source

Defined collections entites are generated to extend/implement their java collection type:

Inlined Entities

The fields from an embedded entity can be 'inlined' into another message or entity. Inlining an embedded entity produces the same result as if all of the fields for the given entity were defined on the entity itself. It is also possible to inline the fields of one message into another message. It is not possible to inline the fields of an entity into an embedded entity or message, nor is it possible to inline the fields of Message into an embedded entity or entity. 

A type that inlines another type will generate code that implements the inlined types's generated interface. 

XML

Restrictions

• An embedded entity cannot be inlined into another embedded entity in a manner that will produce inlining cycles.
• An embedded entity cannot inline a Message or Entity.
• Messages and embedded entities cannot inline non embedded entities. 
• A Message can't inline 2 embedded entities that have conflicting field definitions (e.g. fields with the same name but with attributes that differ in any way other than their doc or required attributes). In the case of required, if an embedded entity defines a field as required then it will be required on the message.
• The isKey and pinned field attributes on an embedded entity are ignored when the entity is inlined into another message or entity. 
• A Message can't define fields that conflict with the any of its inlined embedded entities (e.g. fields with the same name but with attributes that differ in any way other than their doc or required attributes). 
  Exceptions to this rule are:
  • required: In the case of the required attribute, the value set on a message field wins (a particular message can override the required field attribute on an inlined embedded entity)
  • isKeyField: entity or message inlining the embedded entity can set the isKey attribute for a given inlined field. 
  • pinned: the pinned attribute can be overridden by the inlinining entity. 
• SINCE 3.8 The <inline> tags are processed before <field> and <fieldRef> tags. This affects redefining/merging field refs. A directly placed fieldRef will redefine the same fieldRef found in inlined entity, because it is always processed after inlined entity.

Generated Source

Note javadoc is omitted below and only the simple accessors are listed for brevity:

Nested Messages

Required Field Validation

A required field can be declared by setting its required attribute. The required attribute is optional and defaults to false. When required fields are defined checks for their presence will be added to the types isValid() method. Calling getValidationErrors() on a type missing that field will result in an error of the form: "myDoubleField is not set" being added to the provided list. Neither the the Messaging Engine nor the transport layer will invoke isValid() on a given message, validation checks are left to the application code. 

To generate logic for required fields the generateRequiredFieldValidators=true directive should be passed to the AdmCodeGenerator

XML

Generated Code

Generating Source Code From XML

Code generation is accomplished using the ADMGenerator class distributed with the ADM module (nvx-adm-<vesion>.jar or the nvx-core-all-<version>.jar). The xml schema is bundled in these jars at the jar root: /x-adml.xsd. 

Encoding Types

ADM code generation currently supports generating classes in the following encoding types :

• Json: Javascript Object Notation serialization; not very efficient but highly readable
• Protobuf: Google's protobuf serialization format, which can be useful for portability purposes providing a reasonable balance between performance and interoperability.
• Xbuf: Neeve's protobuf implementation which can operate in a wire compatible mode with Google's protobuf, but allows zero garbage operation along with other performance enhancements. 
  

Protobuf IDL

For Xbuf and Protobof the protobuf idl (.proto) is output in the generated source folder using the model's fully name space.

Generated Encoding Output

Invoking the code generator on a model outputs source for objects in a particular encoding: Json, Xbuf or Protobuf to a package that is by default defined as the model's name space. The code generator however can be used to generate source to a namespace specified to the code generator. This makes it possible to generate source for mulitple encodings of the same model by invoking the code generator multiple times with a separate namespace and encoding for each execution. Furthermore it is possible to convert between any of the separate encodings by serializing/deserializing to json which is compatible between all encodings (and in the case of Xbuf/Protobuf to a binary format since the are binary compatible). 

It is perfectly legal to mix separate encoding types at runtime across different models so long as the models have separate factory ids (since the factory id is used to determine which generated factory to use to decode a message). 

Directives

Some advanced properties can be passed to the code generator as directives. The following is a list of supported directives:

Running with java

To run the generator with java you can use:

or you can use the platform jar which contains adm:

Using Maven

To use the code generator with Maven add the following to your pom's plugin section:

Configuration Properties

generateArrayGetterEmptyIfNull

Using the Platform Plugin

If your project is using the nvx-platform-bom and you want to generate code with the same version of the platform you are using, you may also use the nvx-platform-maven-plugin. The advantage of this approach is that you can use the same version of the plugin as the platform bom. Because maven BOMs don't cover plugin versions, using the nvx-core-maven-plugin would mean that the nvx-core-maven-plugin-version would have to be specified separately.

Using ANT

Imported Model Resolution 

To handle multi module and multi project builds, the ADM XML Parser, and Maven Build plugin search for model imports on the classpath using the imported model's fully qualified namespace. The model xml (and .proto for Xbuf/Protobuf) will be copied to the generated source folder in fully qualified form. To allow other models to resolve the model being generated from their classpath, users should ensure that these files are copied to the target classes folder.

SINCE 3.4
ADM Code Generator Maven plugin will copy these files by default to target classes folder, and include them into project artifact's jar.

Import resolution searches both current classpath and OS file system when resolving an import. Recommended is to use relative classpath strings whenever possible. Consider the following example project with two models, model.xml imports other_model.xml:

${basedir}/src/some/package/model.xml [In xml we define namespace = "some.model.namespace"]
${basedir}/src/other/package/other_model.xml [In xml we define namespace = "some.other_model.namespace"]

The recommended way to define import would be one of the following:

Reason why this works is because result of running code generation would be:

${basedir}/target/classes/some/model/namespace/model.xml
${basedir}/target/classes/some/model/namespace/Model.proto
${basedir}/target/classes/some/model/namespace/GeneratedModelClass1.class
${basedir}/target/classes/some/model/namespace/GeneratedModelClass2.class
...

${basedir}/target/classes/some/other_model/namespace/other_model.xml
${basedir}/target/classes/some/other_model/namespace/OtherModel.proto
${basedir}/target/classes/some/other_model/namespace/GeneratedOtherModelClass1.class
${basedir}/target/classes/some/other_model/namespace/GeneratedOtherModelClass2.class
... 

The requirement is that code generation first runs for other_model.xml so that it can be found in classes folder at the time of running model.xml. 

The following modes of import model resolution are provided for advanced use cases but are discouraged. 

Platform Bundled IDLs

The platform bundles the following IDLs in the packaged jars to support custom options used by ADM to enrich handling of enums, and to define additional data types used by ADM. 

Regeneration of Imported Models

The ADM Code generator will be enhanced to allow the code generator to operate in a mode in which it will regenerate all pre-built imported models. This intent of this mode is that it will ensure that all models used by an application are built with the same version of the platform that the importing application is built with. This can be used to work around issues where older dependencies haven't been updated to a newer version of the X platform which is no longer API compatible. 

Incremental Smart Code Regeneration

SINCE 3.4

The Maven Plugin and ADM Code generator take a source model's last modified timestamp or checksum into account. Code generation will be skipped if:

• Model file has been changed since last build.
• Any import model file has changed (checked recursively in imports of imports...)
• Any input option for code generation has changed such as encoding type, namespace, directives etc.

The incremental code generation works by tracking above given changes in an xml file which may be found in output dir. The file has name with pattern like this:

.${model_filename}.xml_${md5checksum}.metadata. model_name is name of model file for which code was generated. md5checksum is a signature calculated from input options given to code generator, so that if any of them changes, the resulting filename no longer represents same code generation. Stored in this file are input options given to code generator and list of models with a number that would be different every time model file is persisted to disk. These files do not go into jar, and may be deleted at any time (which they usually do when clean build is triggered).

Message Engine Independent Serializers

Generated Embedded Entities and Messages can be used independently of the Messaging Engine and support serializers/deserializers to byte [], ByteBuffer and json.

Runtime Compatibility

AdmGenerated Annotation

To validate compatibility of the generated code with the version of the platform jars on which the generated classes are being used, an com.neeve.adm.runtime.annotations.ADMGenerated annotation is added to ADM generated types. The ADMGenerated annotation has runtime retention and is thus available for introspection by applications at runtime.

Generated Code 

AdmGenerated Properties

The ADMGenerated annotation is used by the platform as follows

1. When a message factory is registered with the platform it's compatLevel will be examined to ensure that it is runtime compatible with the platform. The platform will increase it's runtimeCompatibilityVersion each time a change is made that would break compatibility with previously generated code. See Compatibilty Level Rules below.

2. Trace can be enabled to display the version of the platform that generated the factory when it is registered with the runtime to assist with support issues (nv.sma.trace=verbose or nv.ods.trace=verbose).
   Sample output from this trace is:
3. It is also possible for the application to call MessageViewFactoryRegistry.dumpFactoryVersionInfo(StringBuilder) or StoreObjectFactoryRegistry.dumpFactoryVersionInfo(StringBuilder) at any time to collect the above trace at any time. 

Runtime Compatibility Check

The ADM compatLevel is of the form [MAJOR].[MINOR].[PATCH]. This compatLevel is added to the AdmGenerated  annotation's compatLevel property, and is used determine if the generated code is compatible with the platform version being run. Note that the ADM  COMPATIBILITY_LEVEL is versioned and tracked independently of the platform version. 

See the ADM Compatibility Matrix for a listing of the compatibility level changes.

At runtime when Factories are registered with the platform their compatibility will be checked, and an EAdmCompatibilityException will be thrown if the classes are found to be incompatible.

The compatibility rules are enforced as follows:

Bump in patch version

For the same major and minor versions, two differing patch versions must be API compatible at runtime even if the patch version differs. This relation holds true for platform code compiled as 2.0.0 and code generated by a later version of the platform at 2.0.1. The older platform code must be able to operate with the newer platform code.

A bump in patch version may be used by a newer version to indicate that the generated code supports additional APIs or capabilities. For example, perhaps in 2.0.1 messages are generated to implement new formatting in its Json representation or with some new accessor apis that are available by reflection.

	Generated 2.0.0	Generated 2.0.1
Platform 2.0.0
Platform 2.0.1

Bump in minor version

A bump in minor version indicates that newly generated code is not api compatible with previous versions of the platform, but that a newer version of the platform can still operate with previously generated code with the same MAJOR version.

An example of such a case would be when code generated with version 2.0.1 relies on a new platform APIs that is not available in previous versions of the runtime and would thus fail with a NoSuchMethodException  when operating with an older version of the platform. Another example of this case would be if the newer version of the code relies on some semantic behavior change in the existing platform API that would not be provided by an older version of the platform.

	Generated 2.0.0	Generated 2.1.0
Platform 2.0.0
Platform 2.1.0

Sample Compatibility Error

 This error would be thrown running with a older version of the platform that is not compatible with a newer version's generated source (version numbers are hypothetical):

Bump in major version

A bump in major version means that generated code is not backwards or forwards compatible with the platform. A newer version of the platform can't operate with code generated with an older major version and an older version of the platform can't operate with a newer version of generated code.

An example of this case would be where there are breaking API changes in the generated API, for example 3.0.0 might introduce a new method.

	Generated 2.0.0	Generated 3.0.0
Platform 2.0.0
Platform 3.0.0

This error would be thrown running with a new version of the platform that is not compatible with older generated source (version numbers are hypothetical):

Choosing an Encoding Type

When to use each encoding type:

Json

With json encoding fairly simple classes are created that use jackson for serializing to/from json. It is suitable for lightweight applications or for applications that natively work with json (e.g. web applications). 

Pros:

• Memory utilization: Because there isn't much serialization machinery or caching of the backing serialized format, Json generated objects don't use much memory which can be useful for long lived state objects. 

Cons

• Jackson serialization is slow and produces a lot of garbage, and json objects can't be pooled. 

Protobuf

With protobuf endoding objects are create with backing google protobuf generated objects. Protobuf is suitable for applications with higher performance requirements than is afforded by Json encoding. It should be used by applications with moderate to high performance requirements. 

Pros:

• Memory Utilization: Protobuf generated objects are fairly compact in memory compared to Xbuf objects, non repeated field values are store directly in the generated message object, making protobuf encoded objects a good candidate for usage as state entities. 
• Faster serialization than Json. 
• Protobuf is a well known standard, making it easy to interoperate with applications not uses ADM generated code.
• Supports serialization / from json. 

Cons

• Google protobuf generated messages are not zero garbage, and the serialized Protobuf messages are technically reusable so for very low latency applications that aren't using the Platform's zing distribution, this can result in garbage related pauses. 

XBuf

Xbuf generated objects, are backed by the X Platform's high performance implementation of Google protobufs which supports zero garbage operation and cut-through serialization (the ability to read/write fields directly to from a backing buffer). It should be used for applications with the most stringent performance requirements. 

Pros:

• Faster serialization than Json or Protobuf
• Protobuf is a well known standard, making it easy to interoperate with applications not uses ADM generated code.
• Compatible with protobuf on the wire.
• When not interoperating with google protobuf generated recipients, code can also be generated to perform even faster.

Cons:

• Memory Utilization: Currently Xbuf objects can take up more memory than protobuf encoded objects (a memory performance tradeoff). 
• Pooling Complexity: Working with the more advanced message api around pooling take / lend, is harder to achieve. 
• Not currently optimized for use as entities in State Replication. 

Known Limitations:

• Accessors for String and Uuid arrays fields in Xbuf are not currently zero garbage. Consequently, Xbuf may be more expensive than Protobuf for types that use array fields. 

Differences Between Encoding Types

For the most part code generated for the different encoding types behaves the same, but there are some key differences that stem from both the underlying encoding and serialize as well as from the additional features provided by each encoding. 

Unrecognized Field Values

1. For Json encoding unrecognized enum array values are treated as null, and for non array fields an unrecognized array value will be treated as null and hasXXX will return true.

2. For both Protobuf and Xbuf unrecognized fields (those with unrecognized field tags) are preserved when an inbound message is written to a transaction log (although they are inaccessible). If the message is copied by serializing to bytes and deserializing into a new message instance, the unrecognized fields from the original message are sent on the wire. If the message is modified prior to sending, the unrecognized fields may be lost. 
3. For repeated enum fields in protobuf unrecognized enum values are ignored. For Protobuf encoding the underlying protobuf may reorder the unrecognized enum values and put them at the end. Xbuf generated code preserves the order of unrecognized enums. When deserializing from Json unrecognized enum values are treated as null so the effect on a deserialized message or entity is the same as adding an enum array with null values (see below). 

Null Value Handling

1. Message and Entities generated with Json encoding support serializing null values and null values in arrays. 
2. For Xbuf setting a timestamp of -1 or less results in the field value being cleared. 
3. For Xbuf and Protobuf, setting a null value for a String, Date, Enum or Embedded Entity Field results in the field being cleared (protobuf doesn't support null values on the wire).
4. For Xbuf and Protobuf, setting a Date[], String[], or Enum[] with containg a null value results in a NullPointerException being thrown. Setting an Embedded Entity [] array or an enum [] with a null value results in the null value being ignored during serialization. The same holds true when using the XIterator setters or when calling addXXX to add the set of values.
5. For Xbuf and Protobuf, when setting null values in an array field, subsequently calling the getter MAY or MAY NOT result in the null values being returned. Applications are encouraged to use the getXXXIterator accessors, and should be coded to handle either case for maximum portability both between encodings and for handling cases where the null values have been filtered out due to serialization. At present, Xbuf messages generated with protobuf compatibility do not cache a reference to the array passed in, and Protobuf messages do cache values passed in so that nulls are returned ... but this is an implementation detail that could change.  

Pooling Considerations

A major difference between Xbuf and Protobuf or Json encoded entities is that Xbuf messages and entities are pooled by the platform by default. From a coding standpoint this means that when working with Xbuf Encoded Messages or Entities:

1. An application may not hold onto an XBuf encoded message beyond the scope of a message handler. 
2. An application may not hold onto an XString, or Embedded Entity type from a message beyond the scope of a message handler because these objects are pooled along with the message and will be reset once the message is returned to its pool. See Zero Garbage Embedded Entities for detailed usage, but the general rule of thumb is to copy any entity that needs to be retained by applications state, or to use the more advanced 'take' apis. Note that 'take' is not supported for String fields as string fields are not pooled for Xbuf messages. 
3. Setting an XString or Embedded Entity field on a message transfers ownership to the message. If the application wants to retain the entity is application state, then it should copy it into a new entity or use the more advanced 'lend' apis. See  Zero Garbage Embedded Entities for more details. 
4. An application may not mutate a returned array type from a message and should not hold onto to the array beyond the duration of a message handler. See Zero Garbage Array Accessors for more details.