Overview

The X Platform's Application Data Modeling (ADM) framework allows modeling of messages that contain embedded entities. As an embedded part of the message to which the entity belongs, the entity is subject to the same pooling lifecycle and ownership constraints as its parent message: an application handler can not hold onto an embedded entity beyond the scope of a handler. For applications that are highly latency sensitive, and want to transfer the embedded entity as a whole into application state or to outbound messages, it is useful to be able to work with such fields in a zero garbage fashion so that GC pauses can be avoided. Additionally, Embedded entities when used with Xbuf support the notion of pass-through fields. Pass-through fields are fields that are not defined in the entities ADM model, but can be opaquely passed through an application. Understanding pass through fields can be important in certain compatibility scenarios and are discussed here as well.

Lifecycle of Embedded Entities

Embedded entities in messages generated using the XBUF encoding are pooled entities. Therefore, contractually, the lifecycle of an embedded entity is managed and governed by its containing message. A reference to an embedded entity is valid only until the parent message is disposed. To hold onto the contents of an embedded entity post disposal of its containing message, the application can either:

• Copy the embedded entity into another instance (possibly preallocated). ADM generated code provides conveniend getXXXTo accessors for copying an embedded entity into another entity. 
• or 'acquire' a reference to the embedded entity. ADM generated code provides takeXXX accessors which both get and acquire an entity field. 
• or 'serialize' the contents of the embedded entity to a buffer. An embedded entity object can be re-materialized by 'deserializing' it from its serialized form. 

Working with AEP Engines

The above is is particularly relevant for applications working with AEP engines. An AEP engine disposes inbound messages after return from the application's message handler. The engines also dispose outbound messages once stabilized. The act of disposing a message also disposes its embedded entities. Therefore, the following rules apply when working with embedded entities on messages sent and received through an AEP engine

1. An application must not hold onto an embedded entity reference post return from an AEP message handler unless a reference to the container message has been acquired by the application or the embedded entity copied or 'taken' from the message prior to return from the handler. Alternatively (or in-addition), the application can hold onto the serialized form of the message post return from the handler and re-materialize the entity from the serialized form at a later point when structured access to the entity's contents is needed. Note that embedded entities are read-only in received messages and, therefore, cannot be modified.
2. The act of 'setting' an instance of an embedded entity onto an outbound message effectively transfers ownership of the embedded entity to the message (and indirectly to the AEP engine through which the message is being sent). The act of sending a message sent via an AEP engine effectively transfers ownership of the message to the AEP engine. The engine disposes the message once stabilized. Message stabilization occurs in an asynchronous, pipelined manner outside the scope of the AEP engine's sendMessage() method. If the application wants to set an embedded entity onto a message without transferring ownership of the entity to the message, the application needs to 'lend' the entity to the message. When lent, the message will acquire its own reference to the entity assuming that the application wishes to retain its reference to the entity. Regardless of whether an entity is 'set' on or 'lent' to a message, the application must ensure that the entity is not concurrently modified while the entity is in-flight with its container message.  

Working with Embedded Entities

getXXX()

The getXXX() method on a message returns a reference to the XXX embedded entity. 

When using a getXXX() method it is important to keep in mind that the embedded entity is still attached to the message. An application cannot hold on to an entity from a getXXX() call beyond the scope of the message from whence it came since it will be disposed along with the message. 

Copying Embedded Entities from Messages

getXXXTo(XXX copy)

SINCE 3.5.1

ADM generated messages and entities provide convenience accessors for getting and copying an embedded entity field into another entity. 

Note that:

• The entityCopy will be modifiable.
• If a message doesn't have the field set, getXXXTo will clear the contents of the copy. 

XXX.copyInto(XXX copy)

It is also possible to use the standard getter, and use the copyInto method on generated entities. 

This approach is useful in cases where the embedded entity field is an array as ADM doesn't generate copying getters for arrays, But an application can iterate them in a zero garbage fashion and copy the values it needs to retain:

takeXXX()

An embedded entity reference returned by getXXX() is only valid until the entity is disposed i.e. the act of getting an entity does not transfer ownership of the entity to the application A message disposes of its reference to its contained entities when it is disposed itself. An AEP engine disposes a message on return from its message handler. Therefore, applications working with AEP engines that wish to hold onto an embedded entity reference post return from its containing message's handler must either acquire a reference to the container message or 'take' the embedded entity itself from the container message. The act of 'taking' an entity differs from 'getting' an entity in that 'take' does what is necessary to ensure that the entity contents are not cleared and the entity not disposed when the message itself is disposed. For a taken entity to be disposed, the application must explicitly dispose it when done working with it.

Acquire a reference to the container message

Take the embedded entity from the container message

Embedded Entities that are taken from an inbound messages are marked as read only. 

serializeToXXX()

Embedded entities can be serialized to byte array/buffers and native memory using the serializeToXXX() methods present on embedded entities.

Supported Serialized Forms

Currently supported serialized forms are:

• byte[]
• java.nio.ByteBuffer
• Native pointer

deserializeFromXXX()

Embedded entities can be de-serialized to byte array/buffers or native memory using the deserializeFromXXX() methods present on embedded entities. 

setXXX()

The setXXX() method on a message sets an embedded entity on a message. The act of 'setting' an entity on a message transfers ownership of the entity to the message i.e. setXXX() does not acquire its own reference to the entity

It is possible for an application to set an embedded entity retrieved from a message via takeXXX. In that case the application is transferring the ownership that it took to the outbound message:

lendXXX()

It is illegal to hold onto an entity 'set' on a message post the disposal of the container message. This is true even if a reference to the entity is explicitly 'acquired' by the application via acquire(). To hold onto the reference of an entity beyond the lifecycle of its container message, an application must own a reference to the entity (via, for example, 'taking' the entity off an inbound message) and then 'lend' the entity to the message for sending. The act of 'lending' an entity to a message implies that the application continues to own the entity even though the message is holding a reference to it for the duration of the send.

Note then when the field is taken from an inbound message, that it can be set on multiple outbound messages. It is important to keep track of ownership count in this case. Note in the below sample that the last set operation transfers ownership to the last outbound message. 

If in the above code the application were to retain the message, then the code would need to lend the entity to outMessage2.

Some Use Cases

The following describe how to implement certain specific use cases when working with embedded entities

Holding onto Embedded Entities in Inbound Messages for use in Outbound Messages

Reusing Embedded Entities on outbound objects

Because setting an embedded entity on a message sent outbound transfers ownership of the embedded entity to the outbound message, it is not safe to set an embedded entity on multiple outbound sends. To use the embedded entity in multiple outbound messages it must be copied

If the cachedEntity above was not going to be modified, it would also be possible to acquire() the cachedEntity after its creation and lend() it to the each outbound message, but the application must ensure that the entity is never mutated. 

 The copy method for embedded entities was introduced in the 3.5.1 version of the ADM code generator. If using code generated with an older version of the code generator the clone() method can be used, though it doesn't preserve pass through fields. 

Backing Buffer Sizes for Xbuf Entities

SINCE 3.5.1 

Xbuf generated entities are created with a default initial backing buffer size of 256 bytes. If you know that the serialized length of a particular type may be longer than this length, or will always be shorter, you can set the size of the backing buffer globally with the environment variable

<fullclassname>.initialbackingbuffersize=<size-in-bytes>

for example:

com.acme.MyEmbeddedEntity.initialbackingbuffersize=64

Note that backing buffers are sourced from buffer pools managed by the platform ... the actual size of the buffer allocated will be a power of 2. Therefore applications should use a power of 2 when specifying the backing buffer size. 

When an embedded entity is deserialized or copied to another embedded entity with an insufficient backing buffer size, the inadequate buffer is disposed back to its pool and a new backing buffer is created. 

Pass-through Fields 

Xbuf supports the ability for an application to preserve fields received on the wire that are not defined in the ADM model that generated them. These fields are referred to as pass-through fields and they are preserved by in the backing buffer of the embedded entity in their serialized protobuf wire format. Because the backing buffer stores the fields in protobuf wire format it is event possible to serialize and deserialize between different embedded field types that share the same field ids and types. These are called pass-through fields because an intermediate app that receives fields it doesn't recognize can pass them between a sender and receiver application without them being lost. 

Pass through fields are preserved when:

• Taken from inbound to outbound message using take e.g:

• When an embedded entity is deserialized from another embedded entity:

• When an embedded entity is copied from another embedded entity:

Pass through fields are not preserved when:

• An embedded entity is cloned() ... the clone operation currently only copies fields that are known the embedded entity. This may be changed in a future release.

• Using Json encoded enties. 

Serializing/Deserializing with Different Entities

The code sample below provides shows how fields can be passed through types that aren't aware of them. AnotherEmbeddedEntity below doesn't know about field3, but when deserialized from an EmbeddedEntity it can pass the value of field 3 through to AnotherEmbeddedEntity that does define the field:

Usecase: Pass-through fields and model evolution.

Such unknown fields can be passed through from a sender to a receiver by an processor application. One such case where this can be important is if a new field is added to an embedded entity and the sender and receiver applications are updated before the processor application. 

As a concrete example:

In version 2 the a zipcode field is added to Address:

Sender sends an  AddressUpdateMessage with the new zipcode field that the Processor doesn't yet know about. 

The processor application is still on model version 1. It 'takes' the address entity and places it on an address update event. 

The receiver is on model version 2. The processor will have passed through the zip code field.