Overview

The X Platform's Application Data Modeling (ADM) framework allows modeling of messages that contain nested (aka embedded) entities. For applications that are highly latency sensitive, it is useful to be able to work with such fields in a zero garbage fashion so that GC pauses can be avoided. Additionally, there are certain sender-forwarder-receiver type applications in which the sender and receiver need to be able to continually change the contents of a embedded entity passed between them (1) without having the intermediate forwarder entity upgraded each time the entity structure changes and (2) allowing for the intermediate forwarder entity to inspect a subset of the contents of the entity in a structured form using getter type methods. 

Lifecycle of Nested Entities

Nested entities in messages generated using the XBUF encoding are pooled entities. Therefore, contractually, the lifecycle of a nested entity is managed and governed by its containing message. A reference to a nested entity is valid only until the parent message is disposed. To hold onto the contents of a nested entity post disposal of its containing message, the application can either 'serialize' the contents of the nested entity to a buffer or 'acquire' a reference to the nested entity. A nested entity object can be re-materialized by 'deserializing' it from its serialized form. An entity acquired by an application needs to be disposed by the application when not needed any more. 

Working with AEP Engines

The above is is particularly relevant for applications working with AEP engines. An AEP engine disposes inbound messages after return from the application's message handler. The engines also dispose outbound messages once stabilized. The act of disposing a message also disposes its nested entities. Therefore, the following rules apply when working with nested entities on messages sent and received through an AEP engine

1. An application must not hold onto a nested entity reference post return from an AEP message handler unless a reference to the container message has been acquired by the application or the nested entity 'taken' from the message prior to return from the handler. Alternatively (or in-addition), the application can hold onto the serialized form of the message post return from the handler and re-materialize the entity from the serialized form at a later point when structured access to the entity's contents is needed. Note that nested entities are read-only in received messages and, therefore, cannot be modified.
2. The act of 'setting' an instance of a nested entity onto an outbound message effectively transfers ownership of the nested entity to the message (and indirectly to the AEP engine through which the message is being sent). The act of sending a message sent via an AEP engine effectively transfers ownership of the message to the AEP engine. The engine disposes the message once stabilized. Message stabilization occurs in an asynchronous, pipelined manner outside the scope of the AEP engine's sendMessage() method. If the application wants to set a nested entity onto a message without transferring ownership of the entity to the message, the application needs to 'lend' the entity to the message. When lent, the message will acquire its own reference to the entity assuming that the application wishes to retain its reference to the entity. Regardless of whether an entity is 'set' on or 'lent' to a message, the application must ensure that the entity is not concurrently modified while the entity is in-flight with its container message.  

Working with Nested Entities

getXXX()

The getXXX() method on a message returns a reference to the XXX nested entity. 

NestedEntity entity = message.getNestedEntity();
int val = entity.getVal();
.
.
 

takeXXX()

A nested entity reference returned by getXXX() is only valid until the entity is disposed i.e. the act of getting an entity does not transfer ownership of the entity to the application A message disposes of its reference to its contained entities when it is disposed itself. An AEP engine disposes a message on return from its message handler. Therefore, applications working with AEP engines that wish to hold onto a nested entity reference post return from its containing message's handler must either acquire a reference to the container message or 'take' the nested entity itself from the container message. The act of 'taking' an entity differs from 'getting' an entity in that 'take' does what is necessary to ensure that the entity contents are not cleared and the entity not disposed when the message itself is disposed. For a taken entity to be disposed, the application must explicitly dispose it when done working with it.

Acquire a reference to the container message

// acquire a reference to a message. this will prevent the message from
// being disposed upon return from its AEP engine message handler and,
// implicitly, prevent the nested entity from being disposed too.
message.acquire();
.
.
.
// acquired messages must be disposed at later point when the message is 
// not needed by the application any more. disposed messages are returned
// to their message pool. disposal can occur in any thread
message.dispose();

Take the nested entity from the container message

// take the nested entity off the container message. this
// will ensure that the entity is not disposed when the
// message disposes off its reference to the entity. 
NestedEntity entity = message.takeNestedEntity();
.
.
.
// taken entities must be disposed at later point when the entity is 
// not needed by the application any more. disposed entities are returned
// to their entity pool. disposal can occur in any thread
entity.dispose();

serializeToXXX()

Nested entities can be serialized to byte array/buffers and native memory using the serializeToXXX() methods present on nested entities.

// get a reference to a nested entity
NestedEntity entity = message.getNestedEntity();

// serialize the entity to a a ByteBuffer.
// ... note: zero garbage applications need to manage the lifecycle of the byte buffer
entity.serializeToByteBuffer(byteBuffer);

Supported Serialized Forms

Currently supported serialized forms are:

• byte[]
• java.nio.ByteBuffer
• Native pointer

Additional serialized forms such as com.neeve.pkt.PktPacket, com.neeve.pkt.PktIOBuffer (i.e. pooled java.nio.ByteBuffer) will be supported in the future.

deserializeFromXXX()

Nested entities can be de-serialized to byte array/buffers or native memory using the deserializeFromXXX() methods present on nested entities. 

// create a new entity
// ... note: the act of 'creating' a nested entity actually fetches the entity from its pool if entity pooling is enabled           
NestedEntity entity = NestedEntity.create();

// deserialize the entity from a ByteBuffer.
entity.deserializeFromByteBuffer(byteBuffer);

setXXX()

The setXXX() method on a message sets a nested entity on a message. The act of 'setting' an entity on a message transfers ownership of the entity to the message i.e. setXXX() does not acquire its own reference to the entity

NestedEntity entity = <somehow get a reference to a nested entity>
.
.
// set the entity on the message. the entity reference is unchanged
// upon return from this method 
message.setNestedEntity(entity);
.
.
 

lendXXX()

It is illegal to hold onto an entity 'set' on a message post the disposal of the container message. This is true even if a reference to the entity is explicitly 'acquired' by the application via acquire(). To hold onto the reference of an entity beyond the lifecycle of its container message, an application must own a reference to the entity (via, for example, 'taking' the entity off an inbound message) and then 'lend' the entity to the message for sending. The act of 'lending' an entity to a message implies that the application continues to own the entity even though the message is holding a reference to it for the duration of the send.

NestedEntity entity = <somehow get a reference to a nested entity>
.
.
// lend the entity on the message for the send. the entity reference 
// is incremented by 1 upon return from this method 
message.lendNestedEntity(entity);
.
.
// dispose off the entity when done with it
entity.dispose()

Some Use Cases

The following describe how to implement certain specific use cases when working with nested entities

Holding onto Nested Entities in Inbound Messages for use in Outbound Messages

// AEP engine message handler for inbound message
@EventHandler
public void onMessage(Message message) {
   domainObject.setNestedEntity(message.takeNestedEntity());
}
 
// at a later point when the entity needs to be sent in an outbound message
// ... note: it is safe to reuse the same entity reference across multiple 
//           outbound messages. taken entities are read-only and, thus, 
//           prevent concurrent modification while being sent. If a reference
//           to an entity is obtained by means other than 'take', the 
//           application must ensure that the entity is not concurrently 
//           modified while in flight in the send path
outboundMessage.lendNestedEntity(domainObject.getNestedEntity());
 
// at some later point, dispose the entity when done with it
domainObject.getNestedEntity().dispose();
domainObject.setNestedEntity(null);

Reusing Nested Entities on outbound objects

Because setting a nested entity on a message sent outbound transfers ownership of the nested entity to the outbound message, it is not safe to set an nested entity on multiple outbound sends. To use the nested entity in multiple outbound it must be cloned 

private final NestedEntity cachedEntity = NestedEntity .create();
private int count = 0;
 
// AEP engine message handler for inbound message
@EventHandler
public void onMessage(Message message) {
   domainObject.setNestedEntity(message.takeNestedEntity());
	
   Message message1 = Message1.create();
   cachedEntity.setCount(count++);
 
   //clone so message gets a private copy
   message1.setNestedEntity(myEntity.clone()); 
   sendMessage(message1);
   
   Message message2 = Message1.create();
   cachedEntity.setCount(count++);
   //clone so message gets a private copy
   message2.setNestedEntity(myEntity.clone());
   sendMessage(message2);
}

If the cachedEntity above was not going to be modified, it would also be possible to acquire() the cachedEntity after its creation and lend() it to the each outbound message, but the application must ensure that the entity is never mutated. 

Serializing/Deserializing Nested Entities 

// Assume the following
// - NestedEntity has field1, field2 and field3
// - AnotherNestedEntity has field1 and field2 but not field3
// - YetAnotherNestedEntity has field1 and field3 but not field2
 
// get a byte buffer from somewhere
ByteBuffer buffer = <get a byte buffer from somewhere>
 
// serialize NestedEntity using serializeToXXX() methods
// - the serialized form will have field1, field2 and field3 (assuming all were set)
entity.serializeToByteBuffer(buffer);
.
.
.
// re-materialize an instance of NestedEntity from the buffer
NestedEntity entity = NestedEntity.create();
entity.deserializeFromByteBuffer(buffer);
 
// materialize an instance of AnotherNestedEntity
AnotherNestedEntity anotherEntity = AnotherNestedEntity.create();
anotherEntity.deserializeFromByteBuffer(buffer);
anotherEntity.getField1();
anotherEntity.getField3();
 
// serialize AnotherNestedEntity using serializeToXXX() methods
// - the serialized form will have field1, field2 and field3 even though AnotherNestedEntity only has field1 and field2
anotherEntity.serializeToByteBuffer(buffer);
 
// materialize an instance of YetAnotherNestedEntity
YetAnotherNestedEntity yetAnotherEntity = YetAnotherNestedEntity.create();
yetAnotherEntity.deserializeFromByteBuffer(buffer);
yetAnotherEntity.getField1();
yetAnotherEntity.getField2();