Overview

Of Talon's two HIgh Availability models, Event Sourcing requires more discipline to use. In event sourcing replication of application state to a backup is done by replicating only the applications inbound messaging stream. The idea is that by replaying the exact same sequence of events through the same application business logic, the application will produce the same changes to state and produce the same set of outbound messages. Event Sourcing has several performance advantages that are of particular interest to applications concerns with very low latency. With event sourcing, replication is much cheaper because:

1. Received messages don't require re-serialization before replication (they are received in serialized form)
2. Received messages can be replicated in parallel with execution of business logic which can effectively reduce the storage cost to 0. 

While this premise Event Sourcing is simple, it requires discipline on the part of the application developer to ensure that no business logic decisions are made based on any data but that in the inbound messages and the state generated by those message to date, otherwise it would lead to divergent state on the primary and backup members. For example, something as simple as making a logical decision based on System.currentTimeMillis() could lead to the primary instance accepting an order and a backup instance rejecting an order as stale because of clock skew. In the event that the primary instance fails, the order would then not exist on the backup. 

In addition to the risk of state divergence, the Event Sourcing model also has the disadvantage that to recover state, the entire input stream of events needs to be replayed. That means that the longer the application has been running, the longer the recovery time for the application will be from a cold start presuming it was receiving a steady stream of messages. There is a feature on the Talon road map to provide the ability to checkpoint an application's state to reduce recovery time, but it is another consideration that should be taken into account before electing to use the event sourcing model. 

Talon provides several tools to assist applications using Event Sourcing avoid situations such as these, but in general developers should weigh the complexity costs vs. performance requirements. 

Coding For Event Sourcing

A basic event sourcing application is straightforward to create and working with event sourcing is very similar to Working with State Replication with the exception that the application does not need to expose a state factory to the underlying AepEngine. You application state does not event need to be modeled via ADM, because the underlying engine and store does not need to be aware of it. 

A quick way to get started is to use the nvx-talon-es-processor-archetype described in Maven Archetype Quick Starts. The general flow for creating a event sourcing application involves:

• Modeling Messages and State
• Declaring a main class annotated for EventSourcing
• Writing message handlers to perform business logic. 

A Basic App

Model Application Messages

The steps outlined below assume that you have already modeled some messages for your application to use. See the Modeling Message and State sections to get started. 

Annotate App Main for State Replication

Inject Message Sender

If your application will send messages, it can add an injection point for the underlying AEPEngine to inject a message sender.

Declare Message Handlers

When working with an application using EventSourcing you can annotate handler methods that accept the received message type with an EventHandler. With event sourcing you do not need to to make Talon aware of your application state it remains completely private to your application. 

Keep in mind that:

• All message handlers in your application are executed in a single thread ... there is no need for synchronization. 
• You must not do reference any data from the local environment other than your application state when executing business logic that will impact your recoverable state or the outbound messages.

See Preventing Divergence below for some pointers on avoiding divergence. 

Configuring the Engine

Register Messaging Factories

When working with state replication both the ADM and message and state object factories need to be registered with the runtime. Registering the state factory allows the underlying state replication machinery to deserialize replicated state objects based on the ids encoded in the replication stream. The message factories allow are used for deserializing replicated outbound messages as well as messages received from message buses. The state factories can be declared in your config.xml or programmatically:

Declaration in Config DDL

Programmatic Registration

Enable Storage

To actually achieve high availability storage must be configured for the application. The primary means of storage is for Talon apps is through clustered replication to a backup instance. Talon also logs state changes to a disk based transaction log as a fallback mechanism. Storage and persistence can be enabled in the application's configuration xml.

Enabling clustering allows 2 applications of the same name to discover one another and form an HA cluster. When one or more instances of an application connect to one another one instance is elected as the primary via a leadership election algorithm. The primary member will establish messaging connections and begin invoking message handlers in your application. 

Enabling persistence causes the replication stream that is sent to backup instances to also be logged locally on disk to a transaction log file. The transaction log file can be used to recover application state from a cold start, and is also used to initialize new cluster members that connect to it so when clustering is enabled, persistence must be enabled as well. 

There are many configuration knobs that can be used to customize the store's behavior and performance. See DDL Config Reference for a listing of configuration knobs for storage.

Preventing Divergence

Event Sourcing applications must take special care not to reference the local environment to avoid situations that would cause primary and backup state to diverge. Talon provides several facilities to help with this. 

AepEngine Time

The AepEngine provides a getEngineTime() that is intended for use by applications using event sourcing that do time dependent message processing. This method provides the current wall time (in millisecond resolution) as perceived by the engine. If invoked from within a message processor handler and the HA policy is set to event sourcing, this method returns the time stamped on the message event (stamped just before the method is dispatched to the application for processing). Since, for event sourced applications, the message is also replicated for parallel processing on the backup, the backup will receive the same time when invoking this method thus ensuring identical processing. If this method is called on an engine operating in state replication mode or called from outside a message processor, then this method will return the value returned by the java.lang.System.currentTimeMillis method 

Message Injection

Because event sourcing applications replicate the inbound message stream to backup instances, it is possible for an application to inject a message into the processing stream to allow HA processing. This can be used in several ways

• To schedule deferred processing against application state in a fault tolerant fashion. For example if you need to schedule work to be done at a later date, you can inject a message with a delay and it will be queued for deferred execution.
• If processing of a message must be done by another thread, the results of that thread's processing can be injected back into the engine with a message. 
• If you are working with a 3rd party library or other application locally that does processing that may be based on the local environment, you add the results of that processing to message and inject into back into the engine for fault tolerant processing. In this case you might also consider using environment based replication (see below). 

In the contrived example below, injection is used to broadcast the current hostname in a HA safe fashion. 

The above technique can be used to 

See Scheduling and Injection of Messages for more information.

Environment Replication

Above we showed how to inject the host name from the local environment into the replication stream, but that has the drawback of creating a new transaction. Talon provides a mechanism to record and tunnel local environment data into the current transactions replication stream. This feature is called environment replication and allows application developers to write environment providers that record calls into buffers that are serialized for use on the backup so that they can operate on the same data as the primary. With environment replication the above logic would look like:

See Environment Replication for more details and authoring environment providers.