The figure below outlines the high level components that make up a Talon application.
Development Artifacts
From an application developer's perspective, authoring an X application involves:
- Modeling application messages.
- Modeling of application state (optional).
- Writing the event handlers that respond to messages, operate on state and emit outbound messages.
- Configuration that wires together applications via messaging.
The Application Data Modeler (ADM)
The application data modeler provides an XML based modeling language that generates messages and state as plain old java objects. ADM modeled objects are highly optimized and shield application developers from concerns around serialization, transport and persistence of these objects.
See Modeling Message and State
Application Event Handlers
From an application developer perspective, a Talon application is essentially a set of event handlers via @EventHandler
annotated methods that are bound together by an application entry point class. The application's event handlers are invoked in response to messages or notification events dispatched by the Talon runtime. Event Handlers are single threaded and generally must be non-blocking. A typical message handler will consume a message, perform business logic, may update state, and/or send outbound messages. The Talon runtime handles the underlying plumbing to ensure that the operations of receipt, state updates and outbound sends are atomic and reliable.
The following shows a minimal Talon application
For more information on writing event handlers, see the Talon Programming section.
Configuration
X Application code doesn't work directly with the underlying infrastructure components. For example, binding of an application to a particular message bus implementation is a configuration concern. The platform's Domain Descriptor Language (DDL) is the platform's configuration schema. It allows developers to configure a group of related applications in a singe xml-based configuration artifact. Deployment tools such as Robin use this configuration to seed the the platform's configuration repository, where it is then used by Talon VMs when they are launched.
Because the platform handles much of the non-functional operational aspects under the covers, many of the knobs it exposes don't have an impact on application code. Therefore, an application developer often won't need to work with much more than the configuration specified above.
See Understanding Configuration
Runtime Components
Atomic Event Processor (AEP)
Atomic Event Processing refers to the transactional event processing paradigm in Talon application that an AepEngine implements. An AepEngine sits at the heart of every Talon application. An AepEngine orchestrates the processing of inbound messages, business logic processing, replication and persistence of state changes and outbound messaging. This relieves application event (message) handlers from the non-functional concerns of interfacing with messaging providers, persistence layers and transaction coordination.
Simple Messaging Abstraction (SMA)
Talon's Simple Messaging Abstraction layer provides a generic messaging API that encapsulates the set of messaging primitives required by the platform. This allows the platform to inter-operate seamlessly with any messaging provider by proving binding implementations that are simple to write. At a high level, SMA exposes the concepts of buses and channels of an application. An application sends and receives messages over bus channels which are mapped via configuration to an underlying messaging provider destination.
Operational Data Store (ODS)
AEP engines rely on Talon's Operational Data Store. An ODS Store provides HA capabilities for an application via memory-memory multi-peer replication, disk-based persistence, or both. An ODS store provides high performance asynchronous transaction facilities that an AepEngine uses in conjunction with SMA to achieve Atomic Event Processing.
Important ODS Concepts include:
- Replication: The primary means of achieving persistence for an ODS Store is by pipelined, memory-memory replication of transactions to one or more hot backup peers. If a primary application instance fails the backup takes over with no loss of data.
- Transaction Logs: An ODS Store can also log to binary transaction logs as a secondary persistence mechanism if the connection to the backup is lost or a cold restart is required.
- Change Data Capture (CDC): To allow asynchronous yet transactionally consistent syphoning of application state to back end or legacy systems, ODS supports the ability to perform Change Data Capture with a simple callback-based mechanism to push state changes.
- Inter Cluster Replication (ICR): Inter-Cluster Replication allows replication of the store's recovery stream to another data center over messaging, providing an asynchronous and transactional consistent disaster recovery mechanism.
See Application Storage.
XVM
A Talon XVM serves as a container for Talon micro apps and provides management capabilities for the XVMs that it contains. Key features of the Talon XVM include:
- Lifecycle management of applications.
- Alerting and Lifecycle Event emission.
- Emission of heartbeats that contain detailed system, platform and application exposed statistics for low-impact remote application monitoring.
- Emission of application trace for low-impact remote application monitoring.
- Command and Control; XVMs provide capabilities for exposing and invoking commands that are out of band with application messaging.
See The XVM
Discovery
The platform's discovery facilities provides plugin mechanisms for ODS store's to discover one another, and for tooling to discovery running instances of applications. Out of the box, the platform supports discovery of multicast and over messaging.
Application Flow
From an application perspective Talon's application flow is not dissimilar to many other event or message processing architectures: the application exposes message handlers to the platform which in turn passes inbound messages to it for processing. In the act of processing the inbound event, the application will make changes to its state and send some outbound events. The key difference between the Talon architecture and traditional architectures is that application state is stored in memory and resiliency is provided not by synchronous persistence to disk or a data grid, but instead by streaming state changes to a backup's memory in an asynchronous, pipelined fashion. The combination of in memory state and asynchronous 'persistence' allows Talon to operate at extreme performance levels without sacrificing on reliability.
Key aspects of any streaming application platform that are of critical concern for stream oriented applications are:
- Exactly once processing of inbound and outbound messages.
- Atomicity between state updates and the messaging stream: A trading application that thinks it has sent a request to buy 100,000 shares of IBM must be able to rely on that being what is actually sent out. This is particularly important in application or machine failure scenarios. If the application fails after sending out the request to buy 100,000 IBM and the application were to recover and reprocess the event this time buying 200,000 shares it could turn into a costly business problem.
In traditional application architectures the problems of state/messaging atomicity and exactly once processing are often solved using distributed transactions. For example, in J2EE state changes would be committed to a databases and messaging sent over JMS with XA transactions used to coordinate commit on both. Such schemes involve a lot of overhead and kill performance. The application flow in Talon provides the same level of reliability without the synchronous overhead of distributed transaction coordination. The following diagram depicts the application flow in a Talon application:
Elaborating on the diagram above, Talon ensures the same level of reliability as traditional architectures as follows:
- Talon receives an inbound message
- It dispatches the message to the application
- Application updates state (monitored by the Talon)
- Application sends Messages (through Talon)
- Talon effectively replicates the state changes and outbound messages to one or more hot backup instances via memory to memory replication.
- Once the state and oubound effects (messages) are stabilized on a backup (via an asyncrhonous stablitity acknowledgement), outbound messages are released.
- Acknowledgement of outbound messages allows cleanup up indoubt state
- Acknowledgement of inbound message which has not been fully processed.
Key Takeaways
- Stabilization of state changes and outbound effects to the backup before sending outbound messages ensures that in the event of process or machine failure that the backup has the same conception of state as the former primary communicated externally (e.g. I have an outstanding request to buy 100,000 shares of IBM).
- Acknowledgement of the inbound message after replication to the peer ensures that in the event of failure a duplicate can be detected to prevent duplicate dispatch to the application if the message is re-transmitted.
- The entire flow above is pipelined. The application can begin processing the next inbound message before receiving stability from the backup. Nowhere in the flow is there a need to block or perform synchronous operations.
- State and outbound messages can be journalled to a transaction log on disk, but it is not fsync'd and is not a primary recovery mechanism. Disk based journalling is used more for operational tasks, but can also serve as a backup recovery mechanism.
- And best of all ... the above is done transparently to the application which need only concern itself with writing the business logic in message handlers.