In This Section
Abstract
The state of an X application is privately owned by the application and manipulated during the processing of messages by the application. However, there are certain applications in which the state needs to be sourced from and/or changes to state be recorded in external data sources such as an RDBMS or other such database system. This article discussed the mechanisms available in X and best architectural practices to integrate an X application's state with external data sources.
Summary
Gateway Apps
Gateway apps bridge X Platform and non-X Platform portions of an enterprise's application landscape.
As depicted above
- Data in an external source is copied/moved into an app's state repository using messaging via a gateway app
- Data in an app's state repository is copied/moved to an external data source either using messaging via a gateway app or via CDC (change data control) on the application's transaction log
Integrating External Source to App
Data is copied/moved from an external source to an X app state using a Gateway App.
Use Gateway App
The gateway app responsible for copying/moving data between an external data source to an X app would perform the following:
- Connect to the external data source on one side.
- Connect to messaging bus used by the app on the other side
- Use techniques specific to the external data source to read/receive data
- Convert the data received to state management (CRUD) messages understood by the app
- Send the messages to the app via the bus
- The app receives the messages and uses updates its state accordingly
Maven Sender Gateway Archetype
If using Maven as the build tool, such gateway apps can be easily created using the sender gateway maven archetype.
Integrating App to External Source
Data is copied/moved from X app state to an external data source using either a Gateway App or CDC.
Use Gateway App
The gateway app responsible for copying/moving data between an external data source to an X app would perform the following:
- Connect to the external data source on one side.
- Connect to messaging bus used by the app on the other side
- Receive state change messages published by the app
- Use techniques specific to the external data source to write the data in the messages to the external source
Maven Receiver Gateway Archetype
If using Maven as the build tool, such gateway apps can be easily created using the receiver gateway maven archetype.
Use CDC
The use of a gateway app to integrate your apps' state with an external source works well in situations where the gateway app can tap into the application's message stream to detect changes in state. In situations where an app's message stream does not clearly reflect the changes being made to the app's state, additional messages reflecting the changes to the application state would need to be published for the gateway app integration method to work. This can become cumbersome rather quickly and potentially pollute the architecture at play by leaking an application's private state to the outside world. In such situations, integrating the app's state with the external data source via CDC would be the preferred mechanism,
CDC Equivalence to integrate External Source to App?
The pollution of the architecture model and leaking of the application's private state to the outside world would also apply when integrating an external source to the application state via a gateway app as described above if one used a publish-subscribe bus to communicate between the gateway and the app. Therefore, it is recommended that, in application architectures that honor the privacy of application state, a dedicated point-point (direct) bus connection be used between the sending gateway app and the app. The use of such a bus just to feed state changes to the app would contain the visibility of the state changes only to the app and the gateway app.
X provides a CDC API using which one can effectively "tail" an application's transaction log. CDC operates with an awareness of log compaction thus resulting in no loss of data fed to a CDC processor when a log is compacted. When using CDC to integrate an app's state with an external data source, a CDC processor (authored by the user as a custom Talon or Eagle app) would perform the following:
- Connect to the external data source on one side
- Start a transaction log CDC runner on the other side
- The CDC runner would "tail" the log and dispatch entries from the log as and when new entries are added to the log
- On receipt of entries from the CDC runner, use techniques specific to the external data source to write the data in the received entries to the external source
Eagle CDC Runner
Eagle provides a CDC runner designed to CDC data to an RDBMS. It implements all the boilerplate code needed to CDC data to an RDBMS and can be used to integrate an app's state with an RDBMS
Change Conflation and Audit Trail
The CDC runner automatically conflates data in the transaction log. The runner dispatches both the conflated objects and the individual changes to objects to the CDC processor. For example, if the app makes 10 updates to a Customer object in memory, the CDC runner will dispatch all 10 changes as well as the final (conflated) state of the object to the CDC processor. Therefore, the CDC runner's change feed can be used as both an audit trail of changes to the app state objects as well as a conflated stream of coarse grained changes to the app state objects. Since most external data sources, such as RDBMSs, analytical cubes and centralized storage engines have a performance capability that is generally much less than what a X application can process, the ability to only push the conflated changes to objects to the external data source while the app processes all fine grained changes allows for the integration of the app's state and the external source possible and stable. Without this capability, the external source would result in pushing back on the app resulting in queueing of data somewhere in the system effectively causing either a resource exhaustion or a slow down of the app.