Property that can be used to control the default sampling size for series stats.

If the number of datapoints collected in a stats interval exceeds this size, the computation for histographical data will be lossy, increasing . Increasing the value will reduce reduces loss of datapoints, but results in greater overhead in stats collection in terms of both memory usage and pressure on the process caches.

Property that enabled message latency stats tracking.

When set to true, timings for messages are captured as they flow through the system. Enablement of these stats is required to collect message bus latency stats. Enabling this property can increase latency due to the overhead of tracking timestamps.

Property that enabled per message type latency stats tracking.

When set to true, timings for each message type are individually tracked as separate stats. This can be useful in tracking down issues in which a particular message type is problematic (for example, tracking down a high application handler message processing time). However, it results in a higher overhead.

Indicates whether or not timestamps should be stamped on inbound and outbound messages. Disabled by default.

Enabling this setting will allow engines to provide more detail in the form of transaction legs in message latency statistics.

The interval (in seconds) at which engine stats will be traced for a given engine.

Can be set to a postive positive integer indicate the period in seconds at which the engine's stats dump thread will dump recorded engine statistics. Setting a value of 0 disables creation of the stats thread.

When enabled, engine stats are traced to the logger 'nv.aep.engine.stats' at a level of Tracer.Level.INFO; therefore, so to see dumped stats, a trace level of 'nv.aep.engine.stats.trace=info' must be enabled.

NOTE:: disabling the engine stats thread only stops stats from being periodically traced. It does not stop the engine from collecting stats ... ; stats can still be collected by an external thread (such as the Talon Server which reports the stats in server heartbeats). In other words, enabling the stats thread is not a prerequisite for collecting stats, and disabling the stats reporting thread does not stop them from being collected.

NOTE: that while  while collection of engine stats is a zero garbage operation, tracing engine stats is not a zero garbage when performed by this stats thread. For latency sensitive apps, it is recommended to run in a Talon server which can collect engine stats and report them in heartbeats in a zero garbage fashion.

The interval (in seconds) at which engine sys stats will be reported. Set to 0 (the default) to completely disable sys stats tracing for a given engine.

In most cases, AEP sys stats will not be used and system level stats would be recorded in the Server Statistics] from which an AEPEngine is running.

The interval (in seconds) at which multiplexer stats will be traced.

Multiplexer stats can also be reported as part of the overall engine stats from the engine stats thread, so there is no need to set this to a non-zero value if nv.aep.<engine>.stats.interval is greater than zero.

The interval (in seconds) at which message latency stats are traced.

This setting has no effect if nv.msg.latency.stats is false. This allows granular tracing of just message latency stats on a per bus basis. Message latency stats can also be reported as part of the overall engine stats from the engine stats thread, so there is no need to set this to a non-zero value if nv.aep.<engine>.stats.interval is greater than zero.

Metric

Description

NumFlows

Total number of message flows functioning in the engine.

NumMsgsRcvdBestEffort

Total number of messages received by the engine on best-effort channels.

NumMsgsRcvdGuaranteed

Total number of messages received by the engine on guaranteed channels.

NumMsgsSourced

[EventSourcing Only]
Total number of messages sourced from from the recovery log or primary agent in a cluster during agent inialization.

NumDupMsgsRcvd

Total number of duplicate messages received and discarded by an engine. This metric will always be 0 if duplicate detection has been disabled via the nv.aep.duplicate.checking configuration property.

NumMsgsSentBestEffort

Total number of messages sent by the engine on best-effort channels.

NumMsgsSentGuaranteed

Total number of messages sent by the engine on guaranteed channels.

NumMsgsResent

Total number of messages retransmitted by an engine.

When a backup agent assumes the role of a primary on failover, it restransmits all in-doubt messages as part of its first transaction. In-doubt messages are those messages for which positive acknowledgements have not been received from the downstream messaging components. This metric records the number of such restransmitted messages.

NumEventsRcvd

Total number of events received by an engine.

Events received by an engine include message events and the non-message events, e.g. 'channel up event', 'channel down event', etc.

NumFlowEventsRcvd

Total number of flow events received by the engine.

Flow events are synonymous with message events. These include message events received on message channels and message events injected for processing into the engine by the application.

NumFlowEventsProcSuccess

Total number of successfully processed flow events.

A flow event is considered successfully processed if the application did not throw an exception while processing the message associated with the event.

NumFlowEventsProcFail

Total number of failed flow events.

A flow event is considered to have failed processing if the application threw an unchecked exception while procesing the message associated with the event.

NumFlowEventsProcComplete

Total number of flow events whose transactions have completed.

Each successfully processed flow event participates in an AEP transaction. This metric counts the number of successful flow events whose transactions have completed.

NumTransactions

Total number of transactions that have been committed or rolled back.

Note: This metric is equal to NumCommitsStarted + NumRollbacks

NumCommitsStarted

Total number of transactions whose commits have been started.

Note: Transactions are committed in a pipelined manner. Therefore, there can be multiple transactions in the commit pipeline. This metric counts the number of transactions that have been entered into the commit pipeline.

See NumCommitsCompeted

NumCommitsCompleted

Total number of transactions whose commits have completed.

This metric counts the number of transactions that have completed and exited the transaction commit pipeline. The difference between this and the NumCommitsStarted metric is the number of transactions in flight at any point in time.

See NumCommitsStarted

NumSendCommitsStarted

Total number of transactions whose send commits have been started.

A transaction commit pipeline is comprised of a send commit pipeline (the commit of outbound messages in the transaction) and a store commit pipeline (commit to store changes). This metric counts the number of transactions in the send commit pipeline.

NumSendCommitsCompleted

Total number of transactions whose send commits have completed.

This metric counts the number of transactions whose send commits have completed and exited the send commit pipeline.

SendCommitCompletionQueueSize

Number of transaction in the send commit completion queue.

Transactions that complete the send portion of the commit are passed through a send commit completion queue for sequencing before entering into the next phase of a transaction commit. This metric counts the number of transactions held in the post send commit sequencing queue at any point in time.

NumStoreCommitsStarted

Total number of transactions whose store commits have been started.

A transaction commit pipeline is comprised of a send commit pipeline (the commit of outbound messages in the transaction) and a store commit pipeline (commit to store changes). This metric counts the number of transactions in the store commit pipeline.

NumStoreCommitsCompleted

Total number of transactions whose store commits have completed.

This metric counts the number of transactions whose store commits have completed and exited the store commit pipeline.

StoreCommitCompletionQueueSize

Number of transaction in the store commit completion queue.

Transactions that complete the store portion of the commit are passed through a store commit completion queue for sequencing before entering into the next phase of a transaction commit. This metric counts the number of transactions held in the post store commit sequencing queue at any point in time.

NumRollbacks

Number of transactions that have been rolled back.

BackupOutboundQueueSize

[EventSourcing Only]
Number of messages in a backup's outbound queue.

A backup outbound queue holds outbound messages sent during concurrent processing by a backup agent in an Event Sourced cluster. The messages are held in the queue as in-doubt messages until notifications are received from the primary 'acknowledging' the receipt of the messages from downstream messaging components/agents. The messages are flushed from the queue upon receipt of such notifications. This metric counts the number of messages in a backup's outbound queue.

Note: The notifications are piggy-backed on other replication traffic to avoid extra network traffic. Therefore, in the absence of replication traffic, messages can remain in the backup's outbound queue even though acknowledged by the downstream messaging system until some replication traffic occurs to replicate the downstream ack notification.

BackupOutboundLogQueueSize

[EventSourcing Only]
Number of messages in a backup's outbound log queue.

A backup's outbound log queue holds outbound messages for the course of a single transaction. The queue is flushed to the outbound message log upon completion of the transaction. This metric holds the number of messages in the outbound log queue.

OutboundSno

The current outbound sequence number in use by the engine.

Each engine instance maintains a sequence number space for messages outbound by the engine. This metric holds the current outbound sequence number.

OutboundStableSno

The current 'stable' outbound sequence number.

An outbound is considered to be stable when a positive acknowledgement is received for the message from the downstream messaging system. This metric holds the sequence number of the last stable outbound message.

Message Type Specific Statistics

Message type specific statistics.

The engine also (optionally driven by configuration) maintains statistics for each of the different message types flowing through the engine. The following are the different metrics collected for each message type:

• NumMsgsRcvdBestEffort
• NumMsgsRcvdGuaranteed
• NumMsgsSourced
• NumDupMsgsRcvd
• NumMsgsSentBestEffort
• NumMsgsSentGuaranteed
• NumMsgsResent
  
  The semantic meaning of these metrics for each message type is identical to metrics with the same name described above, except that it is local to the message type.

Time spent in AepEngine.sendMessage().

The time in the AepEngine's send call. This latency will be a subset of mproc for solicited sends and it includes msendc.

Time spent in the AepEngine's core send logic.

This leg includes enqueuing the message for delivery in the corresponding bus manager.

Time spent from the point where transaction commit is started to send or store commit, whichever is occurs first.

This latency measures the time taken in any bookkeeping done by the engine prior to commit the transaction to store (or for an engine without a store until outbound messages are released for delivery).

The send commit latency: i.e. time from when send commit is initiated, to receipt of send completion event.

This latency represents the time from when outbound messages for a transaction are released to the time that all acknowledgements for the messages are received.

Because this latency includes acknowledgement time a high value for csend does not necessarily indicate that downstream latency will be affected. The Message Latencies listed below allow this value to be decomposied further.

Time spent from the point the store commit completes to the beginning of the send commit which releases a transaction's outbound messages for delivery.

If the engine doesn't have a store, then this statistic is not captured as messages are released immediately.

The store commit latency i.e. time from when store commit is initiated to receipt of store completion event.

This latency includes the time spent serializing transaction contents, persisting to the store's transaction log, inter cluster replication, and replication to backup members including the replication ack.

High values in cstore will impact downstream message latencies because store commit must complete before outbound messages are released for delivery. The cstore latency is further broken down in the Store Latencies listed below.

Records latencies for the first transaction processing leg.

Transaction Leg One includes time spent from the point where the first message of a transaction is received to submission of send/store commit. It includes message processing and and any overhead associated with transactional book keeping done by the engine.

Each transaction leg is a portion of the overall commit time that is processed on the Aep Engine's thread. The sum of the transaction leg stats are important in that they determine the overall throughput that an application can sustain in terms of transactions per second.

tleg2

Records latencies for the second transaction processing leg.

Transaction Leg Two includes time spent from the point where the send/store commit completion is received to the submission of store/send commit.

Each transaction leg is a portion of the overall commit time that is processed on the Aep Engine's thread. The sum of the transaction leg stats are important in that they determine the overall throughput that an application can sustain in terms of transactions per second.

Records latencies for the third transaction processing leg.

Transaction Leg Three includes time spent from the point where the store/store commit completion is received to the completion of the transaction commit.

Each transaction leg is a portion of the overall commit time that is processed on the Aep Engine's thread. The sum of the transaction leg stats are important in that they determine the overall throughput that an application can sustain in terms of transactions per second.

Records latencies for receipt of a message to transmission of the last outbound message.

NumMsgsInBatches

The number of batch message received by the bus.

The total number of batch messages enqueued for delivery by this bus.

The total number of batch messages enqueued for delivery by this bus.

The number of times this bus has been synchronously flushed.

The number of messages flushed by asynchronous flushes.

The number transactions rolled back for the bus.

The amount of time spent serializing transaction entries in preparation of replication / and persistence.

The time between serializing transaction entries to the last entry being written to the wire (but not stabilized) for replication.

 A high value here can indicate that there is network pushback from backup members.

The time between serializing transaction entries to the time that entries have been passed to the persister for write to disk (but not yet synced).

 A high value here can indicate that there is disk pushback or that there is overhead in preparing entries for persistence or ICR.  

The commit wire time.

For a store in the primary role, wire latency captures the time from the last commit entry being replicated until the last commit ack is received for the transaction. In other words: the round trip time.

For a store in the backup role wire latency capture the amount of time from when the primary wrote the last commit entry to the wire to the time it was received by the backup. When primary and backup are on different hosts, this statistic is subject to clock skew and could even be negative.

 A high value for 'w' on a Primary member may indicate a slow replication connection or can indicate that there is a bottleneck occurring in a backup member. If s2w is low, the latter case is more likely and backup stats should be consulted. A high value for 'w' is not necessarily a problem for applications that are more concerned with throughput provided there is enough bandwidth, but a high value can impact downstream message latency. For a backup member (discounting clock skew) a high value would indicate latency in the replication connection.

The amount of time spent persisting transaction entries to disk.

A high value here indicates that disk is a bottleneck.

The amount of time spent sending transaction entries to via an ICR Sender.

A high value here indicates that intercluster replication is a bottleneck.