The Talon Manual

Versions Compared

Old Version 1

changes.mady.by.user Unknown User (colin)

Saved on

compared with

New Version 2

changes.mady.by.user Unknown User (dave.strom)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

An operational XVM continuously collects raw statistics during the course of its operation. The xvm can also be configured to spin up a background thread that periodically performs the following:

  1. Perform Performs higher level statistical computations such as calculate message rates and average latencies.
  2. Emits heartbeat messages to be processed by handlers.
  3. Optionally outputs server statistics to named trace loggers using the platform's tracing and logging system.
  4. Optionally writes heartbeat messages containing useful server-wide statistics to a binary transaction log (with zero steady-state allocations).
  5. Detect Detects violation/abatement of alert thresholds and dispatch appropriate alert events.

The raw metrics are collected by the server are used by the background statistical thread for its computations and can also be retrieved programatically programmatically by an application for its own use.
In this document, we describe:

  • Describe the various raw metrics captured by the server,
  • Describe how to switch on and configure the statistics thread operation,
  • Describe the higher level statistics calculations performed by the statistics thread,
  • Describe and the format of the output of the statistics thread.

...

When heartbeats are enabled via the nv.server.stats.enable and nv.server.stats.interval properties they can be handled in several ways:, discussed below.

Tracing Heartbeats

By default all server statistics tracers are disabled as trace logging is not zero garbage and introduce introduces cpu overhead in computing statistics. While tracing heartbeats isn't recommended in production, enabling server statistics trace output can be useful for debugging and performance tuning. To enable you will need to configure the appropriate tracers at the debug level. See the Output Trace Loggers section for more detail.

...

(warning) Note that at this time binary heartbeat logs do not support rolling collection ... , so at this time the built-in hearbeat heartbeat logging mechanism is not suitable for long running application instances.

...

  1. Compute higher level statistics such as metric averages.
  2. Output raw and computed statistics to any trace loggers configured at the debug level.
  3. Emit heartbeat messages consumed by handlers and the heartbeat log, if enabled.

The statistics thread can be enabled or disabled via the configuration parameters.

...

To reduce the size of heartbeats, Pool Stats for a given pool are only included when:

...

The server statistics thread uses several named trace loggers to log the raw and computed statistics output. The following is the list of the loggers used by the server stats thread:.

  • nv.server.stats.sys.trace – tracer for System Statistics
  • nv.server.stats.thread.trace – tracer for Thread Statistics
  • nv.server.stats.pool.trace – tracer for Pool Statistics
  • nv.server.stats.app.trace – tracer for Application and Engine Statistics
  • nv.server.stats.userstats.trace – tracer for User Defined Statistics

If you would like to see server statistics trace output, the appropriate tracers need to be configured at the debug level. For example, too to enable System Stats trace output:

...

Appears in trace output when nv.server.stats.enable=true and nv.server.stats.app.trace=debug. 

The below example is output from the Tick To Trade sample application available in GitHub.

Panel
[App (ems) Engine Stats]
MRCV RATE DRATE MRCVB RATE DRATE MRCVG RATE DRATE MSND RATE DRATE MSNDB RATE DRATE MSNDG RATE DRATE EVNT RATE DRATE FEVNT RATE DRATE TXN RATE DRATE TXNSZ RLBK
106K 613 92 106K 613 92 0 0 0 106K 613 92 106K 613 92 0 0 0 320K 1.8K 278 106K 613 92 106K 613 92 0 0
...Disruptor {[0of1,024] 0% (MultiThreadedSufficientCores, BusySpin)}
...Feeder Queues{
......X-Server-ems1-Main (aff=[]): size=0, decongestCount=284178
......X-ODS-StoreEventMultiplexer-1 (aff=[3(s0c3t0)]): size=0, decongestCount=284164
......X-Server-ems1-IOThread-1 (aff=[8(s0c11t0)]): size=0, decongestCount=284162
......X-AEP-BusManager-IO-ems.market (aff=[6(s0c9t0)]): size=0, decongestCount=284160
......X-STEMux-ems-2 (aff=[2(s0c2t0)]): size=0, decongestCount=284160
......X-Client-LinkManagerReader[298b29db-82ae-4cc7-97b7-1143a417e86c] (aff=[7(s0c10t0)]): size=0, decongestCount=284159
...}
[App (ems) User Stats]
...Gauges{
......EMS Messages Received: 142604
......EMS Orders Received: 35651
...}
...Series{
......[In Proc Tick To Trade(sno=35651, #points=150, #skipped=0)
.........In Proc Tick To Trade(interval): [sample=150, min=72 max=84 mean=75 median=75 75%ile=77 90%ile=79 99%ile=83 99.9%ile=84 99.99%ile=84]
.........In Proc Tick To Trade (running): [sample=35651, min=72 max=20008 mean=93 median=76 75%ile=82 90%ile=111 99%ile=227 99.9%ile=805 99.99%ile=11975]
......[In Proc Time To First Slice(sno=35651, #points=150, #skipped=0)
.........In Proc Time To First Slice(interval): [sample=150, min=85 max=98 mean=88 median=88 75%ile=90 90%ile=92 99%ile=95 99.9%ile=98 99.99%ile=98]
.........In Proc Time To First Slice (running): [sample=35651, min=84 max=44695 mean=249 median=88 75%ile=95 90%ile=133 99%ile=283 99.9%ile=36287 99.99%ile=41439]
...}
[App (ems) Store Binding Stats]
CSND RATE DRATE CESND RATE DRATE CRCV RATE DRATE CERCV RATE DRATE CCSND RATE DRATE CCRCV RATE DRATE SIZE
106K 613 88 106K 613 88 0 0 0 0 0 0 0 0 0 106K 613 88 0
[App (ems) Store Binding Persister Stats]
<?>
...no persister
[App (ems) Store Binding ICR Sender Stats]
<?>
...no ICR sender
[App (ems) Bus Binding (market)]
<Detached ,Disruptor {[0of1,024] 0% (SingleThreaded, BusySpin)}>
MRCV RATE DRATE MRCVI RATE DRATE MRCVB RATE DRATE MENQ RATE DRATE MSND RATE DRATE FLS RATE DRATE MFLS RATE DRATE FLA RATE DRATE MFLA RATE DRATE
71302 408 57 0 0 0 0 0 0 35651 204 28 35651 204 28 71302 408 57 35651 204 28 0 0 0 0 0 0
STB RATE DRATE STBIB RATE DRATE STBB RATE DRATE PRCV RATE DRATE COM RATE DRATE RLBK CLNTS CHNLS FAILS
0 0 0 0 0 0 0 0 0 0 0 0 35651 204 28 0 0 2 0
[App (ems) Bus Binding (control-fb0a1b20-3d8f-11e6-abfe-0002c9faaac2)]
<Attached>
MRCV RATE DRATE MRCVI RATE DRATE MRCVB RATE DRATE MENQ RATE DRATE MSND RATE DRATE FLS RATE DRATE MFLS RATE DRATE FLA RATE DRATE MFLA RATE DRATE
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
STB RATE DRATE STBIB RATE DRATE STBB RATE DRATE PRCV RATE DRATE COM RATE DRATE RLBK CLNTS CHNLS FAILS
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0
[App (ems) Bus Binding (ems)]
<Attached>
MRCV RATE DRATE MRCVI RATE DRATE MRCVB RATE DRATE MENQ RATE DRATE MSND RATE DRATE FLS RATE DRATE MFLS RATE DRATE FLA RATE DRATE MFLA RATE DRATE
35651 204 28 0 0 0 0 0 0 71302 408 56 71302 408 56 71302 408 56 71302 408 56 0 0 0 0 0 0
STB RATE DRATE STBIB RATE DRATE STBB RATE DRATE PRCV RATE DRATE COM RATE DRATE RLBK CLNTS CHNLS FAILS
0 0 0 0 0 0 0 0 0 0 0 0 71302 408 56 0 0 2 0

...