Add instrumentation to the CNs for realtime monitoring
Diagnostics on the CNs requires a lot of log grepping, and watching balances between processes is for the most part quite challenging.
The goal of this story is to implement lightweight, configurable instrumentation options to key processes operating on the coordinating nodes so that the state of various services can be visualized in near real time. This is different to the capability offered Nagios which will continue to offer monitoring and alert services.
Instrumentation will take the form of Ganglia  for the real time recording and history display, and JMXetric  for instrumentation. Ganglia will be run on monitor.dataone.org.
JMXetric offers several choices for instrumentation that should be easily integrated into our current environment without being invasive. When properly setup, annotations can be used to indicate methods that should be timed for example.
#3 Updated by Dave Vieglais over 9 years ago
- Status changed from New to In Progress
Using StatsD and Graphite, installing as new VM running on the KU Host hardware.
ip is 188.8.131.52
Sending new metrics for reporting is very simple using UDP. Example in bash:
if [ $# -ne 1 ]
echo "Syntax: $0 ''"
Setup UDP socket with statsd server¶
exec 3<> /dev/udp/$host/$port
printf "$1" >&3
Close UDP socket¶