Task #6021
Bug #6011: Synchronization fails from max_syncobjectqueue_size
Splunk will need to monitor Synchronize
100%
Description
parse this error from
/var/log/dataone/daemon/d1-processing-jsvc.err
java.util.concurrent.ExecutionException: java.lang.Exception: hzSyncObjectQueue has not had more than 50001 remaining capacity for 2 hrs.
simply looking for the string 'Exception: hzSyncObjectQueue' should be enough
If you want more precision, then there are variables in the string, so the string 'than 50001 remaining capacity for 2 hrs.' would have a regular expression like 'than \d+ remaining capacity for \d+ hrs.'
There is a good chance other critical errors will show up in that error log too.
History
#1 Updated by Robert Waltz over 10 years ago
- Description updated (diff)
#2 Updated by David Doyle about 10 years ago
Added prod-cn-dataone index to cluster for this and other future dataone processing data.
Adding d1-processing-jsvc.err to inputs.conf on prod CNs via deployment server.
index = prod-cn-dataone
sourcetype = dataone
source = d1_processing_prod
#3 Updated by David Doyle about 10 years ago
- % Done changed from 0 to 100
- translation missing: en.field_remaining_hours set to 0.0
- Status changed from New to Closed
Added scheduled search per parameters above. Search fires off once per minute over the last minute, emails Robert if more than 0 results, throttles for 30 minutes after alert.
Will go through exceptions in d1-processing-jsvc.err and send off likely candidates for future alerts to coredev for discussion.
#4 Updated by Robert Waltz about 10 years ago
- Target version changed from 2014.30-Block.4.3 to CCI-1.4.2
#5 Updated by Robert Waltz about 10 years ago
- Target version changed from CCI-1.4.2 to CCI-1.4.1
#6 Updated by Dave Vieglais about 10 years ago
- Target version changed from CCI-1.4.1 to CCI-1.5.0
#7 Updated by David Doyle about 10 years ago
- Milestone changed from CCI-1.4 to None
- Estimated time set to 0.00
#8 Updated by Robert Waltz about 10 years ago
- Target version changed from CCI-1.5.0 to CCI-1.5.1
#9 Updated by David Doyle about 10 years ago
- Assignee changed from David Doyle to Robert Waltz
