Project

General

Profile

Task #6021

Bug #6011: Synchronization fails from max_syncobjectqueue_size

Splunk will need to monitor Synchronize

Added by Robert Waltz over 10 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Robert Waltz
Category:
Environment.Production
Target version:
-
Start date:
2014-07-31
Due date:
% Done:

100%

Estimated time:
0.00 h
Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

parse this error from

/var/log/dataone/daemon/d1-processing-jsvc.err

java.util.concurrent.ExecutionException: java.lang.Exception: hzSyncObjectQueue has not had more than 50001 remaining capacity for 2 hrs.

simply looking for the string 'Exception: hzSyncObjectQueue' should be enough

If you want more precision, then there are variables in the string, so the string 'than 50001 remaining capacity for 2 hrs.' would have a regular expression like 'than \d+ remaining capacity for \d+ hrs.'

There is a good chance other critical errors will show up in that error log too.

History

#1 Updated by Robert Waltz over 10 years ago

  • Description updated (diff)

#2 Updated by David Doyle over 10 years ago

Added prod-cn-dataone index to cluster for this and other future dataone processing data.

Adding d1-processing-jsvc.err to inputs.conf on prod CNs via deployment server.
index = prod-cn-dataone
sourcetype = dataone
source = d1_processing_prod

#3 Updated by David Doyle over 10 years ago

  • % Done changed from 0 to 100
  • translation missing: en.field_remaining_hours set to 0.0
  • Status changed from New to Closed

Added scheduled search per parameters above. Search fires off once per minute over the last minute, emails Robert if more than 0 results, throttles for 30 minutes after alert.

Will go through exceptions in d1-processing-jsvc.err and send off likely candidates for future alerts to coredev for discussion.

#4 Updated by Robert Waltz about 10 years ago

  • Target version changed from 2014.30-Block.4.3 to CCI-1.4.2

#5 Updated by Robert Waltz about 10 years ago

  • Target version changed from CCI-1.4.2 to CCI-1.4.1

#6 Updated by Dave Vieglais about 10 years ago

  • Target version changed from CCI-1.4.1 to CCI-1.5.0

#7 Updated by David Doyle about 10 years ago

  • Milestone changed from CCI-1.4 to None
  • Estimated time set to 0.00

#8 Updated by Robert Waltz about 10 years ago

  • Target version changed from CCI-1.5.0 to CCI-1.5.1

#9 Updated by David Doyle about 10 years ago

  • Assignee changed from David Doyle to Robert Waltz

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)