Project

General

Profile

Bug #3549

jsvc logging error and out files grow too large

Added by Robert Waltz almost 9 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
David Doyle
Category:
d1_process_daemon
Target version:
Start date:
2013-02-06
Due date:
2014-04-12
% Done:

100%

Milestone:
CCI-1.3
Product Version:
Story Points:
Sprint:

Description

look at how we redirect io from jsvc.

play around with StdOutErrLog. maybe redirect the stdout/err to nothing, ignore it.

also see http://gabenell.blogspot.com/2009/11/working-around-jsvcs-logging.html

http://www.mooreds.com/wordpress/archives/332

History

#1 Updated by Robert Waltz almost 9 years ago

  • Description updated (diff)

#2 Updated by Robert Waltz almost 9 years ago

  • Target version changed from 2013.6-Block.1.3 to 2013.10-Block.2.1
  • Due date changed from 2013-02-06 to 2013-03-16

#3 Updated by Robert Waltz over 8 years ago

  • Due date changed from 2013-03-16 to 2013-06-22
  • Target version changed from 2013.10-Block.2.1 to 2013.24-Block.3.4

#4 Updated by Chris Jones over 8 years ago

  • Due date changed from 2013-06-22 to 2013-08-24
  • Assignee changed from Robert Waltz to David Doyle
  • Target version changed from 2013.24-Block.3.4 to 2013.33-Block.4.4
  • Milestone changed from CCI-1.1.2 to CCI-1.3

Log files for jsvc in /var/log/dataone/daemon, etc. get too large. Incorporate them into log rotate's schedule.

#5 Updated by David Doyle about 8 years ago

  • Target version changed from 2013.33-Block.4.4 to 2014.2-Block.1.1
  • Due date changed from 2013-08-24 to 2014-01-18

#6 Updated by David Doyle about 8 years ago

The following files have been added to /etc/logrotate.d on the sandbox CNs, with rollout to the remaining environments happening this evening.

(EDIT 2/10/2014: added dataone-processing-jsvc-err below.)


dataone-index-jsvc:

/var/log/dataone/index/*-jsvc.err {
size 50M
copytruncate
compress
delaycompress
rotate 52
}

dataone-processing-jsvc:

/var/log/dataone/daemon/d1-processing-jsvc.log {
size 50M
copytruncate
compress
delaycompress
rotate 52
}

dataone-processing-jsvc-err

/var/log/dataone/daemon/d1-processing-jsvc.err {
size 50M
copytruncate
compress
delaycompress
rotate 52
}

#7 Updated by David Doyle about 8 years ago

  • Status changed from New to In Progress

#8 Updated by David Doyle about 8 years ago

  • % Done changed from 0 to 70

Non-prod environments complete. Will finish out with prod tomorrow.

#9 Updated by David Doyle about 8 years ago

  • % Done changed from 70 to 90
  • Status changed from In Progress to Testing

Prod CNs complete. Leaving this open in testing until a couple of logs rotate out to test.

#10 Updated by David Doyle almost 8 years ago

Testing revealed error in logrotate due to errant wildcard in dataone-processing-jsvc config file. Logs were rotating, then rotating again immediately, which became worse as more logs were created. Config file dataone-index-jsvc appears to be rotating normally.

Changes made across all CNs:

dataone-processing-jsvc changed from catchall to single-file for dataone-processing-jsvc.log
dataone-processing-jsvc-err added to rotate dataone-processing-jsvc.err

To do:

Log directory cleanup to remove/rename bad logs

#11 Updated by David Doyle almost 8 years ago

Cleaned up remaining log directories. Induced log rotation where necessary to test configs. Leaving in testing to remind myself to check periodically.

#12 Updated by David Doyle almost 8 years ago

  • Due date changed from 2014-01-18 to 2014-03-01
  • Target version changed from 2014.2-Block.1.1 to 2014.8-Block.1.4

#13 Updated by David Doyle almost 8 years ago

  • Due date changed from 2014-03-01 to 2014-03-29
  • Target version changed from 2014.8-Block.1.4 to 2014.12-Block.2.2

#14 Updated by David Doyle almost 8 years ago

  • Due date changed from 2014-03-29 to 2014-04-12
  • Target version changed from 2014.12-Block.2.2 to 2014.14-Block.2.3

#15 Updated by David Doyle almost 8 years ago

  • Status changed from Testing to Closed
  • % Done changed from 90 to 100

Done.

#16 Updated by David Doyle over 7 years ago

While working on another issue, I found that these changes weren't pushed to cn-dev-orc-1 after the last rebuild.

Added logrotate configs, forced a rotate.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)