Bug #1366
cn-orc-1 offline
100%
Description
cn-orc-1 was running OK last night, but apparently died shortly after midnight.
Connecting to the root CN service ( http://cn-orc-1.dataone.org/cn/ ) gives a 500 error:
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
java.lang.NoClassDefFoundError: org/springframework/context/i18n/LocaleContextHolder
org.springframework.web.context.request.RequestContextListener.requestInitialized(RequestContextListener.java:60)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
java.lang.Thread.run(Thread.java:662)
Postgress log:
2011-02-15 09:46:53 UTC STATEMENT: INSERT INTO xml_documents (docid, rootnodeid, docname, doctype, user_owner, user_updated, date_created, date_updated, public_access, server_location, rev) VALUES ($1, $2, $3, $4, $5, $6, to_date('2011-02-15', 'YYYY-MM-DD HH24:MI:SS'), to_date('2011-02-15', 'YYYY-MM-DD HH24:MI:SS'), $7, $8, $9)
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:18 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:19 UTC LOG: unexpected EOF on client connection
2011-02-16 00:04:27 UTC ERROR: relation "xml_replication" does not exist at character 65
2011-02-16 00:04:27 UTC STATEMENT: select server, last_checked, replicate, datareplicate, hub from xml_replication
2011-02-16 00:07:09 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:09 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:09 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:10 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:10 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:22 UTC ERROR: relation "xml_replication" does not exist at character 65
2011-02-16 00:07:22 UTC STATEMENT: select server, last_checked, replicate, datareplicate, hub from xml_replication
2011-02-16 00:07:25 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:25 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:25 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:25 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:25 UTC LOG: unexpected EOF on client connection
2011-02-16 00:07:38 UTC ERROR: relation "xml_replication" does not exist at character 65
2011-02-16 00:07:38 UTC STATEMENT: select server, last_checked, replicate, datareplicate, hub from xml_replication
2011-02-16 00:11:08 UTC WARNING: nonstandard use of \ in a string literal at character 99
2011-02-16 00:11:08 UTC HINT: Use the escape string syntax for backslashes, e.g., E'\'.
2011-02-16 00:11:08 UTC WARNING: nonstandard use of \ in a string literal at character 1237
2011-02-16 00:11:08 UTC HINT: Use the escape string syntax for backslashes, e.g., E'\'.
2011-02-16 00:11:15 UTC WARNING: nonstandard use of \ in a string literal at character 99
2011-02-16 00:11:15 UTC HINT: Use the escape string syntax for backslashes, e.g., E'\'.
2011-02-16 00:11:15 UTC WARNING: nonstandard use of \ in a string literal at character 1237
2011-02-16 00:11:15 UTC HINT: Use the escape string syntax for backslashes, e.g., E'\'.
2011-02-16 00:11:39 UTC LOG: unexpected EOF on client connection
2011-02-16 00:11:39 UTC LOG: unexpected EOF on client connection
2011-02-16 00:17:02 UTC LOG: unexpected EOF on client connection
2011-02-16 00:17:02 UTC LOG: unexpected EOF on client connection
2011-02-16 00:17:02 UTC LOG: unexpected EOF on client connection
2011-02-16 00:17:02 UTC LOG: unexpected EOF on client connection
2011-02-16 00:17:02 UTC LOG: unexpected EOF on client connection
History
#1 Updated by Dave Vieglais about 13 years ago
- Status changed from New to In Progress
- Priority changed from Immediate to Urgent
I went in and restarted tomcat:
/etc/init.d/tomcat6 stop
/etc/init.d/tomcat6 start
and the service appears to be back up. Verifying that all is working again. It is not clear why the service stopped working though.
#2 Updated by Robert Waltz about 13 years ago
- Status changed from In Progress to Closed
- Milestone set to CCI-0.5
I think this was due to user tomcat6 running out of the ability to open files. Max Open Files was set to 1024 for tomcat6. I reset that number to 9000+ upon starting the tomcat6 service. edited /etc/init.d/tomcat6
