DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2017-11-29T20:54:32ZDataONE Tasks
Redmine Infrastructure - Bug #8223 (Closed): XMLSchemaService.getInstance().doRefresh not threadsafehttps://redmine.dataone.org/issues/82232017-11-29T20:54:32ZRob Nahfrnahf@epscor.unm.edu
<p>The SMLSchemaService singleton is throwing ConcurrentModificationExcpetions from doRefresh method.</p>
<p>This routine should be synchronized or otherwise made thread safe.</p>
<p>rnahf@cn-sandbox-ucsb-1:/var/metacat/logs$ grep -A10 MetacatHandler metacat.log<br>
metacat 20171128-19:47:48: [ERROR]: MetacatHandler.handleInsertOrUpdateAction - General error when writing the xml object document to the database: null [edu.ucsb.nceas.metacat.MetacatHandler]<br>
java.util.ConcurrentModificationException<br>
at java.util.Vector$Itr.checkForComodification(Vector.java:1184)<br>
at java.util.Vector$Itr.next(Vector.java:1137)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.createRegisteredNameSpaceAndLocationString(XMLSchemaService.java:417)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.doRefresh(XMLSchemaService.java:133)<br>
at edu.ucsb.nceas.metacat.MetacatHandler.handleInsertOrUpdateAction(MetacatHandler.java:1774)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.insertOrUpdateDocument(D1NodeService.java:1551)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.create(D1NodeService.java:483)<br>
at edu.ucsb.nceas.metacat.dataone.CNodeService.create(CNodeService.java:1871)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.putObject(CNResourceHandler.java:709)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.handle(CNResourceHandler.java:257)<br>
at edu.ucsb.nceas.metacat.restservice.D1RestServlet.doPost(D1RestServlet.java:86)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)</p>
<a name="at-orgapachecatalinacoreApplicationFilterChaindoFilterApplicationFilterChainjava208"></a>
<h2 > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<a href="#at-orgapachecatalinacoreApplicationFilterChaindoFilterApplicationFilterChainjava208" class="wiki-anchor">¶</a></h2>
<p>metacat 20171128-20:08:34: [ERROR]: MetacatHandler.handleInsertOrUpdateAction - General error when writing the xml object document to the database: null [edu.ucsb.nceas.metacat.MetacatHandler]<br>
java.util.ConcurrentModificationException<br>
at java.util.Vector$Itr.checkForComodification(Vector.java:1184)<br>
at java.util.Vector$Itr.next(Vector.java:1137)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.createRegisteredNameSpaceAndLocationString(XMLSchemaService.java:417)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.doRefresh(XMLSchemaService.java:133)<br>
at edu.ucsb.nceas.metacat.MetacatHandler.handleInsertOrUpdateAction(MetacatHandler.java:1774)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.insertOrUpdateDocument(D1NodeService.java:1551)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.create(D1NodeService.java:483)<br>
at edu.ucsb.nceas.metacat.dataone.CNodeService.create(CNodeService.java:1871)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.putObject(CNResourceHandler.java:709)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.handle(CNResourceHandler.java:257)<br>
at edu.ucsb.nceas.metacat.restservice.D1RestServlet.doPost(D1RestServlet.java:86)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)</p>
<a name="at-orgapachecatalinacoreApplicationFilterChaindoFilterApplicationFilterChainjava208-2"></a>
<h2 > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<a href="#at-orgapachecatalinacoreApplicationFilterChaindoFilterApplicationFilterChainjava208-2" class="wiki-anchor">¶</a></h2>
<p>metacat 20171128-20:13:58: [ERROR]: MetacatHandler.handleInsertOrUpdateAction - General error when writing the xml object document to the database: null [edu.ucsb.nceas.metacat.MetacatHandler]<br>
java.util.ConcurrentModificationException<br>
at java.util.Vector$Itr.checkForComodification(Vector.java:1184)<br>
at java.util.Vector$Itr.next(Vector.java:1137)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.createRegisteredNameSpaceAndLocationString(XMLSchemaService.java:417)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.doRefresh(XMLSchemaService.java:133)<br>
at edu.ucsb.nceas.metacat.MetacatHandler.handleInsertOrUpdateAction(MetacatHandler.java:1774)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.insertOrUpdateDocument(D1NodeService.java:1551)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.create(D1NodeService.java:483)<br>
at edu.ucsb.nceas.metacat.dataone.CNodeService.create(CNodeService.java:1871)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.putObject(CNResourceHandler.java:709)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.handle(CNResourceHandler.java:257)<br>
at edu.ucsb.nceas.metacat.restservice.D1RestServlet.doPost(D1RestServlet.java:86)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)</p>
<a name="at-orgapachecatalinacoreApplicationFilterChaindoFilterApplicationFilterChainjava208-3"></a>
<h2 > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<a href="#at-orgapachecatalinacoreApplicationFilterChaindoFilterApplicationFilterChainjava208-3" class="wiki-anchor">¶</a></h2>
<p>metacat 20171128-20:20:42: [ERROR]: MetacatHandler.handleInsertOrUpdateAction - General error when writing the xml object document to the database: null [edu.ucsb.nceas.metacat.MetacatHandler]<br>
java.util.ConcurrentModificationException<br>
at java.util.Vector$Itr.checkForComodification(Vector.java:1184)<br>
at java.util.Vector$Itr.next(Vector.java:1137)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.createRegisteredNameSpaceAndLocationString(XMLSchemaService.java:417)<br>
at edu.ucsb.nceas.metacat.service.XMLSchemaService.doRefresh(XMLSchemaService.java:133)<br>
at edu.ucsb.nceas.metacat.MetacatHandler.handleInsertOrUpdateAction(MetacatHandler.java:1774)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.insertOrUpdateDocument(D1NodeService.java:1551)<br>
at edu.ucsb.nceas.metacat.dataone.D1NodeService.create(D1NodeService.java:483)<br>
at edu.ucsb.nceas.metacat.dataone.CNodeService.create(CNodeService.java:1871)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.putObject(CNResourceHandler.java:709)<br>
at edu.ucsb.nceas.metacat.restservice.v2.CNResourceHandler.handle(CNResourceHandler.java:257)<br>
at edu.ucsb.nceas.metacat.restservice.D1RestServlet.doPost(D1RestServlet.java:86)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)<br>
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)<br>
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)<br>
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)</p>
Infrastructure - Bug #8220 (Closed): Patch postgres for https://usn.ubuntu.com/usn/usn-3479-1/https://redmine.dataone.org/issues/82202017-11-21T18:10:55ZDave Vieglaisdave.vieglais@gmail.com
<p>This security bug should not be directly impacting us since all out postgres instances are blocked by the firewall, but we should patch all postgres instances when convenient.</p>
Infrastructure - Task #8219 (Closed): verify proper rendering of http://www.isotc211.org/2005/gmd...https://redmine.dataone.org/issues/82192017-11-21T17:43:26ZDave Vieglaisdave.vieglais@gmail.com
<p>Verify that the pangaea variant of the ISO metadata renders correctly after indexing.</p>
Infrastructure - Task #8217 (Closed): Add formatID : http://www.isotc211.org/2005/gmd-pangaeahttps://redmine.dataone.org/issues/82172017-11-21T17:37:00ZDave Vieglaisdave.vieglais@gmail.com
<p>Add the schemas and format ID for the Pangaea variant of ISO metadata: <a href="http://www.isotc211.org/2005/gmd-pangaea">http://www.isotc211.org/2005/gmd-pangaea</a></p>
Infrastructure - Task #8212 (Closed): Exception message for invalid token when using diagnostic toohttps://redmine.dataone.org/issues/82122017-10-31T16:57:29ZDave Vieglaisdave.vieglais@gmail.com
<p>Bryce provided a patch:</p>
<p><a href="https://gist.github.com/amoeba/b949c174a42effef5b3825bc043ab833">https://gist.github.com/amoeba/b949c174a42effef5b3825bc043ab833</a></p>
Infrastructure - Bug #8183 (Closed): CNDiagnostic.echoIndexedObject() fails on resource mapshttps://redmine.dataone.org/issues/81832017-09-12T19:46:14ZChris Jonescjones@nceas.ucsb.edu
<p>Calling @CNDiagnostic.echoIndexedObject()@ with both known-good and known-bad resource maps fails with a @500 ServiceFailure@ exception. We would expect it to return the list of Solr documents for the known-good resource map, and the original exception content for known-bad resource maps.</p>
<p>Here are two examples:</p>
<p>Good (i.e. has been successfully indexed by DataONE):<br>
<a href="https://knb.ecoinformatics.org/knb/d1/mn/v2/meta/resourceMap_knb.1389.1">https://knb.ecoinformatics.org/knb/d1/mn/v2/meta/resourceMap_knb.1389.1</a></p>
<p>Bad (failed indexing by DataONE):<br>
<a href="https://knb.ecoinformatics.org/knb/d1/mn/v2/meta/urn:uuid:c3cb05ae-0fd7-4ebc-960a-4781521d249d">https://knb.ecoinformatics.org/knb/d1/mn/v2/meta/urn:uuid:c3cb05ae-0fd7-4ebc-960a-4781521d249d</a></p>
<p>The latter ORE has a syntax error with regard to the @<a href="dc:creator">dc:creator</a>@ content, and fails to be parsed by the Jena library with:<br>
<br>
[ERROR] 2017-09-12 12:58:51,900 (RDFDefaultErrorHandler:error:40) (line 9 column 61): {E201} rdf:resource not allowed as attribute here.<br>
[ERROR] 2017-09-12 12:58:51,910 (RDFDefaultErrorHandler:error:40) (line 10 column 70): {E201} Multiple children of property element<br>
[ERROR] 2017-09-12 12:58:51,910 (RDFDefaultErrorHandler:error:40) (line 10 column 70): {E201} rdf:datatype not allowed as attribute here.<br>
[ERROR] 2017-09-12 12:58:51,911 (RDFDefaultErrorHandler:error:40) (line 10 column 82): {E202} Expecting XML start or end element(s). String data "Ocean Spaces" not allowed. Maybe there should be an rdf:parseType='Literal' for embedding mixed XML content in RDF. Maybe a striping error.</p>
<p>So we would expect these errors to trickle up the call stack and be reported to the submitter.</p>
Infrastructure - Story #8175 (Closed): Solr crashing with out of memory errorhttps://redmine.dataone.org/issues/81752017-09-05T16:33:26ZDave Vieglaisdave.vieglais@gmail.com
<p>Solr has been crashing and/or thrashing with out of memory errors on CN-UCSB-1 resulting in poor or no response when using search.dataone.org.</p>
<p>There are several related issues to be addressed here:</p>
<ol>
<li>Poorly written queries generated by the search UI are causing excessive demand on the solr index</li>
<li>Solr is running with 2GB allocated to the JVM. This appears to be insufficient.</li>
<li>Should design and implement a load balancing solution so a single instance of Solr is not used all the time</li>
<li>Memory use is always high on cn-ucsb-1 where processing (and so log aggregation) is running. Review the log aggregation process for memory leaks and performance.</li>
<li>The configuration for Solr is using mostly default parameters. This should be reviewed and revised where appropriate.</li>
</ol>
Infrastructure - Bug #8161 (New): d1-processing Out of Memory Errorhttps://redmine.dataone.org/issues/81612017-08-13T13:51:53ZDave Vieglaisdave.vieglais@gmail.com
<p>Noticed that d1-processing had stopped processing synchronization, last successful log message received 2017-08-11T22:56:00Z</p>
<p>/var/log/dataone/daemon/cn-process-daemon.log reporting:</p>
<p>[ERROR] 2017-08-13 13:36:34,623 (StdOutErrLog$2:print:51) java.lang.OutOfMemoryError: Java heap space</p>
<p>d1-processing is started with:</p>
<p>JAVA_OPTS="-Djava.awt.headless=true -XX:UseParallelGC -Xmx4096M -Xms1024M -Xss1280k -XX:MaxPermSize=512M"</p>
<p>jsvc.exec -home /usr/lib/jvm/java-1.8.0-openjdk-amd64 <br>
-cp /usr/share/java/commons-daemon.jar:/usr/share/java/d1_process_daemon.jar <br>
-debug -outfile /var/log/dataone/daemon/d1-processing-jsvc.log <br>
-errfile /var/log/dataone/daemon/d1-processing-jsvc.err <br>
-pidfile /var/run/d1-processing.pid <br>
-user tomcat7 <br>
-Djava.awt.headless=true <br>
-XX:UseParallelGC <br>
-Xmx4096M <br>
-Xms1024M <br>
-Xss1280k <br>
-XX:MaxPermSize=512M <br>
org.dataone.cn.batch.daemon.SchedulerDaemon</p>
<p>Xmx sets the maximum heap size</p>
<p>Xms sets the initial heap size</p>
<p>Xss sets the java thread stack size</p>
<p>/proc/meminfo reports:<br>
<br>
MemTotal: 32948380 kB<br>
MemFree: 3428124 kB</p>
<p>Restarted d1-processing to continue operations while reviewing options for resolution.</p>
Infrastructure - Story #8158 (Closed): Synchronization appears to fail under high loadhttps://redmine.dataone.org/issues/81582017-08-04T13:50:27ZDave Vieglaisdave.vieglais@gmail.com
<p>When the number of objects that needs to be processed by synchronization is high, the process fails / crashes. For example, the Pangaea node with 325k objects fails on initial sync. The cause of this needs to be evaluated and the synchronization process refactored to allow arbitrarily large numbers of entries to be synchronized.</p>
Infrastructure - Story #8156 (Closed): Member Nodes should advertise in the node document the API...https://redmine.dataone.org/issues/81562017-08-01T16:45:19ZDave Vieglaisdave.vieglais@gmail.com
<p>The purpose of the node document provided by MNs and CNs is to advertise the availability of service types being offered by the node.</p>
<p>It appears that Metacat is not properly advertising the availability of services, in particular the MNView and MNPackage APIs are not present in the node documents for Metacat MNs.</p>
Infrastructure - Bug #8130 (Closed): invalidly formatted dates in listObjects fromDate / toDate p...https://redmine.dataone.org/issues/81302017-07-13T18:29:59ZRob Nahfrnahf@epscor.unm.edu
<p>instead, the filters are not applied, and unfiltered results are returned.<br>
The call should return an InvalidRequest error with explanation.</p>
<p>This is especially devious when someone doesn't escape the '+' in the timezone field, and it gets decoded to space, resulting in an invalid date format, and incorrect filtering.</p>
<p>Checking the svn logs on CNResourceHandler, (added in <a class="changeset" title="Added InvalidToken exceptions to methods as per the documentation change. Removed unwarranted url..." href="https://redmine.dataone.org/projects/d1/repository/revisions/6622">r6622</a>) there isn't a specific reason it was put in, so it should be safe to change.</p>
Infrastructure - Bug #8126 (Closed): Synchronization on cn-ucsb-1 stoppedhttps://redmine.dataone.org/issues/81262017-07-10T10:53:53ZDave Vieglaisdave.vieglais@gmail.com
<p>Noted that sync logs were not being updated with success information. Checking cn-syncrhonization.log:</p>
<p>[ INFO] 2017-07-07 17:58:07,285 (V2TransferObjectTask:notifyReplicaNodes:1293) Task-urn:node:ARCTIC-urn:uuid:463ec0e7-fd06-4137-b379-1051e9f791df - Entering notifyReplicaNodes...<br>
[DEBUG] 2017-07-07 17:58:07,409 (ObjectListHarvestTask:call:221) placed on hzSyncObjectQueue- Task-urn:node:ARCTIC-urn:uuid:3b72fb12-5a25-43ad-a95f-6fbcd3d03151<br>
[ INFO] 2017-07-07 17:58:07,410 (V2TransferObjectTask:processUpdates:1006) Task-urn:node:ARCTIC-urn:uuid:463ec0e7-fd06-4137-b379-1051e9f791df - Completed ProcessUpdate<br>
[ERROR] 2017-07-07 17:58:07,463 (V2TransferObjectTask:call:291) Task-urn:node:ARCTIC-urn:uuid:463ec0e7-fd06-4137-b379-1051e9f791df - Current thread is not owner of the lock! - RuntimeException - Current thread is not owner of the lock!<br>
java.lang.RuntimeException: Current thread is not owner of the lock!<br>
at com.hazelcast.impl.ClientServiceException.readData(ClientServiceException.java:63)<br>
at com.hazelcast.nio.Serializer$DataSerializer.read(Serializer.java:104)<br>
at com.hazelcast.nio.Serializer$DataSerializer.read(Serializer.java:79)<br>
at com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:121)<br>
at com.hazelcast.nio.AbstractSerializer.toObject(AbstractSerializer.java:156)<br>
at com.hazelcast.client.ClientThreadContext.toObject(ClientThreadContext.java:72)<br>
at com.hazelcast.client.IOUtil.toObject(IOUtil.java:34)<br>
at com.hazelcast.client.ProxyHelper.getValue(ProxyHelper.java:186)<br>
at com.hazelcast.client.ProxyHelper.doOp(ProxyHelper.java:146)<br>
at com.hazelcast.client.ProxyHelper.doOp(ProxyHelper.java:140)<br>
at com.hazelcast.client.LockClientProxy.unlock(LockClientProxy.java:68)<br>
at org.dataone.cn.batch.synchronization.tasks.V2TransferObjectTask.call(V2TransferObjectTask.java:233)<br>
at org.dataone.cn.batch.synchronization.tasks.V2TransferObjectTask.call(V2TransferObjectTask.java:113)<br>
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br>
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br>
at java.lang.Thread.run(Thread.java:745)<br>
[ INFO] 2017-07-07 17:58:07,464 (V2TransferObjectTask:call:294) Task-urn:node:ARCTIC-urn:uuid:463ec0e7-fd06-4137-b379-1051e9f791df - exiting with callState: FAILED</p>
<p>Then finally:<br>
<br>
[DEBUG] 2017-07-07 17:58:11,847 (ObjectListHarvestTask:call:221) placed on hzSyncObjectQueue- Task-urn:node:ARCTIC-urn:uuid:f5977766-9b9e-4007-9552-f40ff863c713<br>
[DEBUG] 2017-07-07 17:58:11,951 (ObjectListHarvestTask:call:221) placed on hzSyncObjectQueue- Task-urn:node:ARCTIC-urn:uuid:f766fc31-4334-4b30-9b76-097817a8dfed<br>
[ERROR] 2017-07-07 17:58:11,952 (MemberNodeHarvestJob:execute:79) urn:node:ARCTIC - job-urn:node:ARCTIC died: urn:node:ARCTIC- Disabled<br>
org.dataone.cn.batch.exceptions.ExecutionDisabledException: urn:node:ARCTIC- Disabled<br>
at org.dataone.cn.batch.synchronization.tasks.ObjectListHarvestTask.call(ObjectListHarvestTask.java:176)<br>
at org.dataone.cn.batch.synchronization.jobs.MemberNodeHarvestJob.execute(MemberNodeHarvestJob.java:64)<br>
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)<br>
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)</p>
<p>Sync appeared to proceed normally after a restart of d1-processing.</p>
Infrastructure - Bug #8123 (Testing): portal hazelcast client should load config directly from xmlhttps://redmine.dataone.org/issues/81232017-07-06T16:42:47ZDave Vieglaisdave.vieglais@gmail.com
<p>The hazelcast client for the portal is selectively pulling config values from the configuration file after loading with FileSystemXmlConfig and adding them to a hand crafted ClientConfig instance. Instead, the client should be initiated with the config loaded directly form the XML file since FileSystemXmlConfig is a subclass of ClientConfig.</p>
<p>This will ensure that configuration properties of the hazelcast client can be easily set without changing the code.</p>
<p>In the current version, only address, user, and password are being set - all other configuration is being ignored.</p>
<p><a href="https://repository.dataone.org/software/cicore/trunk/d1_portal_servlet/src/main/java/org/dataone/portal/session/SessionHelper.java">https://repository.dataone.org/software/cicore/trunk/d1_portal_servlet/src/main/java/org/dataone/portal/session/SessionHelper.java</a></p>
Infrastructure - Bug #8115 (New): LDAP logging is set to DEBUG on production CNshttps://redmine.dataone.org/issues/81152017-06-27T16:44:25ZDave Vieglaisdave.vieglais@gmail.com
<p>This is far too verbose for production environment. Needs to be adjusted to INFO or ERROR</p>
Infrastructure - Task #8111 (Closed): D1ResourceHandler.serializeException should not log all exc...https://redmine.dataone.org/issues/81112017-06-07T18:42:28ZRob Nahfrnahf@epscor.unm.edu
<p>Every DataONE exception is logged as a metacat ERROR, which is too high of a log level, and is distracting from log analysis when folks are searching for real system errors. </p>
<p>protected void serializeException(BaseException e, OutputStream out) {<br>
// TODO: Use content negotiation to determine which return format to use<br>
response.setContentType("text/xml");<br>
response.setStatus(e.getCode());</p>
<pre> logMetacat.error("D1ResourceHandler: Serializing exception with code " + e.getCode() + ": " + e.getMessage(), e);
</pre>
<p>the last line should be logMetacat.info(...), or warn, if we always want to be logging exceptions.</p>