DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2019-07-23T16:41:33ZDataONE Tasks
Redmine Member Nodes - Task #8829 (New): get DKAN connected to GMNhttps://redmine.dataone.org/issues/88292019-07-23T16:41:33ZAmy Forresteraforres4@utk.edu
<p>Matt J talked to Carrie -- They recently installed DKAN, and are now having trouble programatically uploading to GMN as Ed had been doing. She was interested in MetacatUI and that there could be a web-based form for uploading content. NKN was unaware that this might be possible with GMN and Metacat. She had lots of good comments about our new service models too. We should definitely follow up.</p>
<p>At a minimum we need to help them get DKAN connected to GMN, as none of their new content is making it into GMN.</p>
Infrastructure - Task #8824 (In Progress): Install Apache 2.4.39 and disable TLSv1.3 on CNshttps://redmine.dataone.org/issues/88242019-06-19T02:10:27ZDave Vieglaisdave.vieglais@gmail.com
<p>Apache 2.4.39 is needed to avoid a lengthy connection time, however it offers TLSv1.3 which seems to break at least python client.</p>
<p>Task is to install 2.4.39 but disable TLSv1.3 for now with:</p>
<pre>SSLProtocol all -TLSv1.3 -SSLv2 -SSLv3
</pre>
<p>in <code>/etc/apache2/mods-enabled/ssl.conf</code></p>
Infrastructure - Story #8823 (New): Recent Apache and OpenSSL combinations break connectivity on ...https://redmine.dataone.org/issues/88232019-06-19T02:03:44ZDave Vieglaisdave.vieglais@gmail.com
<p>The latest Ubuntu 18.04 release of Apache is 2.4.29 and OpenSSL is 1.1.1.</p>
<p>This combination creates a significant delay in TLS renegotiation that results from the Apache config option on the CNs:</p>
<pre>SSLVerifyClient none
<Location "/cn">
<If " ! ( %{HTTP_USER_AGENT} =~ /(windows|chrome|mozilla|safari|webkit)/i )">
SSLVerifyClient optional
</If>
</Location>
</pre>
<p>Which is intended to disable client certificate authentication for web browsers, but allow it for others. This approach worked fine on older Apache / OpenSSL but the new combination creates a several second wait when the server discovers the client is not a web browser and tells it to reconnect with the option of including a client certificate.</p>
<p>The latest released version of Apache is 2.4.39 and this is available through a PPA intended for Debian developers. This has been installed so far on dev-2, sandbox, stage, and stage-2 with the process:</p>
<pre>sudo add-apt-repository ppa:ondrej/apache2
sudo apt update
sudo apt dist-upgrade
sudo systemctl restart apache2
</pre>
<p>This installs Apache 2.4.39 and OpenSSL 1.1.1c which appears to resolve the apparent bug in the 2.4.29 / 1.1.1 combination.</p>
<p>One issue with the update is that by default, Apache now offers TLSv1.3, which is great except that it appears to cause problems with at least Python clients failing to connect and getting a 403 error. For example:</p>
<pre>$ python3
>>> import requests
>>> r = requests.get("https://cn-sandbox-ucsb-1.test.dataone.org/cn/v2/monitor/ping")
>>> r.status_code
403
</pre>
<p>That TLSv1.3 is the problem was verified with cn-stage-unm-2 by configuring Apache with:</p>
<pre> SSLProtocol all -TLSv1.3 -SSLv2 -SSLv3
</pre>
<p>to disable TLSv1.3. After this change the Python client was able to connect as expected.</p>
<p>A workaround has not yet been researched.</p>
<p>It is not clear if this issue applies to other clients such as R and Java, so until we learn one way or the other, TLSv1.3 will be disabled on the CNs.</p>
<p>--This issue will likely apply to Member Nodes as well once TLSv1.3 is generally available or if MNs choose to install Apache 2.4.39.-- CORRECTION: this issue only applies when attempting to renegotiate TLS after headers have been transferred, so will not typically apply to a MN.</p>
Infrastructure - Bug #8722 (New): Object in search index but systemmetadata is not available.https://redmine.dataone.org/issues/87222018-10-01T18:38:25ZDave Vieglaisdave.vieglais@gmail.com
<p>The object: <code>urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2</code> is a CSV file. </p>
<p>Resolve or getSystemMetadata return a 404 for that object. It is however, available in the search index:</p>
<p><a href="https://cn.dataone.org/cn/v2/query/solr/?start=0&rows=10&fl=*&q=id%3A%22urn%5C%3Auuid%5C%3A4923cca4%5C-c155%5C-4edc%5C-b901%5C-f6e3b4f2e7b2%22">https://cn.dataone.org/cn/v2/query/solr/?start=0&rows=10&fl=*&q=id%3A%22urn%5C%3Auuid%5C%3A4923cca4%5C-c155%5C-4edc%5C-b901%5C-f6e3b4f2e7b2%22</a></p>
<p>cn-synchronization.log* reports:</p>
<pre>cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:32,418 [ProcessDaemonTask2] (SyncObjectTask:executeTransferObjectTask:293) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 received
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:32,418 [ProcessDaemonTask2] (SyncObjectTask:executeTransferObjectTask:310) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 submitted for execution
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:32,418 [SynchronizeTask322] (V2TransferObjectTask:call:202) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Locking task, attempt 1
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:32,469 [SynchronizeTask322] (V2TransferObjectTask:call:207) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Processing SyncObject
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:32,575 [SynchronizeTask322] (V2TransferObjectTask:retrieveMNSystemMetadata:317) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Retrieved SystemMetadata Identifier:urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 from node urn:node:KNB for ObjectInfo Identifier urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:36,547 [SynchronizeTask322] (V2TransferObjectTask:createObject:730) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Start CreateObject
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:40,388 [SynchronizeTask322] (V2TransferObjectTask:call:234) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Unlocked Pid.
cn-synchronization.log.32:[ERROR] 2018-09-21 15:37:40,388 [SynchronizeTask322] (V2TransferObjectTask:call:269) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - UnrecoverableException: urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 cn.createObject failed: The identifier is already in use by an existing object. - InvalidRequest - The identifier is already in use by an existing object.
cn-synchronization.log.32:org.dataone.cn.batch.exceptions.UnrecoverableException: urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 cn.createObject failed: The identifier is already in use by an existing object.
cn-synchronization.log.32:[ WARN] 2018-09-21 15:37:40,389 [SynchronizeTask322] (SyncFailedTask:submitSynchronizationFailed:116) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - SynchronizationFailed: detail code: 6001 id:urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 nodeId:urn:node:CNUCSB1 description:Synchronization task of [PID::] urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 [::PID] failed. Cause: InvalidRequest: The identifier is already in use by an existing object.
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:40,460 [SynchronizeTask322] (V2TransferObjectTask:call:294) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - exiting with callState: FAILED
cn-synchronization.log.32:[ INFO] 2018-09-21 15:37:40,461 [ProcessDaemonTask2] (SyncObjectTask:reapFutures:372) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 SyncObjectState: FAILED
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:17,203 [ProcessDaemonTask2] (SyncObjectTask:executeTransferObjectTask:293) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 received
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:17,204 [ProcessDaemonTask2] (SyncObjectTask:executeTransferObjectTask:310) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 submitted for execution
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:17,205 [SynchronizeTask37] (V2TransferObjectTask:call:202) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Locking task, attempt 1
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:17,255 [SynchronizeTask37] (V2TransferObjectTask:call:207) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Processing SyncObject
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:17,462 [SynchronizeTask37] (V2TransferObjectTask:retrieveMNSystemMetadata:317) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Retrieved SystemMetadata Identifier:urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 from node urn:node:KNB for ObjectInfo Identifier urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:21,034 [SynchronizeTask37] (V2TransferObjectTask:createObject:730) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Start CreateObject
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:25,672 [SynchronizeTask37] (V2TransferObjectTask:call:234) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - Unlocked Pid.
cn-synchronization.log.33:[ERROR] 2018-09-21 13:37:25,673 [SynchronizeTask37] (V2TransferObjectTask:call:269) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - UnrecoverableException: urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 cn.createObject failed: The identifier is already in use by an existing object. - InvalidRequest - The identifier is already in use by an existing object.
cn-synchronization.log.33:org.dataone.cn.batch.exceptions.UnrecoverableException: urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 cn.createObject failed: The identifier is already in use by an existing object.
cn-synchronization.log.33:[ WARN] 2018-09-21 13:37:25,674 [SynchronizeTask37] (SyncFailedTask:submitSynchronizationFailed:116) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - SynchronizationFailed: detail code: 6001 id:urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 nodeId:urn:node:CNUCSB1 description:Synchronization task of [PID::] urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 [::PID] failed. Cause: InvalidRequest: The identifier is already in use by an existing object.
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:25,740 [SynchronizeTask37] (V2TransferObjectTask:call:294) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 - exiting with callState: FAILED
cn-synchronization.log.33:[ INFO] 2018-09-21 13:37:25,740 [ProcessDaemonTask2] (SyncObjectTask:reapFutures:372) Task-urn:node:KNB-urn:uuid:4923cca4-c155-4edc-b901-f6e3b4f2e7b2 SyncObjectState: FAILED
</pre>
<p>Need to determine how an object could be added to the search index, apparently be replicated, but not exist in the system metadata.</p>
Member Nodes - Bug #8622 (New): IOE repository is not respondinghttps://redmine.dataone.org/issues/86222018-06-18T19:04:55ZDave Vieglaisdave.vieglais@gmail.com
<p>Possibly related to <a class="issue tracker-4 status-1 priority-4 priority-default child parent" title="Story: Upgrade Member Node to current version of Metacat (IOE) (New)" href="https://redmine.dataone.org/issues/8244">#8244</a></p>
<p>The IOE repository is not responding at the expected baseURL of:</p>
<p><a href="https://data.rcg.montana.edu/catalog/d1/mn">https://data.rcg.montana.edu/catalog/d1/mn</a></p>
<p>and instead returns an error that SSL connection is not available. Attempting to use http instead of https returns a 404 error.</p>
<p>Contact the server administrator and have them check the service availability.</p>
Infrastructure - Bug #8195 (In Progress): Tomcat out of memory errorhttps://redmine.dataone.org/issues/81952017-10-17T22:19:48ZDave Vieglaisdave.vieglais@gmail.com
<p>Tomcat crashed on cn-orc-1 with an out of memory error:</p>
<pre>INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:35336
Oct 17, 2017 8:44:38 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:35336] lost. Reason: java.io.EOFException[null]
Oct 17, 2017 8:46:01 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [76063] operation=GET_MEMBERS in 5 seconds.
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
20171017-20:57:05: [WARN]: [160.36.13.150]:5702 [hzProcess] Master node has timed out its heartbeat and will be removed [com.hazelcast.cluster.ClusterManager]
Oct 17, 2017 8:45:47 PM com.hazelcast.client.ConnectionManager
WARNING: Server didn't respond to client's ping call within 10 seconds!
Oct 17, 2017 8:51:41 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [76063] operation=GET_MEMBERS in 10 seconds.
Oct 17, 2017 8:57:09 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [Address[128.111.54.80]:5703] lost. Reason: java.io.EOFException[null]
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /object
original requestURI: /metacat/d1/cn/v1/object
stripping /metacat/d1/cn/v1 from requestURI
new pathinfo: /object
Exception in thread "Thread-77686" java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ajp-bio-8009-exec-49"
Exception in thread "Timer-3" java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "FileWatchdog"
Exception in thread "hz.dataonePortalInstance.ServiceThread" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ajp-bio-8009-exec-5" java.lang.OutOfMemoryError: Java heap space
Oct 17, 2017 9:03:27 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] Selector exception at hz.dataonePortalInstance.IO.thread-2, cause= java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "ajp-bio-8009-exec-307" java.lang.OutOfMemoryError: Java heap space
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6293" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6289" Oct 17, 2017 9:08:42 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] Selector exception at hz.dataonePortalInstance.IO.thread-1, cause= java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.nio.NodeIOService.onOutOfMemory(NodeIOService.java:56)
at com.hazelcast.nio.AbstractSelectionHandler.handleSocketException(AbstractSelectionHandler.java:58)
at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:92)
at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:142)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "hz.dataonePortalInstance.cached.thread-5" at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
Oct 17, 2017 9:09:03 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] unhandled exception in hz.dataonePortalInstance.IO.thread-2
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.nio.NodeIOService.onOutOfMemory(NodeIOService.java:56)
at com.hazelcast.nio.InOutSelector.handleSelectorException(InOutSelector.java:171)
at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:150)
at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
Oct 17, 2017 9:09:21 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [Address[64.106.40.6]:5703] lost. Reason: java.lang.IllegalArgumentException[null]
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6292" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6285" Oct 17, 2017 9:09:28 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] Selector exception at hz.dataonePortalInstance.IO.thread-1, cause= java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.nio.NodeIOService.disconnectExistingCalls(NodeIOService.java:180)
at com.hazelcast.nio.Connection.close(Connection.java:218)
at com.hazelcast.nio.AbstractSelectionHandler.handleSocketException(AbstractSelectionHandler.java:63)
at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:92)
at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:142)
at java.lang.Thread.run(Thread.java:748)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6288" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6290" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6284" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6286" at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6282" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
Exception in thread "hz._hzInstance_1_DataONE.IO.thread-Acceptor" at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.nio.SocketAcceptor.run(SocketAcceptor.java:66)
at java.lang.Thread.run(Thread.java:748)
Oct 17, 2017 9:11:29 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46066
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
Oct 17, 2017 9:11:33 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46066
Oct 17, 2017 9:11:49 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46311
Oct 17, 2017 9:11:49 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46311
Oct 17, 2017 9:11:49 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:46311] lost. Reason: java.io.EOFException[null]
20171017-21:12:10: [WARN]: [160.36.13.150]:5702 [hzProcess] Master node has timed out its heartbeat and will be removed [com.hazelcast.cluster.ClusterManager]
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46487
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46487
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46489
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46489
Oct 17, 2017 9:12:13 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46488
Oct 17, 2017 9:12:13 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46488
20171017-21:12:22: [WARN]: [160.36.13.150]:5702 [hzProcess] ExecutorService is rejecting an execution. java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@732a0f33 [com.hazelcast.impl.ExecutorManager]
Oct 17, 2017 9:12:26 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:46488] lost. Reason: java.io.EOFException[null]
20171017-21:12:26: [WARN]: [160.36.13.150]:5702 [hzProcess] ExecutorService is rejecting an execution. com.hazelcast.impl.SplitBrainHandler$1@1af304f [com.hazelcast.impl.ExecutorManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state! [com.hazelcast.impl.PartitionManager]
Oct 17, 2017 9:12:36 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:46487] lost. Reason: java.io.EOFException[null]
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
20171017-21:13:09: [WARN]: [160.36.13.150]:5702 [hzProcess] hz.hzProcessInstance.IO.thread-2 Closing socket to endpoint Address[128.111.54.80]:5702, Cause:java.lang.OutOfMemoryError: Java heap space [com.hazelcast.nio.ReadHandler]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at com.hazelcast.nio.DataHolder.<init>(DataHolder.java:28)
at com.hazelcast.nio.Packet.read(Packet.java:379)
at com.hazelcast.nio.SocketPacketReader$DefaultPacketReader.readPacket(SocketPacketReader.java:88)
at com.hazelcast.nio.SocketPacketReader.read(SocketPacketReader.java:62)
at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:85)
at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:142)
at java.lang.Thread.run(Thread.java:748)
20171017-21:13:09: [WARN]: [160.36.13.150]:5702 [hzProcess] hz.hzProcessInstance.IO.thread-1 Closing socket to endpoint Address[64.106.40.6]:5702, Cause:java.lang.OutOfMemoryError: Java heap space [com.hazelcast.nio.ReadHandler]
java.lang.OutOfMemoryError: Java heap space
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
In D1URLFilter.
HTTP Verb: GET
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /object
original requestURI: /metacat/d1/cn/v1/object
stripping /metacat/d1/cn/v1 from requestURI
new pathinfo: /object
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
at edu.ucsb.nceas.metacat.replication.ReplicationService.getURLStream(ReplicationService.java:2341)
at edu.ucsb.nceas.metacat.replication.ReplicationService.getURLContent(ReplicationService.java:2299)
at edu.ucsb.nceas.metacat.replication.ReplicationService.handleForceReplicateDataFileRequest(ReplicationService.java:936)
at edu.ucsb.nceas.metacat.replication.ReplicationServlet.handleGetOrPost(ReplicationServlet.java:157)
at edu.ucsb.nceas.metacat.replication.ReplicationServlet.doGet(ReplicationServlet.java:87)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /object
original requestURI: /metacat/d1/cn/v1/object
stripping /metacat/d1/cn/v1 from requestURI
new pathinfo: /object
</pre> Member Nodes - Task #7686 (Testing): tDAR - Enable URL links out to data from search results https://redmine.dataone.org/issues/76862016-03-23T16:59:05ZLaura Moyerslmoyers1@utk.edu
<p>Adam Brin would like his search results to show links out to his content from DataONE search results. We can do this with EML (see "Online Distribution Info"), so we'd need to develop a stylesheet for DCX (tDAR's metadata) such that search results display the desired link to tDAR content. See attached for a mock-up. </p>
<p>Here are some tDAR DCX identifiers:<br><br>
doi:10.6067:XCV848701_meta<br>
doi:10.6067:XCV848702_meta<br>
doi:10.6067:XCV848525_meta<br>
doi:10.6067:XCV848630_meta</p>
<p>Note they're behind a firewall. </p>
Infrastructure - Bug #7601 (New): CN checksum inconsistencieshttps://redmine.dataone.org/issues/76012016-01-21T20:09:28ZBen Leinfelderleinfelder@nceas.ucsb.edu
<p>While transferring test data from production to the sandbox-2 environment I noticed failures for a group of pids.<br>
I'll use an example to illustrate (doi_10.5066_F71C1TV7)<br>
<a href="https://cn.dataone.org/cn/v2/meta/doi_10.5066_F71C1TV7">https://cn.dataone.org/cn/v2/meta/doi_10.5066_F71C1TV7</a><br>
CN.SystemMetadata reports checksum as:<br>
<br>
46178da6192263921eb755940d716725</p>
<p>Whereas calculating it from disk gives this:<br>
<br>
MD5(/var/metacat/documents/autogen.2013062508395355978.1)= efc11787f789b45db29999fb4bd8d745</p>
<p>The byte size is also off.<br>
<br>
16739<br>
<br>
On disk:<br>
<br>
-rw-r--r-- 1 tomcat7 tomcat7 16529 Jun 25 2013 /var/metacat/documents/autogen.2013062508395355978.1</p>
<p>There are ~70 similar pids that have issues (perhaps more) from our test corpus. They are from the now defunct USGS MN.</p>
<p>I'm not sure what our strategy is since the original MN is not online any longer so we cannot get the "original" bytes from that.</p>
Member Nodes - MNDeployment #6544 (Operational): TERN Australia (powered by AEKOS)https://redmine.dataone.org/issues/65442014-11-03T00:35:54ZLaura Moyerslmoyers1@utk.edu
<p>AEKOS is the data repository component of TERN Eco-informatics. Their metadata standard is RIF-CS (the Australian National Data Service standard), but we intend to use EML for the DataONE MN:</p>
<p><a href="https://releases.dataone.org/online/api-documentation-v1.2.0/design/SearchMetadata_eml.html">https://releases.dataone.org/online/api-documentation-v1.2.0/design/SearchMetadata_eml.html</a></p>
<p>and use GMN as the MN stack.</p>
Member Nodes - MNDeployment #3556 (Operational): ESA Data Registryhttps://redmine.dataone.org/issues/35562013-02-07T03:58:02ZDave Vieglaisdave.vieglais@gmail.comMember Nodes - MNDeployment #3554 (Operational): KNBhttps://redmine.dataone.org/issues/35542013-02-07T03:56:39ZDave Vieglaisdave.vieglais@gmail.comMember Nodes - MNDeployment #3521 (Operational): SEAD Member Nodehttps://redmine.dataone.org/issues/35212013-01-25T21:19:12ZRebecca Koskelarkoskela@unm.edu
<p>SEAD (Sustainable Environment - Actionable Data), another DataNet, would like to become a DataONE Member Node<br>
(<a href="http://sead-data.net/">http://sead-data.net/</a>)</p>
Member Nodes - MNDeployment #3238 (Operational): Idaho Northwest Knowledge Network member node https://redmine.dataone.org/issues/32382012-09-07T01:30:43ZDave Vieglaisdave.vieglais@gmail.com
<p>The Northwest Knowledge network (NKN) at Idaho has been identified as a potential member node, and expressed interest at the 2012 DUG.</p>
<p>Initial contact is Greg Goldberg: CI/NKN/FRAMES Program Manager University of Idaho 208-885-9756 <a href="mailto:gollberg@uidaho.edu">gollberg@uidaho.edu</a></p>
Member Nodes - MNDeployment #3221 (Operational): EDAC member node https://redmine.dataone.org/issues/32212012-09-05T03:39:05ZDave Vieglaisdave.vieglais@gmail.com
<p>This issue is to track the deployment or otherwise of the EDAC (Earth Data and Analysis Center) as a member node.</p>
<p>Base_URL: <a href="http://gstore.unm.edu/apps/dataone">http://gstore.unm.edu/apps/dataone</a><br>
Node_ID: urn:node:EDAC-GSTORE<br>
Deployment_Contact: Soren Scott<br>
Software: Custom python service layer<br>
Target_Tier: 1<br>
Content_Volume_GB: 20,000</p>
<p>Test system<br>
Base_URL: <a href="http://129.24.63.115/dataone">http://129.24.63.115/dataone</a><br>
Node_ID: urn:node:EDACGSTORE ?</p>
Member Nodes - MNDeployment #3118 (Operational): Dryad Member Nodehttps://redmine.dataone.org/issues/31182012-08-05T17:05:51ZDave Vieglaisdave.vieglais@gmail.com
<p>The Dryad MN will operate as a tier 1 member node.</p>
<p>Base_URL: <a href="https://datadryad.org/mn">https://datadryad.org/mn</a><br>
Node_ID: urn:node:DRYAD<br>
Deployment_Contact: Ryan Scherle<br>
Software: Custom on modified DSpace (Dryad)<br>
Target_Tier: 1<br>
Content_Volume_GB: 20</p>