DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2018-07-13T11:24:16ZDataONE Tasks
Redmine Infrastructure - Bug #8655 (New): Synchronization died with OOMhttps://redmine.dataone.org/issues/86552018-07-13T11:24:16ZDave Vieglaisdave.vieglais@gmail.com
<p>d1-processing became unresponsive. cn-synchronization log showed:<br>
<code><br>
[ERROR] 2018-07-12 18:28:26,875 [ProcessDaemonTask1] (SyncObjectTaskManager:run:84) java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space<br>
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space<br>
at java.util.concurrent.FutureTask.report(FutureTask.java:122)<br>
at java.util.concurrent.FutureTask.get(FutureTask.java:192)<br>
at org.dataone.cn.batch.synchronization.SyncObjectTaskManager.run(SyncObjectTaskManager.java:76)<br>
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br>
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br>
at java.lang.Thread.run(Thread.java:748)<br>
Caused by: java.lang.OutOfMemoryError: Java heap space<br>
[ INFO] 2018-07-12 18:28:49,862 [ProcessDaemonTask1] (SyncObjectTaskManager:run:110) SyncObjectTaskManager Complete<br>
[ WARN] 2018-07-12 20:41:15,788 [hz.client.2.Listener] (NodeTopicListener:onMessage:68) urn:node:OTS_NDC- NodeTopicListener Disabl<br>
</code></p>
<p>d1-processing is running with:<br>
<code><br>
-Djava.awt.headless=true -XX:UseParallelGC -Xmx4096M -Xms1024M -Xss1280k -XX:MaxPermSize=512M<br>
</code></p>
Testing MN Management - Task #8464 (New): test: Metadata Validationhttps://redmine.dataone.org/issues/84642018-03-01T21:04:42ZAmy Forresteraforres4@utk.edu
<p>Receive and validate sample metadata records from member node.</p>
Testing MN Management - Story #8463 (New): test: Testing & Developmenthttps://redmine.dataone.org/issues/84632018-03-01T21:04:41ZAmy Forresteraforres4@utk.edu
<p>Install or develop a functional member node to be registered to a non-production environment. </p>
Infrastructure - Bug #7919 (New): unloadable system metadata in CNs by Hazelcasthttps://redmine.dataone.org/issues/79192016-10-26T16:22:56ZRob Nahfrnahf@epscor.unm.edu
<p>Looking through the metacat logs, I found a lot of instances where the HzSystemMetadataMap could not load system metadata for particular pids. Most had dryad in the pid (~1200), but another 130 are from elsewhere.</p>
<p>a random sample showed that it couldn't be retrieved via /meta although the pid could be retrieved from the Dryad MN.</p>
<p>This appears to be another type of half-created content on the CN.</p>
<p>rnahf@cn-ucsb-1:~$ grep 'could not load system metadata' /var/metacat/logs/metacat.log | cut -c60- | sort | uniq | grep -v dryad | wc -l<br>
139<br>
rnahf@cn-ucsb-1:~$ grep 'could not load system metadata' /var/metacat/logs/metacat.log | cut -c60- | sort | uniq | grep dryad | wc -l<br>
1216</p>
Member Nodes - Task #3906 (New): Update malformed Resource Mapshttps://redmine.dataone.org/issues/39062013-08-09T17:50:27ZRob Nahfrnahf@epscor.unm.edu
<p>Update all existing resource maps in Merritt and ONShare MNs so that URIs are used instead of object-literals, to create valid resource maps. </p>
Infrastructure - Task #1334 (New): Completely remove detailCodehttps://redmine.dataone.org/issues/13342011-02-09T22:51:36ZRoger Dahldahl@unm.edu
<p>May want to completely remove DataONEException detailCode: Rationale: For the detailCode to be useful, it needs to carry information that would cause the client to fine tune its reaction to the exception. But most likely, the other information in the DataONEException is sufficient for the client to determine what to do for each exception. If that is not the case, we should consider adding exception types instead.</p>
Infrastructure - Task #1333 (New): traceInformation -> tracehttps://redmine.dataone.org/issues/13332011-02-09T22:46:32ZRoger Dahldahl@unm.edu
<p>Possibly change DataONEException traceInformation to trace. Rationale: All the fields are "information".</p>
Infrastructure - Task #1332 (New): Possibly remove MN_crud.update()https://redmine.dataone.org/issues/13322011-02-09T22:39:32ZRoger Dahldahl@unm.edu
<p>In the current GMN implementation, MN_crud.update() is simply MN_crud.delete() followed by MN_crud.create(). If that will be true for all implementations, we may want to remove MN_crud.update() and have MNs implement only create() and delete(), and, if necessary, support update() in the D1 libraries.</p>
Infrastructure - Task #1331 (New): Possibly combine MN_crud.create() and MN_crud.update()https://redmine.dataone.org/issues/13312011-02-09T22:31:37ZRoger Dahldahl@unm.edu
<p>Maybe we should combine MN_crud.create() and MN_crud.update() because both take a SysMeta object that contains the obsoletes and obsoletedBy fields, and a single call could do the appropriate thing based on that information.</p>
Infrastructure - Task #1236 (New): Highly variable response times from cn.dataone.orghttps://redmine.dataone.org/issues/12362011-01-17T16:34:05ZDave Vieglaisdave.vieglais@gmail.com
<p>Performing listobjects on cn.dataone.org works but the response times are highly variable. </p>
<p>Seems likely to be a caching issue, but likely also related to the query being executed in Metacat to generate the listObjects response.</p>
<p>For example:</p>
<p>Nomadini:~ vieglais$ time curl "<a href="http://cn.dataone.org/cn/object/">http://cn.dataone.org/cn/object/</a>"<br>
<?xml version="1.0" encoding="UTF-8"?>knb:testid:20111111590640eml://ecoinformatics.org/eml-2.1.0D3339CAF019C3EB40811CC191DCDEB122011-01-12T06:40:28.012Z12knb:testid:201111115928161eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:33.441Z12some;id;with;semi;colons;1294775954675eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:30.076Z12knb:testid:201111115733756eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:24.104Z12<a href="/d1:objectList">/d1:objectList</a><br>
real 0m0.392s<br>
user 0m0.006s<br>
sys 0m0.008s<br>
Nomadini:~ vieglais$ time curl "<a href="http://cn.dataone.org/cn/object/">http://cn.dataone.org/cn/object/</a>"<br>
<?xml version="1.0" encoding="UTF-8"?>knb:testid:20111111590640eml://ecoinformatics.org/eml-2.1.0D3339CAF019C3EB40811CC191DCDEB122011-01-12T06:40:28.012Z12knb:testid:201111115928161eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:33.441Z12some;id;with;semi;colons;1294775954675eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:30.076Z12knb:testid:201111115733756eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:24.104Z12<a href="/d1:objectList">/d1:objectList</a><br>
real 1m30.255s<br>
user 0m0.008s<br>
sys 0m0.015s</p>
Infrastructure - Task #1234 (New): Record performance metrics for member nodes https://redmine.dataone.org/issues/12342011-01-15T02:52:53ZDave Vieglaisdave.vieglais@gmail.com
<p>We need to log performance metrics for member nodes for common operations such as get() and listObjects() so that we can obtain a quantitative estimate of how those services scale with alterations to various factors such as the number of documents stored on the node.</p>
<p>Performance will be affected by many factors, so we also need to note the date and time the evaluation was performed, which node was targeted, hardware characteristics (processors, RAM) and the total number of documents in the system.</p>
<p>These metrics should be recorded under svn /documents/Projects/cicore/operations/source/data/nodeperformance.csv in plain text documents formatted as:</p>
<p>Date, nodename, requestURL, CPUs, RAM, totalDocs, operation, duration<br>
...</p>
<p>Date = timestamp in UTC, "%Y%m%dT%H%M%SZ"<br>
nodename = name in registry or domain name if not registered<br>
requestURL = full URL of request<br>
CPUs = reported number of CPUs on system (0=unknown)<br>
RAM = reported RAM in GB on system (0=unknown)<br>
totaldocs = total number of documents accessible from the node<br>
operation = one of "listobjects", "get", "node", "describe"<br>
duration = time taken in seconds (floating point)</p>
<p>Each performance run should append a line to the log.</p>
Infrastructure - Task #1225 (New): Configure Hudson Metacat buildhttps://redmine.dataone.org/issues/12252011-01-13T14:41:27ZDave Vieglaisdave.vieglais@gmail.com
<p>Hudson appears to be having problems building Metacat.</p>
<p>Check build settings and adjust settings or environment as necessary to ensure that a Metacat war file is being buit. </p>
<p>The current configuration is to invoke Ant :</p>
<p>war<br>
runonetest -Dtesttorun=CrudServiceTest</p>
<p>The build appears to fail, perhaps due to a hard coded properties file path. For example: </p>
<p><a href="http://dev-testing.dataone.org:8080/hudson/job/Metacat/4731/console">http://dev-testing.dataone.org:8080/hudson/job/Metacat/4731/console</a></p>
Infrastructure - Task #1001 (New): Link documentation living with projects into ArchitectureDocshttps://redmine.dataone.org/issues/10012010-10-21T14:50:57ZRoger Dahldahl@unm.edu
<p>Inside each of the following projects, there a documentation folder that contains stand-alone Sphinx documentation for that project. Need to integrate those with ArchitectureDocs.</p>
<ul>
<li>GMN</li>
<li>Python DataONE CLI</li>
<li>d1common (Python)</li>
<li>D1 Python ITK</li>
</ul>
Infrastructure - Task #995 (New): implement JSON formatted OLLshttps://redmine.dataone.org/issues/9952010-10-13T22:05:51ZRob Nahfrnahf@epscor.unm.edu
<p>Useful to have responses returned in JSON instead of XML. (Other formats can wait).<br>
Consider using XSLT as secondary filter for the approach. Seems to be the standard.</p>
Infrastructure - Task #925 (New): Test for correct response to invalid input: CCI.ITK.FUSE-clienthttps://redmine.dataone.org/issues/9252010-10-06T19:50:31ZRoger Dahldahl@unm.edu
<p>just one task of many, I'm guessing...</p>