DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2019-01-14T16:46:33ZDataONE Tasks
Redmine CN REST - Story #8757 (New): Fix getChecksum() in MNAuditTask to use dynamic checksum algorithmshttps://redmine.dataone.org/issues/87572019-01-14T16:46:33ZChris Jonescjones@nceas.ucsb.edu
<p>The <code>MNAuditTask.call()</code> method is hardcoded to use <code>MD5</code> checksums on line 277. It requests the Member Node to generate an <code>MD5</code> checksum, and then compares that checksum to the checksum stated in the Coordinating Node<code>s cached</code>SystemMetadata.checksum<code>field for the object. This obviously will fail for objects that submitted objects using</code>SHA-1` or other algorithms.</p>
CN REST - Story #8756 (New): Ensure replica auditor is effectivehttps://redmine.dataone.org/issues/87562019-01-12T20:25:18ZChris Jonescjones@nceas.ucsb.edu
<p>The replication auditor service is currently configured to audit all objects every 90 days. As documented in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="Story: Replica Auditing service is throwing errors (New)" href="https://redmine.dataone.org/issues/8582">#8582</a>, the auditor is not working correctly. While the errors being thrown that are described in that ticket seem to be limited to <code>pid</code>s with certain characters in them, I think the whole auditor process is not keeping up with our content.</p>
<p>Looking at the number of objects on each member node that haven't been audited in the last 90 days, auditing is well behind (if we consider it working at all):</p>
<pre>SELECT sm.authoritive_member_node, count(smr.guid) AS count
FROM systemmetadata sm INNER JOIN smreplicationstatus smr
ON sm.guid = smr.guid
WHERE
smr.member_node != 'urn:node:CN' AND
sm.date_uploaded < (SELECT CURRENT_DATE - interval '90 days') AND
smr.date_verified < (SELECT CURRENT_DATE - interval '90 days')
GROUP BY sm.authoritive_member_node
ORDER BY count DESC;
authoritive_member_node | count
-------------------------+--------
urn:node:ARCTIC | 771872
urn:node:PANGAEA | 507456
urn:node:LTER | 416339
urn:node:DRYAD | 374439
urn:node:CDL | 242115
urn:node:PISCO | 235791
urn:node:KNB | 86075
urn:node:TDAR | 75639
urn:node:NCEI | 50974
urn:node:USGS_SDC | 40290
urn:node:TERN | 31671
urn:node:ESS_DIVE | 28830
urn:node:NMEPSCOR | 16042
urn:node:GOA | 9266
urn:node:IARC | 7677
urn:node:NRDC | 6673
urn:node:TFRI | 6478
urn:node:PPBIO | 3464
urn:node:ORNLDAAC | 3328
urn:node:FEMC | 2430
urn:node:EDI | 2098
urn:node:GRIIDC | 2065
urn:node:mnTestKNB | 2010
urn:node:SANPARKS | 2008
urn:node:ONEShare | 1874
urn:node:R2R | 1787
urn:node:USGSCSAS | 1151
urn:node:EDACGSTORE | 1075
urn:node:US_MPC | 1032
urn:node:RW | 970
urn:node:KUBI | 516
urn:node:NEON | 487
urn:node:LTER_EUROPE | 343
urn:node:IOE | 279
urn:node:RGD | 273
urn:node:ESA | 272
urn:node:NKN | 218
urn:node:OTS_NDC | 126
urn:node:BCODMO | 115
urn:node:SEAD | 90
urn:node:mnTestNKN | 50
urn:node:EDORA | 28
urn:node:ONEShare.pem | 22
urn:node:CLOEBIRD | 17
urn:node:mnTestBCODMO | 11
urn:node:USANPN | 10
urn:node:mnTestTDAR | 10
urn:node:MyMemberNode | 1
</pre>
<p>The table above represents the number of un-audited objects (in the last 90 days), but I get the feeling that the auditor isn't able to audit any of the content it is charged to audit given 1) the frequency, 2) the number of threads allotted, and 3) the configured batch count (seems way too low). <del>Note that this query excludes replicated content - this is just the original objects</del> (After looking at my query again, I think the join is including all replicas - the total is 2,935,787, which is greater than the total objects in the system (2,751,136), so this query needs to be refined).</p>
<p>We need to evaluate the true effectiveness of the auditor. Some strategies may include: 1) looking to see if we may be in an infinite loop on processing a few <code>pid</code>s due to the issues in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="Story: Replica Auditing service is throwing errors (New)" href="https://redmine.dataone.org/issues/8582">#8582</a>, 2) seeing if we can increase the batch size by increasing the total threads allocated in the executor, and 3) decide if we need to offload the process from the CNs and distribute the workload across a cluster of workers that can do the auditing faster. Needs some thought and discussion.</p>
Infrastructure - Story #8639 (New): Replication performance is too slow to service demandhttps://redmine.dataone.org/issues/86392018-07-04T11:17:55ZDave Vieglaisdave.vieglais@gmail.com
<p>The replication process is operating too slowly to service demand resulting in lengthy delays to completion of replication tasks for new and changed content.</p>
<p>This is particularly apparent in the stage environment where perhaps the number of orphaned objects and deprecated / defunct nodes is interfering with expected behaviors.</p>
<p>Goal of this story is to identify and address the immediate issues. Any significant refactoring of the replication process should be captured under another story / epic.</p>
Search UI - Story #8574 (New): PANGAEA Temporary Fix: SID only in Data Citationhttps://redmine.dataone.org/issues/85742018-04-30T13:26:15ZMonica Ihliemail@monicaihli.com
<p>Adjust appearance of data citation in the case of formatid <a href="http://www.isotc211.org/2005/gmd-pangaea">http://www.isotc211.org/2005/gmd-pangaea</a> such that the SID alone appears in the data citation and not the PID. This is intended as a temporary fix to be in place only while a longer term strategy for customizing appearances of data citations is designed and implemented. Once that longer term strategy is in place, whatever temporary code level changes were implemented to accommodate PANGAEA should be removed.</p>
Member Nodes - Story #8535 (In Progress): Neotoma: Story: Discovery & Planninghttps://redmine.dataone.org/issues/85352018-04-09T21:14:39ZAmy Forresteraforres4@utk.edu
<p>Discovery: establish contact and build relationship with a potential new member node. Determine if DataONE and the repository are a good fit for one another and if the repository generally meets the requirements of DataONE member nodes. </p>
<p>Planning: If the repository and DataONE have agreed to proceed with deployment as a member node. Decisions will be made as to how to proceed with development. Node operators will receive training. </p>
<p>This story is complete when a determination is made to either proceed with planning a new deployment, or that joining DataONE is not an option for the repository at this time.</p>
<p><strong>Record initial communication here</strong></p>
Member Nodes - Story #8358 (In Progress): Discovery & Planning (CAFF)https://redmine.dataone.org/issues/83582018-02-12T16:25:06ZAmy Forresteraforres4@utk.edu
<p>Discovery is about establishing contact and building a relationship with a potential new member node. In this phase, it is determined if DataONE and the repository are a good fit for one another and if the repository generally meets the requirements of DataONE member nodes. Broad discussions of deployment options may be reviewed as well.<br>
This story is complete when a determination is made to either proceed with planning a new deployment, or that joining DataONE is not an option for the repository at this time.</p>
Member Nodes - Story #8244 (New): Upgrade Member Node to current version of Metacat (IOE)https://redmine.dataone.org/issues/82442018-01-22T16:38:20ZDave Vieglaisdave.vieglais@gmail.com
<p>The MN operations have been placed in the care of Todd Kipfer. He has requested some assistance with use of Metacat, and especially bringing it to the latest version.</p>
Infrastructure - Story #8038 (In Progress): connect logging output to a log analysis toolhttps://redmine.dataone.org/issues/80382017-03-07T20:42:15ZRob Nahfrnahf@epscor.unm.edu
<p>this would be part of a larger monitoring framework effort</p>
Infrastructure - Story #7859 (New): Add formatID for the STL 3d model file formathttps://redmine.dataone.org/issues/78592016-08-04T19:02:58ZBryce Mecummecum@nceas.ucsb.edu
<p>The STL file format is a domain standard file format for storing 3d models and is the most common way I've managed 3d models used while 3d printing. Given that 3d printing is seeing increased usage in the sciences, I would say this is a good candidate for inclusion in the controlled list of format ids.</p>
<p>Type: DATA<br>
Id: STL<br>
Name: StereoLithography File Format<br>
Media type: application/sla (unofficial)<br>
Extension: .stl</p>
<p>There is an ASCII form and a Binary form of this format. They don't see to be distinguished according to any standard. What do we do in this case?</p>
<p>References: <br>
- <a href="https://en.wikipedia.org/wiki/STL_(file_format)">https://en.wikipedia.org/wiki/STL_(file_format)</a><br>
- <a href="https://reference.wolfram.com/language/ref/format/STL.html">https://reference.wolfram.com/language/ref/format/STL.html</a></p>
Search UI - Story #7754 (In Progress): Support for XSL transform of various metadata formatshttps://redmine.dataone.org/issues/77542016-04-27T14:43:23ZDave Vieglaisdave.vieglais@gmail.com
<p>Currently the DCX and ISO metadata formats are being rendered in the view service using solr output rather than a transform of the XML metadata. This results in a less than satisfactory rendering.</p>
<p>Goal of this story is to implement XSLT for metadata formats that currently are relying on the solr only rendering.</p>
Infrastructure - Story #7358 (In Progress): ContactSubject on NodeList must be valid D1 ldap entryhttps://redmine.dataone.org/issues/73582015-09-16T19:58:13ZRobert Waltz
<p>Before a CN can be started, LDAP must have an approved entry for Contact Subject.</p>
<p>Contact Subject has been defaulted to CN=Robert P Waltz A904,O=Google,C=US,DC=cilogon,DC=org on all of the CN entries in the node list.</p>
<p>Since Robert P Waltz is a developer and not an organizer or director, then the publicized contact on the CNs should be changed to reflect the organizational hierarchy.</p>
<p>The Contact Subject for the CNs should be the PI of the project, or at least, a Co-PI.</p>
<p>Also, The DN of this subject should be derived from the DataONE CA instead of cilogon.</p>
<p>Updating the existing systems should be trivial. The Ldap Entry for each CN node will be modified, and a new LDAP entry for the new Subject will need to be added.</p>
OGC-Slender Node - Story #7166 (New): Create ORE document for an NODC data packagehttps://redmine.dataone.org/issues/71662015-06-08T18:49:41ZDave Vieglaisdave.vieglais@gmail.com
<p>Each accession is treated as a data package. Need to generate a resource map document that describes the contents of an accession.</p>
Infrastructure - Story #6513 (New): Create a Local Cache of all Membernode log records on the sin...https://redmine.dataone.org/issues/65132014-10-07T18:10:26ZRobert WaltzInfrastructure - Story #2488 (New): Changing the authoritativeMembernode will require all replica...https://redmine.dataone.org/issues/24882012-03-14T19:08:19ZRobert Waltz
<p>Modifying the authoritativeMembernode requires human intervention. A tool should be created to perform the modification and then call MNStorage.systemMetadataChanged() on all replicas of the object.</p>
Infrastructure - Story #725 (In Progress): Create Authentication and Access control design specif...https://redmine.dataone.org/issues/7252010-08-02T21:42:24ZChad Berkleyberkley@nceas.ucsb.edu
<p>The metacat <a class="wiki-page new" href="https://redmine.dataone.org/projects/d1/wiki/CrudService">CrudService</a> class contains methods for authentication and access control changes that are not part of the original D1 Crud specification. These services need to be decided on at a higher level and described in the specification so that they can be made to work with any D1 node, not just metacat.</p>