DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2019-01-14T16:46:33ZDataONE Tasks
Redmine CN REST - Story #8757 (New): Fix getChecksum() in MNAuditTask to use dynamic checksum algorithmshttps://redmine.dataone.org/issues/87572019-01-14T16:46:33ZChris Jonescjones@nceas.ucsb.edu
<p>The <code>MNAuditTask.call()</code> method is hardcoded to use <code>MD5</code> checksums on line 277. It requests the Member Node to generate an <code>MD5</code> checksum, and then compares that checksum to the checksum stated in the Coordinating Node<code>s cached</code>SystemMetadata.checksum<code>field for the object. This obviously will fail for objects that submitted objects using</code>SHA-1` or other algorithms.</p>
CN REST - Story #8756 (New): Ensure replica auditor is effectivehttps://redmine.dataone.org/issues/87562019-01-12T20:25:18ZChris Jonescjones@nceas.ucsb.edu
<p>The replication auditor service is currently configured to audit all objects every 90 days. As documented in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="Story: Replica Auditing service is throwing errors (New)" href="https://redmine.dataone.org/issues/8582">#8582</a>, the auditor is not working correctly. While the errors being thrown that are described in that ticket seem to be limited to <code>pid</code>s with certain characters in them, I think the whole auditor process is not keeping up with our content.</p>
<p>Looking at the number of objects on each member node that haven't been audited in the last 90 days, auditing is well behind (if we consider it working at all):</p>
<pre>SELECT sm.authoritive_member_node, count(smr.guid) AS count
FROM systemmetadata sm INNER JOIN smreplicationstatus smr
ON sm.guid = smr.guid
WHERE
smr.member_node != 'urn:node:CN' AND
sm.date_uploaded < (SELECT CURRENT_DATE - interval '90 days') AND
smr.date_verified < (SELECT CURRENT_DATE - interval '90 days')
GROUP BY sm.authoritive_member_node
ORDER BY count DESC;
authoritive_member_node | count
-------------------------+--------
urn:node:ARCTIC | 771872
urn:node:PANGAEA | 507456
urn:node:LTER | 416339
urn:node:DRYAD | 374439
urn:node:CDL | 242115
urn:node:PISCO | 235791
urn:node:KNB | 86075
urn:node:TDAR | 75639
urn:node:NCEI | 50974
urn:node:USGS_SDC | 40290
urn:node:TERN | 31671
urn:node:ESS_DIVE | 28830
urn:node:NMEPSCOR | 16042
urn:node:GOA | 9266
urn:node:IARC | 7677
urn:node:NRDC | 6673
urn:node:TFRI | 6478
urn:node:PPBIO | 3464
urn:node:ORNLDAAC | 3328
urn:node:FEMC | 2430
urn:node:EDI | 2098
urn:node:GRIIDC | 2065
urn:node:mnTestKNB | 2010
urn:node:SANPARKS | 2008
urn:node:ONEShare | 1874
urn:node:R2R | 1787
urn:node:USGSCSAS | 1151
urn:node:EDACGSTORE | 1075
urn:node:US_MPC | 1032
urn:node:RW | 970
urn:node:KUBI | 516
urn:node:NEON | 487
urn:node:LTER_EUROPE | 343
urn:node:IOE | 279
urn:node:RGD | 273
urn:node:ESA | 272
urn:node:NKN | 218
urn:node:OTS_NDC | 126
urn:node:BCODMO | 115
urn:node:SEAD | 90
urn:node:mnTestNKN | 50
urn:node:EDORA | 28
urn:node:ONEShare.pem | 22
urn:node:CLOEBIRD | 17
urn:node:mnTestBCODMO | 11
urn:node:USANPN | 10
urn:node:mnTestTDAR | 10
urn:node:MyMemberNode | 1
</pre>
<p>The table above represents the number of un-audited objects (in the last 90 days), but I get the feeling that the auditor isn't able to audit any of the content it is charged to audit given 1) the frequency, 2) the number of threads allotted, and 3) the configured batch count (seems way too low). <del>Note that this query excludes replicated content - this is just the original objects</del> (After looking at my query again, I think the join is including all replicas - the total is 2,935,787, which is greater than the total objects in the system (2,751,136), so this query needs to be refined).</p>
<p>We need to evaluate the true effectiveness of the auditor. Some strategies may include: 1) looking to see if we may be in an infinite loop on processing a few <code>pid</code>s due to the issues in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="Story: Replica Auditing service is throwing errors (New)" href="https://redmine.dataone.org/issues/8582">#8582</a>, 2) seeing if we can increase the batch size by increasing the total threads allocated in the executor, and 3) decide if we need to offload the process from the CNs and distribute the workload across a cluster of workers that can do the auditing faster. Needs some thought and discussion.</p>
Infrastructure - Story #8639 (New): Replication performance is too slow to service demandhttps://redmine.dataone.org/issues/86392018-07-04T11:17:55ZDave Vieglaisdave.vieglais@gmail.com
<p>The replication process is operating too slowly to service demand resulting in lengthy delays to completion of replication tasks for new and changed content.</p>
<p>This is particularly apparent in the stage environment where perhaps the number of orphaned objects and deprecated / defunct nodes is interfering with expected behaviors.</p>
<p>Goal of this story is to identify and address the immediate issues. Any significant refactoring of the replication process should be captured under another story / epic.</p>
Search UI - Story #8574 (New): PANGAEA Temporary Fix: SID only in Data Citationhttps://redmine.dataone.org/issues/85742018-04-30T13:26:15ZMonica Ihliemail@monicaihli.com
<p>Adjust appearance of data citation in the case of formatid <a href="http://www.isotc211.org/2005/gmd-pangaea">http://www.isotc211.org/2005/gmd-pangaea</a> such that the SID alone appears in the data citation and not the PID. This is intended as a temporary fix to be in place only while a longer term strategy for customizing appearances of data citations is designed and implemented. Once that longer term strategy is in place, whatever temporary code level changes were implemented to accommodate PANGAEA should be removed.</p>
Member Nodes - Story #8535 (In Progress): Neotoma: Story: Discovery & Planninghttps://redmine.dataone.org/issues/85352018-04-09T21:14:39ZAmy Forresteraforres4@utk.edu
<p>Discovery: establish contact and build relationship with a potential new member node. Determine if DataONE and the repository are a good fit for one another and if the repository generally meets the requirements of DataONE member nodes. </p>
<p>Planning: If the repository and DataONE have agreed to proceed with deployment as a member node. Decisions will be made as to how to proceed with development. Node operators will receive training. </p>
<p>This story is complete when a determination is made to either proceed with planning a new deployment, or that joining DataONE is not an option for the repository at this time.</p>
<p><strong>Record initial communication here</strong></p>
Infrastructure - Story #8368 (New): Update backup strategy for jenkins job configurations to subv...https://redmine.dataone.org/issues/83682018-02-15T17:37:09ZDave Vieglaisdave.vieglais@gmail.comInfrastructure - Story #8367 (New): Duplicate jenkins jobs from UNM to UCSBhttps://redmine.dataone.org/issues/83672018-02-15T17:36:01ZDave Vieglaisdave.vieglais@gmail.com
<p>All jenkins jobs need to be duplicated from UNM to UCSB. The goal here is to provide the same functionality as is currently available on the UNM instance.</p>
Member Nodes - Story #8358 (In Progress): Discovery & Planning (CAFF)https://redmine.dataone.org/issues/83582018-02-12T16:25:06ZAmy Forresteraforres4@utk.edu
<p>Discovery is about establishing contact and building a relationship with a potential new member node. In this phase, it is determined if DataONE and the repository are a good fit for one another and if the repository generally meets the requirements of DataONE member nodes. Broad discussions of deployment options may be reviewed as well.<br>
This story is complete when a determination is made to either proceed with planning a new deployment, or that joining DataONE is not an option for the repository at this time.</p>
Member Nodes - Story #8244 (New): Upgrade Member Node to current version of Metacat (IOE)https://redmine.dataone.org/issues/82442018-01-22T16:38:20ZDave Vieglaisdave.vieglais@gmail.com
<p>The MN operations have been placed in the care of Todd Kipfer. He has requested some assistance with use of Metacat, and especially bringing it to the latest version.</p>
Infrastructure - Story #7807 (New): cn.synchronize should support synchronization failure correct...https://redmine.dataone.org/issues/78072016-05-13T16:56:25ZRob Nahfrnahf@epscor.unm.edu
<p>cn.synchronize(session, identifier) works well for its original purpose (supporting MN-driven system metadata updates, and MN-driven push synchronization), but doesn't seem to work for manual synchronization failure workflows. The main problem is that the request can only be made by the MN itself (using the MN client certificate). </p>
<p>As we envision a centralized dashboard for monitoring failed synchronization items, how do we address this situation? </p>
<p>The synchronization processing queue needs both the pid and a nodeId from where to retrieve the object. the NodeId is not specified directly in the method call, but gleaned from the session by a reverse lookup from the certificate. (It uses the first node found in the NodeList where the Node.subject field matches the certificate subject).</p>
<p>Should we allow node.contactSubjects into the algorithm?<br>
Should we add nodeId as a parameter?</p>
Search UI - Story #7754 (In Progress): Support for XSL transform of various metadata formatshttps://redmine.dataone.org/issues/77542016-04-27T14:43:23ZDave Vieglaisdave.vieglais@gmail.com
<p>Currently the DCX and ISO metadata formats are being rendered in the view service using solr output rather than a transform of the XML metadata. This results in a less than satisfactory rendering.</p>
<p>Goal of this story is to implement XSLT for metadata formats that currently are relying on the solr only rendering.</p>
OGC-Slender Node - Story #7146 (In Progress): Determine formatId for content retrievable from NOD...https://redmine.dataone.org/issues/71462015-06-04T20:15:08ZDave Vieglaisdave.vieglais@gmail.com
<p>In order to create system metadata for objects held by NODC, it is necessary to infer the appropriate formatId for each object.</p>
Python GMN - Story #7095 (New): Support HTTP redirect capability for GMN within the Vendor Specif...https://redmine.dataone.org/issues/70952015-05-12T16:20:39ZMark Servillamark.servilla@gmail.com
<p>GMN provides proxy capabilities to integrate existing content residing in a back-end data repository through its Vendor Specific Extension model. Only minimal support exists within the GMN-VSE model for HTTP redirects. Although the use of HTTP redirects is more of an exception within the the GMN-VSE model, it can provide a flexible mechanism for MNs who dynamically manage data and metadata objects.</p>
Python Libraries - Story #5468 (New): Remove dead code from D1 Python stackhttps://redmine.dataone.org/issues/54682014-06-06T18:57:00ZRoger Dahldahl@unm.edu
<p><a href="https://pypi.python.org/pypi/vulture">https://pypi.python.org/pypi/vulture</a></p>
Java Client - Story #3666 (In Progress): D1Client.listUpdateHistory() needs to handle changing ac...https://redmine.dataone.org/issues/36662013-03-15T22:51:23ZRob Nahfrnahf@epscor.unm.edu
<p>the current D1Client.listUpdateHistory() method needs to gracefully handle the situation where a NotAuthorized request is returned. the ObsoletesChain client class may need to be refactored to allow for this exception to be held so it can notify the user where appropriate.</p>
<p>Ostensibly, with a NotAuthorized, the user will not have access to either the tail or head of the chain, so can't return the head or tail, depending on how access changes.</p>