DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2018-11-08T00:21:45ZDataONE Tasks
Redmine CN REST - Bug #8740 (New): CN resolve service returning 404 for some pidshttps://redmine.dataone.org/issues/87402018-11-08T00:21:45ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>A user has reported that certain pids (all for metadata objects) return an http 404 status for the resolve service. Here is the complete list that the user tried that didn't resolve:</p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8">https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23</a></p>
<p>The CNRead.getSystemMetadata() service returns metadata for each of these pids.</p>
CN REST - Bug #8698 (New): CN Performance degradationhttps://redmine.dataone.org/issues/86982018-09-13T21:05:40ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>Calling simple services on the CN can take a long time to execute. For example, calling </p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202">https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202</a></p>
<p>can take about 8-10 seconds, while calling the MN directly (where the resolve service redirects to) takes about 1-2 seconds:</p>
<p><a href="https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202">https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202</a></p>
Infrastructure - Feature #8107 (New): Improved Node Synchronization Feedback - Leveraging MNRead....https://redmine.dataone.org/issues/81072017-06-06T17:30:43ZMonica Ihliemail@monicaihli.com
<p>MN Operators would benefit from clearer feedback from the CN to be aware of when syncs fail for a PID.</p>
<ul>
<li>GMN - currently logs MNRead.synchronizationFailed() requests received as part of the regular GMN logs.</li>
<li>Metacat - also currently just logs MNRead.synchronizationFailed() requests as part of regular logs.</li>
<li>One solution might be to simply log these failures in a separate file to make it easier for node operators to identify when something fails to sync. </li>
</ul>
<p>Additionally, we should confirm what kind of information is getting sent back to MNs with MNRead.synchronizationFailed().</p>
Infrastructure - Story #8049 (In Progress): Support synchronization of system metadata for unhost...https://redmine.dataone.org/issues/80492017-03-21T05:34:38ZRob Nahfrnahf@epscor.unm.edu
<p>As part of mutable-content MN support, allow the MemberNode to keep the system metadata records for all resultant versions of its changeable entities. this allows them to keep accurate system metadata for every version even though they do not have the object bytes anymore for that version.</p>
<p>Benefits:<br>
1. MN does not orphan any objects<br>
2. MN can administer objects from past versions on their own MN. Adjust the access policy of all versions, for example.<br>
3. don't need to call cn.setObsoletedBy or leave that field empty.</p>
<p>Costs:<br>
1. requires new logic for indexing (possibly)<br>
2. requires new logic for registerSystemMEtadata (possibly)<br>
3. require new logic for synchronization </p>
<p>very similar to how we synchronize DATA objects, but don't trigger MN replication.</p>
Infrastructure - Bug #8046 (New): Mutable Member Nodes orphan unhosted objectshttps://redmine.dataone.org/issues/80462017-03-15T17:40:49ZRob Nahfrnahf@epscor.unm.edu
<p>Mutable Member Nodes that only retain the latest version of their entities rarely do the bookkeeping work on orphaned versions they no longer host. This includes:</p>
<ol>
<li>setting the obsoletedBy field of the the orphaned object</li>
<li>transferring authority to another node (CN or MN)</li>
<li> invalidating the replica in the Replica section</li>
</ol>
<p>This leads to problems in search, resolve, and updating system metadata.</p>
<p>I propose we utilize the new CN-carve-out Node Properties to enable the CN to perform these actions on the mutable member node's behalf. This can either be done with a blanket property 'isMutable', or something more specific, because some member nodes might orphan the object, but keep the system metadata. 'autoObsolete', 'autoDeorphan', 'autoReplicaAudit', or such...</p>
Infrastructure - Story #7807 (New): cn.synchronize should support synchronization failure correct...https://redmine.dataone.org/issues/78072016-05-13T16:56:25ZRob Nahfrnahf@epscor.unm.edu
<p>cn.synchronize(session, identifier) works well for its original purpose (supporting MN-driven system metadata updates, and MN-driven push synchronization), but doesn't seem to work for manual synchronization failure workflows. The main problem is that the request can only be made by the MN itself (using the MN client certificate). </p>
<p>As we envision a centralized dashboard for monitoring failed synchronization items, how do we address this situation? </p>
<p>The synchronization processing queue needs both the pid and a nodeId from where to retrieve the object. the NodeId is not specified directly in the method call, but gleaned from the session by a reverse lookup from the certificate. (It uses the first node found in the NodeList where the Node.subject field matches the certificate subject).</p>
<p>Should we allow node.contactSubjects into the algorithm?<br>
Should we add nodeId as a parameter?</p>
Infrastructure - Bug #7698 (New): Exclude SLF4J jars in d1_solr_extensions.jarhttps://redmine.dataone.org/issues/76982016-03-28T16:22:54ZRobert Waltz
<p>Dave reported:<br>
when running “service solr status” on a CN, I see a complaint about multiple SLF4J bindings:<br>
(12:05:53 PM) vieglais: SLF4J: Class path contains multiple SLF4J bindings.<br>
(12:05:53 PM) vieglais: SLF4J: Found binding in <a href="12:05:55 PM" class="external">jar:file:/var/solr/server/solr-webapp/webapp/WEB-INF/lib/d1_solr_extensions.jar!/org/slf4j/impl/StaticLoggerBinder.class</a> vieglais: SLF4J: Found binding in <a href="12:05:56 PM" class="external">jar:file:/var/solr/server/lib/ext/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class</a> vieglais: SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.<br>
(12:05:56 PM) vieglais: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]</p>
<p>looks like a class conflict due to the shaded d1_solr_extensions.jar that includes SLF4J and solr jetty that also includes SLF4J.</p>
<p>try excluding SLF4J jars in d1_solr_extensions.jar.</p>
DataONE API - Bug #7578 (New): Fix 404 link to d1_instance_generator folder in documentationhttps://redmine.dataone.org/issues/75782016-01-08T22:01:20ZBryce Mecummecum@nceas.ucsb.edu
<p>In the MN API documentation for MNStorage.create (<a href="https://jenkins-ucsb-1.dataone.org/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html//apis/MN_APIs.html#MNStorage.create">https://jenkins-ucsb-1.dataone.org/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html//apis/MN_APIs.html#MNStorage.create</a>), I found a the following paragraph contains a broken link to d1_instance_generator:</p>
<blockquote>
<p>"The system metadata included with the create call must contain values for the elements required to be set by clients (see System Metadata). The system metadata document can be crafted by hand or preferably with a tool such as generate_sysmeta.py which is available in the d1_instance_generator Python package. See documentation included with that package for more information on its operation."</p>
</blockquote>
<p>The link to d1_instance_generator was to the SVN folder <a href="https://repository.dataone.org/software/cicore/trunk/d1_instance_generator">https://repository.dataone.org/software/cicore/trunk/d1_instance_generator</a> which is currently a 404. I think the folder moved to /d1_test_utilities_python/src/d1_test/instance_generator.</p>
DataONE API - Bug #7528 (New): Incorrect argument name documented for CNCore.reserveIdentierhttps://redmine.dataone.org/issues/75282015-12-08T19:51:03ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>The docs at <a href="http://jenkins-1.dataone.org/jenkins/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html/apis/CN_APIs.html#CNCore.reserveIdentifier">http://jenkins-1.dataone.org/jenkins/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html/apis/CN_APIs.html#CNCore.reserveIdentifier</a> indicate that the 'key' for the identifier to reserve is 'id', when it should actually be 'pid'. </p>
<p>Specifying a request with id (see attached bash script) produces the error: </p>
<p><?xml version="1.0" encoding="UTF-8"?>7091b0df-df6a-424e-b46c-4c31738f1936<a href="/d1:identifier">/d1:identifier</a><?xml version="1.0" encoding="UTF-8"?><br>
<br>
The given identifier is already in use: null</p>
Infrastructure - Story #7224 (New): push synchronization request status indicator: synchronizeSta...https://redmine.dataone.org/issues/72242015-06-18T08:30:42ZRob Nahfrnahf@epscor.unm.edu
<p>Push synchronization (cn.synchronize, mn.updateSystemMetadata) involves an end-user that might want to have an idea of how long until the queued action is going to take to complete. Something as simple as returning the place in line of the sync request might suffice as the indicator, or a more complete data packet, including the place in line and the queue velocity, could be attempted.</p>
<p>The real-world analogy for this kind of indictor is taking a number at the deli-counter: You don't know when you will be served, but you know how many people are in front of you. </p>
<p>This option is a separate call to the CN to check the status of the sync request, so that the current place in line is returned. The advantage of this is that if the velocity of synchronization changes, the interested party can call again and get an updated value - it has more diagnostic and monitoring power. This could lead to over-use, however.</p>
Infrastructure - Story #2548 (New): recasting untrusted certs to public poses accessibility incon...https://redmine.dataone.org/issues/25482012-03-27T21:55:59ZRob Nahfrnahf@epscor.unm.edu
<p>KNB recasts a connection with an untrusted certificate to public, so that a client does not get "less than public" privileges.<br>
GMN throws an InvalidToken in this situation.<br>
both refuse connections from clients with expired certificates from trusted CAs.</p>
<p>This approach can cause confusion caused when the user unwittingly uses an untrusted certficate and doesn't get what they expected. If these connections were instead refused, the user would be alerted and could reconnect as a public user, if it chose.</p>
<p>brief discussion found at line 97 of : <a href="http://epad.dataone.org/20120131-authn-authz-questions">http://epad.dataone.org/20120131-authn-authz-questions</a></p>
<ul>
<li>when would honest users be in this situation?</li>
<li>elicit advantages of recasting approach</li>
<li>either way, dataone should implement uniform behavior across CN and MNs.</li>
</ul>
Java Client - Task #1647 (New): Update d1_common_java ObjectFormatServiceImpl to include addFormat()https://redmine.dataone.org/issues/16472011-06-15T21:45:55ZChris Jonescjones@nceas.ucsb.edu
<p>The common implementation should include the addFormat() method.</p>
Infrastructure - Task #1236 (New): Highly variable response times from cn.dataone.orghttps://redmine.dataone.org/issues/12362011-01-17T16:34:05ZDave Vieglaisdave.vieglais@gmail.com
<p>Performing listobjects on cn.dataone.org works but the response times are highly variable. </p>
<p>Seems likely to be a caching issue, but likely also related to the query being executed in Metacat to generate the listObjects response.</p>
<p>For example:</p>
<p>Nomadini:~ vieglais$ time curl "<a href="http://cn.dataone.org/cn/object/">http://cn.dataone.org/cn/object/</a>"<br>
<?xml version="1.0" encoding="UTF-8"?>knb:testid:20111111590640eml://ecoinformatics.org/eml-2.1.0D3339CAF019C3EB40811CC191DCDEB122011-01-12T06:40:28.012Z12knb:testid:201111115928161eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:33.441Z12some;id;with;semi;colons;1294775954675eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:30.076Z12knb:testid:201111115733756eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:24.104Z12<a href="/d1:objectList">/d1:objectList</a><br>
real 0m0.392s<br>
user 0m0.006s<br>
sys 0m0.008s<br>
Nomadini:~ vieglais$ time curl "<a href="http://cn.dataone.org/cn/object/">http://cn.dataone.org/cn/object/</a>"<br>
<?xml version="1.0" encoding="UTF-8"?>knb:testid:20111111590640eml://ecoinformatics.org/eml-2.1.0D3339CAF019C3EB40811CC191DCDEB122011-01-12T06:40:28.012Z12knb:testid:201111115928161eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:33.441Z12some;id;with;semi;colons;1294775954675eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:30.076Z12knb:testid:201111115733756eml://ecoinformatics.org/eml-2.1.04d6537f48d2967725bfcc7a9f0d5094ce4088e0975fcd3f1a361f15f46e49f832011-01-12T06:40:24.104Z12<a href="/d1:objectList">/d1:objectList</a><br>
real 1m30.255s<br>
user 0m0.008s<br>
sys 0m0.015s</p>
Infrastructure - Task #1226 (New): Verify that cn buildout is using the hudson generated Metacat ...https://redmine.dataone.org/issues/12262011-01-13T14:48:05ZDave Vieglaisdave.vieglais@gmail.com
<p>knb.war should be generated by hudson.</p>
<p>Ensure that: </p>
<ul>
<li><p>knb.war is being used by the cn-buildout</p></li>
<li><p>The latest build of knb.war is available through a persistent URL via hudson</p></li>
</ul>
Infrastructure - Task #925 (New): Test for correct response to invalid input: CCI.ITK.FUSE-clienthttps://redmine.dataone.org/issues/9252010-10-06T19:50:31ZRoger Dahldahl@unm.edu
<p>just one task of many, I'm guessing...</p>