DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-02-29T01:00:11ZDataONE Tasks
Redmine CN REST - Bug #8860 (New): /token endpoint doesn't set a content-type and character encodinghttps://redmine.dataone.org/issues/88602020-02-29T01:00:11ZBryce Mecummecum@nceas.ucsb.edu
<p>On Firefox only, requests to the /portal/token endpoint (i.e., the one MetacatUI and other clients use to fetch their auth tokens, like <a href="https://cn.dataone.org/portal/token">https://cn.dataone.org/portal/token</a>) result in errors in the browser console.</p>
<p>When you access the URL via an XHR request, you see:</p>
<blockquote>
<p>XML Parsing Error: syntax error<br>
Location: <a href="https://cn-stage.test.dataone.org/portal/token">https://cn-stage.test.dataone.org/portal/token</a><br>
Line Number 1, Column 1:</p>
</blockquote>
<p>When you access the URL directly in Firefox:</p>
<blockquote>
<p>The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.</p>
</blockquote>
<p>I had a hunch that this error would go away if the response simply had the <code>Content-Type</code> header set to <code>text/plain; charset=utf-8</code> so I spun up <code>mitmproxy</code>, made that edit to the intercepted response, and saw that the error does go away.</p>
<p>I think we should modify the portal code to set the <code>Content-Type</code> header like above so the error goes away.</p>
CN REST - Task #8810 (New): Verify configuration of portal certificateshttps://redmine.dataone.org/issues/88102019-05-21T13:00:12ZDave Vieglaisdave.vieglais@gmail.com
<p>Verify that the postinst scripts for dataone-cn-os-core and dataone-cn-portal are correctly setting the locations of the certificates for token signing.</p>
CN REST - Task #8809 (New): Adjust portal.properties for certificate configurationhttps://redmine.dataone.org/issues/88092019-05-21T12:57:41ZDave Vieglaisdave.vieglais@gmail.com
<p>Portal certificates are apparently currently configured in <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code></p>
<p>This should be changed to <code>/etc/dataone/portal/portal.properties</code> to ensure persistence between .war deployments.</p>
CN REST - Bug #8740 (New): CN resolve service returning 404 for some pidshttps://redmine.dataone.org/issues/87402018-11-08T00:21:45ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>A user has reported that certain pids (all for metadata objects) return an http 404 status for the resolve service. Here is the complete list that the user tried that didn't resolve:</p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8">https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23</a></p>
<p>The CNRead.getSystemMetadata() service returns metadata for each of these pids.</p>
CN REST - Bug #8698 (New): CN Performance degradationhttps://redmine.dataone.org/issues/86982018-09-13T21:05:40ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>Calling simple services on the CN can take a long time to execute. For example, calling </p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202">https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202</a></p>
<p>can take about 8-10 seconds, while calling the MN directly (where the resolve service redirects to) takes about 1-2 seconds:</p>
<p><a href="https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202">https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202</a></p>
CN REST - Bug #8630 (New): equivalentIdentity values use uppercase letters when the same person s...https://redmine.dataone.org/issues/86302018-06-26T14:15:00ZLauren Walkerwalker@nceas.ucsb.edu
<p>I would expect a username to appear exactly the same everywhere in the identity service. </p>
<p>Here is an example where the <code>equivalentIdentity</code> uses uppercase letters for the subject (UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org) when the <code>person</code>><code>subject</code> uses lowercase letters (uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org):</p>
<p><a href="https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X">https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X</a></p>
<pre><ns2:subjectInfo xmlns:ns2="http://ns.dataone.org/service/types/v1">
<person>
<subject>http://orcid.org/0000-0002-5944-906X</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>
UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org
</equivalentIdentity>
<verified>false</verified>
</person>
<person>
<subject>
uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org
</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>http://orcid.org/0000-0002-5944-906X</equivalentIdentity>
<verified>false</verified>
</person>
</ns2:subjectInfo>
```~~~
</pre> DataONE API - Task #8529 (New): Add field to Solr index that includes the obsolescence chainhttps://redmine.dataone.org/issues/85292018-03-29T22:36:35ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>When a new PID is added to the Solr index, populate a field that contains the entire obsolescence chain for that PID. This will greatly reduce the amount of time clients have to spend on obtaining this information.</p>
<p>For each PID that precedes the new PID in the obs. chain, update it's Solr document to include the new, complete obs. chain.</p>
<p>The new Solr field that contains the obs. chain should be a multivalued field, if the order of PIDs in the field can be guaranteed. In this scenario, the firsts PID is the lowest in the obs. chain, the lasts pid in the field is the mosts recent, un-obsoleted PID.</p>
<p>If the order cannot be guaranteed in a multi-valued field, then a simple text field should be used, with a reasonable value as a delimiter, one that is not valid for PIDs, such as a blank.</p>
<p>This task is related to <a href="https://redmine.dataone.org/issues/8528">https://redmine.dataone.org/issues/8528</a></p>
DataONE API - Task #8528 (New): Add MNRead.getVersions, CNRead.getVersionshttps://redmine.dataone.org/issues/85282018-03-29T22:18:56ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>These new services would return the obsolescence chain, as a lists of PIDs, for the specified PID. <br>
Both services (for CN and MN) would have the same argument list, i.e. for MNREAD.getVersions:</p>
<p>MNRead.getVersions(Identifier pid, Integer count, String order)</p>
<p>where the arguments are<br>
- pid: the identifier to retrieve the obs. chain for<br>
- count: specifies how much of the lists to return<br>
- if <code>count</code> is <code>null</code> return the whole list<br>
- if <code>count</code> is a positive number, return <code>count</code> pids in the specified order of, say, <code>ASC</code> or <code>DESC</code><br>
- order<br>
- if order is <code>DESC</code>, return <code>count</code> from the <code>HEAD</code>, but from the <code>TAIL</code> if <code>ASC</code>.</p>
<p>Should the presence of multiple SIDs assigned in the obsolescence chain determine the lists of PIDs that is returned? For example, <br>
should the entire chain be returned even if multiple SIDs are encountered. The document at <a href="https://releases.dataone.org/online/api-documentation-v2.0/design/ContentMutability.html#limits-on-the-series">https://releases.dataone.org/online/api-documentation-v2.0/design/ContentMutability.html#limits-on-the-series</a> indicates that multiple SIDs can be defined in an obs.chain.</p>
<p>Should PIDs be included in the list if the requesting user does not have permission to read the sysmeta for that pid? If no, then how is this gap in the chain denoted?</p>
CN REST - Story #8364 (In Progress): Ensure portal uses correct X509 certificateshttps://redmine.dataone.org/issues/83642018-02-13T20:17:25ZChris Jonescjones@nceas.ucsb.edu
<p>We've run into issues where after an upgrade of the <code>dataone-cn-portal</code> package on the CNs, the properties pointing to the public certificate and private key are incorrectly pointing to the old GeoTrust wildcard files rather than the new Lets Encrypt files:<br>
<br>
cn.server.publiccert.filename=/etc/ssl/certs/<em>.test.dataone.org.crt<br>
cn.server.privatekey.filename=/etc/ssl/private/</em>.test.dataone.org.key</p>
<p>These should be (in STAGE):</p>
<p>/etc/letsencrypt/live/cn-stage.test.dataone.org/cert.pem<br>
/etc/letsencrypt/live/cn-stage.test.dataone.org/privkey.pem</p>
<p>The issue might be that these are not being set correctly during the <code>postinst</code> script run. Jing pointed out that these values are taken from the debconf database settings that get set when <code>dataon-cn-os-core</code> is installed. So although the <code>postinst</code> script might be setting the correct values, the old cached values might still be in memory in the debconf database. If so, we'll need to clear those values during installations and upgrades.</p>
<p>Also, knowing where to look for these configuration settings can be challenging. These are referenced from <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code>. These settings should be consolidated into <code>/etc/dataone/portal/portal.properties</code> so they also don't get blown away on war file upgrades in Tomcat.</p>
CN REST - Task #7911 (New): Synchronization allows invalid checksums, preventing corrected synchttps://redmine.dataone.org/issues/79112016-10-17T15:16:07ZChris Jonescjones@nceas.ucsb.edu
<p>Normally, d1_synchronization does checksum validation of objects before registering them in the CN. However, a CHECKSUM_VERIFICATION_SIZE_BYPASS_THRESHOLD flag was introduced into TransferObjectTask that defaults to 10MB. If an object size is greater than this threshold, the checksum won't be verified. In the cn-buildout, this default is not changed in the properties file, but it can be.</p>
<p>As a result, objects below this threshold will throw an exception during sync if the checksum is incorrect, whereas those above the threshold will successfully sync with incorrect system metadata. This becomes a problem later when trying to update the system metadata with the correct checksum because this field is immutable. For example, in the STAGE environment, the following object failed to process the system metadata update:</p>
<p>[ERROR] 2016-10-17 14:54:38,112 (V2TransferObjectTask:call:269) Task-urn:node:mnTestARCTIC-urn:uuid:a3bfef74-f6e9-4ecc-871e-0a3ea764b471 - UnrecoverableException: Failed to update cn with new valid SystemMetadata! - InvalidRequest - The request is trying to modify an immutable field in the SystemMeta: the new system meta's checksum dee03804421bac149371877d2d366abb7c941fba is different to the orginal one bef6df568ed1c713a8323434694319894f25a8b9dfa704f7fe2b7d52592b2b40</p>
<p>Since MN.getChecksum() is normally being called to do the heavy lifting of calculating the actual checksum, I'm not sure why this flag was introduced. Even for muti-gigabyte files, the checksum calculation is pretty quick. To prevent the CN from ingesting incorrect system metadata, I'd suggest we consider removing this threshold, or at a minimum, set the property value to be multi-terabyte. Also, this is a case where the MN is authoritative for the system metadata, but the CN update fails because of the immutable status of the checksum. Ultimately, we shouldn't be sync'ing content with invalid checksums, which allows for the MN operator to correct the checksum and then retry the sync. Needs discussion.</p>
CN REST - Task #7750 (New): apply business rules on the CN that Subject strings will be stripped ...https://redmine.dataone.org/issues/77502016-04-26T17:30:35ZRob Nahfrnahf@epscor.unm.edu
<p>make sure that the business rule is documented</p>
<p>apply to cn_identity_manager and other places in the CN service layer.</p>
CN REST - Task #7571 (New): Description for error code 401, detail code 4957 is misleading in som...https://redmine.dataone.org/issues/75712016-01-05T16:25:01ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>When using CNDiagnostic.echoCredentials() with an expired authentication token, the returned message is:</p>
<p><?xml version="1.0" encoding="UTF-8"?><br>
<br>
No credentials were received in the request. (Session was null)</p>
<p>The description text in this case is misleading, because from the caller's point of view, credentials<br>
were provided, albeit invalid. Is it possible to determine the validity of the credentials, i.e. expiration<br>
status, and indicate that in the description?</p>
<p>BTW, this same token did work correctly before it expired.</p>
<p>A bash script is attached that recreates the error.</p>
CN REST - Bug #7440 (New): Non-discernable error during synchronization affecting (mostly) urn:no...https://redmine.dataone.org/issues/74402015-10-16T17:40:44ZMark Servillamark.servilla@gmail.com
<p>Multiple (366) non-discernable errors (see attachment) with message only stating "Cline is shutdown" occurred on 14 Oct 2015 on cn-stage.test.dataone.org (cn-stage-ucsb-1); sampling of objects/system metadata indicate all is retrievable from the MN <a href="https://dataone-dev.ecoinformatics.org.au/mn">https://dataone-dev.ecoinformatics.org.au/mn</a>. This error had also occurred (24 events) for urn:node:mnTestLTER.</p>
<p>For example:</p>
<p>cn-synchronization.log.1-[ERROR] 2015-10-14 11:33:02,749 (TransferObjectTask:write:606) Task-urn:node:mnTestAEKOS-aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/ABERBALDIE.20150515<br>
cn-synchronization.log.1:Client is shutdown.</p>
CN REST - Bug #7301 (New): CN-stage allows connections to a MN that is operating a self-signed SS...https://redmine.dataone.org/issues/73012015-08-18T18:12:14ZMark Servillamark.servilla@gmail.com
<p>CN-stage supports connections to a MN that is operating a self-signed SSL server certificate - this should not be allowed since the connection could occur with a rogue non-verified server.</p>
<p>This instance occurred with dataone-dev.ecoinformatics.org.au:443 on 18 August 2015:</p>
<p>Certificate chain<br>
0 s:/CN=dataone-dev.ecoinformatics.org.au<br>
i:/CN=dataone-dev.ecoinformatics.org.au</p>
<p>Issuer: CN=dataone-dev.ecoinformatics.org.au<br>
Validity<br>
Not Before: Aug 11 04:56:19 2015 GMT<br>
Not After : Aug 8 04:56:19 2025 GMT<br>
Subject: CN=dataone-dev.ecoinformatics.org.au</p>
CN REST - Bug #7161 (New): TERN object fails to be indexed by Solr, but successfully synchronizedhttps://redmine.dataone.org/issues/71612015-06-05T18:02:10ZMark Servillamark.servilla@gmail.com
<p>TERN object aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/V_ILLAWDB3.20150515 fails to by indexed by Solr on cn.dataone.edu (cn-ucsb-1) even though it was successfully synchronized. Investigation indicates it was not added to the HazelCast ObjectPathMap structure according to Skye Roseboom:</p>
<p>Im not sure why this pid is not appearing in the hazelcast ObjectPathMap structure.</p>
<p>I looked into the metacat database schema a bit and noticed the pid does not seem to appear in the ‘identifier_mapping’ table:</p>
<p>select * from identifier_mapping where guid='aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/V_ILLAWDB3.20150515';</p>
<table><thead>
<tr>
<th>guid</th>
<th>docid</th>
<th>rev</th>
</tr>
</thead><tbody>
</tbody></table>
<p>(0 rows)</p>
<p>Without the ‘docid’ or ‘localid’ as metacat calls them, Im don’t thing the pid could be added to the objectPathMap in hazelcast.</p>