DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-02-29T01:00:11ZDataONE Tasks
Redmine CN REST - Bug #8860 (New): /token endpoint doesn't set a content-type and character encodinghttps://redmine.dataone.org/issues/88602020-02-29T01:00:11ZBryce Mecummecum@nceas.ucsb.edu
<p>On Firefox only, requests to the /portal/token endpoint (i.e., the one MetacatUI and other clients use to fetch their auth tokens, like <a href="https://cn.dataone.org/portal/token">https://cn.dataone.org/portal/token</a>) result in errors in the browser console.</p>
<p>When you access the URL via an XHR request, you see:</p>
<blockquote>
<p>XML Parsing Error: syntax error<br>
Location: <a href="https://cn-stage.test.dataone.org/portal/token">https://cn-stage.test.dataone.org/portal/token</a><br>
Line Number 1, Column 1:</p>
</blockquote>
<p>When you access the URL directly in Firefox:</p>
<blockquote>
<p>The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.</p>
</blockquote>
<p>I had a hunch that this error would go away if the response simply had the <code>Content-Type</code> header set to <code>text/plain; charset=utf-8</code> so I spun up <code>mitmproxy</code>, made that edit to the intercepted response, and saw that the error does go away.</p>
<p>I think we should modify the portal code to set the <code>Content-Type</code> header like above so the error goes away.</p>
CN REST - Task #8810 (New): Verify configuration of portal certificateshttps://redmine.dataone.org/issues/88102019-05-21T13:00:12ZDave Vieglaisdave.vieglais@gmail.com
<p>Verify that the postinst scripts for dataone-cn-os-core and dataone-cn-portal are correctly setting the locations of the certificates for token signing.</p>
CN REST - Task #8809 (New): Adjust portal.properties for certificate configurationhttps://redmine.dataone.org/issues/88092019-05-21T12:57:41ZDave Vieglaisdave.vieglais@gmail.com
<p>Portal certificates are apparently currently configured in <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code></p>
<p>This should be changed to <code>/etc/dataone/portal/portal.properties</code> to ensure persistence between .war deployments.</p>
CN REST - Bug #8740 (New): CN resolve service returning 404 for some pidshttps://redmine.dataone.org/issues/87402018-11-08T00:21:45ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>A user has reported that certain pids (all for metadata objects) return an http 404 status for the resolve service. Here is the complete list that the user tried that didn't resolve:</p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8">https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23</a></p>
<p>The CNRead.getSystemMetadata() service returns metadata for each of these pids.</p>
CN REST - Bug #8698 (New): CN Performance degradationhttps://redmine.dataone.org/issues/86982018-09-13T21:05:40ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>Calling simple services on the CN can take a long time to execute. For example, calling </p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202">https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202</a></p>
<p>can take about 8-10 seconds, while calling the MN directly (where the resolve service redirects to) takes about 1-2 seconds:</p>
<p><a href="https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202">https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202</a></p>
CN REST - Bug #8630 (New): equivalentIdentity values use uppercase letters when the same person s...https://redmine.dataone.org/issues/86302018-06-26T14:15:00ZLauren Walkerwalker@nceas.ucsb.edu
<p>I would expect a username to appear exactly the same everywhere in the identity service. </p>
<p>Here is an example where the <code>equivalentIdentity</code> uses uppercase letters for the subject (UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org) when the <code>person</code>><code>subject</code> uses lowercase letters (uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org):</p>
<p><a href="https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X">https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X</a></p>
<pre><ns2:subjectInfo xmlns:ns2="http://ns.dataone.org/service/types/v1">
<person>
<subject>http://orcid.org/0000-0002-5944-906X</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>
UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org
</equivalentIdentity>
<verified>false</verified>
</person>
<person>
<subject>
uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org
</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>http://orcid.org/0000-0002-5944-906X</equivalentIdentity>
<verified>false</verified>
</person>
</ns2:subjectInfo>
```~~~
</pre> CN REST - Story #8364 (In Progress): Ensure portal uses correct X509 certificateshttps://redmine.dataone.org/issues/83642018-02-13T20:17:25ZChris Jonescjones@nceas.ucsb.edu
<p>We've run into issues where after an upgrade of the <code>dataone-cn-portal</code> package on the CNs, the properties pointing to the public certificate and private key are incorrectly pointing to the old GeoTrust wildcard files rather than the new Lets Encrypt files:<br>
<br>
cn.server.publiccert.filename=/etc/ssl/certs/<em>.test.dataone.org.crt<br>
cn.server.privatekey.filename=/etc/ssl/private/</em>.test.dataone.org.key</p>
<p>These should be (in STAGE):</p>
<p>/etc/letsencrypt/live/cn-stage.test.dataone.org/cert.pem<br>
/etc/letsencrypt/live/cn-stage.test.dataone.org/privkey.pem</p>
<p>The issue might be that these are not being set correctly during the <code>postinst</code> script run. Jing pointed out that these values are taken from the debconf database settings that get set when <code>dataon-cn-os-core</code> is installed. So although the <code>postinst</code> script might be setting the correct values, the old cached values might still be in memory in the debconf database. If so, we'll need to clear those values during installations and upgrades.</p>
<p>Also, knowing where to look for these configuration settings can be challenging. These are referenced from <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code>. These settings should be consolidated into <code>/etc/dataone/portal/portal.properties</code> so they also don't get blown away on war file upgrades in Tomcat.</p>
CN REST - Task #7911 (New): Synchronization allows invalid checksums, preventing corrected synchttps://redmine.dataone.org/issues/79112016-10-17T15:16:07ZChris Jonescjones@nceas.ucsb.edu
<p>Normally, d1_synchronization does checksum validation of objects before registering them in the CN. However, a CHECKSUM_VERIFICATION_SIZE_BYPASS_THRESHOLD flag was introduced into TransferObjectTask that defaults to 10MB. If an object size is greater than this threshold, the checksum won't be verified. In the cn-buildout, this default is not changed in the properties file, but it can be.</p>
<p>As a result, objects below this threshold will throw an exception during sync if the checksum is incorrect, whereas those above the threshold will successfully sync with incorrect system metadata. This becomes a problem later when trying to update the system metadata with the correct checksum because this field is immutable. For example, in the STAGE environment, the following object failed to process the system metadata update:</p>
<p>[ERROR] 2016-10-17 14:54:38,112 (V2TransferObjectTask:call:269) Task-urn:node:mnTestARCTIC-urn:uuid:a3bfef74-f6e9-4ecc-871e-0a3ea764b471 - UnrecoverableException: Failed to update cn with new valid SystemMetadata! - InvalidRequest - The request is trying to modify an immutable field in the SystemMeta: the new system meta's checksum dee03804421bac149371877d2d366abb7c941fba is different to the orginal one bef6df568ed1c713a8323434694319894f25a8b9dfa704f7fe2b7d52592b2b40</p>
<p>Since MN.getChecksum() is normally being called to do the heavy lifting of calculating the actual checksum, I'm not sure why this flag was introduced. Even for muti-gigabyte files, the checksum calculation is pretty quick. To prevent the CN from ingesting incorrect system metadata, I'd suggest we consider removing this threshold, or at a minimum, set the property value to be multi-terabyte. Also, this is a case where the MN is authoritative for the system metadata, but the CN update fails because of the immutable status of the checksum. Ultimately, we shouldn't be sync'ing content with invalid checksums, which allows for the MN operator to correct the checksum and then retry the sync. Needs discussion.</p>
CN REST - Task #7750 (New): apply business rules on the CN that Subject strings will be stripped ...https://redmine.dataone.org/issues/77502016-04-26T17:30:35ZRob Nahfrnahf@epscor.unm.edu
<p>make sure that the business rule is documented</p>
<p>apply to cn_identity_manager and other places in the CN service layer.</p>
CN REST - Task #7571 (New): Description for error code 401, detail code 4957 is misleading in som...https://redmine.dataone.org/issues/75712016-01-05T16:25:01ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>When using CNDiagnostic.echoCredentials() with an expired authentication token, the returned message is:</p>
<p><?xml version="1.0" encoding="UTF-8"?><br>
<br>
No credentials were received in the request. (Session was null)</p>
<p>The description text in this case is misleading, because from the caller's point of view, credentials<br>
were provided, albeit invalid. Is it possible to determine the validity of the credentials, i.e. expiration<br>
status, and indicate that in the description?</p>
<p>BTW, this same token did work correctly before it expired.</p>
<p>A bash script is attached that recreates the error.</p>
CN REST - Bug #7440 (New): Non-discernable error during synchronization affecting (mostly) urn:no...https://redmine.dataone.org/issues/74402015-10-16T17:40:44ZMark Servillamark.servilla@gmail.com
<p>Multiple (366) non-discernable errors (see attachment) with message only stating "Cline is shutdown" occurred on 14 Oct 2015 on cn-stage.test.dataone.org (cn-stage-ucsb-1); sampling of objects/system metadata indicate all is retrievable from the MN <a href="https://dataone-dev.ecoinformatics.org.au/mn">https://dataone-dev.ecoinformatics.org.au/mn</a>. This error had also occurred (24 events) for urn:node:mnTestLTER.</p>
<p>For example:</p>
<p>cn-synchronization.log.1-[ERROR] 2015-10-14 11:33:02,749 (TransferObjectTask:write:606) Task-urn:node:mnTestAEKOS-aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/ABERBALDIE.20150515<br>
cn-synchronization.log.1:Client is shutdown.</p>
CN REST - Bug #7301 (New): CN-stage allows connections to a MN that is operating a self-signed SS...https://redmine.dataone.org/issues/73012015-08-18T18:12:14ZMark Servillamark.servilla@gmail.com
<p>CN-stage supports connections to a MN that is operating a self-signed SSL server certificate - this should not be allowed since the connection could occur with a rogue non-verified server.</p>
<p>This instance occurred with dataone-dev.ecoinformatics.org.au:443 on 18 August 2015:</p>
<p>Certificate chain<br>
0 s:/CN=dataone-dev.ecoinformatics.org.au<br>
i:/CN=dataone-dev.ecoinformatics.org.au</p>
<p>Issuer: CN=dataone-dev.ecoinformatics.org.au<br>
Validity<br>
Not Before: Aug 11 04:56:19 2015 GMT<br>
Not After : Aug 8 04:56:19 2025 GMT<br>
Subject: CN=dataone-dev.ecoinformatics.org.au</p>
CN REST - Bug #7161 (New): TERN object fails to be indexed by Solr, but successfully synchronizedhttps://redmine.dataone.org/issues/71612015-06-05T18:02:10ZMark Servillamark.servilla@gmail.com
<p>TERN object aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/V_ILLAWDB3.20150515 fails to by indexed by Solr on cn.dataone.edu (cn-ucsb-1) even though it was successfully synchronized. Investigation indicates it was not added to the HazelCast ObjectPathMap structure according to Skye Roseboom:</p>
<p>Im not sure why this pid is not appearing in the hazelcast ObjectPathMap structure.</p>
<p>I looked into the metacat database schema a bit and noticed the pid does not seem to appear in the ‘identifier_mapping’ table:</p>
<p>select * from identifier_mapping where guid='aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/V_ILLAWDB3.20150515';</p>
<table><thead>
<tr>
<th>guid</th>
<th>docid</th>
<th>rev</th>
</tr>
</thead><tbody>
</tbody></table>
<p>(0 rows)</p>
<p>Without the ‘docid’ or ‘localid’ as metacat calls them, Im don’t thing the pid could be added to the objectPathMap in hazelcast.</p>
CN REST - Bug #5739 (New): LDAP upgrades fail with purge of dataone-cn-os-corehttps://redmine.dataone.org/issues/57392014-07-16T20:47:31ZRobert Waltz
<p>open ldap should continue to function normally on a host after dataone-cn-os-core has been purged (apt-get remove --purge dataone-cn-os-core)</p>
<p>We should wipe out /etc/ldap/slapd.d, restore it to a simpler configuration, backup and then remove the dataone entries from the openldap database.</p>
<p>We will need to complete this before migrating to 14.04 of ubuntu</p>
CN REST - Task #2414 (New): Implement exceptions for search/solr endointhttps://redmine.dataone.org/issues/24142012-02-27T21:55:19ZSkye Roseboomsroseboo@dataone.unm.edu
<p><a href="http://mule1.dataone.org/ArchitectureDocs-current/apis/CN_APIs.html#CNRead.search">http://mule1.dataone.org/ArchitectureDocs-current/apis/CN_APIs.html#CNRead.search</a></p>
<p>Not currently provided with mod_rewrite</p>