DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-08-06T00:06:07ZDataONE Tasks
Redmine CN REST - Bug #8867 (New): CNCore.listChecksumAlgorithms() returns incorrect listhttps://redmine.dataone.org/issues/88672020-08-06T00:06:07ZMatthew Jonesjones@nceas.ucsb.edu
<p>The definition of the ChecksumAlgorithm type in SystemMetadata allows any checksum algorithm listed in the Library of Congress vocab. But the current CNCore.listChecksumAlgorithms() implementation only returns two, MD5 and SHA-1. Need to correct this to include the full list of supported algorithms (see <a href="http://id.loc.gov/vocabulary/preservation/cryptographicHashFunctions.html">http://id.loc.gov/vocabulary/preservation/cryptographicHashFunctions.html</a>).</p>
<p>The implementation of this is in a property file, which needs to be updated with the correct list. The file (d1_cn_rest/src/test/resources/org/dataone/configuration/node.properties) currently contains:</p>
<p><code>cn.checksumAlgorithmList=SHA-1;MD5</code></p>
<p>But it should contain all of the other valid algorithms as well from the LoC.</p>
CN REST - Bug #8860 (New): /token endpoint doesn't set a content-type and character encodinghttps://redmine.dataone.org/issues/88602020-02-29T01:00:11ZBryce Mecummecum@nceas.ucsb.edu
<p>On Firefox only, requests to the /portal/token endpoint (i.e., the one MetacatUI and other clients use to fetch their auth tokens, like <a href="https://cn.dataone.org/portal/token">https://cn.dataone.org/portal/token</a>) result in errors in the browser console.</p>
<p>When you access the URL via an XHR request, you see:</p>
<blockquote>
<p>XML Parsing Error: syntax error<br>
Location: <a href="https://cn-stage.test.dataone.org/portal/token">https://cn-stage.test.dataone.org/portal/token</a><br>
Line Number 1, Column 1:</p>
</blockquote>
<p>When you access the URL directly in Firefox:</p>
<blockquote>
<p>The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.</p>
</blockquote>
<p>I had a hunch that this error would go away if the response simply had the <code>Content-Type</code> header set to <code>text/plain; charset=utf-8</code> so I spun up <code>mitmproxy</code>, made that edit to the intercepted response, and saw that the error does go away.</p>
<p>I think we should modify the portal code to set the <code>Content-Type</code> header like above so the error goes away.</p>
CN REST - Task #8777 (New): Configure CN to audit objects greater than 1GBhttps://redmine.dataone.org/issues/87772019-03-12T16:47:42ZChris Jonescjones@nceas.ucsb.edu
<p>The replication auditor currently limits auditing of objects at 1GB. There are currently 4 objects greater than 1TB in size, and 3,588 objects greater than 1GB in size, both being very small counts compared to the 2,769,111 objects less than 1GB in size in the network. Nonetheless, they should still be audited if feasible. The limiting factor is likely HTTP timeout limits during the call to <code>MN.getChecksum()</code>. For reference, I'm seeing the following general times for calculating MD5 and SHA-1 checksums:</p>
<pre>Size MD5 SHA-1
---- ------- -------
1GB 00m02.5s 00m02.6s
10GB 00m25.9s 00m30.0s
100GB 03m28.0s 04m01.8s
1TB 50m14.2s 67m38.6s
</pre>
<p>10GB and 100GB objects seem pretty feasible if we set the HTTP client timeout to > 5 minutes, whereas the few > 1TB files may be challenging just due to the timeouts. The other factor is that the <code>AbstractReplicationAuditor</code> sets a default timeout to 60 seconds, and if the task future doesn't return in that time frame, the future gets cancelled. So the HTTP timeout and this timeout need to be increased and coordinated in order to handle larger object auditing.</p>
CN REST - Story #8771 (New): Issue with LDAP when updating `nodeReplicationPolicy`https://redmine.dataone.org/issues/87712019-03-05T19:42:17ZRoger Dahldahl@unm.edu
<p>When a submitting a Node doc update which includes a nodeReplicationPolicy, this section is good:</p>
<pre><nodeReplicationPolicy>
<maxObjectSize>21474836480</maxObjectSize>
<spaceAllocated>1099511627776</spaceAllocated>
</nodeReplicationPolicy>
</pre>
<p>while the same section without <code>maxObjectSize</code> returns error:</p>
<pre> <error detailCode="4822" errorCode="500" name="ServiceFailure">
<description>updateNodeCapabilities failed due to LDAP communication failure:: InvalidAttributeValueException:[LDAP: error code 21 - d1ReplicationPolicyMaxObjectSize: value #0 invalid per syntax]:[LDAP: error code 21 - d1ReplicationPolicyMaxObjectSize: value #0 invalid per syntax]</description>
</error>
</pre>
<p>The schema allows leaving <code>maxObjectSize</code> out, which means that the MN accepts replicas of unlimited size.</p>
<p>Both GMN and Metacat leave <code>maxObjectSize</code> out if the setting is configured to unlimited with <code>-1</code>.</p>
<p>I think it used to work.</p>
CN REST - Story #8770 (New): Issue with CN handling of encoded identifiers in object/ meta/ node/...https://redmine.dataone.org/issues/87702019-03-05T19:37:13ZRoger Dahldahl@unm.edu
<p>Works:<br>
<a href="http://cn.dataone.org/cn/v2/object/doi:10.6073/AA/knb-lter-bes.298.37">http://cn.dataone.org/cn/v2/object/doi:10.6073/AA/knb-lter-bes.298.37</a><br>
<a href="https://cn.dataone.org/cn/v2/node/urn:node:LTER">https://cn.dataone.org/cn/v2/node/urn:node:LTER</a></p>
<p>Does not work:<br>
<a href="http://cn.dataone.org/cn/v2/object/doi%3A10.6073%2FAA%2Fknb-lter-bes.298.37">http://cn.dataone.org/cn/v2/object/doi%3A10.6073%2FAA%2Fknb-lter-bes.298.37</a><br>
<a href="https://cn.dataone.org/cn/v2/node/urn%3Anode%3ALTER">https://cn.dataone.org/cn/v2/node/urn%3Anode%3ALTER</a></p>
<p>Note: Behavior differs between HTTP / HTTPS.</p>
CN REST - Bug #8740 (New): CN resolve service returning 404 for some pidshttps://redmine.dataone.org/issues/87402018-11-08T00:21:45ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>A user has reported that certain pids (all for metadata objects) return an http 404 status for the resolve service. Here is the complete list that the user tried that didn't resolve:</p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8">https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23</a></p>
<p>The CNRead.getSystemMetadata() service returns metadata for each of these pids.</p>
CN REST - Bug #8733 (New): Exception handling in ForesiteResourceMap needs improvementhttps://redmine.dataone.org/issues/87332018-10-12T20:09:55ZChris Jonescjones@nceas.ucsb.edu
<p>We're seeing some errors from the <code>org.dataone.cn.indexer.resourcemap.ForesiteResourceMap</code> constructor when indexing resource maps in Metacat 2.9.0, d1_cn_index_processor 2.3.4. A general <code>Exception</code> is being caught and re-thrown as an <code>OREParserException</code> on line 104:</p>
<pre>metacat 20181012-10:53:02: [INFO]: done queueing doc index for pid urn:uuid:5a860679-159e-45a6-bcb6-beb67f4166c2 [edu.ucsb.nceas.metacat.MetacatHandler:handleReindexAction:2777]
metacat 20181012-10:53:02: [DEBUG]: ================= in the finally statement [edu.ucsb.nceas.metacat.MetacatHandler:handleReindexAction:2808]
metacat 20181012-10:53:02: [DEBUG]: ================= in the finally statement which out is not null [edu.ucsb.nceas.metacat.MetacatHandler:handleReindexAction:2810]
org.dspace.foresite.OREParserException: java.lang.NullPointerException
at org.dataone.cn.indexer.resourcemap.ForesiteResourceMap.<init>(ForesiteResourceMap.java:104)
at org.dataone.cn.indexer.resourcemap.ResourceMapFactory.buildResourceMap(ResourceMapFactory.java:45)
at edu.ucsb.nceas.metacat.index.resourcemap.ResourceMapSubprocessor.processResourceMap(ResourceMapSubprocessor.java:97)
at edu.ucsb.nceas.metacat.index.resourcemap.ResourceMapSubprocessor.processDocument(ResourceMapSubprocessor.java:86)
at edu.ucsb.nceas.metacat.index.SolrIndex.process(SolrIndex.java:236)
at edu.ucsb.nceas.metacat.index.SolrIndex.insert(SolrIndex.java:426)
at edu.ucsb.nceas.metacat.index.SolrIndex.update(SolrIndex.java:637)
at edu.ucsb.nceas.metacat.index.SolrIndex.update(SolrIndex.java:592)
at edu.ucsb.nceas.metacat.index.SystemMetadataEventListener.entryUpdated(SystemMetadataEventListener.java:146)
at edu.ucsb.nceas.metacat.index.SystemMetadataEventListener.entryAdded(SystemMetadataEventListener.java:119)
at com.hazelcast.client.impl.EntryListenerManager.notifyListeners(EntryListenerManager.java:148)
at com.hazelcast.client.impl.EntryListenerManager.notifyListeners(EntryListenerManager.java:130)
at com.hazelcast.client.impl.ListenerManager.customRun(ListenerManager.java:88)
at com.hazelcast.client.ClientRunnable.run(ClientRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
</pre>
<p>The resource map that is failing to index is <a href="https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3A5a860679-159e-45a6-bcb6-beb67f4166c2">https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3A5a860679-159e-45a6-bcb6-beb67f4166c2</a></p>
<p>This resource map parses fine using <code>com.hp.hpl.jena.rdf.model.Model.read()</code> outside of the index processor code.</p>
<p>This <code>ForesiteResourceMap</code> constructor is taking an <code>org.w3c.dom.Document</code> as input, converting it to an in-memory <code>ByteArrayOutputStream</code>, and converting that to an in-memory <code>String</code>, to finally get a <code>ReaderInputStream</code> to be passed to <code>_init()</code>. We may need to address the handling inefficiency as a secondary improvement to the code, if feasible.</p>
<p>We also need to not catch a general <code>Exception</code> on line 117, but rather catch the more specific exception that is thrown, which will help with troubleshooting where the exception is coming from. Debug logging statements would help too.</p>
<p>The exception is a <code>NullPointerException</code>, so we ultimately need to track down where this is coming from.</p>
<p>It doesn't look like this code has changed in the 2.3.6 version of the library, so an upgrade wouldn't make a difference. I did look on <code>cn-orc-1</code> to see if we are having similar issues outside of a Metacat MN context, and see errors like these:</p>
<pre>cn-index-processor-daemon.log.9-[ERROR] 2018-10-12 06:42:44,977 (ErrorHandlerFactory$ErrorLogger:logError:84) {E201} rdf:resource not allowed as attribute here.
cn-index-processor-daemon.log.9:[ERROR] 2018-10-12 06:42:44,977 (ForesiteResourceMap:_init:163) Unable to parse ORE document:
cn-index-processor-daemon.log.9-org.apache.jena.riot.RiotException: {E201} rdf:resource not allowed as attribute here.
cn-index-processor-daemon.log.9- at org.apache.jena.riot.system.ErrorHandlerFactory$ErrorHandlerStd.error(ErrorHandlerFactory.java:128)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.lang.LangRDFXML$ErrorHandlerBridge.error(LangRDFXML.java:241)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.ARPSaxErrorHandler.error(ARPSaxErrorHandler.java:38)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.warning(XMLHandler.java:206)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.warning(XMLHandler.java:183)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.warning(XMLHandler.java:178)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.ParserSupport.warning(ParserSupport.java:199)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.AttributeLexer.error(AttributeLexer.java:187)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.QNameLexer.lookup(QNameLexer.java:80)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.AttributeLexer.processSpecials(AttributeLexer.java:71)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.states.WantDescription.startElement(WantDescription.java:60)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.states.AbsWantLiteralValueOrDescription.startElement(AbsWantLiteralValueOrDescription.java:84)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.states.WantLiteralValueOrDescription.startElement(WantLiteralValueOrDescription.java:56)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.startElement(XMLHandler.java:121)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLNamespaceBinder.handleStartElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLNamespaceBinder.emptyElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.RDFXMLParser.parse(RDFXMLParser.java:151)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.ARP.load(ARP.java:119)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.lang.LangRDFXML.parse(LangRDFXML.java:142)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:142)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:818)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRIOT.java:69)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.model.impl.ModelCom.read(ModelCom.java:274)
cn-index-processor-daemon.log.9- at org.dspace.foresite.jena.JenaOREParser.parseToModel(JenaOREParser.java:119)
cn-index-processor-daemon.log.9- at org.dspace.foresite.jena.JenaOREParser.parse(JenaOREParser.java:69)
cn-index-processor-daemon.log.9- at org.dataone.ore.ResourceMapFactory.parseResourceMap(ResourceMapFactory.java:444)
cn-index-processor-daemon.log.9: at org.dataone.cn.indexer.resourcemap.ForesiteResourceMap._init(ForesiteResourceMap.java:160)
cn-index-processor-daemon.log.9: at org.dataone.cn.indexer.resourcemap.ForesiteResourceMap.<init>(ForesiteResourceMap.java:73)
cn-index-processor-daemon.log.9- at org.dataone.cn.indexer.resourcemap.ResourceMapFactory.buildResourceMap(ResourceMapFactory.java:41)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessor.getNextIndexTask(IndexTaskProcessor.java:641)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessor.processFailedIndexTaskQueue(IndexTaskProcessor.java:229)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessor.processIndexTaskQueue(IndexTaskProcessor.java:191)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessorJob.execute(IndexTaskProcessorJob.java:57)
</pre>
<p>This error is also related to <code>_init</code>, but isn't an NPE, so <code>_init()</code> might be a good place to look first.</p>
CN REST - Bug #8698 (New): CN Performance degradationhttps://redmine.dataone.org/issues/86982018-09-13T21:05:40ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>Calling simple services on the CN can take a long time to execute. For example, calling </p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202">https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202</a></p>
<p>can take about 8-10 seconds, while calling the MN directly (where the resolve service redirects to) takes about 1-2 seconds:</p>
<p><a href="https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202">https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202</a></p>
CN REST - Bug #8630 (New): equivalentIdentity values use uppercase letters when the same person s...https://redmine.dataone.org/issues/86302018-06-26T14:15:00ZLauren Walkerwalker@nceas.ucsb.edu
<p>I would expect a username to appear exactly the same everywhere in the identity service. </p>
<p>Here is an example where the <code>equivalentIdentity</code> uses uppercase letters for the subject (UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org) when the <code>person</code>><code>subject</code> uses lowercase letters (uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org):</p>
<p><a href="https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X">https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X</a></p>
<pre><ns2:subjectInfo xmlns:ns2="http://ns.dataone.org/service/types/v1">
<person>
<subject>http://orcid.org/0000-0002-5944-906X</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>
UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org
</equivalentIdentity>
<verified>false</verified>
</person>
<person>
<subject>
uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org
</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>http://orcid.org/0000-0002-5944-906X</equivalentIdentity>
<verified>false</verified>
</person>
</ns2:subjectInfo>
```~~~
</pre> CN REST - Task #8469 (In Progress): evaluate if ORCID API will continue to work after 1.2 is depr...https://redmine.dataone.org/issues/84692018-03-02T00:37:55ZMatthew Jonesjones@nceas.ucsb.edu
<p>We use ORCID to authenticate users via OAuth. ORCID has announced that it will completely sunset their version 1.2 API on March 1, 2018 (today). See <a href="https://github.com/ORCID/ORCID-Source/blob/master/orcid-model/src/main/resources/README.md">https://github.com/ORCID/ORCID-Source/blob/master/orcid-model/src/main/resources/README.md</a> The API has been deprecated since early 2017. </p>
<p>We use OAuth portions of the API which do not seem to be affected by the XSD version changes in the ORCID API, but we should evaluate whether this will affect us. In particular, I note that they state that the proper endpoints for OAuth are:</p>
<ul>
<li><a href="http://orcid.org/oauth/authorize">http://orcid.org/oauth/authorize</a></li>
<li><a href="https://orcid.org/oauth/token">https://orcid.org/oauth/token</a></li>
</ul>
<p>However, for the second of these, our configuration file (<a href="https://repository.dataone.org/software/cicore/trunk/d1_portal_servlet/src/main/webapp/WEB-INF/portal.properties">https://repository.dataone.org/software/cicore/trunk/d1_portal_servlet/src/main/webapp/WEB-INF/portal.properties</a>) indicates that we use:</p>
<ul>
<li><a href="https://pub.orcid.org/oauth/token">https://pub.orcid.org/oauth/token</a></li>
</ul>
<p>I think the <code>pub</code> endpoints have been deprecated, and we may need to change our configuration to use the established endpoint. Evaluate and possibly change this if needed.</p>
CN REST - Story #8364 (In Progress): Ensure portal uses correct X509 certificateshttps://redmine.dataone.org/issues/83642018-02-13T20:17:25ZChris Jonescjones@nceas.ucsb.edu
<p>We've run into issues where after an upgrade of the <code>dataone-cn-portal</code> package on the CNs, the properties pointing to the public certificate and private key are incorrectly pointing to the old GeoTrust wildcard files rather than the new Lets Encrypt files:<br>
<br>
cn.server.publiccert.filename=/etc/ssl/certs/<em>.test.dataone.org.crt<br>
cn.server.privatekey.filename=/etc/ssl/private/</em>.test.dataone.org.key</p>
<p>These should be (in STAGE):</p>
<p>/etc/letsencrypt/live/cn-stage.test.dataone.org/cert.pem<br>
/etc/letsencrypt/live/cn-stage.test.dataone.org/privkey.pem</p>
<p>The issue might be that these are not being set correctly during the <code>postinst</code> script run. Jing pointed out that these values are taken from the debconf database settings that get set when <code>dataon-cn-os-core</code> is installed. So although the <code>postinst</code> script might be setting the correct values, the old cached values might still be in memory in the debconf database. If so, we'll need to clear those values during installations and upgrades.</p>
<p>Also, knowing where to look for these configuration settings can be challenging. These are referenced from <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code>. These settings should be consolidated into <code>/etc/dataone/portal/portal.properties</code> so they also don't get blown away on war file upgrades in Tomcat.</p>
CN REST - Bug #7918 (New): SEAD object only partially synchronized - missing autogen.201609291601...https://redmine.dataone.org/issues/79182016-10-21T17:20:29ZDave Vieglaisdave.vieglais@gmail.com
<p>While updating <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: SEAD object only partially synchronized - missing autogen.2015061616000265251 document from /var/... (Closed)" href="https://redmine.dataone.org/issues/7222">#7222</a> and <a class="issue tracker-5 status-5 priority-4 priority-default closed" title="Task: SEAD object only partially synchronized in CN production - missing autogen.2016092916013111425 do... (Closed)" href="https://redmine.dataone.org/issues/7914">#7914</a>. Checked for content for those, and it is accessible through CN. May still be an issue however as the SEAD MN reports 69 objects, though<br>
there are 68 reported in the search index.</p>
<p>Counting on the MN:</p>
<p>curl "<a href="http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object">http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object</a>" | xml fo</p>
<p>returns 69 objects.</p>
<p>Counting on the CN::</p>
<p>curl "<a href="https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD">https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD</a>" | xml fo</p>
<p>returns 69 objects.</p>
<p>Counting on the CN using the search index::</p>
<p><a href="https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=10&fl=id%2Ctitle%2CformatId&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22">https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=10&fl=id%2Ctitle%2CformatId&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22</a></p>
<p>returns 68 objects.</p>
<p>List of identifiers in search index::</p>
<p>curl "<a href="https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=100&fl=id&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22">https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=100&fl=id&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22</a>" | xml sel -t -m "//doc/str[@name='id']" -v . -n | sort > SEAD_index_pids.txt</p>
<p>List of identifiers on CN::</p>
<p>curl "<a href="https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD">https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD</a>" | xml sel -t -m "//objectInfo/identifier" -v . -n | sort > SEAD_cn_pids.txt</p>
<p>Missing PID::</p>
<p>diff SEAD_cn_pids.txt SEAD_index_pids.txt<br>
60d59<br>
< seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</p>
<p>System Metadata for PID::</p>
<p><?xml version="1.0"?><br>
<br>
1<br>
seadva-c918e4ff-2861-496a-a907-d2cb382ddb30<br>
FGDC-STD-001-1998<br>
6553<br>
ff3d4641669b4e08c9f8b978b85d6113a4c1bab8<br>
SEAD<br>
CN=urn:node:SEAD,DC=dataone,DC=org<br>
<br>
<br>
public<br>
read<br>
<br>
<br>
<br>
sead-Martin-John-f1dbc3df-c27c-4647-b05a-4b1f05c99a24<br>
2016-08-17T12:41:34.03Z<br>
2016-08-17T14:43:58.677Z<br>
urn:node:SEAD<br>
urn:node:SEAD<br>
<br>
urn:node:SEAD<br>
completed<br>
2016-09-29T23:01:21.235Z<br>
<br>
<br>
urn:node:CN<br>
completed<br>
2016-09-29T23:01:21.241Z<br>
<br>
<a href="/ns1:systemMetadata">/ns1:systemMetadata</a></p>
<p>Confirm object can be retieved::</p>
<p>curl "<a href="http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object/seadva-c918e4ff-2861-496a-a907-d2cb382ddb30">http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object/seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</a>" | xml fo</p>
<p>System Metadata for obsoleted object::</p>
<p><?xml version="1.0"?><br>
<br>
1<br>
sead-Martin-John-f1dbc3df-c27c-4647-b05a-4b1f05c99a24<br>
FGDC-STD-001-1998<br>
7443<br>
ebd481ad8268f9714f23d772c15fb62e61384486<br>
CN=urn:node:SEAD,DC=dataone,DC=org<br>
CN=urn:node:SEAD,DC=dataone,DC=org<br>
<br>
<br>
public<br>
read<br>
<br>
<br>
<br>
seadva-c918e4ff-2861-496a-a907-d2cb382ddb30<br>
2013-10-24T18:41:31.213Z<br>
2016-08-17T14:43:58.677Z<br>
urn:node:SEAD<br>
urn:node:SEAD<br>
<br>
urn:node:CN<br>
completed<br>
2013-10-24T23:00:04.684Z<br>
<br>
<br>
urn:node:SEAD<br>
completed<br>
2013-10-24T23:00:04.579Z<br>
<br>
<a href="/ns1:systemMetadata">/ns1:systemMetadata</a><br></p>
<p>autogenid for <code>seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</code>::</p>
<p>psql metacat<br>
select * from identifier where guid='seadva-c918e4ff-2861-496a-a907-d2cb382ddb30';</p>
<table><thead>
<tr>
<th>guid</th>
<th>docid</th>
<th>rev</th>
</tr>
</thead><tbody>
<tr>
<td>seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</td>
<td>autogen.2016092916012224122</td>
<td>1</td>
</tr>
</tbody></table>
<p>$ ls /var/metacat/data/autogen.2016092916012224122*<br>
ls: cannot access /var/metacat/data/autogen.2016092916012224122: No such file or directory</p>
CN REST - Task #7903 (New): Need to implement/support the default http methods - HEAD and GET ...https://redmine.dataone.org/issues/79032016-10-07T23:59:15ZMatthew Jonesjones@nceas.ucsb.edu
<p>Developers on the Whole Tale project at NCSA reported a bug in the HTTP HEAD request for our resolve service URIs. Example output below to reproduce the error. </p>
<p>Expected: a status code of 200</p>
<p>xarth@shakuras ~ $ curl --head <a href="https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f">https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f</a><br>
HTTP/1.1 500 Internal Server Error<br>
Date: Fri, 07 Oct 2016 23:00:14 GMT<br>
Server: Apache/2.2.22 (Ubuntu)<br>
Content-Length: 260<br>
Access-Control-Allow-Origin: <br>
Access-Control-Allow-Credentials: true<br>
Access-Control-Allow-Headers: Authorization, Content-Type, Location, Content-Length, x-annotator-auth-token<br>
Access-Control-Expose-Headers: Content-Length, Content-Type, Location<br>
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE<br>
Vary: Accept-Encoding<br>
Connection: close<br>
Content-Type: text/xml;charset=UTF-8</p>
<p>xarth@shakuras ~ $ curl --head <a href="https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f">https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f</a><br>
HTTP/1.1 200 OK<br>
Date: Fri, 07 Oct 2016 23:00:48 GMT<br>
Server: Apache/2.4.7 (Ubuntu)<br>
Set-Cookie: JSESSIONID=7DC18368F71D5D9948371B3C33437E8B; Path=/knb/; Secure<br>
DataONE-Checksum: SHA-1,927a11b6e46b771c9922083814f6ee8e5b09f696<br>
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT<br>
DataONE-ObjectFormat: application/octet-stream<br>
DataONE-SerialVersion: 0<br>
Content-Length: 2809655736<br>
Access-Control-Allow-Origin: <br>
Access-Control-Allow-Credentials: true<br>
Content-Type: text/xml</p>
CN REST - Bug #7746 (In Progress): Node registration update fails when <contactSubject> spans mul...https://redmine.dataone.org/issues/77462016-04-19T23:55:09ZMark Servillamark.servilla@gmail.com
<p>Node registration update fails when the string in the node registration document (attached) is on a separate line from the XML element tags. Apparently, parsing this field results in a subject string that does not match the LDAP record correctly, and the result is 401 not authorized exception:</p>
<p>CN=Lisa Stillwell A15851,O=University of North Carolina at Chapel Hill,C=US,DC=cilogon,DC=org<br>
</p>
<p>results in:<br>
<br>
<?xml version="1.0" encoding="UTF-8"?><br>
<br>
<br>
CN=Lisa Stillwell A15851,O=University of North Carolina at Chapel Hill,C=US,DC=cilogon,DC=org<br>
is not a Registered Subject<br>
</p>
<p>Correcting to:<br>
<br>
CN=Lisa Stillwell A15851,O=University of North Carolina at Chapel Hill,C=US,DC=cilogon,DC=org</p>
<p>succeeds.</p>
CN REST - Bug #7489 (New): processing daemon common-logging misconfiguredhttps://redmine.dataone.org/issues/74892015-11-16T16:32:08ZRobert Waltz
<p>the commons-logging.properties file appears misconfigured. It is missing the line:</p>
<p>org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger</p>
<p>Hopefully, adding this line will allow all the apache commons logging calls to go to the log4j loggers.</p>