DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-08-06T00:06:07ZDataONE Tasks
Redmine CN REST - Bug #8867 (New): CNCore.listChecksumAlgorithms() returns incorrect listhttps://redmine.dataone.org/issues/88672020-08-06T00:06:07ZMatthew Jonesjones@nceas.ucsb.edu
<p>The definition of the ChecksumAlgorithm type in SystemMetadata allows any checksum algorithm listed in the Library of Congress vocab. But the current CNCore.listChecksumAlgorithms() implementation only returns two, MD5 and SHA-1. Need to correct this to include the full list of supported algorithms (see <a href="http://id.loc.gov/vocabulary/preservation/cryptographicHashFunctions.html">http://id.loc.gov/vocabulary/preservation/cryptographicHashFunctions.html</a>).</p>
<p>The implementation of this is in a property file, which needs to be updated with the correct list. The file (d1_cn_rest/src/test/resources/org/dataone/configuration/node.properties) currently contains:</p>
<p><code>cn.checksumAlgorithmList=SHA-1;MD5</code></p>
<p>But it should contain all of the other valid algorithms as well from the LoC.</p>
CN REST - Bug #8860 (New): /token endpoint doesn't set a content-type and character encodinghttps://redmine.dataone.org/issues/88602020-02-29T01:00:11ZBryce Mecummecum@nceas.ucsb.edu
<p>On Firefox only, requests to the /portal/token endpoint (i.e., the one MetacatUI and other clients use to fetch their auth tokens, like <a href="https://cn.dataone.org/portal/token">https://cn.dataone.org/portal/token</a>) result in errors in the browser console.</p>
<p>When you access the URL via an XHR request, you see:</p>
<blockquote>
<p>XML Parsing Error: syntax error<br>
Location: <a href="https://cn-stage.test.dataone.org/portal/token">https://cn-stage.test.dataone.org/portal/token</a><br>
Line Number 1, Column 1:</p>
</blockquote>
<p>When you access the URL directly in Firefox:</p>
<blockquote>
<p>The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.</p>
</blockquote>
<p>I had a hunch that this error would go away if the response simply had the <code>Content-Type</code> header set to <code>text/plain; charset=utf-8</code> so I spun up <code>mitmproxy</code>, made that edit to the intercepted response, and saw that the error does go away.</p>
<p>I think we should modify the portal code to set the <code>Content-Type</code> header like above so the error goes away.</p>
CN REST - Task #8778 (New): Ensure SystemMetadata replica auditing updates are saved and broadcasthttps://redmine.dataone.org/issues/87782019-03-12T16:54:08ZChris Jonescjones@nceas.ucsb.edu
<p>In <code>MNAuditTask.call()</code>, we process a batch of pids in need of auditing per Member Node. For each <code>pid</code> in the <code>auditPIDs</code> list, we call <code>MN.getChecksum()</code>. Regardless of success or failure, we set the <code>Replica.replicaVerified</code> date in the <code>SystemMetadata</code>. However, the task has a copy of the system metadata from <code>hzSystemMetadata.get()</code>, and the task doesn't subsequently call <code>hzSystemMetadata.put(pid, sysmeta)</code>. This means that while we are auditing content, we may just not be recording the results! I need to look at the code more to see if we make an API call to <code>CN.updateSystemMetadata()</code> elsewhere, but I would expect the<code>MNAuditTask</code> to do this. Also, if this happens in the task, we also need to broadcast the system metadata change to the authoritative MN and all replica MNS. Lastly, we need to update the <code>serialVersion</code> field to show the other CNs what the most recent replica list is.</p>
CN REST - Task #8776 (New): Set valid replica status to completedhttps://redmine.dataone.org/issues/87762019-03-12T15:57:36ZChris Jonescjones@nceas.ucsb.edu
<p>In <code>MNAuditTask.call()</code> we audit replica checksums, but on success, we only set the <code>Replica.replicaVerified</code> field to the current date. We don't set the <code>Replica.replicationStatus</code> field to <code>COMPLETED</code>. This is an issue because the <code>Replica</code> entry in the <code>SystemMetadata</code> may have been set to <code>FAILED</code> or <code>INVALIDATED</code>, but may now be valid, and so would need to be updated.</p>
CN REST - Story #8771 (New): Issue with LDAP when updating `nodeReplicationPolicy`https://redmine.dataone.org/issues/87712019-03-05T19:42:17ZRoger Dahldahl@unm.edu
<p>When a submitting a Node doc update which includes a nodeReplicationPolicy, this section is good:</p>
<pre><nodeReplicationPolicy>
<maxObjectSize>21474836480</maxObjectSize>
<spaceAllocated>1099511627776</spaceAllocated>
</nodeReplicationPolicy>
</pre>
<p>while the same section without <code>maxObjectSize</code> returns error:</p>
<pre> <error detailCode="4822" errorCode="500" name="ServiceFailure">
<description>updateNodeCapabilities failed due to LDAP communication failure:: InvalidAttributeValueException:[LDAP: error code 21 - d1ReplicationPolicyMaxObjectSize: value #0 invalid per syntax]:[LDAP: error code 21 - d1ReplicationPolicyMaxObjectSize: value #0 invalid per syntax]</description>
</error>
</pre>
<p>The schema allows leaving <code>maxObjectSize</code> out, which means that the MN accepts replicas of unlimited size.</p>
<p>Both GMN and Metacat leave <code>maxObjectSize</code> out if the setting is configured to unlimited with <code>-1</code>.</p>
<p>I think it used to work.</p>
CN REST - Story #8770 (New): Issue with CN handling of encoded identifiers in object/ meta/ node/...https://redmine.dataone.org/issues/87702019-03-05T19:37:13ZRoger Dahldahl@unm.edu
<p>Works:<br>
<a href="http://cn.dataone.org/cn/v2/object/doi:10.6073/AA/knb-lter-bes.298.37">http://cn.dataone.org/cn/v2/object/doi:10.6073/AA/knb-lter-bes.298.37</a><br>
<a href="https://cn.dataone.org/cn/v2/node/urn:node:LTER">https://cn.dataone.org/cn/v2/node/urn:node:LTER</a></p>
<p>Does not work:<br>
<a href="http://cn.dataone.org/cn/v2/object/doi%3A10.6073%2FAA%2Fknb-lter-bes.298.37">http://cn.dataone.org/cn/v2/object/doi%3A10.6073%2FAA%2Fknb-lter-bes.298.37</a><br>
<a href="https://cn.dataone.org/cn/v2/node/urn%3Anode%3ALTER">https://cn.dataone.org/cn/v2/node/urn%3Anode%3ALTER</a></p>
<p>Note: Behavior differs between HTTP / HTTPS.</p>
CN REST - Story #8757 (New): Fix getChecksum() in MNAuditTask to use dynamic checksum algorithmshttps://redmine.dataone.org/issues/87572019-01-14T16:46:33ZChris Jonescjones@nceas.ucsb.edu
<p>The <code>MNAuditTask.call()</code> method is hardcoded to use <code>MD5</code> checksums on line 277. It requests the Member Node to generate an <code>MD5</code> checksum, and then compares that checksum to the checksum stated in the Coordinating Node<code>s cached</code>SystemMetadata.checksum<code>field for the object. This obviously will fail for objects that submitted objects using</code>SHA-1` or other algorithms.</p>
CN REST - Bug #8740 (New): CN resolve service returning 404 for some pidshttps://redmine.dataone.org/issues/87402018-11-08T00:21:45ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>A user has reported that certain pids (all for metadata objects) return an http 404 status for the resolve service. Here is the complete list that the user tried that didn't resolve:</p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1343.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1212.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2">https://cn.dataone.org/cn/v2/resolve/knb-lter-arc.1477.2</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8">https://cn.dataone.org/cn/v2/resolve/knb-lter-bnz.442.8</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.15.5</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1">https://cn.dataone.org/cn/v2/resolve/knb-lter-pie.248.1</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.6.13</a><br>
<a href="https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23">https://cn.dataone.org/cn/v2/resolve/knb-lter-sbc.15.23</a></p>
<p>The CNRead.getSystemMetadata() service returns metadata for each of these pids.</p>
CN REST - Bug #8733 (New): Exception handling in ForesiteResourceMap needs improvementhttps://redmine.dataone.org/issues/87332018-10-12T20:09:55ZChris Jonescjones@nceas.ucsb.edu
<p>We're seeing some errors from the <code>org.dataone.cn.indexer.resourcemap.ForesiteResourceMap</code> constructor when indexing resource maps in Metacat 2.9.0, d1_cn_index_processor 2.3.4. A general <code>Exception</code> is being caught and re-thrown as an <code>OREParserException</code> on line 104:</p>
<pre>metacat 20181012-10:53:02: [INFO]: done queueing doc index for pid urn:uuid:5a860679-159e-45a6-bcb6-beb67f4166c2 [edu.ucsb.nceas.metacat.MetacatHandler:handleReindexAction:2777]
metacat 20181012-10:53:02: [DEBUG]: ================= in the finally statement [edu.ucsb.nceas.metacat.MetacatHandler:handleReindexAction:2808]
metacat 20181012-10:53:02: [DEBUG]: ================= in the finally statement which out is not null [edu.ucsb.nceas.metacat.MetacatHandler:handleReindexAction:2810]
org.dspace.foresite.OREParserException: java.lang.NullPointerException
at org.dataone.cn.indexer.resourcemap.ForesiteResourceMap.<init>(ForesiteResourceMap.java:104)
at org.dataone.cn.indexer.resourcemap.ResourceMapFactory.buildResourceMap(ResourceMapFactory.java:45)
at edu.ucsb.nceas.metacat.index.resourcemap.ResourceMapSubprocessor.processResourceMap(ResourceMapSubprocessor.java:97)
at edu.ucsb.nceas.metacat.index.resourcemap.ResourceMapSubprocessor.processDocument(ResourceMapSubprocessor.java:86)
at edu.ucsb.nceas.metacat.index.SolrIndex.process(SolrIndex.java:236)
at edu.ucsb.nceas.metacat.index.SolrIndex.insert(SolrIndex.java:426)
at edu.ucsb.nceas.metacat.index.SolrIndex.update(SolrIndex.java:637)
at edu.ucsb.nceas.metacat.index.SolrIndex.update(SolrIndex.java:592)
at edu.ucsb.nceas.metacat.index.SystemMetadataEventListener.entryUpdated(SystemMetadataEventListener.java:146)
at edu.ucsb.nceas.metacat.index.SystemMetadataEventListener.entryAdded(SystemMetadataEventListener.java:119)
at com.hazelcast.client.impl.EntryListenerManager.notifyListeners(EntryListenerManager.java:148)
at com.hazelcast.client.impl.EntryListenerManager.notifyListeners(EntryListenerManager.java:130)
at com.hazelcast.client.impl.ListenerManager.customRun(ListenerManager.java:88)
at com.hazelcast.client.ClientRunnable.run(ClientRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
</pre>
<p>The resource map that is failing to index is <a href="https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3A5a860679-159e-45a6-bcb6-beb67f4166c2">https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3A5a860679-159e-45a6-bcb6-beb67f4166c2</a></p>
<p>This resource map parses fine using <code>com.hp.hpl.jena.rdf.model.Model.read()</code> outside of the index processor code.</p>
<p>This <code>ForesiteResourceMap</code> constructor is taking an <code>org.w3c.dom.Document</code> as input, converting it to an in-memory <code>ByteArrayOutputStream</code>, and converting that to an in-memory <code>String</code>, to finally get a <code>ReaderInputStream</code> to be passed to <code>_init()</code>. We may need to address the handling inefficiency as a secondary improvement to the code, if feasible.</p>
<p>We also need to not catch a general <code>Exception</code> on line 117, but rather catch the more specific exception that is thrown, which will help with troubleshooting where the exception is coming from. Debug logging statements would help too.</p>
<p>The exception is a <code>NullPointerException</code>, so we ultimately need to track down where this is coming from.</p>
<p>It doesn't look like this code has changed in the 2.3.6 version of the library, so an upgrade wouldn't make a difference. I did look on <code>cn-orc-1</code> to see if we are having similar issues outside of a Metacat MN context, and see errors like these:</p>
<pre>cn-index-processor-daemon.log.9-[ERROR] 2018-10-12 06:42:44,977 (ErrorHandlerFactory$ErrorLogger:logError:84) {E201} rdf:resource not allowed as attribute here.
cn-index-processor-daemon.log.9:[ERROR] 2018-10-12 06:42:44,977 (ForesiteResourceMap:_init:163) Unable to parse ORE document:
cn-index-processor-daemon.log.9-org.apache.jena.riot.RiotException: {E201} rdf:resource not allowed as attribute here.
cn-index-processor-daemon.log.9- at org.apache.jena.riot.system.ErrorHandlerFactory$ErrorHandlerStd.error(ErrorHandlerFactory.java:128)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.lang.LangRDFXML$ErrorHandlerBridge.error(LangRDFXML.java:241)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.ARPSaxErrorHandler.error(ARPSaxErrorHandler.java:38)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.warning(XMLHandler.java:206)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.warning(XMLHandler.java:183)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.warning(XMLHandler.java:178)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.ParserSupport.warning(ParserSupport.java:199)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.AttributeLexer.error(AttributeLexer.java:187)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.QNameLexer.lookup(QNameLexer.java:80)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.AttributeLexer.processSpecials(AttributeLexer.java:71)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.states.WantDescription.startElement(WantDescription.java:60)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.states.AbsWantLiteralValueOrDescription.startElement(AbsWantLiteralValueOrDescription.java:84)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.states.WantLiteralValueOrDescription.startElement(WantLiteralValueOrDescription.java:56)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.XMLHandler.startElement(XMLHandler.java:121)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLNamespaceBinder.handleStartElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLNamespaceBinder.emptyElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.DTDConfiguration.parse(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
cn-index-processor-daemon.log.9- at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.impl.RDFXMLParser.parse(RDFXMLParser.java:151)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.arp.ARP.load(ARP.java:119)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.lang.LangRDFXML.parse(LangRDFXML.java:142)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFParserRegistry$ReaderRIOTFactoryImpl$1.read(RDFParserRegistry.java:142)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:818)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:258)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:244)
cn-index-processor-daemon.log.9- at org.apache.jena.riot.adapters.RDFReaderRIOT.read(RDFReaderRIOT.java:69)
cn-index-processor-daemon.log.9- at com.hp.hpl.jena.rdf.model.impl.ModelCom.read(ModelCom.java:274)
cn-index-processor-daemon.log.9- at org.dspace.foresite.jena.JenaOREParser.parseToModel(JenaOREParser.java:119)
cn-index-processor-daemon.log.9- at org.dspace.foresite.jena.JenaOREParser.parse(JenaOREParser.java:69)
cn-index-processor-daemon.log.9- at org.dataone.ore.ResourceMapFactory.parseResourceMap(ResourceMapFactory.java:444)
cn-index-processor-daemon.log.9: at org.dataone.cn.indexer.resourcemap.ForesiteResourceMap._init(ForesiteResourceMap.java:160)
cn-index-processor-daemon.log.9: at org.dataone.cn.indexer.resourcemap.ForesiteResourceMap.<init>(ForesiteResourceMap.java:73)
cn-index-processor-daemon.log.9- at org.dataone.cn.indexer.resourcemap.ResourceMapFactory.buildResourceMap(ResourceMapFactory.java:41)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessor.getNextIndexTask(IndexTaskProcessor.java:641)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessor.processFailedIndexTaskQueue(IndexTaskProcessor.java:229)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessor.processIndexTaskQueue(IndexTaskProcessor.java:191)
cn-index-processor-daemon.log.9- at org.dataone.cn.index.processor.IndexTaskProcessorJob.execute(IndexTaskProcessorJob.java:57)
</pre>
<p>This error is also related to <code>_init</code>, but isn't an NPE, so <code>_init()</code> might be a good place to look first.</p>
CN REST - Bug #8698 (New): CN Performance degradationhttps://redmine.dataone.org/issues/86982018-09-13T21:05:40ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>Calling simple services on the CN can take a long time to execute. For example, calling </p>
<p><a href="https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202">https://cn.dataone.org/cn/v2/resolve/aekos.org.au/collection/nsw.gov.au/nsw_atlas/vis_flora_module/KM_CUDM.20160202</a></p>
<p>can take about 8-10 seconds, while calling the MN directly (where the resolve service redirects to) takes about 1-2 seconds:</p>
<p><a href="https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202">https://dataone.tern.org.au/mn/v2/object/aekos.org.au%2Fcollection%2Fnsw.gov.au%2Fnsw_atlas%2Fvis_flora_module%2FKM_CUDM.20160202</a></p>
CN REST - Bug #8630 (New): equivalentIdentity values use uppercase letters when the same person s...https://redmine.dataone.org/issues/86302018-06-26T14:15:00ZLauren Walkerwalker@nceas.ucsb.edu
<p>I would expect a username to appear exactly the same everywhere in the identity service. </p>
<p>Here is an example where the <code>equivalentIdentity</code> uses uppercase letters for the subject (UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org) when the <code>person</code>><code>subject</code> uses lowercase letters (uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org):</p>
<p><a href="https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X">https://cn.dataone.org/cn/v2/accounts/http%3A%2F%2Forcid.org%2F0000-0002-5944-906X</a></p>
<pre><ns2:subjectInfo xmlns:ns2="http://ns.dataone.org/service/types/v1">
<person>
<subject>http://orcid.org/0000-0002-5944-906X</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>
UID=corinalogan,O=unaffiliated,DC=ecoinformatics,DC=org
</equivalentIdentity>
<verified>false</verified>
</person>
<person>
<subject>
uid=corinalogan,o=unaffiliated,dc=ecoinformatics,dc=org
</subject>
<givenName>Corina</givenName>
<familyName>Logan</familyName>
<equivalentIdentity>http://orcid.org/0000-0002-5944-906X</equivalentIdentity>
<verified>false</verified>
</person>
</ns2:subjectInfo>
```~~~
</pre> CN REST - Story #8582 (New): Replica Auditing service is throwing errorshttps://redmine.dataone.org/issues/85822018-05-01T19:15:35ZChris Jonescjones@nceas.ucsb.edu
<p>Replica auditing should be auditing objects every 90 days for fixity, and setting the <code>replicaStatus</code> appropriately. The <code>/var/log/dataone/cn-replication-audit.log*</code> files are showing many errors:</p>
<pre>cjones@cn-ucsb-1:replicate$ grep ERROR cn-replication-audit.log* | grep "Cannot update replica status" | wc -l
437601
</pre>
<p>Determine if this is a configuration issue or a code issue and fix it as needed. Also, fix the code to call <code>Identifier.getValue()</code> when logging these errors to avoid printing the memory location of the object like <code>org.dataone.service.types.v1.Identifier@7e90f2e8</code>. There are multiple places where <code>getValue()</code> needs to be added.</p>
CN REST - Bug #7918 (New): SEAD object only partially synchronized - missing autogen.201609291601...https://redmine.dataone.org/issues/79182016-10-21T17:20:29ZDave Vieglaisdave.vieglais@gmail.com
<p>While updating <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: SEAD object only partially synchronized - missing autogen.2015061616000265251 document from /var/... (Closed)" href="https://redmine.dataone.org/issues/7222">#7222</a> and <a class="issue tracker-5 status-5 priority-4 priority-default closed" title="Task: SEAD object only partially synchronized in CN production - missing autogen.2016092916013111425 do... (Closed)" href="https://redmine.dataone.org/issues/7914">#7914</a>. Checked for content for those, and it is accessible through CN. May still be an issue however as the SEAD MN reports 69 objects, though<br>
there are 68 reported in the search index.</p>
<p>Counting on the MN:</p>
<p>curl "<a href="http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object">http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object</a>" | xml fo</p>
<p>returns 69 objects.</p>
<p>Counting on the CN::</p>
<p>curl "<a href="https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD">https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD</a>" | xml fo</p>
<p>returns 69 objects.</p>
<p>Counting on the CN using the search index::</p>
<p><a href="https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=10&fl=id%2Ctitle%2CformatId&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22">https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=10&fl=id%2Ctitle%2CformatId&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22</a></p>
<p>returns 68 objects.</p>
<p>List of identifiers in search index::</p>
<p>curl "<a href="https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=100&fl=id&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22">https://cn.dataone.org/cn/v1/query/solr/?start=0&rows=100&fl=id&q=datasource%3A%22urn%5C%3Anode%5C%3ASEAD%22</a>" | xml sel -t -m "//doc/str[@name='id']" -v . -n | sort > SEAD_index_pids.txt</p>
<p>List of identifiers on CN::</p>
<p>curl "<a href="https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD">https://cn.dataone.org/cn/v2/object?nodeId=urn:node:SEAD</a>" | xml sel -t -m "//objectInfo/identifier" -v . -n | sort > SEAD_cn_pids.txt</p>
<p>Missing PID::</p>
<p>diff SEAD_cn_pids.txt SEAD_index_pids.txt<br>
60d59<br>
< seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</p>
<p>System Metadata for PID::</p>
<p><?xml version="1.0"?><br>
<br>
1<br>
seadva-c918e4ff-2861-496a-a907-d2cb382ddb30<br>
FGDC-STD-001-1998<br>
6553<br>
ff3d4641669b4e08c9f8b978b85d6113a4c1bab8<br>
SEAD<br>
CN=urn:node:SEAD,DC=dataone,DC=org<br>
<br>
<br>
public<br>
read<br>
<br>
<br>
<br>
sead-Martin-John-f1dbc3df-c27c-4647-b05a-4b1f05c99a24<br>
2016-08-17T12:41:34.03Z<br>
2016-08-17T14:43:58.677Z<br>
urn:node:SEAD<br>
urn:node:SEAD<br>
<br>
urn:node:SEAD<br>
completed<br>
2016-09-29T23:01:21.235Z<br>
<br>
<br>
urn:node:CN<br>
completed<br>
2016-09-29T23:01:21.241Z<br>
<br>
<a href="/ns1:systemMetadata">/ns1:systemMetadata</a></p>
<p>Confirm object can be retieved::</p>
<p>curl "<a href="http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object/seadva-c918e4ff-2861-496a-a907-d2cb382ddb30">http://seadva.d2i.indiana.edu:8081/sead/rest/mn/v1/object/seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</a>" | xml fo</p>
<p>System Metadata for obsoleted object::</p>
<p><?xml version="1.0"?><br>
<br>
1<br>
sead-Martin-John-f1dbc3df-c27c-4647-b05a-4b1f05c99a24<br>
FGDC-STD-001-1998<br>
7443<br>
ebd481ad8268f9714f23d772c15fb62e61384486<br>
CN=urn:node:SEAD,DC=dataone,DC=org<br>
CN=urn:node:SEAD,DC=dataone,DC=org<br>
<br>
<br>
public<br>
read<br>
<br>
<br>
<br>
seadva-c918e4ff-2861-496a-a907-d2cb382ddb30<br>
2013-10-24T18:41:31.213Z<br>
2016-08-17T14:43:58.677Z<br>
urn:node:SEAD<br>
urn:node:SEAD<br>
<br>
urn:node:CN<br>
completed<br>
2013-10-24T23:00:04.684Z<br>
<br>
<br>
urn:node:SEAD<br>
completed<br>
2013-10-24T23:00:04.579Z<br>
<br>
<a href="/ns1:systemMetadata">/ns1:systemMetadata</a><br></p>
<p>autogenid for <code>seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</code>::</p>
<p>psql metacat<br>
select * from identifier where guid='seadva-c918e4ff-2861-496a-a907-d2cb382ddb30';</p>
<table><thead>
<tr>
<th>guid</th>
<th>docid</th>
<th>rev</th>
</tr>
</thead><tbody>
<tr>
<td>seadva-c918e4ff-2861-496a-a907-d2cb382ddb30</td>
<td>autogen.2016092916012224122</td>
<td>1</td>
</tr>
</tbody></table>
<p>$ ls /var/metacat/data/autogen.2016092916012224122*<br>
ls: cannot access /var/metacat/data/autogen.2016092916012224122: No such file or directory</p>
CN REST - Task #7903 (New): Need to implement/support the default http methods - HEAD and GET ...https://redmine.dataone.org/issues/79032016-10-07T23:59:15ZMatthew Jonesjones@nceas.ucsb.edu
<p>Developers on the Whole Tale project at NCSA reported a bug in the HTTP HEAD request for our resolve service URIs. Example output below to reproduce the error. </p>
<p>Expected: a status code of 200</p>
<p>xarth@shakuras ~ $ curl --head <a href="https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f">https://cn.dataone.org/cn/v2/resolve/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f</a><br>
HTTP/1.1 500 Internal Server Error<br>
Date: Fri, 07 Oct 2016 23:00:14 GMT<br>
Server: Apache/2.2.22 (Ubuntu)<br>
Content-Length: 260<br>
Access-Control-Allow-Origin: <br>
Access-Control-Allow-Credentials: true<br>
Access-Control-Allow-Headers: Authorization, Content-Type, Location, Content-Length, x-annotator-auth-token<br>
Access-Control-Expose-Headers: Content-Length, Content-Type, Location<br>
Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, DELETE<br>
Vary: Accept-Encoding<br>
Connection: close<br>
Content-Type: text/xml;charset=UTF-8</p>
<p>xarth@shakuras ~ $ curl --head <a href="https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f">https://knb.ecoinformatics.org/knb/d1/mn/v2/object/urn%3Auuid%3Ae9ff8bfe-f12d-4630-a6f1-f3eab740be6f</a><br>
HTTP/1.1 200 OK<br>
Date: Fri, 07 Oct 2016 23:00:48 GMT<br>
Server: Apache/2.4.7 (Ubuntu)<br>
Set-Cookie: JSESSIONID=7DC18368F71D5D9948371B3C33437E8B; Path=/knb/; Secure<br>
DataONE-Checksum: SHA-1,927a11b6e46b771c9922083814f6ee8e5b09f696<br>
Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT<br>
DataONE-ObjectFormat: application/octet-stream<br>
DataONE-SerialVersion: 0<br>
Content-Length: 2809655736<br>
Access-Control-Allow-Origin: <br>
Access-Control-Allow-Credentials: true<br>
Content-Type: text/xml</p>
CN REST - Bug #7489 (New): processing daemon common-logging misconfiguredhttps://redmine.dataone.org/issues/74892015-11-16T16:32:08ZRobert Waltz
<p>the commons-logging.properties file appears misconfigured. It is missing the line:</p>
<p>org.apache.commons.logging.Log=org.apache.commons.logging.impl.Log4JLogger</p>
<p>Hopefully, adding this line will allow all the apache commons logging calls to go to the log4j loggers.</p>