DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-07-15T18:03:40ZDataONE Tasks
Redmine Infrastructure - Bug #8866 (New): Java client tools should set a custom user agent stringhttps://redmine.dataone.org/issues/88662020-07-15T18:03:40ZBryce Mecummecum@nceas.ucsb.edu
<p>Related to <a href="https://redmine.dataone.org/issues/7047">https://redmine.dataone.org/issues/7047</a></p>
<p>It looks like nowhere in <code>d1_libclient_java</code> do we set a user agent string. Aside from being best practice, it limits our ability to customize our infrastructure around it. For example, OPC is running into HTTP 413s due to overrunning their TLS renegotiation buffer and we can't effectively whitelist their requests, which come from our Java client tools, to allow them to upload large files.</p>
Infrastructure - Story #8862 (New): Deploy a new dataone-cn-rest releasehttps://redmine.dataone.org/issues/88622020-04-23T16:24:46ZJing Taotao@nceas.ucsb.edu
<p>We have a new d1_portal jar release which addresses the issue that restarting tomcat in CNs is needed when the LE certificates are renewed in CNs. The new d1_portal jar file has been deployed to dataone-cn-portal. However, the component dataone-cn-rest was overlooked. We need to deploy it there as well.<br>
Yesterday, we did a hack fix in CNs when we restarted tomcat - dropped the d1_portal-2.3.2.jar file there. So now it should work. But we still need a formal release.</p>
Infrastructure - Task #8858 (New): Update CN Apache configs in version control with directives to...https://redmine.dataone.org/issues/88582020-02-05T20:02:12ZBryce Mecummecum@nceas.ucsb.edu
<p>Sitemaps are located on disk in ${tomcat_webapps_dir}/${context}/sitemaps as <code>sitemap_index.xml</code> and <code>sitemap%d.xml</code> (for each sub-sitemap).</p>
<p>The rule we've come up with is:</p>
<p><code>RewriteRule ^/(sitemap.+) /metacat/sitemaps/$1 [R=303]</code></p>
Infrastructure - Story #8853 (New): Make cn.resolve smarterhttps://redmine.dataone.org/issues/88532019-11-15T16:46:12ZJing Taotao@nceas.ucsb.edu
<p>In this case the cn.resolve() operation should be ignoring the node that is marked as offline, or at least placing it last in the list.</p>
<p>This should be a high priority fix, and should be fairly simple to implement since the information is available in the node document.</p>
<ul>
<li>Dave</li>
</ul>
<blockquote>
<p>On 2019-11-14, at 21:38, Matt Jones <a href="mailto:jones@nceas.ucsb.edu">jones@nceas.ucsb.edu</a> wrote:</p>
<p>FYI, thread form today with Ethan White on ebird replication, and the resolve() api in DataONE. Relates to our conversation today about making resolve() and MetacatUI downloads smarter.</p>
<p>Matt</p>
<p>Ethan White 5:06 PM<br>
What's the right place to report data that if 404ing on DataONE?</p>
<p>Matt Jones 5:07 PM<br>
<a href="mailto:support@dataone.org">support@dataone.org</a> would work</p>
<p>5:08 PM<br>
or let me know</p>
<p>5:08 PM<br>
is it that same data set?</p>
<p>5:08 PM<br>
the Ebird one?</p>
<p>Ethan White 5:09 PM<br>
Yeah, which we had discovered had been reposted and spent a bunch of time gearing up to support again. We were in the middle of testing when it suddenly disappeared again. <a href="http://dataone.ornith.cornell.edu/metacat/d1/mn/v2/object/EOD_CLO_2016.csv.gz">http://dataone.ornith.cornell.edu/metacat/d1/mn/v2/object/EOD_CLO_2016.csv.gz</a></p>
<p>Matt Jones 5:10 PM<br>
yeah. Cornell just gave us permission to replicate the data to other nodes. They haven’t wanted us to do so in the past.</p>
<p>Ethan White 5:13 PM<br>
Thanks. That's good news. So can we expect it to reappear at some point soonish?</p>
<p>Matt Jones 5:14 PM<br>
Yeah, its been replicated. I’m checking to see if it is properly linked to the original.</p>
<p>5:15 PM<br>
<a href="https://knb.ecoinformatics.org/view/EOD_CLO_2016.eml">https://knb.ecoinformatics.org/view/EOD_CLO_2016.eml</a></p>
<p>new messages</p>
<p>Ethan White 5:16 PM<br>
Thanks Matt. FYI that link I posted is the one being returned from a current search of DataONE.</p>
<p>Matt Jones 5:17 PM<br>
Yeah. Because that’s the ‘authoritative’ copy at cornell.</p>
<p>5:17 PM<br>
but Cornell’s node has been going up and down.</p>
<p>5:17 PM<br>
our resolve service lists all copies of a data set</p>
<p>5:17 PM<br>
so if one is down, you can get it from another location:</p>
<p>5:18 PM<br>
<code><br>
$ curl -H "Accept: text/xml" https://cn.dataone.org/cn/v2/resolve/EOD_CLO_2016.eml<br>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><br>
<ns2:objectLocationList xmlns:ns2="http://ns.dataone.org/service/types/v1"><br>
<identifier>EOD_CLO_2016.eml</identifier><br>
<objectLocation><br>
<nodeIdentifier>urn:node:CLOEBIRD</nodeIdentifier><br>
<baseURL>http://dataone.ornith.cornell.edu/metacat/d1/mn</baseURL><br>
<version>v1</version><br>
<version>v2</version><br>
<url>http://dataone.ornith.cornell.edu/metacat/d1/mn/v2/object/EOD_CLO_2016.eml</url><br>
</objectLocation><br>
<objectLocation><br>
<nodeIdentifier>urn:node:CN</nodeIdentifier><br>
<baseURL>https://cn.dataone.org/cn</baseURL><br>
<version>v1</version><br>
<version>v2</version><br>
<url>https://cn.dataone.org/cn/v2/object/EOD_CLO_2016.eml</url><br>
</objectLocation><br>
<objectLocation><br>
<nodeIdentifier>urn:node:KNB</nodeIdentifier><br>
<baseURL>https://knb.ecoinformatics.org/knb/d1/mn</baseURL><br>
<version>v1</version><br>
<version>v2</version><br>
<url>https://knb.ecoinformatics.org/knb/d1/mn/v2/object/EOD_CLO_2016.eml</url><br>
</objectLocation><br>
</ns2:objectLocationList><br>
</code></p>
<p>Ethan White 5:19 PM<br>
OK, thanks. That's why I thought the link in DataONE <a href="https://cn.dataone.org/cn/v2/resolve/EOD_CLO_2016.csv.gz">https://cn.dataone.org/cn/v2/resolve/EOD_CLO_2016.csv.gz</a> would take me to a working version, but clearly I just don't understand the details. We'll just use the the one on KNB at least for the moment. Really appreciate your help as always.</p>
<p>Matt Jones 5:20 PM<br>
No problem. I’d love to make this all work more seamlessly. (edited) </p>
<p>5:20 PM<br>
So suggestions definitely welcome.</p>
<p>5:21 PM<br>
I expect Cornell to take their node offline altogether — so the KNB will likely be the better location.</p>
<p>5:22 PM<br>
Btw, the resolve link when executed in a browser just redirects to the first copy</p>
<p>Ethan White 5:23 PM<br>
Yeah, Cornell's closed approach to things is a pretty big disappointment, especially on data like this that is generated by volunteers. We'll just go to the KNB version permanently.</p>
<p>Matt Jones 5:23 PM<br>
whereas programatically you get the list of locations</p>
<p>5:23 PM<br>
if you ask for XML</p>
<p>Ethan White 5:23 PM<br>
That makes sense. Thanks.</p>
<p>Matt Jones 5:23 PM<br>
and then you can choose to try one or more</p>
</blockquote>
Infrastructure - Story #8849 (New): During sync, the CN does not detect error returned from getCh...https://redmine.dataone.org/issues/88492019-11-05T19:25:48ZRoger Dahldahl@unm.edu
<p>Due to a bug, GMN returned 500 on some getChecksum() calls. The CN did not detect the 500 return status and proceeded with the sync, using "null" as the checksum.</p>
Infrastructure - Story #8848 (New): A minor difference of annotation index between CN and MNhttps://redmine.dataone.org/issues/88482019-11-01T21:37:01ZJing Taotao@nceas.ucsb.edu
<p>The solr index on CN is:</p>
<pre><arr name="sem_annotation">
<str>http://purl.dataone.org/odo/ECSO_00000512</str>
<str>
http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#MeasurementType
</str>
<str>http://purl.dataone.org/odo/ECSO_00001102</str>
<str>http://purl.dataone.org/odo/ECSO_00001243</str>
<str>http://purl.dataone.org/odo/ECSO_00000629</str>
<str>http://purl.dataone.org/odo/ECSO_00000518</str>
<str>http://www.w3.org/2000/01/rdf-schema#Resource</str>
<str>http://purl.dataone.org/odo/ECSO_00000516</str>
<str>http://purl.obolibrary.org/obo/UO_0000301</str>
</arr>
</pre>
<p>The mn is:</p>
<pre><arr name="sem_annotation">
<str>http://purl.dataone.org/odo/ECSO_00000512</str>
<str>
http://ecoinformatics.org/oboe/oboe.1.2/oboe-core.owl#MeasurementType
</str>
<str>http://purl.dataone.org/odo/ECSO_00001102</str>
<str>http://purl.dataone.org/odo/ECSO_00001243</str>
<str>http://purl.dataone.org/odo/ECSO_00000629</str>
<str>http://purl.dataone.org/odo/ECSO_00000518</str>
<str>http://purl.dataone.org/odo/ECSO_00000516</str>
<str>http://purl.obolibrary.org/obo/UO_0000301</str>
</arr>
</pre>
<p>The cn has an extra <code><str>http://www.w3.org/2000/01/rdf-schema#Resource</str></code><br>
Bryce and I discussed it and thought it wouldn't affect the feature. But we still need to figure it out.</p>
Member Nodes - MNDeployment #8847 (In Review): Freshwater Research and Environmental Database (IG...https://redmine.dataone.org/issues/88472019-10-17T19:19:35ZAmy Forresteraforres4@utk.edu
<p>he Freshwater Research and Environmental Database is the central data repository for IGB (Leibniz-Institut of Freshwater Ecology and Inland Fisheries). It is where we store and share environmental data from observations of lakes, rivers, peatlands and other freshwater habitats. In FRED you can find continuous data collected over several decades from our long-term research programme at the lakes Müggelsee, Stechlinsee, Arendsee and the river Spree, as well as environmental data derived from short-term projects in aquatic ecosystems. All data include detailed metadata descriptions in text form to allow reuse of the data. The database can be searched for a range of aspects, such as ecosystem types or abiotic and biotic variables. Data use, where not freely accessible, shall be granted after consulting with the contact person given in the database, and is subject to the IGB Data Policy.</p>
Member Nodes - Bug #8846 (New): TFRI node unresponsivehttps://redmine.dataone.org/issues/88462019-10-08T15:26:52ZDave Vieglaisdave.vieglais@gmail.com
<p>The TFRI node is not responding to the node request, e.g.:</p>
<pre>curl -v "https://172.16.1.243/tfri/d1/mn/v1/node"
</pre>
<p>Verify that the server is operational.</p>
Member Nodes - Bug #8845 (New): Server is returning a 500 error.https://redmine.dataone.org/issues/88452019-10-08T15:20:05ZDave Vieglaisdave.vieglais@gmail.com
<p>Server is returning a 500 error which means something is amiss with the underlying member node application.</p>
<p>Could be as simple as restarting the server, but needs to be looked into.</p>
Member Nodes - Bug #8844 (New): Server certificate is expiredhttps://redmine.dataone.org/issues/88442019-10-08T15:17:01ZDave Vieglaisdave.vieglais@gmail.com
<p>The server certificate has expired and needs to be renewed:</p>
<pre>Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:99:29:51:81:59:be:23:83:e8:a2:2d:9f:78:7c:7d:92:67
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
Validity
Not Before: Jun 19 16:21:39 2018 GMT
Not After : Sep 17 16:21:39 2018 GMT
Subject: CN = dataone.sensor.nevada.edu
</pre> Member Nodes - Bug #8843 (New): Server certificate is expiredhttps://redmine.dataone.org/issues/88432019-10-08T15:14:01ZDave Vieglaisdave.vieglais@gmail.com
<p>The server certificate is expired and needs to be renewed:</p>
<pre>Certificate:
Data:
Version: 3 (0x2)
Serial Number:
29:53:96:6e:cb:a2:bd:76:33:a3:e3:25:e4:bd:65:c8
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon RSA Server CA
Validity
Not Before: Jul 13 00:00:00 2016 GMT
Not After : Jul 13 23:59:59 2019 GMT
Subject: C = US, postalCode = 83844, ST = ID, L = Moscow, street = 875 Perimeter Dr, O = University of Idaho, OU = Information Technology Services, CN = *.northwestknowledge.net
</pre> Infrastructure - Story #8842 (New): Some exceptions in Metacathttps://redmine.dataone.org/issues/88422019-09-19T17:53:22ZJing Taotao@nceas.ucsb.edu
<p>In sandbox, we see some exceptions like. It appears not to hurt function, but we need to take a look at it.<br>
<code><br>
9-Sep-2019 15:19:05.303 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Marking servlet [AxisServlet] as unavailable<br>
19-Sep-2019 15:19:05.304 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.loadOnStartup Servlet [AxisServlet] in web application [/metacat] threw load() exception<br>
java.lang.ClassNotFoundException: org.apache.axis.transport.http.AxisServlet<br>
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1364)<br>
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1185)<br>
at org.apache.catalina.core.DefaultInstanceManager.loadClass(DefaultInstanceManager.java:546)<br>
at org.apache.catalina.core.DefaultInstanceManager.loadClassMaybePrivileged(DefaultInstanceManager.java:527)<br>
at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:150)<br>
at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1044)<br>
at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:983)<br>
at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4956)<br>
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5270)<br>
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)<br>
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)<br>
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)<br>
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)<br>
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:624)<br>
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1834)<br>
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br>
at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br>
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)<br>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)<br>
at java.lang.Thread.run(Thread.java:748)<br>
</code></p>
Member Nodes - MNDeployment #8839 (Planning): CMU KiltHub via Figshare https://redmine.dataone.org/issues/88392019-08-27T16:46:25ZAmy Forresteraforres4@utk.edu
<p>Hannah Gunderman (post-doc at UT) now Research Data Management Consultant @ Carnegie Mellon University. Want to add climate data: air quality research data ~~ dozen datasets from their IR Kilthub via Figshare installation similar to Carrie Institute</p>
<p><a href="https://drive.google.com/file/d/1a3oGIEQk5pa-A-B6iPjur7AsIkf1Wkjz/view?usp=sharing" class="external">MN Description worksheet</a></p>
Member Nodes - Story #8835 (New): Add ability for scanner to stop after a certain number of errorshttps://redmine.dataone.org/issues/88352019-08-12T19:16:39ZJohn Evans
<p>Right now the scanner will try to go thru the entire list of sitemap documents, regardless of whether they all fail or not. We should add the ability to abort further checks if a certain error threshold is crossed.</p>
Member Nodes - Story #8833 (New): Problems utilizing pyshacl within SlenderNodeshttps://redmine.dataone.org/issues/88332019-08-05T19:37:09ZJohn Evans
<p>Opening this ticket to document issues encountered when trying to utilize pyshacl within SlenderNodes.</p>