DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-02-05T20:02:12ZDataONE Tasks
Redmine Infrastructure - Task #8858 (New): Update CN Apache configs in version control with directives to...https://redmine.dataone.org/issues/88582020-02-05T20:02:12ZBryce Mecummecum@nceas.ucsb.edu
<p>Sitemaps are located on disk in ${tomcat_webapps_dir}/${context}/sitemaps as <code>sitemap_index.xml</code> and <code>sitemap%d.xml</code> (for each sub-sitemap).</p>
<p>The rule we've come up with is:</p>
<p><code>RewriteRule ^/(sitemap.+) /metacat/sitemaps/$1 [R=303]</code></p>
Infrastructure - Bug #8821 (New): Object doi:10.18739/A2610VR7Z fails to index on CNhttps://redmine.dataone.org/issues/88212019-06-18T22:43:38ZBryce Mecummecum@nceas.ucsb.edu
<p>I noticed <a href="https://search.dataone.org/cn/v2/meta/doi:10.18739/A2610VR7Z">https://search.dataone.org/cn/v2/meta/doi:10.18739/A2610VR7Z</a> isn't present in the search index and asked about it on DataONEOrg #ci Slack channel. Chris noted that no no task was in the index queue for the PID. He tried to force a reindex which failed. He couldn't find a log entry indicating why it failed so someone needs to look into what's going on. It occurs to me that whatever bug is affecting this Object may have affected more Objects so a second part of fixing this bug should probably be doing some housekeeping or an audit of some kind to make sure the search index isn't missing more content.</p>
Infrastructure - Task #8817 (New): Configure sitemaps on the CNhttps://redmine.dataone.org/issues/88172019-06-06T23:52:23ZBryce Mecummecum@nceas.ucsb.edu
<p>Support for sitemaps landed last fall in Metacat: <a href="https://github.com/NCEAS/metacat/pull/1283">https://github.com/NCEAS/metacat/pull/1283</a>. Sitemaps are good for users but especially for search engines and DataONE's Search Catalog could benefit from having sitemaps enabled. A sitemap could help crawlers discover all of the datasets in DataONE. The CNs already run Metacat and should use Metacat's sitemaps ability to generate sitemaps for all content.</p>
<p>To enable sitemaps on the CNs, a few things seem to be needed. I'm not very familiar with how the CNs get built so I may be wrong or be missing things:</p>
<ul>
<li>Sitemaps rely on two properties in Metacat's <code>metacat.properties</code> file, which should have values: <code>sitemap.location.base=https://search.dataone.org/</code> and <code>sitemap.entry.base=https://search.dataone.org/view</code></li>
<li>The Apache config the CNs are built with need to serve the <code>sitemap_index.xml</code> and individual sitemaps from the Tomcat webapps dir. Metacat generates sitemaps in the <code>sitemaps</code> subfolder (e.g., <code>/usr/lib/tomcat8/webapps/metacat/sitemaps</code>). A <code>Directory</code> directive should work so long as filesystem permissions are set up for Apache to see the files.</li>
<li>We need a robots.txt that points at the sitemap index file at search.dataone.org which provides the entrypoint to <code>sitemap_index.xml</code></li>
<li>Metacat generates sitemaps with a recurring job mechanism that's internal to Metacat. AFAIK this job isn't turned on when Tomcat loads Metacat and a request has to get sent to the admin API which turns this job on as a side-effect. We might want to change this to reduce maintenance burden or chance of having stale sitemaps</li>
</ul>
<p>Dave nominated Jing for this work and has targeted this for the next CCI release. I'm not sure which that is so please select whichever one is appropriate.</p>
<p>Note this relates to <a href="https://redmine.dataone.org/issues/8693">https://redmine.dataone.org/issues/8693</a> which we've delayed because Google's crawler infrastructure has changed and DataONE is now visible by Google. Google staff have indicated we only need to send them a robots.txt that points to our sitemaps for them to begin crawling.</p>
Infrastructure - Task #8755 (New): Expand EML indexing support for EML 2.2https://redmine.dataone.org/issues/87552018-12-19T00:55:12ZBryce Mecummecum@nceas.ucsb.eduInfrastructure - Task #8754 (New): Add EML 2.2 to CN formats listhttps://redmine.dataone.org/issues/87542018-12-19T00:54:05ZBryce Mecummecum@nceas.ucsb.eduInfrastructure - Task #8753 (New): Add support for EML 2.2 (indexing, view)https://redmine.dataone.org/issues/87532018-12-19T00:43:52ZBryce Mecummecum@nceas.ucsb.edu
<p>EML 2.2 is nearly released and, with it, comes changes to indexing and the view service. The key points are that EML now supports Markdown and semantic annotations but some more stuff was added too. See the in-progress What's New in EML 2.2.0 page <a href="https://github.com/NCEAS/eml/blob/BRANCH_EML_2_2/docs/eml-220info.md">https://github.com/NCEAS/eml/blob/BRANCH_EML_2_2/docs/eml-220info.md</a> for details.</p>
<p>I'll track these with sub-tasks but here's an overview:</p>
<p><strong>Formats</strong></p>
<p>We need to expand the list of formats to include EML 2.2.</p>
<p><strong>Indexing</strong></p>
<p>EML 2.2 requires adding new fields and changes to some existing fields. For example, abstracts in EML can now also include a <code>markdown</code> child element which we'll also want to store in Solr. We also want to add in support for semantic annotations and probably structured funding information since that's been requested so much.</p>
<p>Support for semantic annotations will likely use the same approach as our previous work on semantic search where incoming annotations were subject to materialization during the indexing process. To explain what that means, the idea is that, when an annotation for term X comes in, the index processor loads the relevant ontology, materializes the superclass hierarchy for the term, and stores the entire hierarchy in Solr. For EML records with multiple annotations, only the unique set needs to be stored.</p>
<p><strong>View service</strong></p>
<p>As mentioned above, EML 2.2 changes how some existing elements (e.g., abstract) work and adds some new ones (e.g., annotations. The existing EML 2 stylesheets will work for EML 2.2 because 2.2 is backwards compatible but we'll want to extend them to support what's new and changed in EML 2.2</p>
<p>Of note: EML now supports storing Markdown where previously only EML TextType (basically DocBook) was allowed. Our plan on Metacat/MetacatUI to support this is to render the Markdown on the client side which does involve some security concerns (see <a href="https://github.com/NCEAS/metacatui/issues/860">https://github.com/NCEAS/metacatui/issues/860</a> for tracking).</p>
Infrastructure - Bug #8215 (New): Consider how Subjects are compared (e.g. HTTP vs. HTTPS ORCID U...https://redmine.dataone.org/issues/82152017-11-07T01:41:51ZBryce Mecummecum@nceas.ucsb.edu
<p>I know this has come up a few times in the last few years and it has bitten us a lot in day-to-day operations at the Arctic Data Center. Some Subjects in DataONE authentication appear to be compared literally so these two subjects are not considered equivalent:</p>
<p><a href="http://orcid.org/0000-0002-0381-3766">http://orcid.org/0000-0002-0381-3766</a><br>
<a href="https://orcid.org/0000-0002-0381-3766">https://orcid.org/0000-0002-0381-3766</a></p>
<p>even though a reasonable spectator might consider them equivalent.</p>
<p>Where this causes trouble seems to be when System Metadata is authored by users, specifically the rightsHolder and accessPolicy portions, and what's in the System Metadata does not literally match the user's actual Subject. As an example, when a user logs in via ORCID, their DataONE Subject ends up being the <em>http</em> variant of their ORCID URI, so if they use the <em>https</em> variant of their ORCID URI in their System Metadata, API calls requiring read, write, and changePermission permission fail for them because the literal string comparison determines the two Subjects to non-equivalent.</p>
<p>I think this may only affect ORCIDs right now because Subjects such as LDAP DNs may already be compared in a string-insensitive fashion. Though I'm not sure on this point.</p>
<p>A few of us on the NCEAS dev team discussed this and we think it's fair to compare Subjects more intelligently, or at least be more aware of the semantic structure of the Subject string. This would have numerous benefits, such as:</p>
<ul>
<li>Prevent users from becoming hopelessly confused when they can't figure out why they can't read/write Objects (people often don't notice http vs https) which will make DataONE seem friendlier</li>
<li>Make DataONE authentication less dependent upon protocols such as HTTP/HTTPS, both of which my change (i.e., ORCID may disable HTTPS; HTTP(S) may be replaced by a future web protocol) and thus more future-proof</li>
</ul>
<p>This type of change would require changes in one or more software projects:</p>
<ul>
<li>DataONE Portal or libclient Java (I'm not entirely sure where this check is done right now)</li>
<li>Architecture documentation</li>
<li>(Potentially any/all) MN software stacks (At least a review would be needed)</li>
<li>(Potentially) Client tools (e.g., R, Python) (At least a review would be needed)</li>
</ul>
Infrastructure - Decision #8189 (New): Proposal to change the roles mapped to the origin Solr fie...https://redmine.dataone.org/issues/81892017-10-02T18:04:37ZBryce Mecummecum@nceas.ucsb.edu
<p>While discussing changing the behavior of the origin field in the ISO indexing component (<a href="https://redmine.dataone.org/issues/8165">https://redmine.dataone.org/issues/8165</a>) to make it more selective about where in the document originators are pulled, Matt Jones (over email) suggested we revisit the set of roles as well. Let's do that in this Issue.</p>
<p>The current set of roles mapped to the origin field are:</p>
<ul>
<li><em>originator</em>: party who created the resource</li>
<li><em>author</em>: party who authored the resource</li>
<li><em>owner</em>: party that owns the resource</li>
<li><em>principalInvestigator</em>: key party responsible for gathering information and conducting research</li>
</ul>
<p>This current set of roles may be surprising to some/many users so a possible outcome of this Issue is to greatly improve the content in our search index. This would have impacts on the CN and MNs running Metacat.</p>
<p>Key points:</p>
<ul>
<li>Matt's proposal is to exclude principalInvestigator from this list</li>
<li>The Research Workspace Member Node appears to be using the principalInvestigator role for one or more persons they want in their citation so if we follow Matt's proposal we may need to discuss this with them</li>
<li>I would lobby for only including originator and author but my reading of the definitions is a naïve one</li>
</ul>
<p>I'd like us to have a discussion on this, make the relevant change to the codebase, and then bring the discussion back to the MN operators.</p>
<p>Relevant links:</p>
<ul>
<li><a href="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml">http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml</a> (official? definitions for CI_RoleCode)</li>
<li><a href="https://geo-ide.noaa.gov/wiki/index.php?title=ISO_19115_and_19115-2_CodeList_Dictionaries">https://geo-ide.noaa.gov/wiki/index.php?title=ISO_19115_and_19115-2_CodeList_Dictionaries</a> (NOAA wiki entry for the role codes)</li>
</ul>
Infrastructure - Bug #8052 (New): Geohashed value is incorrecthttps://redmine.dataone.org/issues/80522017-03-27T20:43:47ZBryce Mecummecum@nceas.ucsb.edu
<p>Adam Shepherd at BCO-DMO uploaded a test DCX doc here:</p>
<p><a href="https://search-sandbox.test.dataone.org/#view/http://lod.bco-dmo.org/id/dataset-file/682007">https://search-sandbox.test.dataone.org/#view/http://lod.bco-dmo.org/id/dataset-file/682007</a></p>
<p>which has bounding coordinates of:</p>
<p>North<br>
50.4907 degrees<br>
South<br>
20.4907 degrees<br>
East<br>
-120 degrees<br>
West<br>
120.826 degrees</p>
<p>and when we looked at the geohash values stored in the index for the record they appear to be incorrect. The bounding coordinates this DCX record is using are a bit weird but I'm not sure they're invalid. Google's JavaScript maps API calculates the centroid as 35.490700000000004,-179.58700000000002 which, according to this tool, <a href="http://www.movable-type.co.uk/scripts/geohash.html">http://www.movable-type.co.uk/scripts/geohash.html</a>, should have a geohash of 8n23ckusk but has a geohash of sn23ckusr instead, which is nearly 180 degrees longitude away from the expected location.</p>
Infrastructure - Bug #8043 (New): The origin field for EML documents isn't properly extracted whe...https://redmine.dataone.org/issues/80432017-03-10T21:36:25ZBryce Mecummecum@nceas.ucsb.edu
<p>We just ran into this with the following EML record: <a href="https://knb.ecoinformatics.org/#view/doi:10.5063/F15B00CC">https://knb.ecoinformatics.org/#view/doi:10.5063/F15B00CC</a></p>
<p>The EML has six creators (Kiesecker, Fargione, Baruch-Mordo, Trainor, Ryan, Patterson) but the origin field in the Solr index has two (Ryan, Patterson). After some digging, we realized this was likely because the indexing component responsible for EML doesn't respect EML references. The XML for the relevant section is:</p>
<p><code><br>
<creator scope="document"><br>
<references>1484778487589</references><br>
</creator><br>
<creator scope="document"><br>
<references>1484778426939</references><br>
</creator><br>
<creator scope="document"><br>
<references>1484778028081</references><br>
</creator><br>
<creator scope="document"><br>
<references>1484778171131</references><br>
</creator><br>
<creator id="1485385283277" scope="document"><br>
<individualName><br>
<salutation>Dr.</salutation><br>
<givenName>Joe</givenName><br>
<surName>Ryan</surName><br>
</individualName><br>
<organizationName>University of Colorado Boulder</organizationName><br>
<positionName>Professor</positionName><br>
<electronicMailAddress>joseph.ryan@colorado.edu</electronicMailAddress><br>
</creator><br>
<creator id="1484777776976" scope="document"><br>
<individualName><br>
<salutation>Dr.</salutation><br>
<givenName>Lauren</givenName><br>
<surName>Patterson</surName><br>
</individualName><br>
<organizationName>Duke University</organizationName><br>
<positionName>Water Policy Associate</positionName><br>
<address scope="document"><br>
<deliveryPoint>Nicholas Institute for Environmental Policy Solutions, Duke University</deliveryPoint><br>
<city>Durham</city><br>
<administrativeArea>NC</administrativeArea><br>
<postalCode>27708</postalCode><br>
<country>USA</country><br>
</address><br>
<electronicMailAddress>lauren.patterson@duke.edu</electronicMailAddress><br>
</creator><br>
</code></p>
<p>It would be really nice if the origin field got populated with all those referenced creators.</p>
Infrastructure - Story #7859 (New): Add formatID for the STL 3d model file formathttps://redmine.dataone.org/issues/78592016-08-04T19:02:58ZBryce Mecummecum@nceas.ucsb.edu
<p>The STL file format is a domain standard file format for storing 3d models and is the most common way I've managed 3d models used while 3d printing. Given that 3d printing is seeing increased usage in the sciences, I would say this is a good candidate for inclusion in the controlled list of format ids.</p>
<p>Type: DATA<br>
Id: STL<br>
Name: StereoLithography File Format<br>
Media type: application/sla (unofficial)<br>
Extension: .stl</p>
<p>There is an ASCII form and a Binary form of this format. They don't see to be distinguished according to any standard. What do we do in this case?</p>
<p>References: <br>
- <a href="https://en.wikipedia.org/wiki/STL_(file_format)">https://en.wikipedia.org/wiki/STL_(file_format)</a><br>
- <a href="https://reference.wolfram.com/language/ref/format/STL.html">https://reference.wolfram.com/language/ref/format/STL.html</a></p>
Infrastructure - Bug #7858 (New): Obsoleting a resource map clears the resourceMap field for the ...https://redmine.dataone.org/issues/78582016-08-03T21:34:03ZBryce Mecummecum@nceas.ucsb.edu
<p>This is long-standing behavior that I consider a bug. That said, there are likely plenty of design conversations that predate me I'm unaware of.</p>
<p>When I update a Data Package by updating the metadata object and its resource map with new ones, the resourceMap field in the Solr index for the obsoleted metadata object is cleared. I expected it not to be cleared.</p>
<p>Why is this the way it works? The way I see it, clearing out the resourceMap field in the Solr index for the obsoleted metadata object reduces benefit of us versioning objects. When a package is cited by its metadata object's PID and the package is updated after the citation was published, a visitor to the dataset landing page will no longer see the package because the resource map isn't in the index. Of course they will be shown a link to the latest version of the package which does have a resource map but that's not what they cited.</p>
DataONE API - Bug #7684 (New): Call to MNStorage.update() via REST API returns java.lang.StackOve...https://redmine.dataone.org/issues/76842016-03-21T23:07:39ZBryce Mecummecum@nceas.ucsb.edu
<p>I was trying to update an object via the REST API via cURL and forgot to enter the correct URL. The cURL command I used and response is:</p>
<p>$ curl -X PUT -H "Authorization: Bearer $TOKEN" -F "pid=resourceMap_doi:10.5065/D6G44NFV" -F "object=@object.xml" -F "sysmeta=@sysmeta.xml" -F "newPid=resourceMap_doi:10.5065/D6G44NFV_v3" $URL<br>
<?xml version="1.0" encoding="UTF-8"?><br>
java.lang.StackOverflowError<br>
</p>
<p>Where $URL was '<a href="https://arcticdata.io/metacat/d1/mn/v2/object">https://arcticdata.io/metacat/d1/mn/v2/object</a>' instead of '<a href="https://arcticdata.io/metacat/d1/mn/v2/object/resourceMap_doi:10.5065/D6G44NFV">https://arcticdata.io/metacat/d1/mn/v2/object/resourceMap_doi:10.5065/D6G44NFV</a>'</p>
<p>I expected to receive some sort of warning/error that I had forgotten to specify the URL properly for this call but instead saw a StackOverflowError.</p>
Infrastructure - Story #7668 (New): Determine how indexing of data packages should workhttps://redmine.dataone.org/issues/76682016-03-02T00:16:25ZBryce Mecummecum@nceas.ucsb.edu
<p>I've discovered (with Lauren's help) a strange requirement for how the resource maps for nested data packages have to be written. In order to get nested data packages correctly indexed in Solr so that the 'resourceMap' field of the resource map being nested is set to the parent resource map's PID, you have to create the appropriate set of @cito:documents@ statements in addition to the expected @ore:aggregates@ statements.</p>
<p>I expected the following to be sufficient (pardon the highly abstracted RDF, examples are linked below):</p>
<p>parent_resource_map#aggregation ore:aggregates child_resource_map<br>
parent_resource_map#aggregation ore:aggregates metadata_object</p>
<p>but I also had to add a @cito:documents@ statement between the <em>parent resource map's metadata object</em> and the resource maps being nested</p>
<p>parent_resource_map#aggregation ore:aggregates child_resource_map<br>
parent_resource_map#aggregation ore:aggregates metadata_object</p>
<p>parent_metadata_object cito:documents child_resource_map</p>
<p>The documentation does not suggest this and I found it confusing. A real life example of what I expected to work is here: <a href="https://gist.github.com/amoeba/c7a6ba269c5a1f78db1d">https://gist.github.com/amoeba/c7a6ba269c5a1f78db1d</a><br>
What I actually had to insert is here: <a href="https://dev.nceas.ucsb.edu/knb/d1/mn/v2/object/resourceMap_urn:uuid:ab17b047-a341-4d06-b433-92eed90dacec">https://dev.nceas.ucsb.edu/knb/d1/mn/v2/object/resourceMap_urn:uuid:ab17b047-a341-4d06-b433-92eed90dacec</a></p>
<p>Is the need for the @cito:documents@ statement(s) really required and is this the intended behavior? I've made this issue in the hopes we can talk about it.</p>
<p>I suggest updating the API docs with whatever we decide, and hopefully that update will include example RDF for a nested data package.</p>
DataONE API - Bug #7578 (New): Fix 404 link to d1_instance_generator folder in documentationhttps://redmine.dataone.org/issues/75782016-01-08T22:01:20ZBryce Mecummecum@nceas.ucsb.edu
<p>In the MN API documentation for MNStorage.create (<a href="https://jenkins-ucsb-1.dataone.org/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html//apis/MN_APIs.html#MNStorage.create">https://jenkins-ucsb-1.dataone.org/job/API%20Documentation%20-%20trunk/ws/api-documentation/build/html//apis/MN_APIs.html#MNStorage.create</a>), I found a the following paragraph contains a broken link to d1_instance_generator:</p>
<blockquote>
<p>"The system metadata included with the create call must contain values for the elements required to be set by clients (see System Metadata). The system metadata document can be crafted by hand or preferably with a tool such as generate_sysmeta.py which is available in the d1_instance_generator Python package. See documentation included with that package for more information on its operation."</p>
</blockquote>
<p>The link to d1_instance_generator was to the SVN folder <a href="https://repository.dataone.org/software/cicore/trunk/d1_instance_generator">https://repository.dataone.org/software/cicore/trunk/d1_instance_generator</a> which is currently a 404. I think the folder moved to /d1_test_utilities_python/src/d1_test/instance_generator.</p>