Member Nodes

{from Sky Bristol}
* Sending to Ben Wheeler and Drew Ignizio on the USGS side
* my guess is this is coming from a data lifecycle misalignment. You are likely getting a form of CSDGM metadata that only had an onlink to the ScienceBase Item but did not include distinfo links to the data download files from those items. Seems like something that definitely needs to be fixed and can really only be fixed on the USGS source end.
* I'll push on this to see that we come up with a solution for DataONE and other "third party" catalogs.

Background

It is believed that duplicate records are being harvested into DataONE as a consequence of how SDC is handling its relationship with its own source repositories. The background story is that persistent identifiers are inconsistently applied, or in some cases not at all present, across contributors to the SDC. The way that SDC handles this is to essentially wipe and reload their entire set of metadata periodically, rather than identifying updated or new records and only making those changes.

As a consequence of how this process of wiping and reloading is handled, a single object will have been harvested into SDC from a given contributor more than once, but assigned a different arbitrary identifier by SDC each time in the process. This does not result in any ill effect for SDC, but an unintended consequence is that the same object would be harvested into DataONE more than under different identifiers.

Current Action

SDC had taken their member node offline in order to prevent the problem from being compounded by additional duplicate records being harvested. Today sync has been disabled on the CN side for USGS SDC to prevent any such issues. They have thus brought the node at https://usgs.ornl.gov/USGS_MN/mn/v1 back online.

However, download links are still failing to resolve. This is most likely due to that whatever identifier was assigned to a record whenever ingested into DataONE will have changed each time SDC picked the object back up again from its source. And the MN links are tied to the system identifier.

For example, this record in USGS SDC: https://data.usgs.gov/datacatalog/#fq=dataType%3A(collection%20OR%20non-collection)&q=(((title%3A*%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%22*)%5E100%20OR%20(placeKey%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%22)%5E100%20OR%20(keywordsText%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%22)%5E100%20OR%20(abstract%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%22)%5E30%20OR%20fullText%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%22%20AND%20(fileType%3Amarquee%5E30%20OR%20fileType%3Aharvested))%5E30%20OR%20((title%3AMean*annual%20water-budget%20components%20for%20the%20Island%20of%20Maui)%5E100%20OR%20(placeKey%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui)%5E100%20OR%20(keywordsText%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui)%5E100%20OR%20(abstract%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui)%5E30%20OR%20fullText%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%20AND%20(fileType%3Amarquee%5E30%20OR%20fileType%3Aharvested)))%20AND%20(((title%3A*%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20%22*)%5E100%20OR%20(placeKey%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20%22)%5E100%20OR%20(keywordsText%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20%22)%5E100%20OR%20(abstract%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20%22)%5E30%20OR%20fullText%3A%22Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20%22%20AND%20(fileType%3Amarquee%5E30%20OR%20fileType%3Aharvested))%5E30%20OR%20((title%3AMean*annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20)%5E100%20OR%20(placeKey%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20)%5E100%20OR%20(keywordsText%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20)%5E100%20OR%20(abstract%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20)%5E30%20OR%20fullText%3AMean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20%20AND%20(fileType%3Amarquee%5E30%20OR%20fileType%3Aharvested)))%20AND%20(((title%3A*%22%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).%22*)%5E100%20OR%20(placeKey%3A%22%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).%22)%5E100%20OR%20(keywordsText%3A%22%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).%22)%5E100%20OR%20(abstract%3A%22%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).%22)%5E30%20OR%20fullText%3A%22%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).%22%20AND%20(fileType%3Amarquee%5E30%20OR%20fileType%3Aharvested))%5E30%20OR%20((title%3A*Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).)%5E100%20OR%20(placeKey%3A%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).)%5E100%20OR%20(keywordsText%3A%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).)%5E100%20OR%20(abstract%3A%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).)%5E30%20OR%20fullText%3A%20Mean%20annual%20water-budget%20components%20for%20the%20Island%20of%20Maui%2C%20Hawaii%2C%20for%20current%20conditions%2C%202001-10%20rainfall%20and%202001-10%20land%20cover%20(version%202.0).%20AND%20(fileType%3Amarquee%5E30%20OR%20fileType%3Aharvested)))

is found 27 times with 27 different identifiers in DataONE. These are not expected to be different versions of the same object, just 27 duplicate records assigned 27 different arbitrary identifiers. One example identifier among the 27 has the object location URL: https://usgs.ornl.gov/USGS_MN/mn/v1/object/e024bdbf-ae51-4d8b-8327-a94ed207be1c but this object does not exist on the MN and cannot presently be retrieved except by the copy on the CN.

Next Steps

DataONE will attempt to triage the objects already on Production, attempting to determine if obsolescence chains can be constructed to at least tie all duplicate versions of the same object together, so that if anyone has ever cited the data the availability of a new version is not lost.

However, this still leaves up in the air the problem of the URLs not resolving to anything. This must be discussed in further detail.

Future steps will be to redeploy USGS SDC as a v2 Slender Node with GMN. The details of how that will be accomplished, including resolving identifier issues, will be addressed in a separate ticket. This ticket will only address putting a band-aid on the current situation in production as it stands now.

8/13/18: email Aaron Stokes, stokesam@ornl.gov to get re-engaged

Decided in today's maint. call that we will archive the current SDC content before proceeding with GMN.

8/20/18: follow-up email Aaron Stokes, stokesam@ornl.gov to get re-engaged

8/21/18: Mike Frame sent email to USGS team to coordinate F2F meeting with Monica to knock out archiving the data

8/22/18: Call to discuss archiving and next steps
Aaron stokesam@ornl.gov & Tudor garbuletvt@ornl.gov; Monica & Amy

(1) Monica to archive current data
(2) MN start working in Sandbox environment and dev OAI-PMH
(3) Lisa zolly working on USGS PID/DOI issues --> Aaron to see if there is a USGS source repository that can be used for testing while they hash out the issues.

Archival of Existing Objects:
- Monica will archive the 33,000 ish SDC objects currently on the CN.
- Monica will notify USGS once the archival is complete via email.
- USGS will take SDC member node offline temporarily.

10/8/18 - from Aaron

Monica and I met today to discuss our path moving forward, and we have come up with a good plan. Over the next two weeks we plan to archive the existing SDC metadata in the DataONE CN and replace it with a static list of SDC metadata records. These records will be available while we work on setting up the GMN software and introducing DOIs as identifiers to the process.