Member Nodes

And my response to Markus:

Hi Marcus,

It was great to meet you, and thanks for the information. At DataONE, we are very excited about partnering with you, not least because your topical emphasis is extremely complementary to ours. Like you, however, we have to figure out how we might be able to work with you in terms of developer time. One possibility might be for us to use a 'harvest only' strategy for Pangaea to become a Tier 1 member node, which would be a fairly simple deployment for you. That loses the ability for users to contribute data to Pangaea via DataONE APIs, but that could always be added later when it makes sense. Our member team is looking over the materials you sent, and we'll get back to you with some possibilities. Is this the right list of contacts to include in these discussions? Thanks,

Matt

Laura to contact Pangaea week of 30 January to set up an exploratory meeting.

Meeting 20 Feb 2017 at 11am ET. https://epad.dataone.org/pad/p/Pangaea_and_DataONE

DataONE to develop an implementation plan; Pangaea to send examples of metadata and other points of interest. Meet again around 13 March date/time TBD.

Meeting 9 June 2017 (https://epad.dataone.org/pad/p/Pangaea_and_DataONE). Monica demo'd parts of the OAI-PMH harvester. Some questions arose which have been addressed, including (below from email from Monica to Pangaea team 17 June):

• Identifiers - The adapter script now strips the oai-pmh prefix from identifiers as they are loaded into the generic Member Node.

• The modification of record updating logic has been implemented. Previously, if a Pangaea identifier showed up in a harvest, and that identifier already existed in the member node data store, then an .update() upon the record would automatically be processed. What has changed is that now the datestamp of the record from the OAI-PMH harvest is compared to the dateUploaded in the Member Node. An update will only be processed if the date has changed. This means that, should some problem be encountered during a harvest, there will be no consequences from simply starting over. Records will not be needlessly updated/versioned.

• The current version of the adapter script has been installed on one of my test installations of the generic member node software. This particular test installation is not registered to a Coordinating Node. It's just a "stand-alone" test instance for monitoring the activity between the Pangaea service provider, the adapter, and the generic Member Node software.

• The adapter is currently scheduled to run every 2 hours. When this is implemented live for Pangaea, the frequency will probably be much less, but the idea will be the same: use cron to run it periodically. I will leave this running over the next week to keep an eye on things. I've setup rsync to copy over the harvest log and the error log to a subdirectory in the site's document root so you can monitor this too. What I'll be watching out for is any actual processing errors that might be encountered on a record by record basis.

o Harvesting Log:
https://centos7-3gmn.kitty.ninja/pangaea/OAI-PMH_harvest.log
o Error Log:
https://centos7-3gmn.kitty.ninja/pangaea/adapter-errors.log

• Here are some endpoints for the test installation for Pangaea content on my server that you will find helpful:
Description End Point
You can see the current number of objects on this member node installation here: https://centos7-3gmn.kitty.ninja/mn/v2/object?start=0&count=0

You can browse through lists of abbreviated system metadata for harvested records with this end point. Just adjust the start and count parameters to suit you: https://centos7-3gmn.kitty.ninja/mn/v2/object?start=0&count=50

You can grab the identifier field from any of the records in the list and use this endpoint to see the complete system metadata record: https://centos7-3gmn.kitty.ninja/mn/v2/meta/doi:10.1594/PANGAEA.51463_20170617_0026

The science metadata record for an object which was harvested from Pangaea is accessible through this endpoint: https://centos7-3gmn.kitty.ninja/mn/v2/meta/doi:10.1594/PANGAEA.51463_20170617_0026

At this point we are basically ready to move forward with setting up Pangaea. We understand that Pangaea desires some alterations to how their science metadata is presented to end users through the search interface. However, we can still make progress in setting up the member node software and adapter on a Pangaea controlled server. The process must be operational between Pangaea's service provider and the member node software anyways, before the node can be registered to a Coordinating Node in a testing environment.

It would be good for DataONE to know where Pangaea stands on getting an installation of the Generic Member Node active. The instructions for installation are pretty thorough, but there are a lot of steps. So I would like to volunteer to be present with someone from Pangaea during the installation process. That way you can share your screen, and I can provide real-time support as needed.

Next steps: feedback from Pangaea about GMN installation and assist if necessary (Monica), define desired changes to metadata view in search interface (Laura)

Summary of Pangaea thus far:

DataONE is hosting both the development and production installations of GMN software for metadata only implementations of Pangaea. Pangaea's metadata is exposed using their OAI-PMH web service and integrated with GMN using a Python 2.7 adapter.

Metadata Format:Pangaea is using a custom formatId implementation of iso-19115 as a result of a mismatch between the version of GMD referenced within their documents and the version actually needed to validate as a dependency. This custom formatId has been installed and tested in Stage, and we are satisfied with testing that proves the efficacy of this formatId for validating Pangaea metadata content.

An additional issue uncovered during testing Pangaea installation in the Stage environment was limitations to the capacity of synchronization to harvest greater than 50,000 records at a time. A solution to this problem has been developed and is currently installed on Sandbox version of CNs.

Testing: The test GMN is installed as 2.3.8 at host pangaea-dev-orc-1.test.dataone.org. Testing actually started off in stage, but the GMN settings are currently pointed switched to sandbox because that's where the synchronization changes were later being tested.

Production: The production GMN is currently installed as 2.4.0 at host pangaea-orc-1.dataone.org. The data is currently being loaded. The CURRENT status is that the prod GMN is unregistered and being loaded in standalone mode.

Indexing: Indexing the data (which affects how the metadata records are presented to end users in the search interface) was discussed at one point. We have considered whether or not to customize how indexing is performed with relation to the custom formatId. However at this point, we are holding off on any changes, and the gmd-pangaea formatId will follow the same indexing rules as gmd-noaa. We can always reindex at a future point in time if needed.

Moving forward:

Synchronization changes put into production: Confirm schedule for synchronization changes being put into production, as Pangaea going into production will depend on that.

Register in Production: Register in production when synchronization changes are in place.

Work with MNC: To communicate size of node, workflow for news release and other administrative tasks for going live.

Version: 2.4.1 is out. Not sure if we should update again before going live.

Communication Assist about information for testing:
1/8/18: email Michael Diepenbroek mdiepenbroek@pangaea.de to get the approx. order of magnitude of PANGAEA number of metadata records. [some discrepancy between # of objects in our dev instance vs production]

1/9 (Response): We keep around 370 Tsd records
* Total number can be found by search engine: https://www.pangaea.de/?q=*
* All from set “citable”: https://www.pangaea.de/?q=@citable
* All from set “citableWithChilds”: https://www.pangaea.de/?q=@citableWithChilds
Actually, the number is the search engine can be a bit lower, because our search engine does not show our dataset’s older versions (it only shows the newest ones). The total difference is around 1000, as far as I remember.

370,000 + objects loaded in production GMN. We will be waiting on the next CI update to push sync changes to production before registering.

One additional important note: Whereas sandbox/stage used a concatenation of native system identifier (which is a doi) + datetimestamp for assigning PID (the native system id is assigned as our SID), a change has been made that in production, the checksum is assigned as pid. This is to avoid confusion when an end user is looking at record details in the search interface, so they are not presented with two doi-looking values (pid and sid), only one of which would actually resolve.

Update 2.3.7 to CN is deployed to Stage. This update handles the indexing changes that that address synchronization limits that were previously encountered in Stage. We are switching Pangaea's test installation to point back to Stage and will re-enable sync. Last harvest date will be reset and will attempt to allow a complete resync to monitor for success with the new code updates. If all is well, 2.3.7 could be deployed next week to Production.

An additional step is that the Pangaea custom formatID needs to be manually registered in Production.

Once 2.3.7 is is deployed to prod & formatID is registered on the CN, the Pangaea production installation of GMN will be registered in prod. To review: the prod installation of the GMN app is already loaded with 370k plus objects. It's just waiting to be registered. Since we are so close to the finish line, Amy will start getting things in place such as news announcements and other steps to prepare for Pangaea officially going live.

2/16/18: updated POCs about sync optimization and broad overview of steps that will ensue