Member Nodes

Ongoing discussions between Tim and Dave V during late March and early April 2014; some discussion with Bruce Wilson during EU BON meeting in Crete, early April 2014.

Update from Tim Robertson: Has certificates working, but had to put this on hold for some other work. Expects to have significant progress before CCIT,

GBIF's announcement of the MOC with DataONE: http://www.gbif.org/page/8199

GBIF is in a position now to proceed with MN development. We would like to have a GBIF MN in place before the RDA meeting in September. Notes from 6/23/2017:

timrobertson100 [8:02 AM]Hi Folks. GBIF (fmendez leading) will stand up an installation of our D1 MN stack next week to start some tests with you. This is a java software stack built primarily around Dropwizard. We designed it to work with any backend, but we’ll be backing ours by Hadoop HDFS. Our first objective will be to get the 28 snapshot views of all GBIF.org data into D1 (i.e. views of GBIF dating back to 2008). The second objective will be to enable full tier 4 replication. Next week we’ll just run it with an in-memory backend, and we might have some questions on how to run conformance tests. Our codebase is https://github.com/gbif/dataone (edited)

davev [8:11 AM] That’s great Tim - looking forward to seeing GBIF as a participant in DataONE. Approximately how large are the snapshot views?

timrobertson100 [8:29 AM]not entirely sure how we will format this… Worst case: 14:28:10 c5n1 ~ $ hdfs dfs -du -h -s /c4-backup/user/hive/warehouse/snapshot.db 5.8 T 17.3 T /c4-backup/user/hive/warehouse/snapshot.db shows 5.8TB of compressed data (17.3TB because we have 3x replication) but that is probably a fair few more views of the data than we’ll prepare. I expect we’ll store the raw views, and then every quarter we’ll store each view processed to the latest QA routines. So maybe growing at 0.5TB a quarter or so (edited)

davev [8:30 AM] is that 5.8TB / snapshot?

timrobertson100 [8:30 AM] no - that would be all

davev [8:32 AM] would each snapshot be treated as a single blob or file?

timrobertson100 [8:33 AM] We’ve just set up a new HDFS cluster, which is showing 153TB used with 365TB free. I’d expect we’ll be offering 50-100TB of replication space, and would target ecological datasets if at all possible. That way we can start trying to data mine for tabular data with species scientific names, and try and build the taxa+space+time indexes of tabular data

davev [8:33 AM] Nice

timrobertson100 [8:33 AM]each snapshot would be a compressed CSV of verbatim data and another of the same data after QA

davev [8:34 AM]what metadata format is used to describe the snapshots?

timrobertson100 [8:34 AM]Specifically each is about 30 columns of Darwin Core. We’d craft that metadata - Dublin Core, EML or probably more applicable would be DataCite. We’ll DOI everything, so DataCite for sure. But your guidance appreciated there.

davev [8:36 AM]so DataCite is fairly light on the details. It’s ok to assist with discovery but falls short of a rich description of a dataset

timrobertson100 [8:38 AM] The issue though, is that these represent a derived few of 10,000+ datasets. Each of those has EML though

davev [8:38 AM] yeah, I was just pondering that

timrobertson100 [8:38 AM] we could archive each dataset independently as well, but I had thought we might start with the snapshots first (to enable people to do large scale analysis and change in GBIF.org across time) (edited)

[8:39]
https://api.gbif.org/v1/image/unsafe/http://www.gbif.org/sites/default/files/gbif_analytics/global/figure/occ_kingdom.png (125kB)
shows the data growth on those snapshots

davev [8:40 AM]
an EML description of a snapshot would at least enable description of the individual columns

timrobertson100 [8:41 AM]
Yeah, we could do that.

davev [8:41 AM]
that’s a lot of growth and not slowing down

timrobertson100 [8:42 AM]
The DataCite kernel though, allows us to link the DOIs of each individual contributing dataset using the (or something like this) so you can trace those back to each source and it’s EML. we’re anticipating 30% growth this year. The citations are growing quickly too. you can find those here [[https://demo.gbif.org/resource/search?contentType=literature]] - I think it was 78 peer reviewed citations last month (edited)

davev [8:44 AM] that is awesome.so each collection has a DOI, and each snapshot, which is a collection of collections, has a DOI as well, right?

timrobertson100 [8:45 AM] We’d like to chat with you and DataCite at the RDA meeting about this. We’re finding our way a bit here, and would like to align what we do with you. Yes. and then we crawl for papers. So paperDOI -> downloadDOI -> “many source dataset DOIs”. This allows us to show publishers which papers made use of parts of their data. We offer filtered search across all records, so normally a use spans parts of many datasets. https://demo.gbif.org/occurrence/search is a SOLR Cloud index across 782M records (download temporarily disabled as we move datacenters today)

Follow up notes from web testing:

Need to confirm the time zone of the server being used: System metadata is being stamped with +2:00 such as "2017-10-03T00:00:00.000+02:00".

Need to consult with MN for testing/implementation of updates.

+Tier 1:+

• MNCore - All tests pass except for "getLogRecords - test event filtering". This is supposed to test ability to filter on event type. But https://labs.gbif.org:7005/mn/v1/log?event=read successfully does so. The actual url used per stack trace was https://labs.gbif.org:7005/mn/v1/log?toDate=2017-10-06T19:10:02.452%2B00:00&event=read&start=0&count=0, from which the test expected 0 results returned. The issue may be due to appearance of discrepancy between server time zones.

• MNRead - All tests pass except for "getReplica - tests getReplica returns a valid object" However comments indicate that if node is not yet registered, then this error can be ignored. (Node is not currently registered)

• Authentication - Complaining about "Authentication - test with self-signed certificate" failing to return a Not Authorized exception.

• Content Integrity - passes

+Tier 2:+
* MNAuthorization passes - Note: originally there were some issues with the online version but Rob was able to successfully run it from his local machine, which populated the test objects needed for it to pass now. This points to the need for some updating to what's on the server now.

+Tier 3:+

• MNStorage API - "update - tests that update works" fails. This specifically attempts to create an object and then update it. The test was unable to successfully perform an update and note of the update-related pids (such as mNodeTier3TestUpdate201727993310335) were found to have obsoletes/obsoleted properties. Furthermore, update - tests with bad obsoletedBy info, and update - tests with bad obsoletes failed to return invalid system metadata exceptions.

+Tier 4:+

• MNReplication - All tests pass.

Overall there seems to be some outstanding issues with the web tester version online which were confirmed to affect MN Authorization and MN read (different errors in stack trace), suggesting that what's on the server is outdated in some places.

Last communication w/ Federico on 12/5:
GBIF is working on a T1/T2 implementation in small project called DataRepo (https://github.com/gbif/data-repo/) which is the back-end of their DataOne MN, and is almost ready to be released. Their plan is to have the DataOne MN working initially as Tier1 or Tier2, and then support tiers 3 and 4.

From their side they are pretty much ready to start testing operations of tiers 3 and 4 but prefer to do it incrementally.

GBIF on 12/14: We have a testing public environment now for our member node, https://ws.gbif-uat.org:7000/mn/v1/, it took us a while to make it available because of firewall restrictions, this MN is backed by another service that we putting on production that is called the GBIF Data Repo which can be found here http://api.gbif-uat.org/v1/data_packages/ (staging/user acceptance test environment)

12/15:
* Asked Federico an estimation of how many data objects will be exposed via the MN. Response: amount of objects will be, initially, no more 20, we'd like to expose GBIF snasphots , each is about 4 GB.

• Checking the MN, I observed that a count of 20 is being returned when there is only 1 object being exposed by the MN. For example only one object was being returned from https://ws.gbif-uat.org:7000/mn/v1/object?start=0 on the morning of 12/15, but the list objects response is:

42592444-b1c7-4086-9625-66153b8c90b5
ba0fb20240817bab7ef68667c1915962
2017-12-14T13:58:43.590Z
6821285

/ns2:objectList

Provided a link to type documentation at http://jenkins-1.dataone.org/jenkins/job/API_Documentation_trunk/ws/api-documentation/build/html/apis/Types.html#Types.Slice . GBIF is excited about their progress and eager to get registered in Sandbox. This will probably take place after the new year as the listObjects() functionality is central to harvesting and should be responding predictably before we attempt to harvest content.

Approaching GBIF w/ invitation to consider using GMN with OAI-PMH as a temporary stand-in to get GBIF content online until they get further with custom development.

1/29/2018: {Amy} email tim & Frederico with the idea.
* Tim out of office until 2/26

Continue --> Task #5836

Contacted by Federico to report that they are testing MN on web tester, but some now deleted identifiers seem to have been cached.

Notes on the MN API: I am testing their node as a v1 Tier 1. They chose NOT to implement the optional pidFilter parameter to getLogRecords(), and formatId parameter to listObjects().

Their packages are zip acrhives of 6-month snapshots of their collection, and are composed of 1 ORE, 1 EML metadata doc, and 1 zip archive. The zip archive is multiple GB, so may be difficult to get without adjusting client timeouts and possible memory .