Infrastructure

I wiped off the data objects and the system metadata in the sandbox env CNs and kicked off the synchronization of data objects and system metadata objects from member nodes.
Over a weekend, cns only harvested 160 data objects. We found some system metadata having wrong information in those objects ( I didn't check every one, just a couple):

The cns say that the originMemberNode and authoritativeMemberNode are ucsb-2
https://cn-sandbox-ucsb-1.test.dataone.org/cn/v1/meta/doi:10.5072/FK2/LTER/knb-lter-gce.100.15

But the member nodes say that the originMemberNode and authoritativeMemberNode are ucsb-1.
https://mn-sandbox-ucsb-1.test.dataone.org/knb/d1/mn/v1/meta/doi:10.5072/FK2/LTER/knb-lter-gce.100.15
https://mn-sandbox-ucsb-2.test.dataone.org/knb/d1/mn/v1/meta/doi:10.5072/FK2/LTER/knb-lter-gce.100.15

So the system metadata on the cn doesn't match the one on the member nodes.

Here is another example:
https://cn-sandbox-ucsb-1.test.dataone.org/cn/v1/meta/doi:10.5072/FK2/LTER/knb-lter-gce.102.2
https://mn-sandbox-ucsb-2.test.dataone.org/knb/d1/mn/v1/meta/doi:10.5072/FK2/LTER/knb-lter-gce.102.2
https://mn-sandbox-ucsb-1.test.dataone.org/knb/d1/mn/v1/meta/doi:10.5072/FK2/LTER/knb-lter-gce.102.2

Skye and I looked at the reason why the synchronization didn't finish and found the hazelcast server on the mn-ucsb-2 was down. We restarted the tomcat on the mn-ucsb-2 and kicked off the synchronization again. This time the synchronization was done.
I checked the system metadata about some pid and found the cn's and mn's matches. So currently cns has a mixture of matching and miss-matching system metadata.