Infrastructure

There are some objects that appear in the cn-sandbox object list (cn/v1/object) but do not seem to exist in the hazelcast system metadata map.

For example esa.89.1, knb-lter-sbc.28.14, peggym.28.5, nikkis.34.18, knb-lter-sbc.27.8

These document ids were discovered in the hzIdentifiers hzcast set and then confirmed to appear in the object list.

Investigating esa.89.1 - it appears in the backing database. "SELECT * FROM systemmetadata where guid='esa.89.1' does return one row on cn-sandbox-unm-1. But I cannot get it to load into the hazelcast system metadata map or to display on the /cn/v1/meta or object endpoints. It does appear in the object list however.

Not sure if the error is that they appear in the object list and in hzIdentifiers OR if the error is that they do not appear in system metadata hz map.

The sandbox CNs were scrubbed on Mar 21, 2012 11:47:43 PM
esa.89.1 was synchronized on cn-sandbox-unm-1 at 2012-03-22 00:09:17,320 from urn:node:mnStageUCSB1.

Replication occurs and the document appears on urn:node:mnStageORC1.

Synchronization runs again and picks up the record on cn-sandbox-orc-1. However, instead of going through the normal update process as it would with a replica that it finds in the hzSystemMetadata map, it receives a null object and so determines that esa.89.1 should be created again. However, upon the call to CNCore.create(), a IdentifierNotUnique error is returned.

As noted above, esa.89.1 is in the database in systemmetadata, in xml_documents and the sciMetadata can be found in /var/metacat/Documents as 'autogen.2012032116091821001.1'. The record was found on all the CNs. But failed to load on all the CNs.

hz_peek confirms that hzSystemMetadata does not contain the Identifier key for esa.89.1. However a call to D1NodeService.getSystemMetadata will throw an NotFound error in D1NodeService.isAuthorized at line 938 indicating that HazelcastService.getInstance().getSystemMetadataMap().get(pid) returns null.