Infrastructure

going to need to know what type of exception is the cause: It seems to be a generic RuntimeException. see the source code: https://github.com/hazelcast/hazelcast/blob/maintenance-2.x/hazelcast/src/main/java/com/hazelcast/impl/ClientServiceException.java

Just noticed that in /etc/dataone/storage/hazelcast.xml the number of backups is set to 3, which means HZ is trying to make 4 copies of everything (self + 3 copies) which of course doesn't work since there's only 3 CNs.

backup-count should be set to "2" for both hzSystemMetadata and hzObjectPath (actually all structures).

Additional suggestions:

Allow read from the backup copies. Backups are created synchronously, so a backup copy will be consistent with the owner's copy. Hence it should be safe to read a backup copy instead of always requesting the owner's copy which is the default.

<read-backup-data>true</read-backup-data>

The eviction policy is currently set to dump 25% of the entries when the maximum of 3,000,000 is reached. This seems completely arbitrary and is likely using a huge amount of memory. Instead, we could set the limit much lower and rely on efficient population of the map on demand from postgres. Suggest keeping 5000 entries in memory:

<max-size policy="cluster_wide_map_size">5000</max-size>

On startup, MapLoader.loadAllKeys is used to pre-populate the hazelcast map. However, entries are also loaded on demand when a key (i.e. PID) is requested. Is there any benefit to pre-populating the map when there is little chance that the populated entries will be relevant to the current operations? Suggest changing MapLoader.loadAllKeys to return NULL. Then entries will be loaded on demand. This would be a change in Metacat.

See: https://docs.hazelcast.org/docs/2.4.1/manual/single_html/index.html#Persistence

WAN replication is available in the version of hazelcast we are using (2.4.1) though becomes an enterprise "feature" on later versions. If we are planning to drop hazelcast for inter-CN sync, then enabling WAN replication may buy us some time for stability before needing to change technologies.

https://docs.hazelcast.org/docs/2.4.1/manual/single_html/index.html#WanReplication

wrt to MapLoader.loadAllKeys, it appears this has already been done:

https://github.com/NCEAS/metacat/blob/master/src/edu/ucsb/nceas/metacat/dataone/hazelcast/SystemMetadataMap.java#L114

Reducing the number of backups seems like a no-brainer.

Regarding pre-population and how many to hold in memory, I think we need to consider the impact on lookup performance and also the hit distribution. Reducing from 750k in memory to 5k in memory I think really ups the odds that a db lookup + serialization will be needed.

If 95% of the time an HZ map lookup is going to take the same amount of time as a CN.getSystemMetadata call, it may not be worth using hazelcast at all.

Ideally we could pre-populate the map (upon startup) with items likely to be needed, based on archive status and modification date.

I also think because of how we index OREs, we need a larger cache, or else a large ORE will cycle the cache for clients. We didn't seem to have pre-loading or stability issues that long ago, so I would suggest keeping 100k-200k in memory. that's still at least 35% of what we are currently using, and would scale going from 3 CNs to 2 CNs.

Given the current analysis of xerces processing time for the view service, maybe there's a similar performance drag affecting systemMetadata serialization as well. JAXB serialization may also be slower than JiBX, as well.

The following changes have been deployed on production:

1. <backup-count>2</backup-count>
2. <max-size policy="cluster_wide_map_size">50000</max-size>
3. <read-backup-data>true</read-backup-data>

After tomcat restart sync appears stable. (previously sync was failing with timeouts after an hour or so).

Performance of getSystemMetadata, getObject, and the view service appear unaffected.

wrt. Rob's comments - the reduction was from 3,000,000 entries held in memory. I dropped it down to 50k which may be on the low side, but the primary goal is to get synchronization stable first.

The system metadata map is used to mirror sysmeta between the CNs. An alternative is to use postgres replication but that would involve major refactoring.

We need to do more profiling to determine what the bottlenecks are for basic operations like getSystemMetadata. I suspect authorization as the main blocker, but really need to measure.

wrt. JAXB / JiBX and xerces - this may well be a bottleneck as well. moving in and out of xml is pretty slow.