Task #2208
Reconcile the difference between count returned by CN.listObjects() and hzSystemMetadata.size()
100%
Description
A call to CN.listObjects() on cn-dev-2 currently returns 5210 objects. A call to hzSsytemMetadata.size() returns 4210. Determine if this is caused by map entry eviction. We know that the map will not reflect all keys in the backing store (section 2.2.3 of the Hazelcast documentation). This has implications on d1_indexer.
History
#1 Updated by Chris Jones over 12 years ago
- Status changed from In Progress to Closed
I've confirmed that entries that are evicted from the hzSystemMetadata map will not subsequently show up in a call to map.size(), i.e. the size of the map will always reflect the in-memory size, not the total size when map.getAll() is called. On that note, when map.get() is called on a key that has been evicted, the MapLoader implementation pulls the entry from the data store and puts it into the map, which generates an entry added event. This explains why we have seen these events on calls to map.get() in the past.
To reconcile the difference between the size from CN.listObjects() and map.size(), I've created a new Hazelcast structure called 'hzIdentifiers'. This is a distributed Set of unique identifiers in the storage cluster. It will always reflect the size of the backing system metadata postgresql table (and thus be the same as the size from CN.listObjects()). It is a low-expense means of getting all identifiers, and may be useful to the d1_indexer.
The memory footprint of this structure is fairly low since it only contains Identifier objects. On Metacat startup, the hzIdentifiers Set is populated, and it is updated on add/update/evict events on hzSystemMetadata.
