Infrastructure

The timeouts may be due to load on the servers. Due to the increased number of objects in the maps when moving from 5K objects on sandbox to 130+K objects on staging, the network utilization may increase due to the number of objects managed by each hazelcast partition. Once a partition changes on a member, it will be synchronized with the rest of the cluster due to the backup policy of the cluster. I assume that the size of any partition increases with the addition of objects to a distributed map, since the number of partitions default at 271. The increase in the number of the partitions should decrease the number of objects stored per partition and limit the amount of bandwidth needed when synchronizing cluster members for any changed partition.

To test the number of partitions as related to the issue of timeouts on sandbox, I decreased the partition count from 271 to 10.

hazelcast.map.partition.count = 10.

I then ran a program that would evict all hzSystemMetadata objects, get them all, then put them all. Thus creating a lot of transactions on the hazelcast map across the cluster.

network usage was monitored using tcpstat