Infrastructure

If a CN joins a cluster, it will need to build &/or rebuild its index.

I think i should look at the last identifier that the CN has indexed. And then start up and start receiving entries. It should not immediately processing MNs for log entries. It should only listen until the first entries received after startup. After the first new entries are received then we know the gap that needs to be recovered.

To assist ordering, the unique identifier should be a Double composed of the system's datetime as a long + . + Hazelcast.getAtomicNumber.incrementAndGet()

This should guarantee uniqueness, and a certain degree of sequentiality (depending upon system clock skew it can not be guaranteed). because sequentiality may not be absolute, any recovery must start at a sequences before the start of the gap and end after the gap. Thus, the unique identifier field should in solr terms overwritable.

Of course clock skew between the CN's can not be too great anyhow or else other facets of CN operation will be affected.