Infrastructure

With the current mechanism, it is possible that not all objects will be synchronized if a failure should occur. There is a very real possibility that if a failure should stop synchronization, the next time it starts, it will not begin from where it left off. Spring Batch has persistence and fail over capabilities.

Create a persistent mechanism that will determine if a failure is repeated due do to a problem with the system or a problem with a document. If an object is unable to be synchronized do to object format failure, then a message needs to be sent to administrator.

Determining batch size may be an issue as well. If search services do not order result sets, then more robust persistence measures should be taken to insure recovery after failure. Currently, ordering of search results is unspecified. It may be that MNs will not implement such a measure. In fact, this problem may well extend beyond simple ordering of sets based on dataSysModified...

We also will need to create a background process that double checks every doc in metacat that can be index has been indexed. This is incase a message is lost or an error occurs that allows a document to be created/updated in metacat but not pushed to the index triggering mechanism. or, if the triggering mechanism fails.