Infrastructure

It appears that the indexing process is far too slow to keep up with content additions and changes. Since the version 2.3 upgrade which includes support for multiple indexing threads, the performance appears improved, but it falls far short of what is needed to provide reasonable currency.

In particular, it appears that large resource maps such as those provided by the ARCTIC node are very slow to evaluate.

Some optimization may be possible without major refactoring of the indexing process.

A few possible options:

1. Check that changes to properties such as ownership do not trigger an entire re-index of the package. If permissions change, then there is no need to reindex the entire package since other properties are unchanged. This should be in place now since content is immutable, and only mutable metadata fields should be updated.

2. Dedicate a single thread to resource map processing, expanding to more threads when there is no backlog of other content. This would allow efficient processing of content on which the resource map indexing may depend.

3. Refactor the index so that resource maps may be processed independently, without the need for all other objects to be loaded and processed.

4. Refactor the indexing of resource maps so that a partially processed resource map is persisted so that processing may continue as content becomes available rather than starting from scratch each time.