Infrastructure

The index processor stops processing or processes extremely slowly, failing to drain the task queue.

Attempting to stop using "service d1-index-task-processor stop" fails, with the process continuing to run very slowly for hours.

<pre>
SELECT status,deleted,COUNT(*) AS cnt FROM index_task GROUP BY status,deleted ORDER BY status;

status | deleted | cnt
------------+---------+-------
FAILED | f | 3269
IN PROCESS | f | 2
NEW | t | 1621
NEW | f | 64348
</pre>

cn-index-processor-daemon.log:
<pre>
[ INFO] 2016-11-14 13:36:10,874 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
[ INFO] 2016-11-14 13:36:15,876 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
[ INFO] 2016-11-14 13:36:18,972 (IndexTaskProcessor:isResourceMapReadyToIndex:197) Not all map resource references indexed for map: resource_map_doi:10.18739/A25G76. Marking new and continuing...
[ INFO] 2016-11-14 13:36:18,979 (IndexTaskProcessor:getNextIndexTask:178) Task for pid: resource_map_doi:10.18739/A25G76 not processed.
[ INFO] 2016-11-14 13:36:18,982 (IndexTaskProcessor:getNextIndexTask:161) Start of indexing pid: resource_map_doi:10.18739/A25G8K
[ INFO] 2016-11-14 13:36:20,876 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
[ INFO] 2016-11-14 13:36:25,877 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
[ INFO] 2016-11-14 13:36:30,877 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
[ INFO] 2016-11-14 13:36:35,878 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
[ INFO] 2016-11-14 13:36:40,878 (IndexTaskProcessorScheduler:stop:88) Job executing, waiting for it to complete.....
</pre>