Infrastructure

calling shutdown resulted in a hang, then exception thrown.

It seems like an ordinary exception that should be handled.

[ WARN] 2018-04-12 16:12:32,208 [main] (IndexTaskProcessorScheduler:stop:118) 1 jobs executing,  waiting for them to complete...
[ WARN] 2018-04-12 16:12:37,208 [main] (IndexTaskProcessorScheduler:stop:118) 1 jobs executing,  waiting for them to complete...
[ INFO] 2018-04-12 16:12:40,478 [QuartzScheduler_Worker-1] (HTTPService:doRequest:381) REQUEST URI= http://localhost:8983/solr/search_core/select/?q=seriesId%3A11333%2F1366838+OR+seriesId%3A11333%2F1366852+OR+seriesId%3A11333%2F1366836+OR+seriesId%3A11333%2F1366870+OR+seriesId%3A11333%2F1366864+OR+seriesId%3A11333%2F1366844+OR+seriesId%3A11333%2F1366874+OR+seriesId%3A11333%2F1366830+OR+seriesId%3A11333%2F1366866+OR+seriesId%3A11333%2F1366854+OR+seriesId%3A11333%2F1366860+OR+seriesId%3A11333%2F1366872+OR+seriesId%3A11333%2F1366834+OR+seriesId%3A11333%2F1366858+OR+seriesId%3A11333%2F1366846+OR+seriesId%3A11333%2F1366862+OR+seriesId%3A11333%2F1366828+OR+seriesId%3A11333%2F1366832+OR+seriesId%3A11333%2F1366878+OR+seriesId%3A11333%2F1366876+OR+seriesId%3A11333%2F1906394+OR+seriesId%3A11333%2F1366868+OR+seriesId%3A11333%2F1366850+OR+seriesId%3A11333%2F1366856+OR+seriesId%3A11333%2F1366848&start=0&rows=25&indent=on
[ INFO] 2018-04-12 16:12:40,487 [QuartzScheduler_Worker-1] (HTTPService:doRequest:381) REQUEST URI= http://localhost:8983/solr/search_core/select/?q=id%3A11333%2F1366838+OR+id%3A11333%2F1366852+OR+id%3A11333%2F1366836+OR+id%3A11333%2F1366870+OR+id%3A11333%2F1366864+OR+id%3A11333%2F1366844+OR+id%3A11333%2F1366874+OR+id%3A11333%2F1366830+OR+id%3A11333%2F1366866+OR+id%3A11333%2F1366854+OR+id%3A11333%2F1366860+OR+id%3A11333%2F1366872+OR+id%3A11333%2F1366834+OR+id%3A11333%2F1366858+OR+id%3A11333%2F1366846+OR+id%3A11333%2F1366862+OR+id%3A11333%2F1366828+OR+id%3A11333%2F1366832+OR+id%3A11333%2F1366878+OR+id%3A11333%2F1366876+OR+id%3A11333%2F1906394+OR+id%3A11333%2F1366868+OR+id%3A11333%2F1366850+OR+id%3A11333%2F1366856+OR+id%3A11333%2F1366848&start=0&rows=25&indent=on
[ INFO] 2018-04-12 16:12:40,506 [QuartzScheduler_Worker-1] (IndexTaskProcessor:getNextIndexTask:664) the original index task - IndexTask [id=8123478, pid=11333/1906395, formatid=http://www.openarchives.org/ore/terms, objectPath=/var/metacat/data/autogen.2016071500060240859.1, dateSysMetaModified=1407716100000, deleted=false, taskModifiedDate=1523549371987, priority=3, status=FAILED]
[ INFO] 2018-04-12 16:12:40,506 [QuartzScheduler_Worker-1] (IndexTaskProcessor:getNextIndexTask:670) the new index task is a ResourceMapIndexTask
[ INFO] 2018-04-12 16:12:40,506 [QuartzScheduler_Worker-1] (IndexTaskProcessor:getNextIndexTask:671) the new index task - IndexTask [id=8123478, pid=11333/1906395, formatid=http://www.openarchives.org/ore/terms, objectPath=/var/metacat/data/autogen.2016071500060240859.1, dateSysMetaModified=1407716100000, deleted=false, taskModifiedDate=1523549371987, priority=3, status=FAILED]
[ INFO] 2018-04-12 16:12:40,506 [QuartzScheduler_Worker-1] (IndexTaskProcessor:processTaskOnThread:260) using multiple threads to process index and the size of the thread pool is 5
[ERROR] 2018-04-12 16:12:40,507 [QuartzScheduler_Worker-1] (JobRunShell:run:222) Job d1-cn-index-processor.d1-index-processor-job threw an unhandled Exception: 
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ce719d9 rejected from java.util.concurrent.ThreadPoolExecutor@7e212803[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 222]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at org.dataone.cn.index.processor.IndexTaskProcessor.processTaskOnThread(IndexTaskProcessor.java:267)
    at org.dataone.cn.index.processor.IndexTaskProcessor.processIndexTaskQueue(IndexTaskProcessor.java:186)
    at org.dataone.cn.index.processor.IndexTaskProcessorJob.execute(IndexTaskProcessorJob.java:57)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[ERROR] 2018-04-12 16:12:40,508 [QuartzScheduler_Worker-1] (ErrorLogger:schedulerError:2360) Job (d1-cn-index-processor.d1-index-processor-job threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ce719d9 rejected from java.util.concurrent.ThreadPoolExecutor@7e212803[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 222]]
    at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ce719d9 rejected from java.util.concurrent.ThreadPoolExecutor@7e212803[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 222]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at org.dataone.cn.index.processor.IndexTaskProcessor.processTaskOnThread(IndexTaskProcessor.java:267)
    at org.dataone.cn.index.processor.IndexTaskProcessor.processIndexTaskQueue(IndexTaskProcessor.java:186)
    at org.dataone.cn.index.processor.IndexTaskProcessorJob.execute(IndexTaskProcessorJob.java:57)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
    ... 1 more
[ WARN] 2018-04-12 16:12:42,209 [main] (IndexTaskProcessorScheduler:stop:128) Job scheduler [org.dataone.cn.index.processor.IndexTaskProcessorScheduler@52538a70] finished executing all jobs. The d1-index-processor shut down sucessfully.============================================