Task #3162
Enhance index task processing to limit re-tries.
100%
Description
Update index task processing to detect when a task has attempted sufficient retries and move the task to a failure status.
Consider how limited retries will effect processing ORE docs that are 'waiting' for data to appear. Probaby best to limit retries and allow auditing to handle it once a day or so.
Consider listener on objectPath to create index tasks for science metadata and resource maps? Indicates document is present on one CN file system - not necessarily the local. So limited help. Possible to listen to metacat replication complete events for local filesystem?
Create second quartz taks in the index-processor daemon that handles failed tasks daily or so.
redmine3162.txt 
(26.3 KB)
History
#1 Updated by Skye Roseboom over 11 years ago
- Milestone changed from CCI-1.0.4 to CCI-1.0.5
#2 Updated by Skye Roseboom over 11 years ago
- Milestone changed from CCI-1.0.5 to CCI-1.1
#3 Updated by Skye Roseboom over 11 years ago
- Milestone changed from CCI-1.1 to CCI-1.0.4
#4 Updated by Skye Roseboom over 11 years ago
Implementing a processing back off strategy for IndexTask processing. After 2 retries or 1 failure the index task will be marked FAILED and a 'nextExecution' timestamp is set onto the task. So no index task will be retried more than 2 times without failing.
Once in failure state, the task will not process again until the 'nextExecution' time has passed. The back-off strategy in IndexTask will move the execution time for first failure at 20 minutes, then 2 hours, then 8 hours, then 24 hours. After 3 attempts at the 24hour retry level, the task it moved to its final state - once a week retry.
#6 Updated by Skye Roseboom over 11 years ago
also need to move peggym.130.4-archived science metadata file for delete unit test.
#7 Updated by Skye Roseboom over 11 years ago
- Tracker changed from Story to Task
#8 Updated by Skye Roseboom over 11 years ago
- Status changed from In Progress to Testing
Moved to 1.0.0 branch for testing in RC of 1.0.4
#9 Updated by Skye Roseboom over 11 years ago
- Status changed from Testing to Closed
Tested on stage environment with 1.0.4 RC.
Ran index rebuild process and index-process daemon.
Index log less chatty about tasks that cannot process yet or at all as the retry back-off strategy takes effect. Failed index tasks are eventually (as determined by back-off strategy) retried as part of normal index processing.
