Task #7848: SEAD in Stage - slow indexing - Member Nodes - DataONE Tasks

Task #7848

MNDeployment #3521: SEAD Member Node

SEAD in Stage - slow indexing

Added by Laura Moyers over 8 years ago. Updated about 8 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Dave Vieglais

Target version:

Operational

Start date:

2016-07-18

Due date:

% Done:

100%

Story Points:

Sprint:

Description

On 7/13/16, Chris, Dave and others discovered a configuration problem in stage where the CNs at UNM and ORC didn't recognize the CN at UCSB:

cn-stage-ucsb-1 had a server.name property in metacat set to ‘cn-stage.test.dataone.org’ instead of cn-stage-ucsb-1.test.dataone.org

Chris sorted this out, and indexing of SEAD content began.

The current issue is that on Thursday, 7/14/16, only 61 of 208 possible objects had indexed. On Friday, 7/15/16, 61 of 209 possible objects had indexed. On Monday, 7/18/16, 97 objects had indexed of 210 possible objects.

Can someone please investigate why this process is going so slowly? This doesn't seem normal.

History

#1 Updated by Dave Vieglais over 8 years ago

Status changed from New to In Progress
% Done changed from 0 to 30

Indexing appears to be slow on stage because there are a large number of documents that are not present on the ORC node (which is where indexing is being run). There are 59271 docs in /var/metacat/data on UCSB, but 58786 on ORC

On ORC, cn-index-processor-load.log shows a lot of failed entries:
2016-07-21 13:40:20 new:0, failed: 12571

Switching off indexing on ORC and starting up on UCSB.

On UCSB, cn-index-processor-load.log shows:

2016-07-21 13:43:30 new:43, failed: 2402

Conclusion:
1. There is a lot of content that has not been replicated by metacat
2. There are a lot of index failed errors
3. The slow indexing on ORC is likely a result of the very high number of failed index operations (which get added back onto the queue)

Recommendation:
Recommend a complete purge and re-harvest of the stage environment, adding back one node at a time and verifying sync, replication and indexing tasks proceed as expected.