Project

General

Profile

Task #3141

Increase production CN Metacat replication timeout

Added by Ben Leinfelder over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Ben Leinfelder
Category:
Support Operations
Start date:
2012-08-16
Due date:
% Done:

100%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

There are ~14k extra documents on UCSB that have yet to be replicated to UNM and ORC (see related bugs for more analysis).

When manually triggering Metacat-Metacat replication, I see that retrieving the list of documents to replicate (from UCSB to ORC and UNM) times out. It is currently set to do this in 60 seconds.

Using curl to retrieve this same list, shows that it takes ~1:02 minutes! So we just miss the window. I'd like to increase this timeout to 5 minutes which will require editing the metacat.properties file and restarting the Tomcat instances. The proposed procedure follows:
0. Stop daemons on all nodes by executing /usr/local/bin/stopDaemons.sh
1. Set DNS RR to UCSB-only
2. Edit metacat.properties file on ORC and UNM, setting the timeout to 5 minutes:

replication.client.timeout=300000

3. Restart Tomcat on ORC and UNM
4. restart indexing daemons on ORC and UNM by executing /usr/local/bin/startDaemons.sh
5. Set DNS RR to point to ORC and UNM
6. Edit metacat.properties file on UCSB
7. Restart Tomcat on UCSB
8. restart indexing daemons & d1-processing on UCSB by executing /usr/local/bin/startDaemons.sh


Related issues

Related to Infrastructure - Task #3146: Stream Metacat replication update list Closed 2012-08-17
Related to Infrastructure - Story #3136: Inconsistent data in production 1.0.3 release Closed 2012-08-15

History

#1 Updated by Ben Leinfelder over 8 years ago

I've changed this in the cn-buildout, so it will be included for subsequent releases of the dataone-cn-metacat package.

#2 Updated by Ben Leinfelder over 8 years ago

  • Status changed from New to In Progress
  • Assignee set to Ben Leinfelder

We have made the change on UNM and ORC. Content is still replicating from UCSB to those servers. When that completes, we will add UNM and ORC to the RR.

#3 Updated by Ben Leinfelder over 8 years ago

  • Parent task deleted (#3136)

#4 Updated by Ben Leinfelder over 8 years ago

  • Status changed from In Progress to Closed

I believe UCSB has been restarted at this point (though it's a moot point with the pending 1.0.4 updates)

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)