Increase production CN Metacat replication timeout
There are ~14k extra documents on UCSB that have yet to be replicated to UNM and ORC (see related bugs for more analysis).
When manually triggering Metacat-Metacat replication, I see that retrieving the list of documents to replicate (from UCSB to ORC and UNM) times out. It is currently set to do this in 60 seconds.
Using curl to retrieve this same list, shows that it takes ~1:02 minutes! So we just miss the window. I'd like to increase this timeout to 5 minutes which will require editing the metacat.properties file and restarting the Tomcat instances. The proposed procedure follows:
0. Stop daemons on all nodes by executing /usr/local/bin/stopDaemons.sh
1. Set DNS RR to UCSB-only
2. Edit metacat.properties file on ORC and UNM, setting the timeout to 5 minutes:
3. Restart Tomcat on ORC and UNM
4. restart indexing daemons on ORC and UNM by executing /usr/local/bin/startDaemons.sh
5. Set DNS RR to point to ORC and UNM
6. Edit metacat.properties file on UCSB
7. Restart Tomcat on UCSB
8. restart indexing daemons & d1-processing on UCSB by executing /usr/local/bin/startDaemons.sh