Infrastructure

An email from Matt:

There are 4 production hosts in this list and 1 development machine that need to be handled separately from the others:

knbvm.nceas.ucsb.edu
mn-ucsb-1.dataone.org
mn-ucsb-2.dataone.org
mn-ucsb-3.dataone.org
dev.nceas.ucsb.edu

For all of the other dataone development machines in your list:
1) Jing will do a backup on each host before you do the upgrade
a) ensure that we don't lose our client certificates
2) do an in-place, offline upgrade so we don't lose our existing services as configured, downtime is ok, but should be announced to the dataone developers so they know when each host is being upgraded. Jing can confirm that all services are working after the upgrade, and if not, get them fixed. Best to try this on one host first, and if it all works well, then proceed with the others.

For the KNB and dev.nceas.ucsb.edu:
1) Minimize downtime to less than 15 minutes by:
a) setting up duplicate VMs with postgres replication and rsync, where the new VMs have upgraded OS's
b) prepare a script that turns off Tomcat on the live machine, then confirms a final postgres/file sync, then switches to the new host as the live VM

Jing will be developing a plan for this, and we plan to first do it on dev.nceas.ucsb.edu, and once that works well, then on the KNB. So both of these will need new virtual machines configured. I'll call you to talk about where those should go.

For the other production machines, longer downtime is less of an issue, so please just coordinate with Jing to upgrade these like we are the dev machines:
1) backup, including keys
2) shutdown tomcat and in-place upgrade

The knbvm.nceas.ucsb.edu is done.

The mn-ucsb-2.dataone.org (hosting ESA and GOA) has been done.

It took much longer than i anticipated. The reason is that Tomcat couldn't be up after the upgrading. It turned out the service of registering schema was stuck. In the xml_catalog table, there some records like:
12 | Schema | | | eml://ecoinformatics.org/eml-2.0.1 | http://data.esa.org:80/esa/schema/eml-2.0.1/eml.xsd
Metacat tried to download the schema from data.esa.org/esa/schema/eml-2.0.1/eml.xsd. However the tomcat was not up so the url was not accessible. The service stopped there in a dead loop.

After i changed the system_id to /schema/eml-2.0.1/eml.xsd to indicate is a local file, it works.

The mn-ucsb-1.dataone.org was done.

Only knb is left.

KNB was done.

Please see: https://epad.dataone.org/pad/p/KNB_upgrade_2015