Project

General

Profile

Task #6530

Story #6528: Upgrade MNs operated by DataONE to Ubuntu 14.04

Upgrade the OS of the production member nodes operated by UCSB

Added by Jing Tao over 7 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
2014-10-10
Due date:
% Done:

100%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

Nick and Jing will work on that:

knbvm.nceas.ucsb.edu
mn-ucsb-1.dataone.org
mn-ucsb-2.dataone.org
mn-ucsb-3.dataone.org

History

#1 Updated by Jing Tao over 7 years ago

An email from Matt:

There are 4 production hosts in this list and 1 development machine that need to be handled separately from the others:

knbvm.nceas.ucsb.edu
mn-ucsb-1.dataone.org
mn-ucsb-2.dataone.org
mn-ucsb-3.dataone.org
dev.nceas.ucsb.edu

For all of the other dataone development machines in your list:
1) Jing will do a backup on each host before you do the upgrade
a) ensure that we don't lose our client certificates
2) do an in-place, offline upgrade so we don't lose our existing services as configured, downtime is ok, but should be announced to the dataone developers so they know when each host is being upgraded. Jing can confirm that all services are working after the upgrade, and if not, get them fixed. Best to try this on one host first, and if it all works well, then proceed with the others.

For the KNB and dev.nceas.ucsb.edu:
1) Minimize downtime to less than 15 minutes by:
a) setting up duplicate VMs with postgres replication and rsync, where the new VMs have upgraded OS's
b) prepare a script that turns off Tomcat on the live machine, then confirms a final postgres/file sync, then switches to the new host as the live VM

Jing will be developing a plan for this, and we plan to first do it on dev.nceas.ucsb.edu, and once that works well, then on the KNB. So both of these will need new virtual machines configured. I'll call you to talk about where those should go.

For the other production machines, longer downtime is less of an issue, so please just coordinate with Jing to upgrade these like we are the dev machines:
1) backup, including keys
2) shutdown tomcat and in-place upgrade

#2 Updated by Jing Tao over 7 years ago

  • Target version set to Maintenance Backlog

#3 Updated by Dave Vieglais about 7 years ago

  • Target version changed from Maintenance Backlog to CCI-1.5.1

#4 Updated by Jing Tao almost 7 years ago

The knbvm.nceas.ucsb.edu is done.

#5 Updated by Jing Tao almost 7 years ago

The mn-ucsb-2.dataone.org (hosting ESA and GOA) has been done.

It took much longer than i anticipated. The reason is that Tomcat couldn't be up after the upgrading. It turned out the service of registering schema was stuck. In the xml_catalog table, there some records like:
12 | Schema | | | eml://ecoinformatics.org/eml-2.0.1 | http://data.esa.org:80/esa/schema/eml-2.0.1/eml.xsd
Metacat tried to download the schema from data.esa.org/esa/schema/eml-2.0.1/eml.xsd. However the tomcat was not up so the url was not accessible. The service stopped there in a dead loop.

After i changed the system_id to /schema/eml-2.0.1/eml.xsd to indicate is a local file, it works.

#6 Updated by Jing Tao almost 7 years ago

The mn-ucsb-1.dataone.org was done.

#7 Updated by Rob Nahf almost 7 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

#8 Updated by Jing Tao almost 7 years ago

Only knb is left.

#10 Updated by Jing Tao almost 7 years ago

  • translation missing: en.field_remaining_hours set to 0.0
  • % Done changed from 30 to 100
  • Status changed from In Progress to Closed

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)