Infrastructure

Laura pointed out a number of replication request failures. I've looked at some of the logs, and all requests to @mn-orc-1@ are failing. After looking at the Metacat installation, while the PostgreSQL database is fine, the filesystem is not. There should be thousands of objects stored in @/var/metacat@, but the subdirectories are gone. Either they were deleted, corrupted, or perhaps the filesystem didn't mount correctly - it's hard to say. Nevertheless, Metacat thinks that the three sub directories of @/var/metacat/{documents, data, temporary}@ should be available, and they're not. Requests for replication are attempting to spool the MMP object for the system metadata to the temporary directory, and can't:

[ERROR] 2017-05-09 02:19:32,247 (MNCommunication:requestReplication:45) Caught base exception attempting to call replicate for pid:
resource_map_urn:uuid:dc4b2a33-a83d-4232-85c8-9dbb5e7c2516 at target: urn:node:mnORC1with exception:
Could not resolve multipart files: Processing of multipart/form-data request failed.
/var/metacat/temporary/upload_3961ca52_15bcf40708e8000_00000783.tmp (No such file or directory) and message:
Could not resolve multipart files: Processing of multipart/form-data request failed. /var/metacat/temporary/upload3961ca52_15bcf40708e_8000_00000783.tmp (No such file or directory)

It looks to me like the @/var/metacat@ directory needs to be restored in order to get this Metacat installation back up and running.

Since mn-orc-1 had an issue without enough spaces, Dave created new partition and the metadata data directory was moved from /var/metacat to /var/data/metacat. I remember the metacat.properties was configured correctly. However, when I looked the file, it isn't. This is weird. I will correct it during the upgrade.

Jing -- the config probably ended back at defaults during an upgrade. While metacat supports reconfiguring the location of /var/metacat, I highly recommend that you don't, and instead provide a symbolic link from /var/metacat to /var/data/metacat to prevent this problem in the future.

Matt -- Sounds good. I will use a symbolic link from /var/metacat to /var/data/metacat.