Synchronization of mnTestSEAD objects results in "NULL" exception on cn-stage.test.dataone.org in cn-syncrhonization.log. The "NULL" exception limits diagnosing the issue.

Observation of mnTestSEAD MN as of 6 April:

Science metadata is FGDC-STD-001-1998 format¶

There are no resource map or data objects¶

There are no relevant log messages in either /var/lib/tomcat7/logs or /var/metacat/logs¶

See attached file for a list of the mnTestSEAD object list and relevant section of the cn-synchronization.log file from 6 April.

Additional information from Skye on 17 April:

One SEAD error found today, regarding their newest content:

[DEBUG] 2015-04-16 22:51:46,010 (TransferObjectTask:processSystemMetadata:402) Task-urn:node:mnTestSEAD-seadva-MyersJames41411794-ca00-4bef-9d23-bf99441cfa1d Added CN as replica because formatType METADATA is sciMetadata

[DEBUG] 2015-04-16 22:51:46,297 (TransferObjectTask:createObject:651) Task-urn:node:mnTestSEAD-seadva-MyersJames41411794-ca00-4bef-9d23-bf99441cfa1d getting ScienceMetadata

[ INFO] 2015-04-16 22:51:46,479 (TransferObjectTask:createObject:705) Task-urn:node:mnTestSEAD-seadva-MyersJames41411794-ca00-4bef-9d23-bf99441cfa1d Creating Object

[ERROR] 2015-04-16 22:51:52,291 (TransferObjectTask:write:585) Task-urn:node:mnTestSEAD-seadva-MyersJames41411794-ca00-4bef-9d23-bf99441cfa1d
<?xml version="1.0" encoding="UTF-8"?>

Error inserting or updating document: <?xml version="1.0"?><error>cvc-complex-type.2.4.b: The content of element 'timeinfo' is not complete. One of '{sngdate, mdattim, rngdates}' is expected.</error>

[DEBUG] 2015-04-16 22:51:52,461 (TransferObjectTask:call:233) Task-urn:node:mnTestSEAD-seadva-MyersJames41411794-ca00-4bef-9d23-bf99441cfa1d Unlocked task

Looks like an FGDC schema validation failure for this particular object.

However I also looked at some other objects that Mark noted and see the same error Mark saw - for example:

cn-synchronization.log.1:[ERROR] 2015-04-07 00:30:51,911 (TransferObjectTask:write:606) Task-urn:node:mnTestSEAD-seadva-PowerMaryE.d6664a9f-322a-4551-980b-62b44c222162
cn-synchronization.log.1-null

The ’null' line corresponds to a logging message where the exception should be printed out, but we are just seeing ‘null’ in the log.

I think Robert may need to take another look at these sync failures. When I first started playing with the test SEAD yesterday I was getting a bunch of timeouts on REST endpoints - then at some point they started working. Im wondering if some of the sync failures may be related to timeouts? Although I think Mark has tried harvesting from this test SEAD node a few times?