Task #4148: Troubleshoot getting system metadata and objects on the production USANPN - Member Nodes - DataONE Tasks

Task #4148

MNDeployment #3186: USA-NPN (USA National Phenology Network)

Troubleshoot getting system metadata and objects on the production USANPN

Added by Chris Jones about 11 years ago. Updated over 10 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Ben Leinfelder

Target version:

Operational

Start date:

2013-10-31

Due date:

% Done:

100%

Story Points:

Sprint:

Description

Lee has deployed the USANPN production Metacat Node, and Ben has started the synchronization process in the production environment. Before, and as of 10-30-2013, we are seeing failures in synchronizing content, with many issues revolving around the CN's inability to call MNRead.get() and MNRead.getSystemMetadata() on the USANPN member node. The following identifiers have failed:

10.5066/F7028PJS.0
10.5066/F73R0QWG.0
10.5066/F77H1GKR.0
10.5066/F7C82781.0
10.5066/F7H1300R.0
10.5066/F7H1300R.1
10.5066/F7MS3QRT.0
10.5066/F7MS3QRT.1
doi:10.5066/F7MS3QRT.0
resourceMap_10.5066/F7H1300R.0
resourceMap_10.5066/F7MS3QRT.0
resourceMap_10.5066/F7MS3QRT.1
resourceMap_doi:10.5066/F7MS3QRT.0

One thing to note is that calling get() or getSystemMetadata() in a browser works for the above identifiers, but not when they are URL-encoded:

https://mynpn.usanpn.org/knb/d1/mn/v1/object/10.5066/F7028PJS.0
vs.
https://mynpn.usanpn.org/knb/d1/mn/v1/object/10.5066%2FF7028PJS.0

Note that all of the identifiers in the above list contain a '/'. When the CN makes these calls, it correctly URL-encodes the identifiers.

My suggestion here is that we may have a misconfiguration on the USANPN Metacat installation that wasn't caught during stage testing perhaps because the identifiers in stage didn't have slashes in them, although I'm not sure about this.

We need to check that both Apache and Tomcat are configured correctly to handle slashes in the URLs. In particular, these settings should be confirmed:

Apache:
AllowEncodedSlashes On
AcceptPathInfo On

Tomcat:
org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
org.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true

See the Metacat installation instructions:
http://knb.ecoinformatics.org/knb/docs/dataone.html#apache-configuration-details
http://knb.ecoinformatics.org/knb/docs/dataone.html#configure-tomcat-to-allow-dataone-identifiers

History

#1 Updated by Ben Leinfelder about 11 years ago

I sent a similar msg to Lee regarding this. Apparently that did not resolve it?

Ben,

Hey buddy, I mucked with things last night, but to no avail. Looking at the logs on this side doesn't seem to bear any fruitful clues as to why the synchronization is not coming through. Do you have any clues on your side?

Lee

On Tue, Oct 29, 2013 at 12:50 PM, Lee Marsh npnlee85@gmail.com wrote:
Ben,

Yea, that was most likely the cause. To be sure, I'll have to look again tonight when I can restart the server, to see if it worked, but I strongly suspect that was it.

Thanks!

Lee

On Tue, Oct 29, 2013 at 12:39 PM, Benjamin Leinfelder leinfelder@nceas.ucsb.edu wrote:
Hi Lee,
I see some processing errors -- seem to do with identifier URL escaping/encoding.

Compare these results:
https://mynpn.usanpn.org/knb/d1/mn/v1/meta/resourceMap_10.5066%2FF7MS3QRT.0
https://mynpn.usanpn.org/knb/d1/mn/v1/meta/resourceMap_10.5066/F7MS3QRT.0

We are getting an error with the first URL. Can you double check these settings?
http://knb.ecoinformatics.org/knb/docs/dataone.html#configure-tomcat-to-allow-dataone-identifiers
If you have those set, then I'll dig deeper on our CN side.

Thanks,
-ben

#2 Updated by Chris Jones about 11 years ago

Status changed from New to In Progress

From Lee Marsh:

So, I don't think I can login to redmine, so I'll just give an update here. In any case, good news, I think. I've done the following so far:

1) Updated Apache configuration as follows:

    AllowEncodedSlashes On
    AcceptPathInfo On

Made no perceivable difference after a 'reload' command. I had installed these same changes Tuesday evening while I was working on this, but I was forced to roll back the changes. It was conflated with some other, unrelated issues that were causing the server to crash.

2) I noticed that while I had added:

org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true
org.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true

to catalina.properties, upon closer inspection, I see I had done it to the wrong version of this file! I've since made the same change to the applicable version of that file, which the server should actually be using. Unfortunately, I cannot right now restart the server to verify if the change made any difference, but I feel confident this should resolve the problem. We can see tomorrow, or maybe later this afternoon, if all active sessions timeout on the other app running on tomcat.