Story #7586
LogAggregation fails for MN in production with handshake alert
100%
Description
For 5 MNs (urn:node:NKN, urn:node:NMEPSCOR, urn:node:IOE, urn:node:USANPN, urn:node:TFRI) log aggregation is failing with the following message:
org.dataone.service.exceptions.ServiceFailure: class org.dataone.client.exception.ClientSideException: /Remote host closed connection during handshake
The cause is due to Java support of SNI and a misconfigured apache web server that supports SNI. With Java 7, a java client will send a request for the hostname that it expects a response from. If, on the apache web server, the ServerName or ServerAlias is not set properly in the virtual host directive to respond with the correct hostname requested, then apache will respond with TLS Alert (Level: Warning, Description: Unrecognized name). The TLS Alert causes the java client to abort. The alert number is 112.
To confirm that each of the 5 failing MNs produced the TLS Alert from their server, I ran a java program that would execute with the d1_libclient_java a request for listObjects. The network traffic was monitored with tcpdump and then analyzed with wireshark. Each of the MNs responded with the TLS Alert of Unrecognized name.
I was able to test a solution on sandbox by modifying the virtual host file in apache of the MN mn-sandbox-orc-1.test.dataone.org. in /etc/apache2/sites-available/gmn-ssl. I added the directive- ServerName wrong.hostname.org - under the VirtualHost directive. Restarting apache and then using the test program, I was able to recreate the Unrecognized name error. I then added the directive- ServerAlias * - directly beneath the ServerName directive. After restarting apache, the test program was successful.
urn:node:NKN, urn:node:USANPN and urn:node:IOE all have alternative hostnames that respond correctly when running the test program:
For urn:node:NKN, override the base url of https://dataone.northwestknowledge.net/mn with https://dataone.nkn.uidaho.edu/mn .
For urn:node:USANPN, override the base url of https://mynpn.usanpn.org/knb/d1/mn with https://mynpn.npn.arizona.edu/knb/d1/mn .
For urn:node:IOE, override the base url of https://data.rcg.montana.edu/catalog/d1/mn with https://metacat2.rcg.montana.edu/catalog/d1/mn .
Because these MemberNodes may respond to two different hostname requests, I propose that we ask the MemberNode operators to include in the virtual host file of their MN site the directive, ServerAlias * .
https://bz.apache.org/bugzilla/show_bug.cgi?id=56241
https://community.qualys.com/thread/13532
http://stackoverflow.com/questions/7615645/ssl-handshake-alert-unrecognized-name-error-since-upgrade-to-java-1-7-0
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=7127374
Subtasks
Related issues
History
#1 Updated by Robert Waltz over 8 years ago
- Related to Task #6787: Ensure Java 7 Client code can communicate over SSL added
#2 Updated by Robert Waltz over 8 years ago
Apache Server 2.4.10 and 2.2.30 reportedly fix the behaviour so that the TLS Alert is not sent.
#3 Updated by Robert Waltz over 8 years ago
- Description updated (diff)
#4 Updated by Rob Nahf over 8 years ago
- Related to Bug #7504: Investigate support for TLS Server Name Indication during CN-to-MN communications added
#5 Updated by Robert Waltz over 8 years ago
- Tracker changed from Bug to Story
#6 Updated by Robert Waltz over 8 years ago
Here is the generic text for the email that will be customized and sent to each individual Member Node.
DataONE has found a configuration issue with your Apache webserver that prohibits harvesting of log records and synchronization of objects. The configuration issue affects how DataONE services interact with your Member Node. While establishing a TLS connection to your Member Node, A DataONE service produces the following error:
org.dataone.service.exceptions.ServiceFailure: class org.dataone.client.exception.ClientSideException: /Remote host closed connection during handshake
The cause is due to Java support of SNI and a misconfigured apache web server that supports SNI. With Java 7, a java client will send a request with a specific hostname that it expects a response from. If, on the apache web server, the ServerName or ServerAlias is not set properly in the virtual host directive to respond with the correct hostname requested, then apache will respond with TLS Alert (Level: Warning, Description: Unrecognized name). The TLS Alert causes the java client to abort. The alert number is 112.
To confirm that your webserver was sending the TLS Alert, DataONE tested the connection to your webserver by running a java program that would execute with the d1_libclient_java a request for listObjects. The network traffic was monitored with tcpdump and then analyzed with wireshark. DataONE did find that your Member Node webserver responded with the TLS Alert of Unrecognized name after analysis of the network traffic.
Apache Web Server 2.4.10 and 2.2.30 reportedly fix the behaviour so that the TLS Alert is no longer sent as a response. However, we do not recommend upgrading at this time. A simpler approach will be to either edit or add a ServerName directive (https://httpd.apache.org/docs/2.2/mod/core.html#servername) or add a ServerAlias (https://httpd.apache.org/docs/2.2/mod/core.html#serveralias) directive to the virtual host file that serves requests for your Member Node.
If you are using a debian distribution of linux, your Member Node virtual host file should be in the directory- /etc/apache2/sites-available. You would add the ServerName or ServerAlias directive under the VirtualHost directive.
If your webserver only is configured to run as a Member Node and servers no other websites, then adding in 'ServerAlias *' will allow the server to respond appropriately to any hostname that is associated with your Member Node.
#7 Updated by Robert Waltz about 8 years ago
- % Done changed from 0 to 100
- Status changed from New to Closed
