Infrastructure

Using Safari Version 5.1.3 (7534.53.10) I can just hit "Cancel" when it asks me which client certificate I want to use when connecting and then just lets me through to the onemercury UI as expected. I have a Self-signed certificate as well as an old CILogon certificate in my keychain.
When I remove the CILogon certificate, I get the behavior Dave described.
Odd!

I've run into the same problem on Mac OS X 10.9.2 (Mavericks) on Safari 7.0.2 while connecting to https://cn.dataone.org/onemercury, and having a single certificate installed in my Keychain (my Apple ID certificate for the App Store [com.apple.idms.appleid.prd...]).

Instead of using the 'Acceptable client certificate' list provided by the server, Safari looks to be sending the Apple ID certificate anyway, and the server rejects it (correctly). Safari's behavior should be to continue to the site unauthenticated as other browsers do.

We have hit this same Safari error on https://arcticdata.io, and found a workaround. In the workaround, we simply configure Apache to check the user agent, and if it is Safari, then to set @SSLVerifyClient none@. The consequences of this are that, for Safari, the only way to log into DataONE is with the new AuthTokens -- X.509 certificates from CILogon will be ignored. But, as nobody except developers uses CILogon certs in browser user agents, this works well for most end users, and avoids the Safari failures to connect. Other user agents should be unaffected, and CN-MN and MN-MN communication should be fine as well. Here's the apache config inside of our SSL virtual host directive used to set this up:

SSLVerifyClient none


SSLVerifyClient optional

With the Arctic Data Center deployment, we've found a side-effect of adding the conditional SSLVerifyClient code Matt mentioned above.

Basically, after the change, POSTing data to the repository with a request larger than around 128K was causing a buffer overflow during the Apache SSL renegotiation phase with the client. the error was:

[Tue Apr 19 14:11:18.334162 2016] [ssl:error] [pid 22897:tid 140098161288960] [client 128.111.54.100:37784] AH02018: request body exceeds maximum size (131072) for SSL buffer
[Tue Apr 19 14:11:18.334203 2016] [ssl:error] [pid 22897:tid 140098161288960] [client 128.111.54.100:37784] AH02257: could not buffer message body to allow SSL renegotiation to proceed

After reading http://stackoverflow.com/questions/14281628/ssl-renegotiation-with-client-certificate-causes-server-buffer-overflow we decided to change the client code to add a 'Expect: 100-continue' header to each upload request to let Apache know to not to download the whole request body during the renegotiation. This seems to be working, but it's something to consider for DataONE when making this SSL config change.

would configuring DataONE servers to trust Apple CAs solve the problem? (Assuming if you are using Safari, you will at least have one Apple-signed certificate?

https://www.apple.com/certificateauthority/

So, while the fixes described in #note-6 and #note-7 above helped, it also introduced an unwanted side effect for POSTing to the server. With that configuration, the entire body of the POST is forced to be buffered on the server during the SSL renegotiation phase. The default @SSLRenegBufferSize@ is 128K, so for uploads of files of any size above 128K (most!) will cause the renegotiation to fail since it can't buffer the file. We increased that size to 100MB, but obviously it doesn't scale - GB range files will fail, or if you keep increasing the buffer size, you'll just run out of RAM, particularly in concurrent connection scenarios.

We've since changed our configuration on the MNs. The trick is to default to @SSLVerifyClient none@, and then only ask for optional certificates when the user agent doesn't match @webkit@. On the MN, we used:

SSLVerifyClient none


SSLVerifyClient optional

This works great, doesn't require buffering the body, and allows cert-based auth for CN synchronization and MN-MN replication.

However, token-based auth is failing in Safari still because the CNs are still set to @SSLVerifyClient optional@ when the browser calls @/portal/oath@ on the CN (that then redirects to orcid.org).

I'd recommend we change the CN configurations to:

SSLVerifyClient none


SSLVerifyClient optional

In this setup, all web requests default to not requesting a certificate, but calls to the DataONE API endpoints under @/cn@ will be asked for an optional certificate when they don't match @webkit@, case insensitive. This allows curl, java, R, and python clients to use certificates when desired, along with CN sync and MN repl.

Due to issues with other clients, we've updated the configuration to not ask for client certs for any browser-based clients:

SSLVerifyClient none


SSLVerifyClient optional


SSLVerifyDepth 10