Infrastructure

In the tomcat log file (thanks rob for finding it - /var/log/tomcat8/localhost.2019-05-15.log), we found:

15-May-2019 18:45:37.268 WARNING [ajp-nio-8009-exec-22] 
org.apache.catalina.core.ApplicationContext.getRequestDispatcher An application attempted to obtain 
a request dispatcher with an illegal path [/d1/cn/v2/meta/some%2F%2Fjunk] that was rejected because 
it contained an encoded directory traversal attempt

Here is the information from the tomcat web site:

Fixed in Apache Tomcat 8.5.16
Important: Security Constraint Bypass CVE-2017-7675

The HTTP/2 implementation bypassed a number of security checks that prevented directory traversal attacks. It was therefore possible to bypass security constraints using an specially crafted URL.

This was fixed in revision 1796091.

Our tomcat version is 8.5.39.0

Verified that the issue is after apache. Accessing tomcat directly gives the same response. e.g.:

curl "http://localhost:8080/cn/v2/resolve/some%2F%2Fjunk"

gives a 500 error, whereas:

curl -v "http://localhost:8080/cn/v2/resolve/some//junk"

gives the expected 404 response.

it's interesting to note that NOT URL encoding the identifier (some//junk) works.
https://cn-stage-ucsb-1.test.dataone.org/cn/v2/resolve/some//junk

The issue arises from a security fix in tomcat8.5.16, whereby encoded slashes are interpretted as a directory traversal attack.

The solution may have to be url decoding the pid in either ResolveForwarder or ResolveFilter, although in principle it seems like it could lead to issues, too. For example, consider the Identifier some%2F%2Fjunk which is distinct from some//junk. The former would have to be escaped to avoid service failure, while the latter would have to NOT be escaped to avoid serviceFailure...

Given the PID: some%2F%2Fjunk

It is URL encoded as: some%252F%252Fjunk

The request: curl "http://localhost:8080/cn/v2/resolve/some%252F%252Fjunk" actually works as expected:

<?xml version="1.0" encoding="UTF-8"?>
<error detailCode="4140" errorCode="404" name="NotFound">
<description>getSystemMetadata failed: NotFound - 1420: No system metadata could be found for given PID: some%2F%2Fjunk</description>
</error>


Edited /var/lib/tomcat8/webapps/cn/WEB-INF/config/log4j.properties, adding:

log4j.logger.org.dataone.cn=INFO


When calling resolve, results in:

[ INFO] 2019-05-16 00:25:42,148 (ResolveForwarder:forward:79) proxy resolve: /metacat/d1/cn/v2/meta/some%2F%2Fjunk
[ INFO] 2019-05-16 00:25:42,148 (AbstractProxyForwarder:forwardRequest:103) CN Getting proxy...
[ INFO] 2019-05-16 00:25:42,148 (AbstractProxyForwarder:forwardRequest:106) CN Dispatching: /d1/cn/v2/meta/some%2F%2Fjunk
[ INFO] 2019-05-16 00:25:42,168 (ExceptionController:get:67) received error of 500


A successful call is like:

[ INFO] 2019-05-16 00:26:29,819 (ResolveForwarder:forward:79) proxy resolve: /metacat/d1/cn/v2/meta/some//junk
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:103) CN Getting proxy...
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:106) CN Dispatching: /d1/cn/v2/meta/some//junk
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:111) CN Dispatching proxy...
[ INFO] 2019-05-16 00:26:29,860 (AbstractProxyForwarder:forwardRequest:113) CN Completed proxy...


This means the failure is at line 107 of AbstractProxyForwarder.java in d1_cn_rest:

    protected void forwardRequest(ServletContext servletContext, String contextName, ProxyServletRequestWrapper proxyServletWrapper, HttpServletResponse response) throws ServletException, IOException, NotFound, ServiceFailure {
        ServletContext proxyServletContext = servletContext.getContext(contextName);
        if (proxyServletContext != null) {
            logger.info("CN Getting proxy...");

            try {
                logger.info("CN Dispatching: " + proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo());
                RequestDispatcher dispatch =
                        proxyServletContext.getRequestDispatcher(proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo());

                if (dispatch != null) {
                    logger.info("CN Dispatching proxy...");
                    dispatch.forward(proxyServletWrapper, response);
                    logger.info("CN Completed proxy...");

                } else {
                    throw new NotFound("n/a", "Servlet path " + proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo() + " of contextName " + contextName + " not found! "
                            + servletContext.getServerInfo());
                }

The tomcat implementation of ServletContext seems to be ApplicationContext.java

With the problem apparently at:
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/core/ApplicationContext.java#L436-L443

and:
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/tomcat/util/http/RequestUtil.java#L76

Per Rob's suggestion:

I think we might also create a custom applicationContext subclass.

https://stackoverflow.com/questions/25612730/tomcat-7-0-how-to-launch-with-a-custom-applicationcontext
Stack Overflow
Tomcat 7.0+ How to launch with a custom ApplicationContext
I'm trying to figure out a way to ask Tomcat(nicely) to launch the server with my custom ApplicationContext for example: public class MyApplicationContext extends ApplicationContext ...

I found a context parameter (dispatchersUseEncodedPaths) that lets us bypass the directory traversal check. The solution was to add a second line to webapps/cn/META-INF/context.xml for the /metacat path. It doesn't seem to be used, however, but I'm not sure why.

<?xml version="1.0" encoding="UTF-8"?>
<Context antiJARLocking="true" crossContext="true" path="/cn"/>
<Context antiJARLocking="true" crossContext="true" dispatchersUseEncodedPaths="false" path="/metacat"/>

I've tried different paths as well, '/d1/cn/v1', '/d1/cn/v2', '/d1/cn', but they don't work either.

I copy-pasted the RequestUtils.normalize code to test it, and was able to determine that it leaves the encoded paths intact. If we want to use the existing ApplicationContext, we need to know how to apply the 'dispatchersUseEncodedPaths'.

tried path=

/metacat
/cn
/d1/cn
/d1/cn/v1
/d1/cn/v2

None seem to have any effect.