Project

General

Profile

Bug #8797

Story #8796: Various issues with service access after upgrade to 18.04

Resolve is reporting 500 error instead of 404 for missing objects

Added by Dave Vieglais over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
2019-05-21
Due date:
% Done:

100%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

On 18.04 the following request fails with 500 error:

curl "https://cn-stage-unm-1.test.dataone.org/cn/v2/resolve/some%2F%2Fjunk"
<?xml version="1.0" encoding="UTF-8"?>
<error detailCode="500" errorCode="500" name="ServiceFailure">
    <description>Internal Server Error: The server encountered an unexpected condition which prevented it from fulfilling the request.</description>
</error>

Whereas a 404 error is expected and still returned by cn-stage-unm-2 which is currently on Ubuntu 14.04.

Note that the double slash is important. With a single slash, the correct 404 error is returned


Subtasks

Task #8805: Adjust context configuration for CN Metacat installsClosedJing Tao

History

#1 Updated by Jing Tao over 5 years ago

In the tomcat log file (thanks rob for finding it - /var/log/tomcat8/localhost.2019-05-15.log), we found:

15-May-2019 18:45:37.268 WARNING [ajp-nio-8009-exec-22] 
org.apache.catalina.core.ApplicationContext.getRequestDispatcher An application attempted to obtain 
a request dispatcher with an illegal path [/d1/cn/v2/meta/some%2F%2Fjunk] that was rejected because 
it contained an encoded directory traversal attempt

Here is the information from the tomcat web site:

Fixed in Apache Tomcat 8.5.16
Important: Security Constraint Bypass CVE-2017-7675

The HTTP/2 implementation bypassed a number of security checks that prevented directory traversal attacks. It was therefore possible to bypass security constraints using an specially crafted URL.

This was fixed in revision 1796091.

Our tomcat version is 8.5.39.0

#2 Updated by Dave Vieglais over 5 years ago

Verified that the issue is after apache. Accessing tomcat directly gives the same response. e.g.:

curl "http://localhost:8080/cn/v2/resolve/some%2F%2Fjunk"

gives a 500 error, whereas:

curl -v "http://localhost:8080/cn/v2/resolve/some//junk"

gives the expected 404 response.

#3 Updated by Rob Nahf over 5 years ago

it's interesting to note that NOT URL encoding the identifier (some//junk) works.
https://cn-stage-ucsb-1.test.dataone.org/cn/v2/resolve/some//junk

The issue arises from a security fix in tomcat8.5.16, whereby encoded slashes are interpretted as a directory traversal attack.

The solution may have to be url decoding the pid in either ResolveForwarder or ResolveFilter, although in principle it seems like it could lead to issues, too. For example, consider the Identifier some%2F%2Fjunk which is distinct from some//junk. The former would have to be escaped to avoid service failure, while the latter would have to NOT be escaped to avoid serviceFailure...

#4 Updated by Dave Vieglais over 5 years ago

Given the PID: some%2F%2Fjunk

It is URL encoded as: some%252F%252Fjunk

The request: curl "http://localhost:8080/cn/v2/resolve/some%252F%252Fjunk" actually works as expected:

<?xml version="1.0" encoding="UTF-8"?>
<error detailCode="4140" errorCode="404" name="NotFound">
<description>getSystemMetadata failed: NotFound - 1420: No system metadata could be found for given PID: some%2F%2Fjunk</description>
</error>

#5 Updated by Jing Tao over 5 years ago

  • Assignee changed from Jing Tao to Rob Nahf

#6 Updated by Dave Vieglais over 5 years ago

Edited /var/lib/tomcat8/webapps/cn/WEB-INF/config/log4j.properties, adding:

log4j.logger.org.dataone.cn=INFO

When calling resolve, results in:

[ INFO] 2019-05-16 00:25:42,148 (ResolveForwarder:forward:79) proxy resolve: /metacat/d1/cn/v2/meta/some%2F%2Fjunk
[ INFO] 2019-05-16 00:25:42,148 (AbstractProxyForwarder:forwardRequest:103) CN Getting proxy...
[ INFO] 2019-05-16 00:25:42,148 (AbstractProxyForwarder:forwardRequest:106) CN Dispatching: /d1/cn/v2/meta/some%2F%2Fjunk
[ INFO] 2019-05-16 00:25:42,168 (ExceptionController:get:67) received error of 500

A successful call is like:

[ INFO] 2019-05-16 00:26:29,819 (ResolveForwarder:forward:79) proxy resolve: /metacat/d1/cn/v2/meta/some//junk
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:103) CN Getting proxy...
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:106) CN Dispatching: /d1/cn/v2/meta/some//junk
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:111) CN Dispatching proxy...
[ INFO] 2019-05-16 00:26:29,860 (AbstractProxyForwarder:forwardRequest:113) CN Completed proxy...

#7 Updated by Dave Vieglais over 5 years ago

This means the failure is at line 107 of AbstractProxyForwarder.java in d1_cn_rest:

    protected void forwardRequest(ServletContext servletContext, String contextName, ProxyServletRequestWrapper proxyServletWrapper, HttpServletResponse response) throws ServletException, IOException, NotFound, ServiceFailure {
        ServletContext proxyServletContext = servletContext.getContext(contextName);
        if (proxyServletContext != null) {
            logger.info("CN Getting proxy...");

            try {
                logger.info("CN Dispatching: " + proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo());
                RequestDispatcher dispatch =
                        proxyServletContext.getRequestDispatcher(proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo());

                if (dispatch != null) {
                    logger.info("CN Dispatching proxy...");
                    dispatch.forward(proxyServletWrapper, response);
                    logger.info("CN Completed proxy...");

                } else {
                    throw new NotFound("n/a", "Servlet path " + proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo() + " of contextName " + contextName + " not found! "
                            + servletContext.getServerInfo());
                }

#9 Updated by Dave Vieglais over 5 years ago

Per Rob's suggestion:

I think we might also create a custom applicationContext subclass.

https://stackoverflow.com/questions/25612730/tomcat-7-0-how-to-launch-with-a-custom-applicationcontext
Stack Overflow
Tomcat 7.0+ How to launch with a custom ApplicationContext
I'm trying to figure out a way to ask Tomcat(nicely) to launch the server with my custom ApplicationContext for example: public class MyApplicationContext extends ApplicationContext ...

#10 Updated by Rob Nahf over 5 years ago

I found a context parameter (dispatchersUseEncodedPaths) that lets us bypass the directory traversal check. The solution was to add a second line to webapps/cn/META-INF/context.xml for the /metacat path. It doesn't seem to be used, however, but I'm not sure why.

<?xml version="1.0" encoding="UTF-8"?>
<Context antiJARLocking="true" crossContext="true" path="/cn"/>
<Context antiJARLocking="true" crossContext="true" dispatchersUseEncodedPaths="false" path="/metacat"/>

I've tried different paths as well, '/d1/cn/v1', '/d1/cn/v2', '/d1/cn', but they don't work either.

#11 Updated by Rob Nahf over 5 years ago

I copy-pasted the RequestUtils.normalize code to test it, and was able to determine that it leaves the encoded paths intact. If we want to use the existing ApplicationContext, we need to know how to apply the 'dispatchersUseEncodedPaths'.

#12 Updated by Dave Vieglais over 5 years ago

tried path=

/metacat
/cn
/d1/cn
/d1/cn/v1
/d1/cn/v2

None seem to have any effect.

#13 Updated by Dave Vieglais over 5 years ago

  • % Done changed from 0 to 30
  • Status changed from New to In Progress

Reading through the code, documentation for Tomcat contexts plus a couple articles results in:

  1. the call proxyServletContext.getRequestDispatcher() is being made in the metacat servlet context

  2. Tomcat reads [engine-name]/[server-name]/[app-name].xml after the war is deployed.

  3. There can only be one path context.

The result is /etc/tomcat8/Catalina/localhost/metacat.xml:

<Context path="/metacat/" antiResourceLocking="false" unpackWAR="false" crossContext="true" dispatchersUseEncodedPaths="false"/>

After editing and restarting Tomcat8, the resolve service works as expected.

#14 Updated by Dave Vieglais over 5 years ago

  • % Done changed from 30 to 100
  • Status changed from In Progress to Closed

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)