Bug #8797
Story #8796: Various issues with service access after upgrade to 18.04
Resolve is reporting 500 error instead of 404 for missing objects
100%
Description
On 18.04 the following request fails with 500 error:
curl "https://cn-stage-unm-1.test.dataone.org/cn/v2/resolve/some%2F%2Fjunk" <?xml version="1.0" encoding="UTF-8"?> <error detailCode="500" errorCode="500" name="ServiceFailure"> <description>Internal Server Error: The server encountered an unexpected condition which prevented it from fulfilling the request.</description> </error>
Whereas a 404 error is expected and still returned by cn-stage-unm-2 which is currently on Ubuntu 14.04.
Note that the double slash is important. With a single slash, the correct 404 error is returned
Subtasks
History
#1 Updated by Jing Tao over 5 years ago
In the tomcat log file (thanks rob for finding it - /var/log/tomcat8/localhost.2019-05-15.log), we found:
15-May-2019 18:45:37.268 WARNING [ajp-nio-8009-exec-22] org.apache.catalina.core.ApplicationContext.getRequestDispatcher An application attempted to obtain a request dispatcher with an illegal path [/d1/cn/v2/meta/some%2F%2Fjunk] that was rejected because it contained an encoded directory traversal attempt
Here is the information from the tomcat web site:
Fixed in Apache Tomcat 8.5.16 Important: Security Constraint Bypass CVE-2017-7675 The HTTP/2 implementation bypassed a number of security checks that prevented directory traversal attacks. It was therefore possible to bypass security constraints using an specially crafted URL. This was fixed in revision 1796091.
Our tomcat version is 8.5.39.0
#2 Updated by Dave Vieglais over 5 years ago
Verified that the issue is after apache. Accessing tomcat directly gives the same response. e.g.:
curl "http://localhost:8080/cn/v2/resolve/some%2F%2Fjunk"
gives a 500 error, whereas:
curl -v "http://localhost:8080/cn/v2/resolve/some//junk"
gives the expected 404 response.
#3 Updated by Rob Nahf over 5 years ago
it's interesting to note that NOT URL encoding the identifier (some//junk) works.
https://cn-stage-ucsb-1.test.dataone.org/cn/v2/resolve/some//junk
The issue arises from a security fix in tomcat8.5.16, whereby encoded slashes are interpretted as a directory traversal attack.
The solution may have to be url decoding the pid in either ResolveForwarder or ResolveFilter, although in principle it seems like it could lead to issues, too. For example, consider the Identifier some%2F%2Fjunk
which is distinct from some//junk
. The former would have to be escaped to avoid service failure, while the latter would have to NOT be escaped to avoid serviceFailure...
#4 Updated by Dave Vieglais over 5 years ago
Given the PID: some%2F%2Fjunk
It is URL encoded as: some%252F%252Fjunk
The request: curl "http://localhost:8080/cn/v2/resolve/some%252F%252Fjunk"
actually works as expected:
<?xml version="1.0" encoding="UTF-8"?>
<error detailCode="4140" errorCode="404" name="NotFound">
<description>getSystemMetadata failed: NotFound - 1420: No system metadata could be found for given PID: some%2F%2Fjunk</description>
</error>
#5 Updated by Jing Tao over 5 years ago
- Assignee changed from Jing Tao to Rob Nahf
#6 Updated by Dave Vieglais over 5 years ago
Edited /var/lib/tomcat8/webapps/cn/WEB-INF/config/log4j.properties
, adding:
log4j.logger.org.dataone.cn=INFO
When calling resolve, results in:
[ INFO] 2019-05-16 00:25:42,148 (ResolveForwarder:forward:79) proxy resolve: /metacat/d1/cn/v2/meta/some%2F%2Fjunk
[ INFO] 2019-05-16 00:25:42,148 (AbstractProxyForwarder:forwardRequest:103) CN Getting proxy...
[ INFO] 2019-05-16 00:25:42,148 (AbstractProxyForwarder:forwardRequest:106) CN Dispatching: /d1/cn/v2/meta/some%2F%2Fjunk
[ INFO] 2019-05-16 00:25:42,168 (ExceptionController:get:67) received error of 500
A successful call is like:
[ INFO] 2019-05-16 00:26:29,819 (ResolveForwarder:forward:79) proxy resolve: /metacat/d1/cn/v2/meta/some//junk
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:103) CN Getting proxy...
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:106) CN Dispatching: /d1/cn/v2/meta/some//junk
[ INFO] 2019-05-16 00:26:29,820 (AbstractProxyForwarder:forwardRequest:111) CN Dispatching proxy...
[ INFO] 2019-05-16 00:26:29,860 (AbstractProxyForwarder:forwardRequest:113) CN Completed proxy...
#7 Updated by Dave Vieglais over 5 years ago
This means the failure is at line 107 of AbstractProxyForwarder.java in d1_cn_rest:
protected void forwardRequest(ServletContext servletContext, String contextName, ProxyServletRequestWrapper proxyServletWrapper, HttpServletResponse response) throws ServletException, IOException, NotFound, ServiceFailure { ServletContext proxyServletContext = servletContext.getContext(contextName); if (proxyServletContext != null) { logger.info("CN Getting proxy..."); try { logger.info("CN Dispatching: " + proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo()); RequestDispatcher dispatch = proxyServletContext.getRequestDispatcher(proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo()); if (dispatch != null) { logger.info("CN Dispatching proxy..."); dispatch.forward(proxyServletWrapper, response); logger.info("CN Completed proxy..."); } else { throw new NotFound("n/a", "Servlet path " + proxyServletWrapper.getServletPath() + proxyServletWrapper.getPathInfo() + " of contextName " + contextName + " not found! " + servletContext.getServerInfo()); }
#8 Updated by Dave Vieglais over 5 years ago
The tomcat implementation of ServletContext seems to be ApplicationContext.java
With the problem apparently at:
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/core/ApplicationContext.java#L436-L443
and:
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/tomcat/util/http/RequestUtil.java#L76
#9 Updated by Dave Vieglais over 5 years ago
Per Rob's suggestion:
I think we might also create a custom applicationContext subclass.
https://stackoverflow.com/questions/25612730/tomcat-7-0-how-to-launch-with-a-custom-applicationcontext
Stack Overflow
Tomcat 7.0+ How to launch with a custom ApplicationContext
I'm trying to figure out a way to ask Tomcat(nicely) to launch the server with my custom ApplicationContext for example: public class MyApplicationContext extends ApplicationContext ...
#10 Updated by Rob Nahf over 5 years ago
I found a context parameter (dispatchersUseEncodedPaths) that lets us bypass the directory traversal check. The solution was to add a second line to webapps/cn/META-INF/context.xml for the /metacat path. It doesn't seem to be used, however, but I'm not sure why.
<?xml version="1.0" encoding="UTF-8"?> <Context antiJARLocking="true" crossContext="true" path="/cn"/> <Context antiJARLocking="true" crossContext="true" dispatchersUseEncodedPaths="false" path="/metacat"/>
I've tried different paths as well, '/d1/cn/v1', '/d1/cn/v2', '/d1/cn', but they don't work either.
#11 Updated by Rob Nahf over 5 years ago
I copy-pasted the RequestUtils.normalize code to test it, and was able to determine that it leaves the encoded paths intact. If we want to use the existing ApplicationContext, we need to know how to apply the 'dispatchersUseEncodedPaths'.
#12 Updated by Dave Vieglais over 5 years ago
tried path=
/metacat /cn /d1/cn /d1/cn/v1 /d1/cn/v2
None seem to have any effect.
#13 Updated by Dave Vieglais over 5 years ago
- % Done changed from 0 to 30
- Status changed from New to In Progress
Reading through the code, documentation for Tomcat contexts plus a couple articles results in:
the call
proxyServletContext.getRequestDispatcher()
is being made in the metacat servlet contextTomcat reads
[engine-name]/[server-name]/[app-name].xml
after the war is deployed.There can only be one path context.
The result is /etc/tomcat8/Catalina/localhost/metacat.xml
:
<Context path="/metacat/" antiResourceLocking="false" unpackWAR="false" crossContext="true" dispatchersUseEncodedPaths="false"/>
After editing and restarting Tomcat8, the resolve service works as expected.
#14 Updated by Dave Vieglais over 5 years ago
- % Done changed from 30 to 100
- Status changed from In Progress to Closed