Bug #7570: Search UI requests cause Out of Memory Error - Infrastructure - DataONE Tasks

Bug #7570

Search UI requests cause Out of Memory Error

Added by Chris Jones over 8 years ago. Updated about 7 years ago.

Status:

In Progress

Priority:

Normal

Assignee:

Jing Tao

Category:

d1_indexer

Target version:

CCI-2.4.0

Start date:

2016-01-05

Due date:

% Done:

30%

Milestone:

None

Product Version:

Story Points:

Sprint:

Description

https://search.dataone.org/cn/v2/query/solr/?q=++++-obsoletedBy:*+formatType:METADATA+++beginDate:[1800-01-01T00:00:00Z TO NOW]&rows=1&fl=beginDate&sort=beginDate+asc&wt=json

https://search.dataone.org/cn/v2/query/solr/?q=++++-obsoletedBy:*+formatType:METADATA+++endDate:[1800-01-01T00:00:00Z TO NOW]+-obsoletedBy:*&rows=1&fl=endDate&sort=endDate+desc&wt=json

https://search.dataone.org/cn/v2/query/solr/?q=++++-obsoletedBy:*+formatType:METADATA++&rows=0&facet=true&facet.sort=count&facet.mincount=1&facet.limit=-1&facet.field=keywords&facet.field=origin&facet.field=family&facet.field=species&facet.field=genus&facet.field=kingdom&facet.field=phylum&facet.field=order&facet.field=class&facet.field=site&facet.field=attributeName&facet.field=attributeLabel&wt=json&json.wrf=jQuery19109610712484859317_1452005239758&_=1452005239759

Here's the stacktrace reported back to the Search UI:

"error":{"msg":"Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space","trace":"org.apache.solr.common.SolrException: Error while processing facet fields: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:659)
at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:285)
at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:102)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:255)
at org.dataone.solr.handler.component.SolrSearchHandler.handleRequestBody(SolrSearchHandler.java:101)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.dataone.solr.servlet.SessionAuthorizationUtil.handleNoCertificateManagerSession(SessionAuthorizationUtil.java:131)
at org.dataone.solr.servlet.SearchServiceSessionAuthorizationFilter.handleNoCertificateManagerSession(SearchServiceSessionAuthorizationFilter.java:45)
at org.dataone.solr.servlet.SessionAuthorizationFilterStrategy.doFilter(SessionAuthorizationFilterStrategy.java:264)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
","code":500}

To temporarily alleviate the issue, I restarted Solr on the 3 CNs. That had no effect, so I modified the Solr configuration in /var/solr/solr.in.sh to request 2048m of Java heap space (up from 512m that was in the file), and restarted Solr. That, too, had no effect. I then bumped it to 4096m, restarted, and all queries started to work again.

Related issues

History

#1 Updated by Chris Jones over 8 years ago

Description updated (diff)

#2 Updated by Dave Vieglais about 8 years ago

Assignee changed from Skye Roseboom to Andrei Buium

reassigning.

#3 Updated by Andrei Buium about 8 years ago

Status changed from New to In Progress
% Done changed from 0 to 30

Lauren has split the UI query most likely to be the culprit (the last one containing many facets) into multiple queries.
Other options to consider:
Using docValues="true" in the schema for fields often sorted/faceted on (would speed up query time and lower memory use at query time but increase total index size due to a forward map being created).
Checking if store="true" is absolutely necessary on all the current schema fields (NOTE: displaying formats without an existing xslt relies on solr stored fields).
Delivering a cached result (preferably also created from multiple queries).
Using SolrCloud to shard the index and split memory use more evenly across different nodes.
Further increasing memory allowed. (May be necessary regardless. Is it currently possible for a user to kill solr with a complicated query?)