Project

General

Profile

Bug #8027

Heap memory errors from solr

Added by Dave Vieglais over 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
d1_cn_buildout
Target version:
Start date:
2017-02-27
Due date:
% Done:

100%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

Solr is configured to use only 512mb for heap memory. Numerous out of heap memory exceptions were showing in the admin console (port 8983).

Edited /var/solr/solr.in.sh and changed SOLR_HEAP to "2048m"

The admin console shows typical heap use around 1GB, so 2GB is probably high, but will monitor and reduce if necessary.

This task is to update the install config to ensure that SOLR_HEAP setting is updated or verified during solr upgrades.


Related issues

Related to Infrastructure - Bug #7570: Search UI requests cause Out of Memory Error In Progress 2016-01-05

Associated revisions

Revision 18648
Added by Rob Nahf over 5 years ago

refs #8027. Increased SOLR_HEAP (jetty) to 2048m.

Revision 18648
Added by Rob Nahf over 5 years ago

refs #8027. Increased SOLR_HEAP (jetty) to 2048m.

History

#1 Updated by Dave Vieglais over 5 years ago

To access the admin console, use an SSH tunnel, like: ssh -L8983:localhost:8983 cn-ucsb-1.dataone.org then http://localhost:8983

#2 Updated by Rob Nahf over 5 years ago

  • Category changed from d1_cn_solr_extensions to d1_cn_buildout
  • % Done changed from 0 to 30
  • Status changed from New to In Progress

also noticing very steady heap utilization at 1Gb. Found an article http://java-monitor.com/forum/showthread.php?t=427 which says not to worry too much about the initial head size - that GC will kick it down if you sett it too high, and it will still grab from the system as needed up to the maximum.

So, I'm going to set it to '2048m', with the understanding that the GC will help make sure that jetty's not being a memory hog.

First commit is in trunk. Need to do the 2.3 branch.

#3 Updated by Rob Nahf over 5 years ago

  • % Done changed from 30 to 50
  • Status changed from In Progress to Testing

there are two solr.in.sh files, one in /var/solr/bin/init.d/solr.in.sh and the other at /usr/share/dataone-solr/debian/server_configs/solr.in.sh. I updated both, in 2.3 branch and trunk.

#4 Updated by Dave Vieglais about 5 years ago

  • Related to Bug #7570: Search UI requests cause Out of Memory Error added

#5 Updated by Dave Vieglais about 5 years ago

  • Target version changed from CCI-2.3.2 to CCI-2.3.3

Heap use by solr is now being monitored - see https://measure-unm-1.dataone.org

#6 Updated by Rob Nahf about 5 years ago

comparing the heap peaks on measure-unm-1 with the logs on cn-ucsb-1, there is no correlation between the number of queries per minute and heap usage.

Heap most likely due to the demands of individual queries, eitherthe query itself, or the authorization part of it. (authorization use recursion to parse subject info).

Another possibility is solr updates happening during periods of reading.

#7 Updated by Rob Nahf about 5 years ago

I'm observing that periods of high activity (rapid deltas of heap utilization) on cn-unm-1 and cn-orc-1 are usually reflected as peaks that bump cn-ucsb-1 from 60% (baseline) to 80%.

These correlated patterns don't hold for the spikes to 100%, but does help narrow the field of what to look at...

#8 Updated by Rob Nahf about 5 years ago

need to bump version of DATAONE-SOLR_v2.3 in the DEBIAN control file.

#9 Updated by Rob Nahf about 5 years ago

  • Status changed from Testing to Closed
  • % Done changed from 50 to 100

updated version in dataone-solr to 2.3.2

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)