Bug #8027
Heap memory errors from solr
100%
Description
Solr is configured to use only 512mb for heap memory. Numerous out of heap memory exceptions were showing in the admin console (port 8983).
Edited /var/solr/solr.in.sh and changed SOLR_HEAP to "2048m"
The admin console shows typical heap use around 1GB, so 2GB is probably high, but will monitor and reduce if necessary.
This task is to update the install config to ensure that SOLR_HEAP setting is updated or verified during solr upgrades.
Related issues
Associated revisions
refs #8027. Increased SOLR_HEAP (jetty) to 2048m.
refs #8027. Increased SOLR_HEAP (jetty) to 2048m.
History
#1 Updated by Dave Vieglais over 7 years ago
To access the admin console, use an SSH tunnel, like: ssh -L8983:localhost:8983 cn-ucsb-1.dataone.org then http://localhost:8983
#2 Updated by Rob Nahf over 7 years ago
- Category changed from d1_cn_solr_extensions to d1_cn_buildout
- % Done changed from 0 to 30
- Status changed from New to In Progress
also noticing very steady heap utilization at 1Gb. Found an article http://java-monitor.com/forum/showthread.php?t=427 which says not to worry too much about the initial head size - that GC will kick it down if you sett it too high, and it will still grab from the system as needed up to the maximum.
So, I'm going to set it to '2048m', with the understanding that the GC will help make sure that jetty's not being a memory hog.
First commit is in trunk. Need to do the 2.3 branch.
#3 Updated by Rob Nahf over 7 years ago
- % Done changed from 30 to 50
- Status changed from In Progress to Testing
there are two solr.in.sh files, one in /var/solr/bin/init.d/solr.in.sh and the other at /usr/share/dataone-solr/debian/server_configs/solr.in.sh. I updated both, in 2.3 branch and trunk.
#4 Updated by Dave Vieglais over 7 years ago
- Related to Bug #7570: Search UI requests cause Out of Memory Error added
#5 Updated by Dave Vieglais over 7 years ago
- Target version changed from CCI-2.3.2 to CCI-2.3.3
Heap use by solr is now being monitored - see https://measure-unm-1.dataone.org
#6 Updated by Rob Nahf over 7 years ago
comparing the heap peaks on measure-unm-1 with the logs on cn-ucsb-1, there is no correlation between the number of queries per minute and heap usage.
Heap most likely due to the demands of individual queries, eitherthe query itself, or the authorization part of it. (authorization use recursion to parse subject info).
Another possibility is solr updates happening during periods of reading.
#7 Updated by Rob Nahf over 7 years ago
I'm observing that periods of high activity (rapid deltas of heap utilization) on cn-unm-1 and cn-orc-1 are usually reflected as peaks that bump cn-ucsb-1 from 60% (baseline) to 80%.
These correlated patterns don't hold for the spikes to 100%, but does help narrow the field of what to look at...
#8 Updated by Rob Nahf over 7 years ago
need to bump version of DATAONE-SOLR_v2.3 in the DEBIAN control file.
#9 Updated by Rob Nahf over 7 years ago
- Status changed from Testing to Closed
- % Done changed from 50 to 100
updated version in dataone-solr to 2.3.2
