Heap memory errors from solr
Solr is configured to use only 512mb for heap memory. Numerous out of heap memory exceptions were showing in the admin console (port 8983).
Edited /var/solr/solr.in.sh and changed SOLR_HEAP to "2048m"
The admin console shows typical heap use around 1GB, so 2GB is probably high, but will monitor and reduce if necessary.
This task is to update the install config to ensure that SOLR_HEAP setting is updated or verified during solr upgrades.
#2 Updated by Rob Nahf over 5 years ago
- Category changed from d1_cn_solr_extensions to d1_cn_buildout
- % Done changed from 0 to 30
- Status changed from New to In Progress
also noticing very steady heap utilization at 1Gb. Found an article http://java-monitor.com/forum/showthread.php?t=427 which says not to worry too much about the initial head size - that GC will kick it down if you sett it too high, and it will still grab from the system as needed up to the maximum.
So, I'm going to set it to '2048m', with the understanding that the GC will help make sure that jetty's not being a memory hog.
First commit is in trunk. Need to do the 2.3 branch.
#6 Updated by Rob Nahf about 5 years ago
comparing the heap peaks on measure-unm-1 with the logs on cn-ucsb-1, there is no correlation between the number of queries per minute and heap usage.
Heap most likely due to the demands of individual queries, eitherthe query itself, or the authorization part of it. (authorization use recursion to parse subject info).
Another possibility is solr updates happening during periods of reading.
#7 Updated by Rob Nahf about 5 years ago
I'm observing that periods of high activity (rapid deltas of heap utilization) on cn-unm-1 and cn-orc-1 are usually reflected as peaks that bump cn-ucsb-1 from 60% (baseline) to 80%.
These correlated patterns don't hold for the spikes to 100%, but does help narrow the field of what to look at...