Solr crashing with out of memory error
Solr has been crashing and/or thrashing with out of memory errors on CN-UCSB-1 resulting in poor or no response when using search.dataone.org.
There are several related issues to be addressed here:
- Poorly written queries generated by the search UI are causing excessive demand on the solr index
- Solr is running with 2GB allocated to the JVM. This appears to be insufficient.
- Should design and implement a load balancing solution so a single instance of Solr is not used all the time
- Memory use is always high on cn-ucsb-1 where processing (and so log aggregation) is running. Review the log aggregation process for memory leaks and performance.
- The configuration for Solr is using mostly default parameters. This should be reviewed and revised where appropriate.