Story #3760: Index Maintenance Story
Speeding up index build time
Investigate alternative solr commit strategies while building the index to decrease time to takes to build index.
- For example 'commitWithin' seconds instead of hard commits on every update.
Possibly also substitute out the hazelcast hzIdentifiers iteration strategy for looping over all documents and move to a DAO/postgres query paging solution. Decrease network traffic/wait time.
Committed a change to d1_cn_index_tool to allow it to output a performance log file with the same logging statements as the index task processor.
#3 Updated by Andrei Buium almost 7 years ago
- Start date set to 2016-04-01
- % Done changed from 0 to 30
- Status changed from New to In Progress
Adding support to do batch adds from the index build tool has sped up tests significantly.
We also had commit=true hard-coded into the solr update url. Removing this speeds up performance as well, but it appears autoCommit isn't taking over in its absence. This needs investigating, because having the commit happen outside of the index tool code execution path would be ideal.