Task #3766
Story #3760: Index Maintenance Story
Speeding up index build time
100%
Description
Investigate alternative solr commit strategies while building the index to decrease time to takes to build index.
- For example 'commitWithin' seconds instead of hard commits on every update.
Possibly also substitute out the hazelcast hzIdentifiers iteration strategy for looping over all documents and move to a DAO/postgres query paging solution. Decrease network traffic/wait time.
Associated revisions
Committed a change to d1_cn_index_tool to allow it to output a performance log file with the same logging statements as the index task processor.
refs #3766
Committed a change to d1_cn_index_tool to allow it to output a performance log file with the same logging statements as the index task processor.
refs #3766
History
#1 Updated by Skye Roseboom almost 11 years ago
- Parent task set to #3760
#2 Updated by Dave Vieglais about 8 years ago
- Assignee changed from Skye Roseboom to Andrei Buium
reassigning for review.
#3 Updated by Andrei Buium about 8 years ago
- Start date set to 2016-04-01
- % Done changed from 0 to 30
- Status changed from New to In Progress
Adding support to do batch adds from the index build tool has sped up tests significantly.
We also had commit=true hard-coded into the solr update url. Removing this speeds up performance as well, but it appears autoCommit isn't taking over in its absence. This needs investigating, because having the commit happen outside of the index tool code execution path would be ideal.
#4 Updated by Dave Vieglais about 7 years ago
- % Done changed from 30 to 100
- Status changed from In Progress to Closed
