Project

General

Profile

Task #3766

Story #3760: Index Maintenance Story

Speeding up index build time

Added by Skye Roseboom over 9 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Andrei Buium
Category:
-
Target version:
-
Start date:
2016-04-01
Due date:
% Done:

100%

Estimated time:
0.00 h
Story Points:
Sprint:

Description

Investigate alternative solr commit strategies while building the index to decrease time to takes to build index.
- For example 'commitWithin' seconds instead of hard commits on every update.

Possibly also substitute out the hazelcast hzIdentifiers iteration strategy for looping over all documents and move to a DAO/postgres query paging solution. Decrease network traffic/wait time.

Associated revisions

Revision 17774
Added by Andrei Buium almost 7 years ago

Committed a change to d1_cn_index_tool to allow it to output a performance log file with the same logging statements as the index task processor.
refs #3766

Revision 17774
Added by Andrei Buium almost 7 years ago

Committed a change to d1_cn_index_tool to allow it to output a performance log file with the same logging statements as the index task processor.
refs #3766

History

#1 Updated by Skye Roseboom over 9 years ago

  • Parent task set to #3760

#2 Updated by Dave Vieglais almost 7 years ago

  • Assignee changed from Skye Roseboom to Andrei Buium

reassigning for review.

#3 Updated by Andrei Buium almost 7 years ago

  • Start date set to 2016-04-01
  • % Done changed from 0 to 30
  • Status changed from New to In Progress

Adding support to do batch adds from the index build tool has sped up tests significantly.
We also had commit=true hard-coded into the solr update url. Removing this speeds up performance as well, but it appears autoCommit isn't taking over in its absence. This needs investigating, because having the commit happen outside of the index tool code execution path would be ideal.

#4 Updated by Dave Vieglais almost 6 years ago

  • % Done changed from 30 to 100
  • Status changed from In Progress to Closed

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)