Project

General

Profile

Story #8158

Synchronization appears to fail under high load

Added by Dave Vieglais over 4 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
d1_synchronization
Target version:
Start date:
2017-08-04
Due date:
% Done:

100%

Story Points:

Description

When the number of objects that needs to be processed by synchronization is high, the process fails / crashes. For example, the Pangaea node with 325k objects fails on initial sync. The cause of this needs to be evaluated and the synchronization process refactored to allow arbitrarily large numbers of entries to be synchronized.

History

#1 Updated by Dave Vieglais over 4 years ago

  • Assignee changed from Robert Waltz to Dave Vieglais

#2 Updated by Dave Vieglais almost 4 years ago

  • Sprint set to Infrastructure backlog

#3 Updated by Dave Vieglais almost 4 years ago

  • Target version changed from CCI-2.4.0 to CCI-2.3.7

#4 Updated by Rob Nahf almost 4 years ago

harvest was refactored to better keep the queue under 50k items. (It doesn't harvest everything if the queue is too long). Synchronizing in stage only shows sync failing due to Hz errors (connection issues, CONCURRENT_MAP_REMOVES).

So, I believe the sync code is robust with respect to high load.

#5 Updated by Rob Nahf almost 4 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

#6 Updated by Dave Vieglais over 3 years ago

  • % Done changed from 30 to 100
  • Status changed from In Progress to Closed

Appears resolved with Rob's changes to synchronization process.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)