Project

General

Profile

Task #2623

Story #2622: d1_replication should prioritize MN replication tasks based on load, failures, and bandwidth factors

Modify Metacat's MNodeService.replicate() to queue requests

Added by Chris Jones almost 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Ben Leinfelder
Category:
Metacat
Start date:
2012-04-20
Due date:
% Done:

100%

Milestone:
CCI-1.0.0
Product Version:
*
Story Points:
Sprint:

Description

At the moment calls to replicate() are handled immediately regardless of how many requests have been made or the size of the objects being replicated. It's possible to saturate the bandwidth with a dozen or so of large replica requests, or possibly overwhelm tomcat/apache's ability to handle requests because of sheer volume.

Add a queue to Metacat (possibly via hazelcast) that is used to manage incoming calls to replicate(). Processing of the queue should have a configurable maximum based on the number of requests currently being processed, or the load caused by the requests (bandwidth in particular).

History

#1 Updated by Chris Jones almost 9 years ago

  • Assignee changed from Chris Jones to Ben Leinfelder

#2 Updated by Ben Leinfelder almost 9 years ago

We're doing a simpler implementation shift now - keeping the initial queue idea on the back burner if all else fails.
The plan:
1. Share an ExecutorService with a fixed number of threads available for replicate() tasks.
2. each replicate() call is submitted for execution.
3. That's it.
Number of threads in the pool is set to available processors minus 1.

#3 Updated by Ben Leinfelder almost 9 years ago

  • Status changed from New to Closed

changes are in MNResourceHandler in Metacat

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)