Task #2623
Story #2622: d1_replication should prioritize MN replication tasks based on load, failures, and bandwidth factors
Modify Metacat's MNodeService.replicate() to queue requests
100%
Description
At the moment calls to replicate() are handled immediately regardless of how many requests have been made or the size of the objects being replicated. It's possible to saturate the bandwidth with a dozen or so of large replica requests, or possibly overwhelm tomcat/apache's ability to handle requests because of sheer volume.
Add a queue to Metacat (possibly via hazelcast) that is used to manage incoming calls to replicate(). Processing of the queue should have a configurable maximum based on the number of requests currently being processed, or the load caused by the requests (bandwidth in particular).
History
#1 Updated by Chris Jones about 12 years ago
- Assignee changed from Chris Jones to Ben Leinfelder
#2 Updated by Ben Leinfelder about 12 years ago
We're doing a simpler implementation shift now - keeping the initial queue idea on the back burner if all else fails.
The plan:
1. Share an ExecutorService with a fixed number of threads available for replicate() tasks.
2. each replicate() call is submitted for execution.
3. That's it.
Number of threads in the pool is set to available processors minus 1.
#3 Updated by Ben Leinfelder about 12 years ago
- Status changed from New to Closed
changes are in MNResourceHandler in Metacat
