Project

General

Profile

Feature #4038

change default replication implementation

Added by Matthew Jones over 8 years ago. Updated over 7 years ago.

Status:
New
Priority:
Normal
Assignee:
Category:
d1_replication
Target version:
-
Start date:
Due date:
% Done:

0%

Milestone:
CCI-2.0
Product Version:
*
Story Points:
Sprint:

Description

For objects less than a threshold size, change the default number of replicas for objects to two (instead of the current 0) within the replication management system on CNs.

During today's MN Forum call, the MN operators discussed the impact of the government shutdown on data availability and noted that DataONE's replication system could effectively improve data availability in the midst of massive service shutdowns like we have seen this week. We discussed the pros and cons of changing the default replication policy so that, in the absence of an explicit ReplicationPolicy for an object, DataONE Coordinating Nodes would create two replicas of any object that was smaller than a certain threshold size, probably around 500 MB today. This threshold would increase over time as network bandwidth improved, so should be configurable on the CN without code changes. This would ensure that the replication capabilities of DataONE are more fully utilized, while keeping the control of replication firmly in the hands of Member Nodes (if they don't want replicas, they simply set replicationAllowed=false in the replication policy in system metadata).

I have created a design document that describes our envisioned system here:

http://mule1.dataone.org/ArchitectureDocs-current/design/ReplicationOverview.html

History

#1 Updated by Matthew Jones over 8 years ago

  • Description updated (diff)

#2 Updated by Dave Vieglais over 8 years ago

My preference would be to make no changes and use a tool to assist MNs with updating the replication policy on their content through the CNReplication.setReplicationPolicy method. If it doesn't do so already, the CLI would seem to be the right tool for a generic implementation.

#3 Updated by Dave Vieglais about 8 years ago

  • Due date changed from 2013-10-05 to 2014-02-01
  • Target version changed from 2013.39-Block.5.3 to 2014.4-Block.1.2

#4 Updated by Chris Jones almost 8 years ago

  • Target version changed from 2014.4-Block.1.2 to 2014.14-Block.2.3
  • Due date changed from 2014-02-01 to 2014-04-12
  • Assignee changed from Chris Jones to Matthew Jones

This policy decision needs concensus from MN operators. Reassigning to Matt for now for discussion at the CCIT 2014 meeting.

#5 Updated by Robert Waltz over 7 years ago

  • Due date deleted (2014-04-12)
  • Target version deleted (2014.14-Block.2.3)
  • Start date deleted (2013-10-04)
  • Milestone changed from None to CCI-2.0

#6 Updated by Dave Vieglais over 7 years ago

  • Due date set to 2014-10-01
  • Target version set to Maintenance Backlog
  • Start date set to 2014-10-01

#7 Updated by Dave Vieglais over 7 years ago

  • Start date deleted (2014-10-01)
  • Target version deleted (Maintenance Backlog)
  • Due date deleted (2014-10-01)

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)