Story #3729

Member Nodes should be the authoritative source of System Metadata

Added by Chris Jones about 11 years ago. Updated about 8 years ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 280.00 h)
Story Points:


After gaining some experience with CN-based authority for system metadata, we've realized that there are many use cases that require system metadata to be managed by the MNs authoritatively, and by the CNs secondarily as a cached version. The main use case involves access control. When an ITK client creates an object through MN.create(), control of the system metadata is transferred to the CN once synchronization happens. After that point, the ITK client (and scientist) has to make CN.setAccessPolicy() calls to make any changes. If the MN is set to sync once a week, this is problematic.

Ultimately, the CN stores and manages system metadata in order to track replicas of an object, and to perform auditing on those replicas. This change would really just require that the CN remains the authority of the ReplicaList, whereas the MN would become the authority of all other fields. By doing so, ITK clients will be able to interact with their MN without delay, and the CNs will work off of cached versions of the system metadata (in Hazelcast and persisted in Metacat).

To effect these changes, specific sub tasks include:

1) Design of the system - sequence diagram and use case changes
2) Update architecture and guide docs to express that the MN is the authoritative source for system metadata
3) Ensure MN API calls reflect that the MN is responsible for incrementing serial version
3.1) Serial version will be used to track all fields except the ReplicaList
3.2) Consider adding an optional serialVersion attribute to a a ReplicaList, and use it to track versions of the list on the CN
4) Include a push notification of system metadata change (CN.systemMetadataChanged())
5) Change the CN stack to accommodate the MN-based authority
5.1) Replication
5.2) Synchronization
5.3) DAO layer for replication
6) Change all MN stacks to implement the new features (Metacat, GMN, Mercury, EDAC, Dryad, Merritt)
7) Add MN.updateSystemMetadata() interfaces (d1_common_{java|python}, d1_libclient_{java|python})
8) Refactor CN sysmeta methods to delegate to MN.


Task #3743: Change system metadata design documents to reflect MN authorityClosedChris Jones

Task #3744: Add MN API calls to update system metadataClosedChris Jones

Task #3745: Add Replica version attributeRejectedChris Jones

Task #3746: Add CNCore.systemMetadataChanged() callRejectedChris Jones

Task #7175: update operations documentation ClosedDave Vieglais

Task #3787: CN should detect changed sysmeta and resync objects when neededClosedRob Nahf

Task #3788: Add CN.updateSystemMetadata methodClosedRob Nahf

Task #7179: add API documentation for CN.updateSystemMetadataClosedRob Nahf

Task #7180: add d1_libclient_java implementationClosedRob Nahf

Task #7225: Add cn.updateSystemMetadata implementation in MetacatClosed


#1 Updated by Chris Jones about 11 years ago

  • Description updated (diff)

#2 Updated by Robert Waltz over 9 years ago

  • Target version set to CCI-2.0.0
  • Start date set to 2014-12-01
  • Due date set to 2014-12-01

#3 Updated by Chris Jones over 8 years ago

  • Assignee changed from Chris Jones to Dave Vieglais

Outstanding tasks in this story can be moved to other stories since the V2.0.0 release is complete.

#4 Updated by Dave Vieglais about 8 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)