Project

General

Profile

Task #3787

Story #3729: Member Nodes should be the authoritative source of System Metadata

CN should detect changed sysmeta and resync objects when needed

Added by Matthew Jones about 9 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
d1_synchronization
Target version:
Start date:
2013-06-04
Due date:
% Done:

100%

Milestone:
CCI-2.0
Product Version:
2.0.0
Story Points:
Sprint:

Description

The current CN assumes that certain fields in system metadata are immutable, and so does not notice when those fields are updated. This can arise when, for example, the formatid of an object changes, which change the object from DATA to either METADATA or RESOURCE types, which means the object needs to be re-synced (for an example, see ticket #3719).

The -CNCore.systemMetadataChanged- CN.synchronize call will be pushed to the CoordinatingNode from the MemberNode. Given the timing of the push call to -systemMetadataChanged- CN.synchronize and the regular harvesting of content from the MemberNode, a race condition may appear on the CN in which older data overwrites a newer update. Management of the sequence of regular synchronization harvesting and asychronous systemMetadataChanged will need to be designed for d1 processing and the cn rest interface.

To close this bug we need the following changes:

1) System metadata changes on the CN are detected when they are made and inspected to see if any operations need to be taken
2) if a new sync is required, the sync gets scheduled and executed, and the object is inserted into the CN metacat store as needed
3) metacat replication notices that the object needs to be replicated to the other CNs and does it
4) ensure order of synchronous and asynchronous sync events


Related issues

Related to Infrastructure - Story #7364: update the sync logic for basic v1-v2 interoperability Closed 2015-09-22
Related to Infrastructure - Bug #7371: A systemMetadata update that changes the formatID from DATA to METADATA or RESOURCE_MAP will not upload the object to the CN New 2015-09-21

History

#1 Updated by Matthew Jones about 9 years ago

  • Assignee changed from Skye Roseboom to Robert Waltz

#2 Updated by Robert Waltz over 8 years ago

  • Milestone changed from None to CCI-2.0
  • Product Version set to 2.0.0

#3 Updated by Robert Waltz almost 8 years ago

  • Assignee changed from Robert Waltz to Ben Leinfelder

#4 Updated by Robert Waltz almost 8 years ago

  • Description updated (diff)

#5 Updated by Robert Waltz over 7 years ago

  • Target version set to CCI-2.0.0

#6 Updated by Rob Nahf about 7 years ago

  • % Done changed from 0 to 30
  • Status changed from New to In Progress
  • Assignee changed from Ben Leinfelder to Rob Nahf

Implementing CN.synchronize(pid) instead of CN.systemMetadataChanged(pid), with the behavior of CN.synchronize(pid) adding a SyncObjectTask to the existing queue should eliminate the race condition, except if duplicate SyncObjectTasts exist in the sync queue, and are processed at the same time.

We need to either make sure there's a lock on the pid for sync processing, or ensure that the sync queue doesn't already have the SyncObjectTask (look for duplicates).

#7 Updated by Rob Nahf about 7 years ago

  • Description updated (diff)

#8 Updated by Rob Nahf almost 7 years ago

synchronization uploads non-DATA objects and persists them using cn.create. For situations where the formatID is changed from a DATA format to a METADATA format, for example, the systemMetadata already exists on the CN, and the cn.create will fail due to existing checks on the existence of the identifier. Disabling those checks might impact the ban on reusing identifiers of deleted objects.

A new CN method may be needed to accomplish persisting the uploaded METADATA....

#9 Updated by Rob Nahf almost 7 years ago

  • Related to Story #7364: update the sync logic for basic v1-v2 interoperability added

#10 Updated by Rob Nahf almost 7 years ago

  • Related to Bug #7371: A systemMetadata update that changes the formatID from DATA to METADATA or RESOURCE_MAP will not upload the object to the CN added

#11 Updated by Rob Nahf over 6 years ago

  • Status changed from In Progress to Testing
  • % Done changed from 30 to 50

#12 Updated by Rob Nahf over 6 years ago

  • Status changed from Testing to Closed
  • % Done changed from 50 to 100
  • translation missing: en.field_remaining_hours set to 0.0

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)