Project

General

Profile

Task #7935

Use a dedicated hazelcast queue as the listener to generate index task

Added by Jing Tao about 5 years ago. Updated almost 5 years ago.

Status:
New
Priority:
Normal
Assignee:
Category:
d1_indexer
Target version:
Start date:
2016-11-16
Due date:
% Done:

0%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

Currently the index task generator listens to the events from the systemmetadata map. But the events from the system metadata map can come from different scenarios. For example, when tomcat starts up, hazelcast will load the system metadata from the database into the systemmetadata map. If the index generator is running, it can generate lots of index tasks which we really don't need.

If we have a dedicate queue, we can avoid this scenario. But we should call a method to put the system metadata into the queue specifically during the synchronization.

History

#1 Updated by Jing Tao about 5 years ago

Besides the synchronization, what other scenario do we need to put the system metadata into the dedicated queue?

#2 Updated by Dave Vieglais almost 5 years ago

  • Target version changed from CCI-2.3.1 to CCI-2.3.2

#3 Updated by Rob Nahf almost 5 years ago

The current operational structure (sync on UCSB and index on ORC) doesn't allow us to have sync generate index tasks in the local postgres DB.

In an orderly start, the index generator is started after tomcat (owner of HZ system metadata map), but tomcat restarts, and maybe even network partitioning issues might lead to a the index generator hearing a lot of spurious map add events.

Gaining this independence might be accomplished by creating a Hazelcast "SystemMetadataUpdated" list/set or map that is only added to by synchronization and the API update methods The indexTaskGenerator would be listening to this data structure on ORC. Not sure if it solves the network partitioning issue.

#4 Updated by Jing Tao almost 5 years ago

  • Target version changed from CCI-2.3.2 to CCI-2.4.0

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)