Project

General

Profile

Task #8080

Story #8061: develop queue-based processing system for the CN

ioslate queue creation logic from processing logic from the queue definition logic

Added by Rob Nahf over 4 years ago. Updated over 4 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
d1_indexer
Target version:
-
Start date:
2017-04-27
Due date:
% Done:

30%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

Use factory methods to define the queues and channels that will be shared by multiple processors (that represent steps in the pipeline).

d1_cn_index_common should hold the creation methods (Factories) and the queue name for the starting queue.
d1_cn_index_processor should hold the queue names specific to queue processing.


Subtasks

Task #8086: upgrade Spring dependenciesIn ProgressRob Nahf

Associated revisions

Revision 18838
Added by Rob Nahf over 4 years ago

refs #8080: first prototype of d1_index_processing using messaging queues. Classes in the new org.dataone.cn.index.messaging hold the configurations and manage the ansynchronous consumers, which are for now in o.d.c.index.processor package. Integration test IndexProcessingPrioritizationIT#testConsumerContainerSetup has full branching workflow "test". Imported Hazelcast test configurations; abstracted HttpService behind an interface. More work needed on test configurations - production config hijacked for now until I can get a clean separation.

Revision 18838
Added by Rob Nahf over 4 years ago

refs #8080: first prototype of d1_index_processing using messaging queues. Classes in the new org.dataone.cn.index.messaging hold the configurations and manage the ansynchronous consumers, which are for now in o.d.c.index.processor package. Integration test IndexProcessingPrioritizationIT#testConsumerContainerSetup has full branching workflow "test". Imported Hazelcast test configurations; abstracted HttpService behind an interface. More work needed on test configurations - production config hijacked for now until I can get a clean separation.

History

#1 Updated by Rob Nahf over 4 years ago

  • Assignee changed from Dave Vieglais to Rob Nahf

#2 Updated by Rob Nahf over 4 years ago

if we are going to utilize message queues for other process pipelines, then putting the factories in d1_cn_common makes more sense.

#3 Updated by Rob Nahf over 4 years ago

It turns out that Connections and Channels are pretty well encapsulated, so should be passed into Consumers, and maybe Publishers. The actual durable queues can be created through code, or through configuration (or even through the command line. So, queue creation should probably happen in the postinst of dataone-cn-index-processor debian package. (and d1_cn_common for the newIndexTask)

http://stackoverflow.com/questions/18418936/rabbitmq-and-relationship-between-channel-and-connection

#4 Updated by Rob Nahf over 4 years ago

For the d1_cn_common parts, Spring already has some pretty nice encapsulations that would both simplify application code (especially in publishing) and being able to push all of the application configuration stuff into xml..

http://docs.spring.io/spring-amqp/reference/htmlsingle/#compatibility
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-messaging.html

Spring also has a high-level class called RabbitTemplate:
http://docs.spring.io/spring-amqp/docs/latest_ga/api/org/springframework/amqp/rabbit/core/RabbitTemplate.html
http://stackoverflow.com/questions/34125116/spring-rabbittemplate-how-to-get-hold-of-the-published-message-for-nacks-in-pub

using spring would require upgrade of Spring from 3.1.4 to 4.1.x

#5 Updated by Rob Nahf over 4 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 30

org.dataone.cn.messaging.QueueAccess is a new class in d1_cn_common that gives the user client access to the rabbitMQ queue. It's in d1_cn_common because it is more general purpose than indexing, and it only needs the name of a legitimate queue to work.

Indexing configuration will go into d1_cn_index_common (where the IndexTask definitions will be as well).

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)