https://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2017-04-26T05:56:00ZDataONE TasksInfrastructure - Story #8061: develop queue-based processing system for the CNhttps://redmine.dataone.org/issues/8061?journal_id=288042017-04-26T05:56:00ZRob Nahfrnahf@epscor.unm.edu
<ul></ul><p>RabbitMQ uses the terms queues, exchanges, channels, brokers, consumers, publishers.</p>
<p>Our processing consumers will connect to named queues via channels, and we will likely be using their high-level framework which sets up handlers in the consumers, and exception handlers in the channel (I believe).Does it make sense to abstract the channels?</p>
Infrastructure - Story #8061: develop queue-based processing system for the CNhttps://redmine.dataone.org/issues/8061?journal_id=295112018-01-17T18:50:14ZDave Vieglaisdave.vieglais@gmail.com
<ul><li><strong>Sprint</strong> set to <i>Infrastructure backlog</i></li></ul> Infrastructure - Story #8061: develop queue-based processing system for the CNhttps://redmine.dataone.org/issues/8061?journal_id=295302018-01-17T19:00:18ZRob Nahfrnahf@epscor.unm.edu
<ul></ul><p>I recently came across Apache Flink, which is a stream-based messaging system with deliver-exactly-once guarantees, and could be a simpler system than RabbitMQ, depending on its robustness across the WAN. It looks like it is coupled with Kafka. </p>
<p>Keep as a possible alternative, although development work with RabbitMQ is mostly complete.</p>