Project

General

Profile

Story #2067

Refactor the index task generator and processor to improve scalability and performance

Added by Dave Vieglais about 13 years ago. Updated almost 13 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Skye Roseboom
Category:
d1_indexer
Target version:
Sprint-2012.05-Block.1.3
Start date:
2011-11-29
Due date:
% Done:

100%

Estimated time:
(Total: 2.00 h)
Story Points:
Sprint:

Description

The current implementation of the index task generator and processor is unsuitable for a production environment. It needs to be refactored to improve scalability and to support concurrent access by a single task generator and one or more task processors.

Subtasks

Task #2068: Define the index task data modelClosedSkye Roseboom

Task #2069: Select a persistence technology for the index task queueClosedSkye Roseboom

Task #2070: Implement the refactored index task generatorClosedSkye Roseboom

Task #2071: Update the index task processor to read from task queueClosedSkye Roseboom

Task #2072: Update the index task generator to listen to the hzSystemMetadata map for updated entriesClosedSkye Roseboom

Task #2119: Create command line tool to build/update entire solr indexClosedSkye Roseboom

Task #2130: Build-out for solr indexing componentsClosedSkye Roseboom

Task #2143: Refactor static nodeRegistryMapClosedSkye Roseboom

Task #2173: Create solr index test harnessClosedSkye Roseboom

Task #2174: Solr installation cleanupClosedSkye Roseboom

Task #2184: Archived objects must be removed from the indexClosedSkye Roseboom

Task #2202: Remove unneeded fields from SOLR schemaClosedSkye Roseboom

Task #2220: Update solr index refresh tool to use new hzIdentifiers SetClosedSkye Roseboom

Task #2221: Update Task Generation to act only on 'update' eventsClosedSkye Roseboom

Task #2222: indexerapi generating slightly incorrect data_urlClosedSkye Roseboom

Task #2223: Remove obsoletedBy field from SOLR indexClosedSkye Roseboom

Task #2224: Set value of fileId to resolve URL if object is science metadataClosedSkye Roseboom

Task #2225: Remove the contains field form the solr index.ClosedSkye Roseboom

Task #2248: Hudson continuous buildsClosedSkye Roseboom

Task #2270: Restrict access to solr update to localhostClosedSkye Roseboom

History

#1 Updated by Dave Vieglais about 13 years ago

  • Target version changed from Sprint-2011.48-Block.6 to Sprint-2011.49-Block.6
  • Position set to 325

#2 Updated by Skye Roseboom about 13 years ago

  • Status changed from New to In Progress

#3 Updated by Dave Vieglais about 13 years ago

  • Position deleted (324)
  • Position set to 13
  • Target version changed from Sprint-2011.49-Block.6 to Sprint-2011.51-Block.6

#4 Updated by Skye Roseboom almost 13 years ago

  • Target version changed from Sprint-2011.51-Block.6 to Sprint-2012.01-Block.1.1
  • Position deleted (18)
  • Position set to 3

#5 Updated by Skye Roseboom almost 13 years ago

  • Status changed from In Progress to Closed

Closing this story as all sub-tasks have been completed.

#6 Updated by Skye Roseboom almost 13 years ago

  • Status changed from Closed to In Progress

re-opening story with new tasks

#7 Updated by Dave Vieglais almost 13 years ago

  • Target version changed from Sprint-2012.01-Block.1.1 to Sprint-2012.03-Block.1.2
  • Position deleted (41)
  • Position set to 1

#8 Updated by Skye Roseboom almost 13 years ago

  • Status changed from In Progress to Closed

Closing story as all tasks completed.

#9 Updated by Skye Roseboom almost 13 years ago

  • Status changed from Closed to In Progress

opening with new sub-task.

#10 Updated by Skye Roseboom almost 13 years ago

  • Target version changed from Sprint-2012.03-Block.1.2 to Sprint-2012.05-Block.1.3

moving forward to current sprint, for new sub-task (2270).

#11 Updated by Skye Roseboom almost 13 years ago

  • Status changed from In Progress to Closed

closing again with final task complete.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)