Project

General

Profile

Task #7770

Story #7769: Improve the performance on solr index

Profile the index process

Added by Jing Tao over 4 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
d1_cn_index_processor
Start date:
2016-05-04
Due date:
% Done:

100%

Estimated time:
0.00 h
Story Points:
Sprint:

Description

Solr Index profiling

EML:
1. Create a index task queue 328 (first one is very high, now it drops dramatically. Generally it is about 10 ~ 30)
2. Fetch a task from the queue 10
3. Total processing time 1076

- SolrIndexService.processObject - 536

*process system metadata - 66 ( each field is from 0 to 1, but the id took 32)

*process by science metadata - 255 (each field take 2,3 and 4. abstract 12, keyword 10, title 10, project 8, CommonRootSolrField("attribute") 32, text 11,southBoundCoord 9, northBoundCoord 7, west 10, east 8, begin date 10, enddate 7)

*process by BaseReprocessSubprocessor 71 (series id and resource map)

*process merging - 141

- sending to the solr server 532

A resourceMap:
1. Create a index task queue 233
2. Fetch a task from the queue 12
3. Total processing time 1652

- SolrIndexService.processObject - 980

*process system metadata - 38

* process by ResourceMapSubprocessor.processResourceMap - 416

ResourceMapFactory.buildResourceMap() create ResourceMap from Document, 95

ResourceMap.getAllDocumentIDs() referenced in ResourceMap, 134

ResourceMapSubprocessor.clearSidChain() removing obsoletes chain from Solr index, 99

HttpService.getDocumentsById() get existing referenced ids' Solr docs, 26

*process by RdfXmlSubprocessor.processDocument() 400

RdfXmlSubprocess.process gets a dataset from tripe store service , 116

RdfXmlSubprocess.process adds ont-model , 21

RdfXmlSubprocess.process process the fields total , 238 (prov stuff)

*process merging - 125

- sending to the solr server 661

History

#1 Updated by Jing Tao over 4 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100
  • translation missing: en.field_remaining_hours set to 0.0

#2 Updated by Jing Tao over 4 years ago

  • Category set to d1_cn_index_processor

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)