Project

General

Profile

Bug #7816

After reindex cn-dev, there are more than 20,000 index tasks keep the "IN PROCESS" status

Added by Jing Tao almost 8 years ago. Updated almost 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
d1_cn_index_processor
Start date:
2016-05-23
Due date:
% Done:

100%

Story Points:
Sprint:

Description

When the index finished, it should be failed or removed from the index_task table. However, there are 20,000 index tasks keep "IN PROCESS" status.

History

#1 Updated by Jing Tao almost 8 years ago

  • Target version set to CCI-2.3.0
  • Assignee set to Jing Tao

After further investigation, we found all of the documents come from the two type:
http://www.isotc211.org/2005/gmd
http://www.isotc211.org/2005/gmd-noaa

#2 Updated by Jing Tao almost 8 years ago

Example ids:
iso19119_looselyCoupled_201611154258815
{A2AE9572-A192-4FFF-9A28-3D2FD0387FF0}

#3 Updated by Jing Tao almost 8 years ago

It turned out that the convert property on the distributServiceType blocked the index without any error message:




The serviceType bean has a converter as well. So duplicated converter doesn't work, I guess.

It is weird the 2.1 branch has the converter, but there is no index_task with "IN PROCESS" status for http://www.isotc211.org/2005/gmd or http://www.isotc211.org/2005/gmd-noaa on both sandbox and stage.

#5 Updated by Jing Tao almost 8 years ago

After further investigation, I found the problem is not the context file. The problem is in the org.dataone.cn.indexer.convert.MemberNodeServiceRegistrationTypeDocumentService:

public Document getMemberNodeServiceRegistrationTypeDocument() {
Document doc = null;
InputStream stream = fetchServiceTypeDoc();
if (stream != null) {
try {
doc = XmlDocumentUtility.generateXmlDocument(stream);
} catch (SAXException e) {
log.error("Unable to create w3c Document from input stream", e);
e.printStackTrace();
} finally {
IOUtils.closeQuietly(stream);
}
}
return doc;
}

The IOUtils class was imported from org.apache.solr.common.util.IOUtils. Now it doesn't work any more. When I change it to org.apache.commons.io.IOUtils. It works.

The 2.1 branch of d1_cn_index_processor has the same issue. However, it works. I guess it is caused by different version of solr jar file.

#6 Updated by Jing Tao almost 8 years ago

  • Target version changed from CCI-2.3.0 to CCI-2.2.0
  • Status changed from New to Closed
  • % Done changed from 0 to 100

The ticket is moved to 2.2 release.

Since we will create d1_cn_index_processor branch 2.2 from branch 2.1. So the file src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java should be copied from the trunk to branch 2.2.

I compared the file on the trunk with the branch 2.1 and found the only change is the import part:

svn diff src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java https://repository.dataone.org/software/cicore/branches/D1_CN_INDEX_PROCESSOR_v2.1/src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java

Index: src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java

--- src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java (.../trunk/cn/d1_cn_index_processor/src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java) (working copy)
+++ src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java (.../branches/D1_CN_INDEX_PROCESSOR_v2.1/src/main/java/org/dataone/cn/indexer/convert/MemberNodeServiceRegistrationTypeDocumentService.java) (revision 18015)
@@ -24,7 +24,6 @@
import java.io.IOException;
import java.io.InputStream;

-import org.apache.commons.io.IOUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
@@ -31,6 +30,7 @@
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.log4j.Logger;
+import org.apache.solr.common.util.IOUtils;
import org.dataone.cn.indexer.XmlDocumentUtility;
import org.dataone.configuration.Settings;
import org.springframework.beans.factory.annotation.Autowired;

#7 Updated by Jing Tao almost 8 years ago

  • Project changed from Infrastructure to CN Index
  • Category set to d1_cn_index_generator

#8 Updated by Jing Tao almost 8 years ago

  • Category changed from d1_cn_index_generator to d1_cn_index_processor

#9 Updated by Jing Tao almost 8 years ago

  • % Done changed from 100 to 30
  • Status changed from Closed to In Progress

We need to merge it to the 2.2 branch.

#10 Updated by Jing Tao almost 8 years ago

  • % Done changed from 30 to 100
  • Status changed from In Progress to Closed

Merge the change to the 2.2 branch.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)