https://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2015-08-26T20:14:59ZDataONE TasksInfrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=259512015-08-26T20:14:59ZMark Servillamark.servilla@gmail.com
<ul><li><strong>Subject</strong> changed from <i>mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PID</i> to <i>mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDs</i></li></ul> Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=260052015-09-09T16:53:21ZRob Nahfrnahf@epscor.unm.edu
<ul><li><strong>Assignee</strong> set to <i>Rob Nahf</i></li></ul><p>Is the metacat deployed to stage using the v2 dataone libclient?</p>
<p>ObjectPathMap.load(Identiifer) method relies on ReplicationService.getDocumentInfoMap() to function, and swallows all exceptions thrown from that service method, logging those exceptions only when log.debug is enabled.</p>
<p>My hunch is that there is a problem with ReplicationService.getDocumentInfoMap(). Code for ObjectPathMap has been stable since 2012. ReplicationService has had many commits in 2014-15, the latest one in May, 2015. A lot related to use of new HttpClient.</p>
Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=260072015-09-09T17:34:38ZRob Nahfrnahf@epscor.unm.edu
<ul></ul><p>we should probably enable debugging on ObjectPathMap class to check if something is wrong with ReplicationService.</p>
<p>And then refactor the logging to Info or Warn, because we're relying on a service outside the package. There's no logging in ReplicationService....</p>
<pre>/**
* Implementation of hazelcast MapLoader interface method.
* For the provided Identifier (as key), returns the path to the
* document on the local filesystem. Returns null if it can't
* create the path.
*/
@Override
public String load(Identifier key)
{
String docid = null;
String path = null;
try {
docid = im.getLocalId(key.getValue());
path = pathToDocid(docid);
} catch (Exception e) {
if (logMetacat.isDebugEnabled()) {
e.printStackTrace();
}
return null;
}
return path;
}
</pre> Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=260372015-09-14T18:35:08ZDave Vieglaisdave.vieglais@gmail.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-9 status-9 priority-5 priority-high3 parent" href="/issues/6544">MNDeployment #6544</a>: TERN Australia (powered by AEKOS)</i> added</li></ul> Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=261722015-09-29T14:42:08ZMark Servillamark.servilla@gmail.com
<ul></ul><p>This same issue is affecting urn:node:mnTestNRDC1 on cn-stage.test.dataone.org:</p>
<p>[ INFO] 2015-09-28 08:06:13,601 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_6<br>
0326--v1.xml<br>
[ INFO] 2015-09-28 08:06:13,707 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60<br>
326--v1.xml is not available. Task will be retried.<br>
[ INFO] 2015-09-28 08:06:13,711 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.<br>
xml not processed.</p>
Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=262582015-10-09T21:17:36ZMark Servillamark.servilla@gmail.com
<ul></ul><p>Attempted re-indexing of two mnTestNRDC1 objects:</p>
<a name="NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1xml"></a>
<h1 >NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml<a href="#NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1xml" class="wiki-anchor">¶</a></h1>
<a name="NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1xml"></a>
<h1 >NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml<a href="#NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1xml" class="wiki-anchor">¶</a></h1>
<p>Indexing again failed due to "Object path ... is not available" error.</p>
<p>[ INFO] 2015-10-09 21:10:32,942 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml<br>
[ INFO] 2015-10-09 21:10:33,854 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml is not available. Task will be retried.<br>
[ INFO] 2015-10-09 21:10:33,861 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml not processed.<br>
[ INFO] 2015-10-09 21:10:33,865 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml<br>
[ INFO] 2015-10-09 21:10:33,939 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml is not available. Task will be retried.<br>
[ INFO] 2015-10-09 21:10:33,945 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml not processed.</p>
Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=262592015-10-09T21:28:51ZMark Servillamark.servilla@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/download/396/opnf.txt">opnf.txt</a><a href="/attachments/396/opnf.txt"><img src="/images/magnifier.png" alt="Magnifier" /></a> added</li></ul><p>Performing a simple grep on cn-stage.test.dataone.org (cn-stage-ucsb-1) for "Object path for pid" (unique string for this error) in /var/log/dataone/index/cn-index-processor-daemon.log.* reveals 63 occurrences from both GMN and Metacat MNs in less than a single day - see attached file. I <em>have not verified</em> existence of complete synchronization for all listed objects.</p>
Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=262742015-10-12T19:56:02ZLaura Moyerslmoyers1@utk.edu
<ul><li><strong>Related to</strong> <i><a class="issue tracker-9 status-9 priority-4 priority-default parent" href="/issues/6957">MNDeployment #6957</a>: NRDC - Nevada Research Data Center</i> added</li></ul> Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=262762015-10-12T19:56:16ZLaura Moyerslmoyers1@utk.edu
<ul><li><strong>Subject</strong> changed from <i>mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDs</i> to <i>Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDs</i></li></ul><p>NRDC (in stage) has been identified as having this problem. Content is successfully synchronized but not indexed. It is possible that the autogen document isn't being generated, so indexing cannot occur. Sample messages follow: </p>
<pre>[ INFO] 2015-10-09 21:10:32,942 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml
[ INFO] 2015-10-09 21:10:33,854 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml is not available. Task will be retried.
[ INFO] 2015-10-09 21:10:33,861 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml not processed.
[ INFO] 2015-10-09 21:10:33,865 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml
[ INFO] 2015-10-09 21:10:33,939 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml is not available. Task will be retried.
[ INFO] 2015-10-09 21:10:33,945 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml not processed.
</pre>
<p>This is occurring <em>across environments</em> and for multiple MNs, but it is not the same content which is not indexed in the different environments. </p>
Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=262872015-10-13T23:14:37ZBen Leinfelderleinfelder@nceas.ucsb.edu
<ul></ul><p>A number of us worked to track this down (Jing, Skye, Chris) and saw that we could index ltern.21.28 on cn-stage-ucsb-1 when the other two CNs were offline (Tomcat stopped). This is because the object had not [Metacat] replicated to the other two CNs in the environment, but the Object Path Map (Hazelcast) is shared amongst the three when they are all running. There's no telling which CN will 'lookup' the path for the given pid and we were getting the null path for this object (either unm or orc).<br>
Then I noticed that many of the certificates had expired, particularly the ones used for Metacat replication from CN-to-CN. Chris generated new ones and I have installed them and begun a forced replication for all content in that environment. Not sure how long that will take, but new content that is harvested should behave as expected. Let us know if not!</p>
Infrastructure - Bug #7318: Indexing failure with ObjectPath not found / Was: mnTestAEKOS PIDs fail to be indexed by Solr in cn-stage due to lack of "Object Path" for the PIDshttps://redmine.dataone.org/issues/7318?journal_id=277552016-06-13T23:22:23ZRob Nahfrnahf@epscor.unm.edu
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>Assume there were no problems upon reindexing.</p>