we should probably enable debugging on ObjectPathMap class to check if something is wrong with ReplicationService.

And then refactor the logging to Info or Warn, because we're relying on a service outside the package. There's no logging in ReplicationService....

/**
 *  Implementation of hazelcast MapLoader interface method.
 *  For the provided Identifier (as key), returns the path to the
 *  document on the local filesystem.  Returns null if it can't 
 *  create the path. 
 */
@Override
public String load(Identifier key) 
{

    String docid = null;
    String path = null;
    try {
        docid = im.getLocalId(key.getValue());
        path = pathToDocid(docid);          
    } catch (Exception e) {
        if (logMetacat.isDebugEnabled()) {
                         e.printStackTrace();
                   }
                   return null;
    }
    return path;
}

This same issue is affecting urn:node:mnTestNRDC1 on cn-stage.test.dataone.org:

[ INFO] 2015-09-28 08:06:13,601 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_6
0326--v1.xml
[ INFO] 2015-09-28 08:06:13,707 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60
326--v1.xml is not available. Task will be retried.
[ INFO] 2015-09-28 08:06:13,711 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.
xml not processed.

Attempted re-indexing of two mnTestNRDC1 objects:

NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml¶

NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml¶

Indexing again failed due to "Object path ... is not available" error.

[ INFO] 2015-10-09 21:10:32,942 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml
[ INFO] 2015-10-09 21:10:33,854 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml is not available. Task will be retried.
[ INFO] 2015-10-09 21:10:33,861 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_04_4241_60326--v1.xml not processed.
[ INFO] 2015-10-09 21:10:33,865 (IndexTaskProcessor:getNextIndexTask:155) Start of indexing pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml
[ INFO] 2015-10-09 21:10:33,939 (IndexTaskProcessor:isObjectPathReady:263) Object path for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml is not available. Task will be retried.
[ INFO] 2015-10-09 21:10:33,945 (IndexTaskProcessor:getNextIndexTask:164) Task for pid: NRDC_NCCP_SCIENCE_METADATA_Spring1_Met_OneMin_2011_05_4241_60326--v1.xml not processed.

A number of us worked to track this down (Jing, Skye, Chris) and saw that we could index ltern.21.28 on cn-stage-ucsb-1 when the other two CNs were offline (Tomcat stopped). This is because the object had not [Metacat] replicated to the other two CNs in the environment, but the Object Path Map (Hazelcast) is shared amongst the three when they are all running. There's no telling which CN will 'lookup' the path for the given pid and we were getting the null path for this object (either unm or orc).
Then I noticed that many of the certificates had expired, particularly the ones used for Metacat replication from CN-to-CN. Chris generated new ones and I have installed them and begun a forced replication for all content in that environment. Not sure how long that will take, but new content that is harvested should behave as expected. Let us know if not!