Infrastructure

Change indexing of provenance-related fields to use field names with a 'prov_' prefix, and drop the dynamic field suffixes. Also change hadExecution to two fields: prov_wasExecutedByExecution and prov_wasExecutedByUser. refs #6843

Rename hadExecution bean to prov.wasExecutedByExecution and prov.wasExecutedByUser. refs #6843

Modify the provRdfXmlSubprocessor bean to index the prov_hasDerivations field, which indexes science metadata documents that describe data entities that were sources of other data entities. Also fix the SPARQL query logic in the prov_hasSources bean. refs #6843

copy the provenance application context beans from d1_cn_index_processor to update the most recent field name changes in the SPARQL queries that get deployed. refs #6843

Pilot error - I missed adding this file to the last commit. refs #6843

Update the fixed SPARQL query in the buildout. refs #6843

Fix attribute name typo: multiValued, not multivalued. refs #6843

Fix attribute name typo: multiValued, not multivalued. refs #6843

Uncomment the provRdfXmlSubprocessor. refs #6843

Add minor debugging to help track down identifiers referenced in resource maps not found in the Solr index. refs #6843

Add the application-context-annotator bean definition into the tests to get the index processor tests working on jenkins. refs #6843

Modify the RdfXmlSubprocessor to use mergeWithIndexedDocuments() rather than mergeDocs() to do the merging of content with content already in the index. The SolrIndexService iterates through the results provided by processDocument(), and so will index all of the document identifiers extracted out of the triple statements found in the RDF/XML document. I've removed mergeDocs() (not needed now) and fleshed out mergeWithIndexedDocument(). refs #6843

Minor changes to indexing prov_used and prov_wasGeneratedBy. refs #6843

After troubleshooting issues that Lauren pointed out:

• Remove the prov_wasGeneratedBy Solr field (in favor of just prov_wasGeneratedByExecution and prov_wasGeneratedByProgram)
• Add the prov_generated field as the inverse, where a program generates an entity
• Update the testProvenanceFields() test to reflect the above
• For now, ignore the testInsertProvResourceMap() test due to HZ conflicts

refs #6843

• Remove the prov_wasGeneratedBy Solr field (in favor of just prov_wasGeneratedByExecution and prov_wasGeneratedByProgram)
• Add the prov_generated field as the inverse, where a program generates an entity

refs #6843

We've found a bug during index processing that causes documents to not be indexed because of a mismatch between a 'beginDate' or 'endDate' field being added, and an existing 'beginDate' or 'endDate' field. This is a temporary fix that converts potential date strings to Date objects, and compares them. If they match, it is a dupe, and we don't add the field. Otherwise we add it. See and refs #6843

Update the v1_1 Solr schema file with the prov_ field changes.

refs #6843

Update the other various versions of the Solr schema to reflect the prov_ changes, and add prov_generated and prov_used to the query field descriptions file. Again, add all prov_* to the list of default return fields forSolr queries.

refs #6843

Fix merging issues in the RdfxmlSubprocessor where, if mergeDocs() is called twice, and the existing map is larger than the pending map, we don't drop all of the existing documents, but rather add them to the merged map. Also, implement mergeWithIndexeddocument() in the same way as the AnnotatorSubprocessor (note that by following the API, we are cumulatively merging all existing docs, which seems a bit inefficient). refs #6843