Bug #7957
Incorrect or missing PID escaping when deleting records from Solr
100%
Description
The CN is not correctly escaping the PID before passing it to the indexer we use (Apache Solr) in delete calls.
Example:
(HTTPService:writeError:255) <?xml version="1.1" encoding="utf-8"?>
doi:10.6067:XCV843020_meta&v=1483015376289
(TemporalPeriodSolrField:getFields:79) Couldn't extract 'start' or 'end' date for pid
doi:10.6067:XCV843020_meta&v=1483015376289. Temporal pattern of type period needs to contain at least one of these.
Value was:
History
#1 Updated by Roger Dahl almost 8 years ago
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,411 (HTTPService:writeError:253) URL: http://localhost:8983/solr/search_core/update?commit=true
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,411 (HTTPService:writeError:254) Post:
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,411 (HTTPService:writeError:255) <?xml version="1.1" encoding="utf-8"?>
./index/cn-index-processor-daemon.log.2:doi:10.6067:XCV843020_meta&v=1483015376289
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,412 (HTTPService:writeError:256)
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-Response:
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,412 (HTTPService:writeError:253) URL: http://localhost:8983/solr/search_core/update?commit=true
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,412 (HTTPService:writeError:254) Post:
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,412 (HTTPService:writeError:255) <?xml version="1.1" encoding="utf-8"?>
./index/cn-index-processor-daemon.log.2-doi:10.6067:XCV843019_meta&v=1482979301264
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,412 (HTTPService:writeError:256)
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-Response:
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-[ INFO] 2016-12-29 13:31:39,418 (IndexTaskProcessor:saveTask:794) IndexTaskProcess.saveTask save the index task testV1CreateV2ListObjects_20163400441223
./index/cn-index-processor-daemon.log.2-[ INFO] 2016-12-29 13:31:39,419 (IndexTaskProcessor:getNextIndexTask:579) Task for pid: testV1CreateV2ListObjects_20163400441223 not processed since the object path is not ready.
./index/cn-index-processor-daemon.log.2-[ INFO] 2016-12-29 13:31:39,419 (IndexTaskProcessor:processFailedIndexTaskQueue:211) IndexTaskProcessor.processFailedIndexTaskQueue with size 0
./index/cn-index-processor-daemon.log.2-[ERROR] 2016-12-29 13:31:39,421 (HTTPService:writeError:259) <?xml version="1.0" encoding="UTF-8"?>
./index/cn-index-processor-daemon.log.2-
./index/cn-index-processor-daemon.log.2-500311[com.ctc.wstx.exc.WstxLazyException] Unexpected character '=' (code 61); exp
ected a semi-colon after the reference for entity 'v'
./index/cn-index-processor-daemon.log.2- at [row,col {unknown-source}]: [2,41][com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '=' (code 61); expected a semi-c
olon after the reference for entity 'v'
./index/cn-index-processor-daemon.log.2- at [row,col {unknown-source}]: [2,41]
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:45)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:671)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3505)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:804)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:363)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:281)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:177)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:98)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
./index/cn-index-processor-daemon.log.2- at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.Server.handle(Server.java:497)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
./index/cn-index-processor-daemon.log.2- at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
./index/cn-index-processor-daemon.log.2- at java.lang.Thread.run(Thread.java:745)
./index/cn-index-processor-daemon.log.2-Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character '=' (code 61); expected a semi-colon after the reference for entity 'v'
./index/cn-index-processor-daemon.log.2- at [row,col {unknown-source}]: [2,41]
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:600)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.StreamScanner.parseEntityName(StreamScanner.java:1936)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1438)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4496)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:3964)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3543)
./index/cn-index-processor-daemon.log.2- at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3503)
./index/cn-index-processor-daemon.log.2- ... 32 more
./index/cn-index-processor-daemon.log.2-500
./index/cn-index-processor-daemon.log.2-
#2 Updated by Jing Tao almost 8 years ago
- Target version set to CCI-2.3.1
#3 Updated by Jing Tao almost 8 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
The pid in the solr delete command (xml format) was not escaped. After escaped the id, it works well. The fixed code was committed to the d1_cn_index_processor trunk and 2.3 branch.
