https://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2019-02-21T23:03:10ZDataONE TasksCN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=312252019-02-21T23:03:10ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Since those events should be filtered out. So maybe we just delete them? The criteria is the subject is a CN and IP address is 10.42.x.x.</p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=312262019-02-21T23:12:38ZChris Jonescjones@nceas.ucsb.edu
<ul></ul><p>I think it's fine to delete them Jing, since we know they are CN events. They can be deleted from Elastic Search as well, so ask Rushi or Dave about that if need be.</p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=312272019-02-22T00:12:40ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Run this query<br>
<code>curl "http://localhost:8983/solr/event_core/select?q=subject:CN=urn\:node\:CN*%20AND%20ipAddress:10.42.*&fl=subject,ipAddress"</code></p>
<p>It returned 32642 records.</p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=312282019-02-22T03:02:22ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Proposed three three delete command:</p>
<p><code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete>(subject:CN=urn\:node\:CNUCSB1*)AND(ipAddress:10.42*)</delete>"</code></p>
<p><code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete>(subject:CN=urn\:node\:CNORC1*)AND(ipAddress:10.42*)</delete>"</code></p>
<p><code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete>(subject:CN=urn\:node\:CNUNM1*)AND(ipAddress:10.42*)</delete>"</code></p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=312312019-02-22T18:36:50ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>This query <code>curl -d "q=(subject:CN=urn\:node\:CNUCSB1*)AND(ipAddress:10.42*)&fl=subject,ipAddress" http://localhost:8983/solr/event_core/select</code> returns 32462 records;</p>
<p><code>curl -d "q=(subject:CN=urn\:node\:CNORC1*)AND(ipAddress:10.42*)&fl=subject,ipAddress" http://localhost:8983/solr/event_core/select</code> returns 0 records.</p>
<p><code>curl -d "q=(subject:CN=urn\:node\:CNUNM*)AND(ipAddress:10.42*)&fl=subject,ipAddress" http://localhost:8983/solr/event_core/select</code> returns 180 records.</p>
<p>So the delete queries will totally remove 32,642 records. Chris, does it sounds reasonable number?</p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=313142019-04-19T17:31:21ZChris Jonescjones@nceas.ucsb.edu
<ul></ul><p>Hi Jing - we discussed this with ESS-DIVE yesterday, and it reminded me of this ticket - sorry for the delayed response.</p>
<p>I wanted to get a sense of how many <code>read</code> events your query entailed, so I issued this query:</p>
<pre>curl -d "q=(subject:CN=urn\:node\:CNUCSB1*)AND(ipAddress:10.42*)&rows=0&facet=true&facet.field=event&facet.limit=1000000" http://localhost:8983/solr/event_core/select | xmlstarlet fo
</pre>
<p>This summarizes the count of each event name, and we get:</p>
<pre><int name="updateSystemMetadata">79878</int>
<int name="read">13937</int>
<int name="synchronization_failed">221</int>
<int name="INSERT">0</int>
<int name="UPDATE">0</int>
<int name="create">0</int>
<int name="delete">0</int>
<int name="replicate">0</int>
<int name="unknown">0</int>
<int name="update">0</int>
<int name="upload">0</int>
</pre>
<p>So a large part of the query deletes <code>updateSystemMetadata</code> events and it also catches the <code>synchronization_failed</code> events. I don't think we want to delete those events since they are there for reference, but we also don't want them to have the wrong IP address.</p>
<p>To clean this up, I'd probably say your delete query should be <code>(subject:CN=urn\:node\:CNUCSB1*)AND(ipAddress:10.42*)AND(event:read)</code>, and then we probably want to update the remaining Solr documents where <code>subject:CN=urn\:node\:CNUCSB1*</code> and change the IP address to the actual IP address, and do the same for the other CN's records as well.</p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=313302019-05-01T20:06:34ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Proposed three three delete command:</p>
<p><code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete><query>(subject:CN=urn\:node\:CNUCSB1*)AND(ipAddress:10.42*)AND(event:read)</query></delete>"</code></p>
<p><code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete><query>(subject:CN=urn\:node\:CNORC1*)AND(ipAddress:10.42*)AND(event:read)</query></delete>"</code></p>
<p><code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete><query>(subject:CN=urn\:node\:CNUNM1*)AND(ipAddress:10.42*)AND(event:read)</query></delete>"</code></p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=313312019-05-01T20:54:30ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>This page give some information to update a document:<br>
<a href="https://solr.pl/en/2012/07/09/solr-4-0-partial-documents-update/">https://solr.pl/en/2012/07/09/solr-4-0-partial-documents-update/</a></p>
CN REST - Story #8749: Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/8749?journal_id=313332019-05-01T22:25:05ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Chris and I used the command to delete those records:<br>
<code>curl http://localhost:8983/solr/event_core/update/?commit=true -H "Content-Type: text/xml" -d "<delete><query>(subject:CN=urn\:node\:CN*)AND(ipAddress:10.42*)AND(event:read)</query></delete>"</code></p>