DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2017-10-24T16:09:06ZDataONE Tasks
Redmine Infrastructure - Story #8208 (New): Review and adjust memory allocated to d1-index-task-generator...https://redmine.dataone.org/issues/82082017-10-24T16:09:06ZDave Vieglaisdave.vieglais@gmail.com
<p>Configuration for these services is located at:<br>
<br>
/etc/default/d1-index-task-generator<br>
/etc/default/d1-index-task-processor</p>
<p>d1-index-task-generator is probably fine, however d1-index-task-processor may benefit from increased allocation especially for processing resource maps.</p>
Infrastructure - Story #8207 (New): review and adjust memory allocated to d1-processinghttps://redmine.dataone.org/issues/82072017-10-24T16:06:24ZDave Vieglaisdave.vieglais@gmail.com
<p>d1-processing settings are under @/etc/default/d1-processing@</p>
<p>Currently runs like: <br>
<br>
jsvc.exec -home /usr/lib/jvm/java-1.8.0-openjdk-amd64 -cp /usr/share/java/commons-daemon.jar:/usr/share/j<br>
ava/d1_process_daemon.jar -debug -outfile /var/log/dataone/daemon/d1-processing-jsvc.log -errfile /var/log/dataone/daemon/d1-process<br>
ing-jsvc.err -pidfile /var/run/d1-processing.pid -user tomcat7 -Djava.awt.headless=true -XX:UseParallelGC -Xmx4096M -Xms1024M -Xss12<br>
80k -XX:MaxPermSize=512M org.dataone.cn.batch.daemon.SchedulerDaemon</p>
Infrastructure - Story #8204 (New): Adjust memory allocation for services running under JVM on CNshttps://redmine.dataone.org/issues/82042017-10-24T15:29:28ZDave Vieglaisdave.vieglais@gmail.com
<p>Most services on CNs run in JVM instances, each with separate restrictions on memory use.</p>
<p>Current configurations were mostly based on defaults and have not changed much despite significantly higher memory use for systems such as Solr and Hazelcast.</p>
<p>The goal of this story is to evaluate the memory configuration of each JVM instance on the CNs and tune as necessary with the primary goal of avoiding out of memory errors, secondary goal is improving performance.</p>
<p>Note that with increased heap allocation, garbage collection may become a significant bottleneck, with application freeze in the order of several minutes possible or likely with higher heap allocations (e.g. 16GB)</p>
Infrastructure - Story #8049 (In Progress): Support synchronization of system metadata for unhost...https://redmine.dataone.org/issues/80492017-03-21T05:34:38ZRob Nahfrnahf@epscor.unm.edu
<p>As part of mutable-content MN support, allow the MemberNode to keep the system metadata records for all resultant versions of its changeable entities. this allows them to keep accurate system metadata for every version even though they do not have the object bytes anymore for that version.</p>
<p>Benefits:<br>
1. MN does not orphan any objects<br>
2. MN can administer objects from past versions on their own MN. Adjust the access policy of all versions, for example.<br>
3. don't need to call cn.setObsoletedBy or leave that field empty.</p>
<p>Costs:<br>
1. requires new logic for indexing (possibly)<br>
2. requires new logic for registerSystemMEtadata (possibly)<br>
3. require new logic for synchronization </p>
<p>very similar to how we synchronize DATA objects, but don't trigger MN replication.</p>
Infrastructure - Story #7807 (New): cn.synchronize should support synchronization failure correct...https://redmine.dataone.org/issues/78072016-05-13T16:56:25ZRob Nahfrnahf@epscor.unm.edu
<p>cn.synchronize(session, identifier) works well for its original purpose (supporting MN-driven system metadata updates, and MN-driven push synchronization), but doesn't seem to work for manual synchronization failure workflows. The main problem is that the request can only be made by the MN itself (using the MN client certificate). </p>
<p>As we envision a centralized dashboard for monitoring failed synchronization items, how do we address this situation? </p>
<p>The synchronization processing queue needs both the pid and a nodeId from where to retrieve the object. the NodeId is not specified directly in the method call, but gleaned from the session by a reverse lookup from the certificate. (It uses the first node found in the NodeList where the Node.subject field matches the certificate subject).</p>
<p>Should we allow node.contactSubjects into the algorithm?<br>
Should we add nodeId as a parameter?</p>
Infrastructure - Story #7224 (New): push synchronization request status indicator: synchronizeSta...https://redmine.dataone.org/issues/72242015-06-18T08:30:42ZRob Nahfrnahf@epscor.unm.edu
<p>Push synchronization (cn.synchronize, mn.updateSystemMetadata) involves an end-user that might want to have an idea of how long until the queued action is going to take to complete. Something as simple as returning the place in line of the sync request might suffice as the indicator, or a more complete data packet, including the place in line and the queue velocity, could be attempted.</p>
<p>The real-world analogy for this kind of indictor is taking a number at the deli-counter: You don't know when you will be served, but you know how many people are in front of you. </p>
<p>This option is a separate call to the CN to check the status of the sync request, so that the current place in line is returned. The advantage of this is that if the velocity of synchronization changes, the interested party can call again and get an updated value - it has more diagnostic and monitoring power. This could lead to over-use, however.</p>
Infrastructure - Story #6883 (New): support for ESRI Shapefile object formathttps://redmine.dataone.org/issues/68832015-03-11T01:53:35ZMatthew Jonesjones@nceas.ucsb.edu
<p>We are missing shapefiles from the format list, and need to add them as a data type. The identifier of a shapefile is unclear, but here is the entry from UDFR:</p>
<p><a href="http://udfr.org/ontowiki/view/r/u1f590">http://udfr.org/ontowiki/view/r/u1f590</a></p>
<p>Also see details:</p>
<p><a href="http://en.wikipedia.org/wiki/Shapefile">http://en.wikipedia.org/wiki/Shapefile</a></p>
Member Nodes - Story #5895 (New): GBIF: Transition to productionhttps://redmine.dataone.org/issues/58952014-07-17T21:07:29ZRoger Dahldahl@unm.edu
<p>Tasks related to moving the new MN into production.</p>
Member Nodes - Story #5867 (New): GBIF: Registration in Production environmenthttps://redmine.dataone.org/issues/58672014-07-17T21:06:34ZRoger Dahldahl@unm.edu
<p>Verify that Log Records can be accessed only as allowed by the MN policy.</p>
Member Nodes - Story #5838 (New): GBIF: Testinghttps://redmine.dataone.org/issues/58382014-07-17T21:05:40ZRoger Dahldahl@unm.edu
<p>If a new MN is being developed from scratch, tickets in this section will track progress.</p>
Member Nodes - Story #5833 (In Progress): GBIF: Developinghttps://redmine.dataone.org/issues/58332014-07-17T21:05:32ZRoger Dahldahl@unm.edu
<p>Determine which software stack to use, etc.</p>
Infrastructure - Story #4091 (New): ESRI GeoPortal MN stackhttps://redmine.dataone.org/issues/40912013-10-15T13:36:56ZBruce Wilsonbwilso27@utk.edu
<p>The objective is to design, develop, and implement a MN Stack to integrate with the ESRI GeoPortal server (<a href="http://www.esri.com/software/arcgis/geoportal">http://www.esri.com/software/arcgis/geoportal</a>).</p>
Infrastructure - Story #4052 (In Progress): OPeNDAP MN Storyhttps://redmine.dataone.org/issues/40522013-10-06T20:07:50ZBruce Wilsonbwilso27@utk.edu
<p>There are a large number of possible DataONE MN's running Data Access Protocol (DAP) compliant servers, particularly servers based on the OPeNDAP Hyrax and UCAR THREDDS servers. The objective of this work is to develop a MN stack that can be used with at least one of these DAP-compliant software stacks as a low-barrier route to becoming a Tier 1 MN.</p>
Infrastructure - Story #2548 (New): recasting untrusted certs to public poses accessibility incon...https://redmine.dataone.org/issues/25482012-03-27T21:55:59ZRob Nahfrnahf@epscor.unm.edu
<p>KNB recasts a connection with an untrusted certificate to public, so that a client does not get "less than public" privileges.<br>
GMN throws an InvalidToken in this situation.<br>
both refuse connections from clients with expired certificates from trusted CAs.</p>
<p>This approach can cause confusion caused when the user unwittingly uses an untrusted certficate and doesn't get what they expected. If these connections were instead refused, the user would be alerted and could reconnect as a public user, if it chose.</p>
<p>brief discussion found at line 97 of : <a href="http://epad.dataone.org/20120131-authn-authz-questions">http://epad.dataone.org/20120131-authn-authz-questions</a></p>
<ul>
<li>when would honest users be in this situation?</li>
<li>elicit advantages of recasting approach</li>
<li>either way, dataone should implement uniform behavior across CN and MNs.</li>
</ul>
DataONE API - Story #1644 (New): Develop an object format creation policyhttps://redmine.dataone.org/issues/16442011-06-14T16:25:11ZChris Jonescjones@nceas.ucsb.edu
<p>The object format list in d1_common_java is thus far an ad hoc list of known object formats needed in the D1 software. Additions will be needed. We need to develop a policy on who will have write access to the realtime version of this list, when the on-disk version will be periodically updated, etc. New object formats need to be vetted, and that process should be put into place. This process should align with the object format creation process with the UDFR group when their registry is operational.</p>