DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2019-11-22T18:31:19ZDataONE Tasks
Redmine Infrastructure - Story #8856 (New): Put the system metadata part ahead of the object part when d1...https://redmine.dataone.org/issues/88562019-11-22T18:31:19ZJing Taotao@nceas.ucsb.edu
<p>When a client calls the mn.create/update methods, it usually constructs a multipart which contains the sys part (containing the system metadata information), object part (containing the object itself) and other parts. There is no requirement about the order of those parts.<br>
Metacat will use a new streaming multipart handler which will calculate the checksum when it stores the object part into a file. This requires we should know the checksum algorithm before the serialization of the object. So Metacat has to digest the system metadata first in order to improve the performance.<br>
In order to take the advantage, we recommend clients should put the system metadata part ahead of the object part when it is constructing the multipart to be sent to the server.<br>
Note: event though the client doesn't use the recommended order, the process still works but the performance will be poor.</p>
Infrastructure - Story #8736 (New): Decouple the index generator to the hazelcast system metadata...https://redmine.dataone.org/issues/87362018-10-23T20:01:34ZJing Taotao@nceas.ucsb.edu
<p>Currently, the index generator is a listener of the system metadata map. During the tomcat starting, the map will load data from the db table and will generate a lot of unneeded index tasks. Currently we try to use filters to filter out them but it is very expensive. </p>
CN REST - Story #8364 (In Progress): Ensure portal uses correct X509 certificateshttps://redmine.dataone.org/issues/83642018-02-13T20:17:25ZChris Jonescjones@nceas.ucsb.edu
<p>We've run into issues where after an upgrade of the <code>dataone-cn-portal</code> package on the CNs, the properties pointing to the public certificate and private key are incorrectly pointing to the old GeoTrust wildcard files rather than the new Lets Encrypt files:<br>
<br>
cn.server.publiccert.filename=/etc/ssl/certs/<em>.test.dataone.org.crt<br>
cn.server.privatekey.filename=/etc/ssl/private/</em>.test.dataone.org.key</p>
<p>These should be (in STAGE):</p>
<p>/etc/letsencrypt/live/cn-stage.test.dataone.org/cert.pem<br>
/etc/letsencrypt/live/cn-stage.test.dataone.org/privkey.pem</p>
<p>The issue might be that these are not being set correctly during the <code>postinst</code> script run. Jing pointed out that these values are taken from the debconf database settings that get set when <code>dataon-cn-os-core</code> is installed. So although the <code>postinst</code> script might be setting the correct values, the old cached values might still be in memory in the debconf database. If so, we'll need to clear those values during installations and upgrades.</p>
<p>Also, knowing where to look for these configuration settings can be challenging. These are referenced from <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code>. These settings should be consolidated into <code>/etc/dataone/portal/portal.properties</code> so they also don't get blown away on war file upgrades in Tomcat.</p>
Testing MN Management - Story #8352 (New): Move to Productionhttps://redmine.dataone.org/issues/83522018-02-08T15:28:13ZMonica Ihliemail@monicaihli.comTesting MN Management - Story #8347 (New): Testing & Developmenthttps://redmine.dataone.org/issues/83472018-02-08T15:28:11ZMonica Ihliemail@monicaihli.com
<p>Install or develop a functional member node to be registered to a non-production environment. </p>
Testing MN Management - Story #8340 (New): Discoveryhttps://redmine.dataone.org/issues/83402018-02-08T15:28:09ZMonica Ihliemail@monicaihli.com
<p>Discovery is about establishing contact and building a relationship with a potential new member node. In this phase, it is determined if DataONE and the repository are a good fit for one another and if the repository generally meets the requirements of DataONE member nodes. Broad discussions of deployment options may be reviewed as well.<br>
This story is complete when a determination is made to either proceed with planning a new deployment, or that joining DataONE is not an option for the repository at this time.</p>
Infrastructure - Story #8207 (New): review and adjust memory allocated to d1-processinghttps://redmine.dataone.org/issues/82072017-10-24T16:06:24ZDave Vieglaisdave.vieglais@gmail.com
<p>d1-processing settings are under @/etc/default/d1-processing@</p>
<p>Currently runs like: <br>
<br>
jsvc.exec -home /usr/lib/jvm/java-1.8.0-openjdk-amd64 -cp /usr/share/java/commons-daemon.jar:/usr/share/j<br>
ava/d1_process_daemon.jar -debug -outfile /var/log/dataone/daemon/d1-processing-jsvc.log -errfile /var/log/dataone/daemon/d1-process<br>
ing-jsvc.err -pidfile /var/run/d1-processing.pid -user tomcat7 -Djava.awt.headless=true -XX:UseParallelGC -Xmx4096M -Xms1024M -Xss12<br>
80k -XX:MaxPermSize=512M org.dataone.cn.batch.daemon.SchedulerDaemon</p>
Infrastructure - Story #8049 (In Progress): Support synchronization of system metadata for unhost...https://redmine.dataone.org/issues/80492017-03-21T05:34:38ZRob Nahfrnahf@epscor.unm.edu
<p>As part of mutable-content MN support, allow the MemberNode to keep the system metadata records for all resultant versions of its changeable entities. this allows them to keep accurate system metadata for every version even though they do not have the object bytes anymore for that version.</p>
<p>Benefits:<br>
1. MN does not orphan any objects<br>
2. MN can administer objects from past versions on their own MN. Adjust the access policy of all versions, for example.<br>
3. don't need to call cn.setObsoletedBy or leave that field empty.</p>
<p>Costs:<br>
1. requires new logic for indexing (possibly)<br>
2. requires new logic for registerSystemMEtadata (possibly)<br>
3. require new logic for synchronization </p>
<p>very similar to how we synchronize DATA objects, but don't trigger MN replication.</p>
Infrastructure - Story #7807 (New): cn.synchronize should support synchronization failure correct...https://redmine.dataone.org/issues/78072016-05-13T16:56:25ZRob Nahfrnahf@epscor.unm.edu
<p>cn.synchronize(session, identifier) works well for its original purpose (supporting MN-driven system metadata updates, and MN-driven push synchronization), but doesn't seem to work for manual synchronization failure workflows. The main problem is that the request can only be made by the MN itself (using the MN client certificate). </p>
<p>As we envision a centralized dashboard for monitoring failed synchronization items, how do we address this situation? </p>
<p>The synchronization processing queue needs both the pid and a nodeId from where to retrieve the object. the NodeId is not specified directly in the method call, but gleaned from the session by a reverse lookup from the certificate. (It uses the first node found in the NodeList where the Node.subject field matches the certificate subject).</p>
<p>Should we allow node.contactSubjects into the algorithm?<br>
Should we add nodeId as a parameter?</p>
Infrastructure - Story #7224 (New): push synchronization request status indicator: synchronizeSta...https://redmine.dataone.org/issues/72242015-06-18T08:30:42ZRob Nahfrnahf@epscor.unm.edu
<p>Push synchronization (cn.synchronize, mn.updateSystemMetadata) involves an end-user that might want to have an idea of how long until the queued action is going to take to complete. Something as simple as returning the place in line of the sync request might suffice as the indicator, or a more complete data packet, including the place in line and the queue velocity, could be attempted.</p>
<p>The real-world analogy for this kind of indictor is taking a number at the deli-counter: You don't know when you will be served, but you know how many people are in front of you. </p>
<p>This option is a separate call to the CN to check the status of the sync request, so that the current place in line is returned. The advantage of this is that if the velocity of synchronization changes, the interested party can call again and get an updated value - it has more diagnostic and monitoring power. This could lead to over-use, however.</p>
Infrastructure - Story #6883 (New): support for ESRI Shapefile object formathttps://redmine.dataone.org/issues/68832015-03-11T01:53:35ZMatthew Jonesjones@nceas.ucsb.edu
<p>We are missing shapefiles from the format list, and need to add them as a data type. The identifier of a shapefile is unclear, but here is the entry from UDFR:</p>
<p><a href="http://udfr.org/ontowiki/view/r/u1f590">http://udfr.org/ontowiki/view/r/u1f590</a></p>
<p>Also see details:</p>
<p><a href="http://en.wikipedia.org/wiki/Shapefile">http://en.wikipedia.org/wiki/Shapefile</a></p>
Member Nodes - Story #5833 (In Progress): GBIF: Developinghttps://redmine.dataone.org/issues/58332014-07-17T21:05:32ZRoger Dahldahl@unm.edu
<p>Determine which software stack to use, etc.</p>
Infrastructure - Story #4091 (New): ESRI GeoPortal MN stackhttps://redmine.dataone.org/issues/40912013-10-15T13:36:56ZBruce Wilsonbwilso27@utk.edu
<p>The objective is to design, develop, and implement a MN Stack to integrate with the ESRI GeoPortal server (<a href="http://www.esri.com/software/arcgis/geoportal">http://www.esri.com/software/arcgis/geoportal</a>).</p>
Infrastructure - Story #4052 (In Progress): OPeNDAP MN Storyhttps://redmine.dataone.org/issues/40522013-10-06T20:07:50ZBruce Wilsonbwilso27@utk.edu
<p>There are a large number of possible DataONE MN's running Data Access Protocol (DAP) compliant servers, particularly servers based on the OPeNDAP Hyrax and UCAR THREDDS servers. The objective of this work is to develop a MN stack that can be used with at least one of these DAP-compliant software stacks as a low-barrier route to becoming a Tier 1 MN.</p>
Infrastructure - Story #2548 (New): recasting untrusted certs to public poses accessibility incon...https://redmine.dataone.org/issues/25482012-03-27T21:55:59ZRob Nahfrnahf@epscor.unm.edu
<p>KNB recasts a connection with an untrusted certificate to public, so that a client does not get "less than public" privileges.<br>
GMN throws an InvalidToken in this situation.<br>
both refuse connections from clients with expired certificates from trusted CAs.</p>
<p>This approach can cause confusion caused when the user unwittingly uses an untrusted certficate and doesn't get what they expected. If these connections were instead refused, the user would be alerted and could reconnect as a public user, if it chose.</p>
<p>brief discussion found at line 97 of : <a href="http://epad.dataone.org/20120131-authn-authz-questions">http://epad.dataone.org/20120131-authn-authz-questions</a></p>
<ul>
<li>when would honest users be in this situation?</li>
<li>elicit advantages of recasting approach</li>
<li>either way, dataone should implement uniform behavior across CN and MNs.</li>
</ul>