DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2019-06-19T01:44:19ZDataONE Tasks
Redmine Infrastructure - Bug #8822 (New): Account queries in STAGE-2 failing from web browserhttps://redmine.dataone.org/issues/88222019-06-19T01:44:19ZBryce Mecummecum@nceas.ucsb.edu
<p>Steps to reproduce:</p>
<ol>
<li>Visit <a href="https://search-stage-2.test.dataone.org/">https://search-stage-2.test.dataone.org/</a></li>
<li>Log in</li>
<li>Navigate to <a href="https://search-stage-2.test.dataone.org/data">https://search-stage-2.test.dataone.org/data</a></li>
<li>Observe an HTTP 500 in the Network pane of whichever browser you're using to <a href="https://search-stage-2.test.dataone.org/cn/v2/accounts/?query=%7BYOUR_DN%7D">https://search-stage-2.test.dataone.org/cn/v2/accounts/?query={YOUR_DN}</a></li>
</ol>
<p>Response body:</p>
<pre><?xml version="1.0" encoding="UTF-8"?>
<error detailCode="500" errorCode="500" name="ServiceFailure">
<description>Internal Server Error: The server encountered an unexpected condition which prevented it from fulfilling the request.</description>
</error>
</pre>
<p>Some notes:</p>
<ul>
<li>I notice this in multiple browsers</li>
<li>I notice this only when the request is issued from MetacatUI, not when I visit the URL in my browser or hit it with curl</li>
<li>I don't notice this error on search.dataone.org</li>
<li>MetacatUI on search.dataone.org doesn't even issue this request</li>
<li>MetacatUI on stage-2 is at v2.4.2 and on search is at 2.6.1</li>
</ul>
<p>It seems like a bug to me that we see a service failure but I'm not sure if this is a MetacatUI bug or an issue in the CN stack (i.e., Apache config or something) but I wanted to file it for someone to take a look.</p>
CN REST - Task #8809 (New): Adjust portal.properties for certificate configurationhttps://redmine.dataone.org/issues/88092019-05-21T12:57:41ZDave Vieglaisdave.vieglais@gmail.com
<p>Portal certificates are apparently currently configured in <code>/var/lib/tomcat7/webapps/portal/WEB-INF/portal.properties</code></p>
<p>This should be changed to <code>/etc/dataone/portal/portal.properties</code> to ensure persistence between .war deployments.</p>
Infrastructure - Decision #8774 (New): Add new CN format for MPEG-2 or update video/mpeg format t...https://redmine.dataone.org/issues/87742019-03-08T02:04:18ZBryce Mecummecum@nceas.ucsb.edu
<p>We already have this format:</p>
<pre><objectFormat>
<formatId>video/mpeg</formatId>
<formatName>MPEG-1 Video</formatName>
<formatType>DATA</formatType>
<mediaType name="video/mpeg"/>
<extension>mpg</extension>
</objectFormat>
</pre>
<p>It clearly says MPEG-1 Video but there is also an MPEG-2 format. It'd be useful to have a proper format ID to use for MPEG-2 video. The info for MPEG-2 is largely the same as for MPEG-1 (format type, mediaType, extension). Should we add a new format or just update the description of the existing one? We also have an MP4 format type so maybe that'd be a reason to just add another to cover MPEG-2.</p>
Infrastructure - Story #8762 (New): Add new formats to CNhttps://redmine.dataone.org/issues/87622019-02-07T19:58:28ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>Add a new formatId for Python notebooks: application/x-ipynb+json. This is the current popular Media Type for Python notebooks (<a href="https://jupyter.readthedocs.io/en/latest/reference/mimetype.html">https://jupyter.readthedocs.io/en/latest/reference/mimetype.html</a>), but as it has the 'x-' prefix it is 'unregistered' and<br>
not recognized by IANA (<a href="https://www.iana.org/assignments/media-types/media-types.xhtml">https://www.iana.org/assignments/media-types/media-types.xhtml</a>).</p>
<p>The new entry should be:<br>
<objectFormat><br>
<formatId>application/x-ipynb+json</formatId><br>
<formatName>Jupyter Notebook</formatName><br>
<formatType>DATA</formatType><br>
<mediaType name="application/x-ipynb+json"/><br>
<extension>ipynb</extension><br>
</objectFormat></p>
DataONE API - Task #8529 (New): Add field to Solr index that includes the obsolescence chainhttps://redmine.dataone.org/issues/85292018-03-29T22:36:35ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>When a new PID is added to the Solr index, populate a field that contains the entire obsolescence chain for that PID. This will greatly reduce the amount of time clients have to spend on obtaining this information.</p>
<p>For each PID that precedes the new PID in the obs. chain, update it's Solr document to include the new, complete obs. chain.</p>
<p>The new Solr field that contains the obs. chain should be a multivalued field, if the order of PIDs in the field can be guaranteed. In this scenario, the firsts PID is the lowest in the obs. chain, the lasts pid in the field is the mosts recent, un-obsoleted PID.</p>
<p>If the order cannot be guaranteed in a multi-valued field, then a simple text field should be used, with a reasonable value as a delimiter, one that is not valid for PIDs, such as a blank.</p>
<p>This task is related to <a href="https://redmine.dataone.org/issues/8528">https://redmine.dataone.org/issues/8528</a></p>
DataONE API - Task #8528 (New): Add MNRead.getVersions, CNRead.getVersionshttps://redmine.dataone.org/issues/85282018-03-29T22:18:56ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>These new services would return the obsolescence chain, as a lists of PIDs, for the specified PID. <br>
Both services (for CN and MN) would have the same argument list, i.e. for MNREAD.getVersions:</p>
<p>MNRead.getVersions(Identifier pid, Integer count, String order)</p>
<p>where the arguments are<br>
- pid: the identifier to retrieve the obs. chain for<br>
- count: specifies how much of the lists to return<br>
- if <code>count</code> is <code>null</code> return the whole list<br>
- if <code>count</code> is a positive number, return <code>count</code> pids in the specified order of, say, <code>ASC</code> or <code>DESC</code><br>
- order<br>
- if order is <code>DESC</code>, return <code>count</code> from the <code>HEAD</code>, but from the <code>TAIL</code> if <code>ASC</code>.</p>
<p>Should the presence of multiple SIDs assigned in the obsolescence chain determine the lists of PIDs that is returned? For example, <br>
should the entire chain be returned even if multiple SIDs are encountered. The document at <a href="https://releases.dataone.org/online/api-documentation-v2.0/design/ContentMutability.html#limits-on-the-series">https://releases.dataone.org/online/api-documentation-v2.0/design/ContentMutability.html#limits-on-the-series</a> indicates that multiple SIDs can be defined in an obs.chain.</p>
<p>Should PIDs be included in the list if the requesting user does not have permission to read the sysmeta for that pid? If no, then how is this gap in the chain denoted?</p>
Infrastructure - Task #8362 (New): Add the indication that the cn account list is sliced https://redmine.dataone.org/issues/83622018-02-12T17:48:02ZJing Taotao@nceas.ucsb.edu
<p>Today, I registered an account on dataone network. But I didn't find it on this page:<br>
<a href="https://cn.dataone.org/cn/v2/accounts">https://cn.dataone.org/cn/v2/accounts</a></p>
<p>The reason is that it is a list, so can be sliced. We should include the page info in there like we did for other lists. Otherwise, makes it much harder to navigate.</p>
Infrastructure - Task #8205 (New): Adjust memory allocation to Tomcat7 on CNshttps://redmine.dataone.org/issues/82052017-10-24T15:33:33ZDave Vieglaisdave.vieglais@gmail.com
<p>Heap allocated to Tomcat7 on the CNs is currently configured for 12GB.</p>
<p>With the large number of System Metadata entries being stored in Hazelcast, we are starting to see issues of out of memory exceptions that seem to be triggered during periods of high activity.</p>
<p>Once the VM memory allocation has been increased, the maximum heap size allocation for Tomcat should be increased to 24GB.</p>
<p><em>Note:</em> A likely consequence of this will be extended periods of GC activity, so this needs to be carefully evaluated before deploying to production CNs, otherwise delays of potentially minutes long may occur during GC.</p>
Infrastructure - Story #8204 (New): Adjust memory allocation for services running under JVM on CNshttps://redmine.dataone.org/issues/82042017-10-24T15:29:28ZDave Vieglaisdave.vieglais@gmail.com
<p>Most services on CNs run in JVM instances, each with separate restrictions on memory use.</p>
<p>Current configurations were mostly based on defaults and have not changed much despite significantly higher memory use for systems such as Solr and Hazelcast.</p>
<p>The goal of this story is to evaluate the memory configuration of each JVM instance on the CNs and tune as necessary with the primary goal of avoiding out of memory errors, secondary goal is improving performance.</p>
<p>Note that with increased heap allocation, garbage collection may become a significant bottleneck, with application freeze in the order of several minutes possible or likely with higher heap allocations (e.g. 16GB)</p>
Infrastructure - Feature #8065 (New): Add service to re-populate Hazelcast map for node attributeshttps://redmine.dataone.org/issues/80652017-04-11T18:01:08ZMark Servillamark.servilla@gmail.com
<p>We are requesting a new administrative service that will allow the re-population of the Hazelcast map for node attributes obtained from LDAP. It is often the case that direct modification of node attributes in LDAP do not affect the operational state of the CN because attribute changes do not flow into the Hazelcast data structure map unless the node makes the update directly or D1 processing is restarted. It would be very helpful if a service existed that would re-populate node attributes in the Hz map.</p>
DUG - Task #7800 (New): abstract reviews (https://docs.google.com/document/d/1iZXCN4AWqYi5Z7wqMKb...https://redmine.dataone.org/issues/78002016-05-10T15:18:24ZDave Vieglaisdave.vieglais@gmail.comInfrastructure - Task #4675 (New): Administratively delete those pids with the white spaceshttps://redmine.dataone.org/issues/46752014-03-31T18:06:15ZJing Taotao@nceas.ucsb.edu
<p>This step should be done after finish task 4647. Do we have a tool to delete them? Or we will develop the tool?</p>
Infrastructure - Task #3171 (New): Add d1_ common_java d1_libclient_java and their dependency jar...https://redmine.dataone.org/issues/31712012-08-31T17:27:33ZJing Taotao@nceas.ucsb.edu
<p>In order to introduce dataone into morpho, we need include the d1_ common_java d1_libclient_java and their dependency jar files.</p>
<p>We may manually check in those released version jar files into the morpho library. However, during the implementaion of morpho, developers may also make some modification on d1_common_java and d1_libclient_jar as well. In order to bring the changes made locally or just committed to trunk (not in the release tag), we may need moven-ant-task.jar file to the morpho ant build.xml. </p>
Infrastructure - Task #3166 (New): Add d1_ common_java and d1_libclient_java and their depency jarhttps://redmine.dataone.org/issues/31662012-08-31T17:15:18ZJing Taotao@nceas.ucsb.eduInfrastructure - Task #1646 (New): Add addFormat() to the CNCore APIhttps://redmine.dataone.org/issues/16462011-06-15T21:43:55ZChris Jonescjones@nceas.ucsb.edu
<p>Once the object format policy has been established, modify CNCore to include addFormat(). This should require a Session object, an ObjectFormatIdentifier, and an ObjectFormat as arguments.</p>