DataONE Tasks: Issueshttps://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2020-06-17T21:49:55ZDataONE Tasks
Redmine CN REST - Story #8864 (New): Sychronization does not register authoritative replica entry correctlyhttps://redmine.dataone.org/issues/88642020-06-17T21:49:55ZChris Jonescjones@nceas.ucsb.edu
<p>When objects are synchronized to the CN, the <code>d1_synchronization</code> component will fetch the system metadata <br>
for each object and will add a <code><replica></code> entry for the origin node (like <code>urn:node:ESS_DIVE</code>, <br>
as well as entries for other copies (for instance for science metadata copied to the CN, <br>
a <code><replica>urn:node:CN</replica></code> will be added.</p>
<p>In some instances, the origin replica instance is not added to the replica list.<br><br>
This causes downstream problems for the <code>d1_replication</code> component because it relies on the origin node <br>
replica entry to be present in order to set up a replication request to a target node. I'm seeing errors like:</p>
<pre>/var/log/dataone/replicate/cn-replication.log.90:[ERROR] 2020-06-04 05:18:30,179 [pool-15-thread-1] (MNCommunication:requestReplication:34) Could not determine replication source node for replication request for pid: ess-dive-eb6cbb22c605506-20200122T170607966. Replication request failed.
</pre>
<p>Looking back in the logs, this is the case for the following objects:</p>
<pre>ess-dive-3947e68e9825233-20180621T213650539
ess-dive-3b8d9f4513e02f9-20180621T214221437
ess-dive-467a6c3dda4dc88-20180621T211148554
ess-dive-51f345daca126f7-20180328T160350610716
ess-dive-53b37ae5d8c0f51-20200219T211634419654
ess-dive-6b688fab5524c46-20200121T210154766
ess-dive-7a31346c154f02b-20200127T155012488
ess-dive-a1fb05cbd903309-20200130T190835651
ess-dive-b420b097851c716-20180523T161714606
ess-dive-ba81a8a8e0bef31-20180727T200828345
ess-dive-bfaf3d6d6fd038c-20180716T154005175903
ess-dive-c2ef5f3af108c9c-20180621T220020545
ess-dive-eb6cbb22c605506-20200122T170607966
ess-dive-f3238db16593de5-20180621T215956950
</pre>
<p>We need to fix this issue in <code>d1_synchronization</code> so replication runs correctly and monthly <br>
replica auditing (done by ESS_DIVE) doesn't flag these issues.</p>
Infrastructure - Story #8791 (New): Complete deprecation of ONEMercury interfacehttps://redmine.dataone.org/issues/87912019-05-01T14:00:51ZDave Vieglaisdave.vieglais@gmail.com
<p>ONEMercury is still deployed on CNs and there are a number of links on the DataONE web site that link to the ONEMercury UI.</p>
<p>On production systems, the page: </p>
<p><a href="https://cn.dataone.org/onemercury/">https://cn.dataone.org/onemercury/</a> </p>
<p>should redirect to:</p>
<p><a href="https://search.dataone.org/">https://search.dataone.org/</a></p>
<p>It may be useful to have an informational page that continues the redirect after a few seconds to help inform users of the change.</p>
CN REST - Story #8771 (New): Issue with LDAP when updating `nodeReplicationPolicy`https://redmine.dataone.org/issues/87712019-03-05T19:42:17ZRoger Dahldahl@unm.edu
<p>When a submitting a Node doc update which includes a nodeReplicationPolicy, this section is good:</p>
<pre><nodeReplicationPolicy>
<maxObjectSize>21474836480</maxObjectSize>
<spaceAllocated>1099511627776</spaceAllocated>
</nodeReplicationPolicy>
</pre>
<p>while the same section without <code>maxObjectSize</code> returns error:</p>
<pre> <error detailCode="4822" errorCode="500" name="ServiceFailure">
<description>updateNodeCapabilities failed due to LDAP communication failure:: InvalidAttributeValueException:[LDAP: error code 21 - d1ReplicationPolicyMaxObjectSize: value #0 invalid per syntax]:[LDAP: error code 21 - d1ReplicationPolicyMaxObjectSize: value #0 invalid per syntax]</description>
</error>
</pre>
<p>The schema allows leaving <code>maxObjectSize</code> out, which means that the MN accepts replicas of unlimited size.</p>
<p>Both GMN and Metacat leave <code>maxObjectSize</code> out if the setting is configured to unlimited with <code>-1</code>.</p>
<p>I think it used to work.</p>
CN REST - Story #8770 (New): Issue with CN handling of encoded identifiers in object/ meta/ node/...https://redmine.dataone.org/issues/87702019-03-05T19:37:13ZRoger Dahldahl@unm.edu
<p>Works:<br>
<a href="http://cn.dataone.org/cn/v2/object/doi:10.6073/AA/knb-lter-bes.298.37">http://cn.dataone.org/cn/v2/object/doi:10.6073/AA/knb-lter-bes.298.37</a><br>
<a href="https://cn.dataone.org/cn/v2/node/urn:node:LTER">https://cn.dataone.org/cn/v2/node/urn:node:LTER</a></p>
<p>Does not work:<br>
<a href="http://cn.dataone.org/cn/v2/object/doi%3A10.6073%2FAA%2Fknb-lter-bes.298.37">http://cn.dataone.org/cn/v2/object/doi%3A10.6073%2FAA%2Fknb-lter-bes.298.37</a><br>
<a href="https://cn.dataone.org/cn/v2/node/urn%3Anode%3ALTER">https://cn.dataone.org/cn/v2/node/urn%3Anode%3ALTER</a></p>
<p>Note: Behavior differs between HTTP / HTTPS.</p>
CN REST - Story #8757 (New): Fix getChecksum() in MNAuditTask to use dynamic checksum algorithmshttps://redmine.dataone.org/issues/87572019-01-14T16:46:33ZChris Jonescjones@nceas.ucsb.edu
<p>The <code>MNAuditTask.call()</code> method is hardcoded to use <code>MD5</code> checksums on line 277. It requests the Member Node to generate an <code>MD5</code> checksum, and then compares that checksum to the checksum stated in the Coordinating Node<code>s cached</code>SystemMetadata.checksum<code>field for the object. This obviously will fail for objects that submitted objects using</code>SHA-1` or other algorithms.</p>
CN REST - Story #8756 (New): Ensure replica auditor is effectivehttps://redmine.dataone.org/issues/87562019-01-12T20:25:18ZChris Jonescjones@nceas.ucsb.edu
<p>The replication auditor service is currently configured to audit all objects every 90 days. As documented in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="Story: Replica Auditing service is throwing errors (New)" href="https://redmine.dataone.org/issues/8582">#8582</a>, the auditor is not working correctly. While the errors being thrown that are described in that ticket seem to be limited to <code>pid</code>s with certain characters in them, I think the whole auditor process is not keeping up with our content.</p>
<p>Looking at the number of objects on each member node that haven't been audited in the last 90 days, auditing is well behind (if we consider it working at all):</p>
<pre>SELECT sm.authoritive_member_node, count(smr.guid) AS count
FROM systemmetadata sm INNER JOIN smreplicationstatus smr
ON sm.guid = smr.guid
WHERE
smr.member_node != 'urn:node:CN' AND
sm.date_uploaded < (SELECT CURRENT_DATE - interval '90 days') AND
smr.date_verified < (SELECT CURRENT_DATE - interval '90 days')
GROUP BY sm.authoritive_member_node
ORDER BY count DESC;
authoritive_member_node | count
-------------------------+--------
urn:node:ARCTIC | 771872
urn:node:PANGAEA | 507456
urn:node:LTER | 416339
urn:node:DRYAD | 374439
urn:node:CDL | 242115
urn:node:PISCO | 235791
urn:node:KNB | 86075
urn:node:TDAR | 75639
urn:node:NCEI | 50974
urn:node:USGS_SDC | 40290
urn:node:TERN | 31671
urn:node:ESS_DIVE | 28830
urn:node:NMEPSCOR | 16042
urn:node:GOA | 9266
urn:node:IARC | 7677
urn:node:NRDC | 6673
urn:node:TFRI | 6478
urn:node:PPBIO | 3464
urn:node:ORNLDAAC | 3328
urn:node:FEMC | 2430
urn:node:EDI | 2098
urn:node:GRIIDC | 2065
urn:node:mnTestKNB | 2010
urn:node:SANPARKS | 2008
urn:node:ONEShare | 1874
urn:node:R2R | 1787
urn:node:USGSCSAS | 1151
urn:node:EDACGSTORE | 1075
urn:node:US_MPC | 1032
urn:node:RW | 970
urn:node:KUBI | 516
urn:node:NEON | 487
urn:node:LTER_EUROPE | 343
urn:node:IOE | 279
urn:node:RGD | 273
urn:node:ESA | 272
urn:node:NKN | 218
urn:node:OTS_NDC | 126
urn:node:BCODMO | 115
urn:node:SEAD | 90
urn:node:mnTestNKN | 50
urn:node:EDORA | 28
urn:node:ONEShare.pem | 22
urn:node:CLOEBIRD | 17
urn:node:mnTestBCODMO | 11
urn:node:USANPN | 10
urn:node:mnTestTDAR | 10
urn:node:MyMemberNode | 1
</pre>
<p>The table above represents the number of un-audited objects (in the last 90 days), but I get the feeling that the auditor isn't able to audit any of the content it is charged to audit given 1) the frequency, 2) the number of threads allotted, and 3) the configured batch count (seems way too low). <del>Note that this query excludes replicated content - this is just the original objects</del> (After looking at my query again, I think the join is including all replicas - the total is 2,935,787, which is greater than the total objects in the system (2,751,136), so this query needs to be refined).</p>
<p>We need to evaluate the true effectiveness of the auditor. Some strategies may include: 1) looking to see if we may be in an infinite loop on processing a few <code>pid</code>s due to the issues in <a class="issue tracker-4 status-1 priority-4 priority-default child" title="Story: Replica Auditing service is throwing errors (New)" href="https://redmine.dataone.org/issues/8582">#8582</a>, 2) seeing if we can increase the batch size by increasing the total threads allocated in the executor, and 3) decide if we need to offload the process from the CNs and distribute the workload across a cluster of workers that can do the auditing faster. Needs some thought and discussion.</p>
CN REST - Story #8749 (New): Fix log aggregation events from the CN without associated CN IPshttps://redmine.dataone.org/issues/87492018-11-16T20:39:55ZChris Jonescjones@nceas.ucsb.edu
<p>The robots list used to filter out usage events includes the IP addresses of the CNs, so events logged during synchronization don't show up as true hits. Because of the SSL infrastructure at lbl.gov, the ESS-DIVE group doesn't see the public IP of an incoming request, but rather an internal private IP assigned by lbl.gov infrastructure. You can see the impact of this on the <a href="https://data.ess-dive.lbl.gov/#profile" class="external">ESS-DIVE profile page</a>. The spike of 11,000+ downloads in August 2018 was the CN synchronizing content.</p>
<p>Rushiraj summarized these events in a <a href="https://gist.github.com/rushirajnenuji/847d8239acf68a108bda30e04af0406b" class="external">gist</a></p>
<p>There are multiple <code>10.42.x.x</code> IP associated with the CN requests. These events all need to be updated in the <code>logsolr</code> core and changed to an actual CN IP. For future synchronizations, perhaps we need to add <code>10.42.0.0/16</code> to the robots list? </p>
CN REST - Story #8582 (New): Replica Auditing service is throwing errorshttps://redmine.dataone.org/issues/85822018-05-01T19:15:35ZChris Jonescjones@nceas.ucsb.edu
<p>Replica auditing should be auditing objects every 90 days for fixity, and setting the <code>replicaStatus</code> appropriately. The <code>/var/log/dataone/cn-replication-audit.log*</code> files are showing many errors:</p>
<pre>cjones@cn-ucsb-1:replicate$ grep ERROR cn-replication-audit.log* | grep "Cannot update replica status" | wc -l
437601
</pre>
<p>Determine if this is a configuration issue or a code issue and fix it as needed. Also, fix the code to call <code>Identifier.getValue()</code> when logging these errors to avoid printing the memory location of the object like <code>org.dataone.service.types.v1.Identifier@7e90f2e8</code>. There are multiple places where <code>getValue()</code> needs to be added.</p>
Infrastructure - Story #8470 (New): Make the spring context of the d1_index_processor daemon more...https://redmine.dataone.org/issues/84702018-03-02T20:56:14ZJing Taotao@nceas.ucsb.edu
<p>In the IndexTaskProcessorDaemon class, which is the start point of d1-indexx-processor, the context is initialized from a file in the class path:</p>
<p><code>context = new ClassPathXmlApplicationContext("processor-daemon-context.xml");</code></p>
<p>The file, processor-daemon-context.xml, imports all the index sub-processor application files. This make very hard for us to add a new sub-processor since we have to rebuild a new jar file in order to include a new file. Instead, if we use the <code>FileSystemXmlApplicationContext</code> class and read the file path from a properties file, we can get more flexibility.</p>
Infrastructure - Story #8367 (New): Duplicate jenkins jobs from UNM to UCSBhttps://redmine.dataone.org/issues/83672018-02-15T17:36:01ZDave Vieglaisdave.vieglais@gmail.com
<p>All jenkins jobs need to be duplicated from UNM to UCSB. The goal here is to provide the same functionality as is currently available on the UNM instance.</p>
Infrastructure - Story #8363 (New): indexer shutdown generates index taskshttps://redmine.dataone.org/issues/83632018-02-12T21:42:22ZRob Nahfrnahf@epscor.unm.edu
<p>Seen in STAGE, somehow the index processor generated about 15k tasks (after processing 215k tasks over the weekend) during a service stop. It also created about 12.5 failures. Before trying to stop services, this the status of postgres:</p>
<pre>d1-index-queue=# select status, count(*) from index_task group by status;
status | count
------------+-------
NEW | 5
FAILED | 1659
IN PROCESS | 367
(3 rows)
</pre>
<p>Execution of <code>/etc/init.d/d1-index-task-processor stop</code> timed out.<br>
I performed <code>/etc/init.d/d1-index-task-generator stop</code> successfully, getting an <code>[OK]</code><br>
then I performed <code>/etc/init.d/d1-processing stop</code> on UCSB, also getting an '<code>[OK]</code></p>
<p>examination of the indexing log file a couple minuted later showed this:</p>
<pre>[ INFO] 2018-02-12 20:36:08,975 (IndexTaskProcessor:logProcessorLoad:245) new tasks:0, tasks previously failed: 1661
[ INFO] 2018-02-12 20:36:09,361 (IndexTaskProcessor:processFailedIndexTaskQueue:226) IndexTaskProcessor.processFailedIndexTaskQueue with size 0
[ WARN] 2018-02-12 20:36:09,361 (IndexTaskProcessorJob:execute:58) processing job [org.dataone.cn.index.processor.IndexTaskProcessorJob@515de84e] finished execution of index task processor [org.dataone.cn.index.processor.IndexTaskProcessor@2062
1d44]
[ WARN] 2018-02-12 20:36:26,571 (IndexTaskProcessorScheduler:stop:99) stopping index task processor quartz scheduler [org.dataone.cn.index.processor.IndexTaskProcessorScheduler@103bbd22] ...
[ INFO] 2018-02-12 20:36:26,572 (QuartzScheduler:standby:572) Scheduler QuartzScheduler_$_NON_CLUSTERED paused.
[ INFO] 2018-02-12 20:36:26,572 (IndexTaskProcessorScheduler:stop:111) Scheuler.interrupt method can't succeed to interrupt the d1 index job and the static method IndexTaskProcessorJob.interruptCurrent() will be called.
[ WARN] 2018-02-12 20:36:26,572 (IndexTaskProcessorJob:interruptCurrent:92) IndexTaskProcessorJob class [1806183035] interruptCurrent called, shutting down processor [org.dataone.cn.index.processor.IndexTaskProcessor@20621d44]
[ WARN] 2018-02-12 20:36:26,573 (IndexTaskProcessor:shutdownExecutor:952) processor [org.dataone.cn.index.processor.IndexTaskProcessor@20621d44] Shutting down the ExecutorService. Will allow active tasks to finish; will cancel submitted tasks
and return them to NEW status, wait for active tasks to finish, then return any remaining task not yet submitted to NEW status....
[ WARN] 2018-02-12 20:36:26,573 (IndexTaskProcessor:shutdownExecutor:955) ...1.) closing ExecutorService to new tasks...
[ WARN] 2018-02-12 20:36:26,574 (IndexTaskProcessor:shutdownExecutor:957) ...2.) cancelling cancellable futures...
[ WARN] 2018-02-12 20:36:26,575 (IndexTaskProcessor:shutdownExecutor:958) ...number of futures: 591344
[ WARN] 2018-02-12 20:36:26,575 (IndexTaskProcessor:shutdownExecutor:959) ... number of tasks in futures map: 591344
</pre>
<p>15 minutes or so later, the log showed this:</p>
<pre>[ WARN] 2018-02-12 20:36:26,573 (IndexTaskProcessor:shutdownExecutor:955) ...1.) closing ExecutorService to new tasks...
[ WARN] 2018-02-12 20:36:26,574 (IndexTaskProcessor:shutdownExecutor:957) ...2.) cancelling cancellable futures...
[ WARN] 2018-02-12 20:36:26,575 (IndexTaskProcessor:shutdownExecutor:958) ...number of futures: 591344
[ WARN] 2018-02-12 20:36:26,575 (IndexTaskProcessor:shutdownExecutor:959) ... number of tasks in futures map: 591344
[ WARN] 2018-02-12 20:52:30,811 (IndexTaskProcessor:shutdownExecutor:988) ...number of (cancellable) runnables/tasks reset to new: 0
[ WARN] 2018-02-12 20:52:30,811 (IndexTaskProcessor:shutdownExecutor:989) ...number of (cancellable) runnables not mapped to tasks: 0
[ WARN] 2018-02-12 20:52:30,811 (IndexTaskProcessor:shutdownExecutor:990) ...number of uncancellable runnables: 591344 (completed or in process)
[ WARN] 2018-02-12 20:52:30,812 (IndexTaskProcessor:shutdownExecutor:993) ...3.) waiting (with timeout) for active futures to finish...
[ WARN] 2018-02-12 20:52:30,812 (IndexTaskProcessor:shutdownExecutor:998) ...4.) Reviewing remaining uncancellables to check for completion, returning incomplete ones to NEW status...
[ WARN] 2018-02-12 20:52:30,835 (IndexTaskProcessor:shutdownExecutor:1026) ...5.) Calling shutdownNow on the executor service.
[ WARN] 2018-02-12 20:52:30,835 (IndexTaskProcessor:shutdownExecutor:1028) ... .... number of runnables still waiting: 0
[ WARN] 2018-02-12 20:52:30,835 (IndexTaskProcessor:shutdownExecutor:1030) ...6.) returning preSubmitted tasks to NEW status...
[ WARN] 2018-02-12 20:52:30,835 (IndexTaskProcessor:shutdownExecutor:1031) ... .... number of preSubmitted tasks: 34735
[ INFO] 2018-02-12 20:52:30,835 (IndexTask:markNew:454) Even tough it was masked new, it is still considered failed for id testGetPackage_2017119234441164 since it was tried to many times.
[ERROR] 2018-02-12 20:52:30,891 (IndexTaskProcessor:shutdownExecutor:1038) ....... Exception thrown trying to return task to NEW status for pid: testGetPackage_2017119234441164
org.springframework.orm.hibernate3.HibernateOptimisticLockingFailureException: Object of class [org.dataone.cn.index.task.IndexTask] with identifier [13071797]: optimistic locking failed; nested exception is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect): [org.dataone.cn.index.task.IndexTask#13071797]
...
[ INFO] 2018-02-12 20:54:19,618 (IndexTask:markNew:454) Even tough it was masked new, it is still considered failed for id P3_201622214921901 since it was tried to many times.
[ WARN] 2018-02-12 20:54:19,621 (IndexTaskProcessor:shutdownExecutor:1036) ... preSubmittedTask for pid P3_201622214921901returned to NEW status.
[ WARN] 2018-02-12 20:54:19,623 (IndexTaskProcessor:shutdownExecutor:1036) ... preSubmittedTask for pid resource_map_doi:10.5065/D6VD6WFPreturned to NEW status.
[ INFO] 2018-02-12 20:54:19,623 (IndexTask:markNew:454) Even tough it was masked new, it is still considered failed for id testGetPackage_NotAuthorized_201710605522454 since it was tried to many times.
[ WARN] 2018-02-12 20:54:19,626 (IndexTaskProcessor:shutdownExecutor:1036) ... preSubmittedTask for pid testGetPackage_NotAuthorized_201710605522454returned to NEW status.
[ WARN] 2018-02-12 20:54:19,628 (IndexTaskProcessor:shutdownExecutor:1036) ... preSubmittedTask for pid resource_map_urn:uuid:d3606ccb-2d50-4723-ae45-c0d01b817e48returned to NEW status.
[ WARN] 2018-02-12 20:54:19,631 (IndexTaskProcessor:shutdownExecutor:1036) ... preSubmittedTask for pid resource_map_doi:10.18739/A2165Freturned to NEW status.
[ WARN] 2018-02-12 20:54:19,631 (IndexTaskProcessor:shutdownExecutor:1041) ............7.) DONE with shutting down IndexTaskProcessor.
[ INFO] 2018-02-12 20:54:19,631 (IndexTaskProcessorScheduler:stop:113) The scheuler.interrupt method seems not interrupt the d1 index job and the static method IndexTaskProcessorJob.interruptCurrent() was called.
[ WARN] 2018-02-12 20:54:19,632 (IndexTaskProcessorScheduler:stop:128) Job scheduler [org.dataone.cn.index.processor.IndexTaskProcessorScheduler@103bbd22] finished executing all jobs. The d1-index-processor shut down sucessfully.============================================
</pre>
<p>but postgres yielded this:</p>
<pre>d1-index-queue=# select status, count(*) from index_task group by status;
status | count
--------+-------
NEW | 15367
FAILED | 14032
(2 rows)
</pre>
<p>indexer shutdowns are a stubborn problem...</p>
Infrastructure - Story #8234 (New): Use University of Kansas ORCID membership to support authenti...https://redmine.dataone.org/issues/82342018-01-09T02:00:28ZDave Vieglaisdave.vieglais@gmail.com
<p><a href="https://orcid.org/members/001G000001CAkZgIAL-university-of-kansas" class="external">KU is a premium ORCID member</a> as a member of the Greater Western Library Alliance (GWLA). As a result, KU has access to five ORCID API keys. One is currently in use for the KU DSpace instance.</p>
<p>Goal of this story is to leverage on of the remaining API keys to support ORCID authentication in the DataONE production environment.</p>
Infrastructure - Story #7611 (New): ORNLDAAC fails during log aggregationhttps://redmine.dataone.org/issues/76112016-01-26T16:35:10ZRobert Waltz
<p>ORNLDAAC is able to respond to the following rest call:</p>
<p><a href="http://mercury-ops2.ornl.gov/ornldaac/mn/v1/log?fromDate=2015-12-02T20:26:54.001%2B00:00&toDate=2016-01-26T00:00:00.000%2B00:00&start=0&count=0">http://mercury-ops2.ornl.gov/ornldaac/mn/v1/log?fromDate=2015-12-02T20:26:54.001%2B00:00&toDate=2016-01-26T00:00:00.000%2B00:00&start=0&count=0</a></p>
<p>The response is</p>
<p>So, if ORNLDAAC is queried to not return any event records, but just the total available records over a time period, then the MN response just fine.</p>
<p>However, if ORNLDAAC is queried to return event records for the same timeframe:</p>
<p><a href="http://mercury-ops2.ornl.gov/ornldaac/mn/v1/log?fromDate=2015-12-02T20:26:54.001%2B00:00&toDate=2016-01-26T00:00:00.000%2B00:00&start=0&count=1000">http://mercury-ops2.ornl.gov/ornldaac/mn/v1/log?fromDate=2015-12-02T20:26:54.001%2B00:00&toDate=2016-01-26T00:00:00.000%2B00:00&start=0&count=1000</a></p>
<p>then the MN responds with an "Internal Server Error" that is transmitted in HTML.</p>
DataONE API - Story #6759 (New): ObjectFormat Managementhttps://redmine.dataone.org/issues/67592015-01-13T20:12:14ZRob Nahfrnahf@epscor.unm.edu
<p>There currently are not any API methods for managing the collection of objectFormats registered to a dataone environment. There is a "bootstrap" resource that constitutes a the list in either d1_libclient_java or d1_common_java that can be used in testing environments. There's also a different resource in the cn-os-core project that is used in production.</p>
<p>These 2 resources are difficult to maintain (keep synchronized), and there isn't a defined process for adding formats to production.</p>
<p>We discussed the inclusion of an "addFormat(...) method in V2, but it is not currently in the API. (It would be part of the CNCore API).</p>
<p>It would be good to review the situation with a focused discussion to at least troubleshoot the existing informal management practices and formalize them; and then consider if more infrastructure is needed.</p>
DataONE API - Story #1644 (New): Develop an object format creation policyhttps://redmine.dataone.org/issues/16442011-06-14T16:25:11ZChris Jonescjones@nceas.ucsb.edu
<p>The object format list in d1_common_java is thus far an ad hoc list of known object formats needed in the D1 software. Additions will be needed. We need to develop a policy on who will have write access to the realtime version of this list, when the on-disk version will be periodically updated, etc. New object formats need to be vetted, and that process should be put into place. This process should align with the object format creation process with the UDFR group when their registry is operational.</p>