https://redmine.dataone.org/https://redmine.dataone.org/favicon.ico2012-07-17T16:43:06ZDataONE TasksInfrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=130052012-07-17T16:43:06ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Matt also suggest to separate out the meta, ore and data files from data directory into different subdirectories in order to overcome potential file system limits on number of files.</p>
Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=130922012-07-29T22:56:40ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Structure in the .morpho:<br>
/user-home-directory/.morpho/profiles/profile-name/cache/data<br>
/user-home-directory/.morpho/profiles/profile-name/cache/data/sysmeta<br>
/user-home-directory/.morpho/profiles/profile-name/cache/metadata<br>
/user-home-directory/.morpho/profiles/profile-name/cache/metadata/sysmeta<br>
/user-home-directory/.morpho/profiles/profile-name/cache/ore<br>
/user-home-directory/.morpho/profiles/profile-name/cache/ore/sysmeta</p>
<p>Traditionally, morpho keeps the unsaved data file in temp. However, if user clear the temp directory even though he may have incomplete package. This can cause the package miss data files. So I propose the unsaved data file will be stored in incomplete directory which can be persistent.<br><br>
/user-home-directory/.morpho/profiles/profile-name/incomplete/data<br>
/user-home-directory/.morpho/profiles/profile-name/incomplete/data/sysmeta<br>
/user-home-directory/.morpho/profiles/profile-name/incomplete/metadata<br>
/user-home-directory/.morpho/profiles/profile-name/incomplete/metadata/sysmeta<br>
/user-home-directory/.morpho/profiles/profile-name/incomplete/ore<br>
/user-home-directory/.morpho/profiles/profile-name/incomplete/ore/sysmeta</p>
<p>Some other files wil be stored in temp ( i am not sure if we will need it any more).<br>
/user-home-directory/.morpho/profiles/profile-name/temp/</p>
<p>After user specifies a local store, the directory structure will look like:<br>
/user-specified-directory/data<br>
/user-specified-directory/data/sysmeta<br>
/user-specified-directory/metadata<br>
/user-specified-directory/metadata/sysmeta<br>
/user-specified-directory/ore<br>
/user-specified-directory/ore/sysmeta</p>
Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=130932012-07-29T23:00:38ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>I am not sure if the user specified data store will be base on the profile or whole morpho user. But i prefer to base on each profile.</p>
Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=131512012-07-31T18:43:06ZChris Jonescjones@nceas.ucsb.edu
<ul></ul><p>In order to allow data managers to have more control over file names for their data files and metadata files, morpho needs a means of tracking local filenames against the object identiifier, and optionally tracking a 'network copy' of the object. For instance, morpho could maintain an internal h2 SQL database with a table such as:</p>
<p>pid local_uri dataone_uri</p>
<hr>
<p>doi:10.6085/AA/CMRX00_XXXITBDXLSR02_20060618.50.5 file:///Users/frenockm/PISCO/metadata/CMRX00_XXXITBDXLSR02_20060618.50.5.xml <a href="https://cn.dataone.org/cn/v1/resolve/doi:10.6085/AA/CMRX00_XXXITBDXLSR02_20060618.50.5">https://cn.dataone.org/cn/v1/resolve/doi:10.6085/AA/CMRX00_XXXITBDXLSR02_20060618.50.5</a><br>
...</p>
<p>This may also be implemented as a SOLR index as opposed to SQL tables. When the user changes a local URI outsie of Morpho (renames a file), Morpho may need to bring up a dialog asking for the location of the missing file (likely on startup).</p>
Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=131612012-08-01T23:39:29ZJing Taotao@nceas.ucsb.edu
<ul></ul><p>Chris' suggestion is great. We will address the issue on the task 3074.</p>
<p>The user specified directory only store the eml documets generating by morpho, and the downloaded eml and data files from network by Morpho. <br>
The local file which is imported to morpho will keep at the original location. Morpho wouldn't copy it to its local store, just assign it an identifier and the identifer-filename.mapping will redirect the identifier to the local file location. If user modify the local file by an external editor, then the next time morpho open the package and it will prompt the user either to assign a new id to the local file or ask user to import the data file again depending on the nature of the change. If user modify the file in morpho, morpho will copy the old version to the central store with the hash string of the old identifier and overwrite the orginal file. Of course a new identifier will be assigned to the new file. </p>
<p>Open issue:<br>
Where we store the system metadata and ore information?</p>
<ol>
<li><p>Store them in the file system described above. <br>
advantage - they persist in a safer way than in an embedded database. It is easy to take a look for the trouble shooting.<br>
disadvantage - every time we need to parse them to get the information. But those files are pretty small, it maybe is not very expensive.</p></li>
<li><p>Store them in a embedded database.<br>
advantage - we don't need to parse the documents and only do sql commands to get information.<br>
disadvantage - if the database crashes, it will be a disaster. It is hard to take a look for the trouble shooting since it is a binary format.</p></li>
</ol>
Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=140332012-10-11T15:33:27ZDave Vieglaisdave.vieglais@gmail.com
<ul><li><strong>Target version</strong> changed from <i>Sprint-2012.37-Block.5.3</i> to <i>Sprint-2012.41-Block.6.1</i></li></ul> Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=140852012-10-14T15:00:01ZBen Leinfelderleinfelder@nceas.ucsb.edu
<ul><li><strong>Subject</strong> changed from <i>Morpho's new data file system.</i> to <i>Allow user to specify Morpho storage directory</i></li><li><strong>Category</strong> set to <i>Morpho</i></li></ul><p>Moving this to 2.x.y target. I think the structure of the filesystem looks like a fine idea, but allowing users to directly edit these files opens up a huge cn of worms in terms of consistency and versioning.</p>
Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=142512012-10-24T18:20:05ZBen Leinfelderleinfelder@nceas.ucsb.edu
<ul><li><strong>Target version</strong> changed from <i>Sprint-2012.41-Block.6.1</i> to <i>Sprint-2012.44-Block.6.2</i></li></ul> Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=145492012-12-12T16:51:00ZChris Jonescjones@nceas.ucsb.edu
<ul><li><strong>Target version</strong> changed from <i>Sprint-2012.44-Block.6.2</i> to <i>Sprint-2012.50-Block.6.4</i></li></ul> Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=154642013-03-01T18:33:38ZBen Leinfelderleinfelder@nceas.ucsb.edu
<ul><li><strong>Target version</strong> changed from <i>Sprint-2012.50-Block.6.4</i> to <i>2013.10-Block.2.1</i></li></ul> Infrastructure - Task #3076: Allow user to specify Morpho storage directoryhttps://redmine.dataone.org/issues/3076?journal_id=156852013-03-02T05:29:48ZBen Leinfelderleinfelder@nceas.ucsb.edu
<ul><li><strong>Parent task</strong> changed from <i>#3075</i> to <i>#3635</i></li></ul>