Project

General

Profile

Bug #7322

D1Object stores data in memory, causes out of memory errors

Added by Chris Jones over 6 years ago. Updated over 6 years ago.

Status:
Testing
Priority:
Normal
Assignee:
Category:
d1_libclient_java
Target version:
Start date:
2015-08-27
Due date:
% Done:

50%

Story Points:
Sprint:

Description

When assembling DataPackage instances and populating them, the DataPackage class relies on the underlying D1Object.download() method to store members of the DataPackage locally. The current implementation calls IOUtils.toByteArray(inputstream), which of course stores all bytes in memory. With large data files, this effectively renders DataPackage useless because of OutOfMemory exceptions. The move towards using the javax.activation.DataSource interface helps with this since it provide in memory and on disk implemetations.

Change download() to default to the on-disk DataSource, and make the storage location configurable in d1client.properties.

History

#1 Updated by Chris Jones over 6 years ago

  • Status changed from In Progress to Testing
  • % Done changed from 30 to 50

I've updated the 1.3 branch code, as well as the trunk, to use a configurable on-disk storage location. Some review would be helpful. I've tested this code while populating the DEV2 environment, and it seemed to work well with multi-gigabyte DataPackages. More testing would be good.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)