Bug #7322
D1Object stores data in memory, causes out of memory errors
50%
Description
When assembling DataPackage instances and populating them, the DataPackage class relies on the underlying D1Object.download() method to store members of the DataPackage locally. The current implementation calls IOUtils.toByteArray(inputstream), which of course stores all bytes in memory. With large data files, this effectively renders DataPackage useless because of OutOfMemory exceptions. The move towards using the javax.activation.DataSource interface helps with this since it provide in memory and on disk implemetations.
Change download() to default to the on-disk DataSource, and make the storage location configurable in d1client.properties.
History
#1 Updated by Chris Jones about 9 years ago
- Status changed from In Progress to Testing
- % Done changed from 30 to 50
I've updated the 1.3 branch code, as well as the trunk, to use a configurable on-disk storage location. Some review would be helpful. I've tested this code while populating the DEV2 environment, and it seemed to work well with multi-gigabyte DataPackages. More testing would be good.
