[VFS] OutOfMemory Exception on large files

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[VFS] OutOfMemory Exception on large files

Michael Melhem
Hi Guys,

We have a problem with the DefaultFileContent-OutputStream upon handling
of large files. We get an OutOFMemory Exception on any upload of a file
 > than approx 13 Meg, but never happens on the download (ie
inputstream) of the same file.

The Exception happens with the latest VFS code from svn, plus all
earlier snap shots I have tried.

We are using HTTP protocol. We have code something like this (in a
simple junit test):-

...
final FileObject toDownload = fsManager.resolveFile(sourceURI.toString());
final InputStream is = toDownload.getContent().getInputStream();

final FileObject toUpload = fsManager.resolveFile(targetURI.toString());
final OutPutStream os = toUpload.getContent().getOutputStream();


org.apache.commons.io.IOUtils.copy(is, os);

os.close() // this close will always cause an OutOfMemory Exception
(when file is approx larger than 13 M on our system)
is.close()
...

Note we use the commons.io.IOUtils for the copy which handles
stream-copy using 4k buffers AFIAK, so i dont think the problem is there.

Any help on this will be much appreciated!!

Regards,
Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VFS] OutOfMemory Exception on large files

Mario Ivankovits
Michael Melhem wrote:
> We are using HTTP protocol. We have code something like this (in a
> simple junit test):-
Neither "http" nor "url" supports output streams, so I guess your target
URL is a "webdav://" resource, isnt it?

webdav buffers the output until close and send the whole stuff at once.
Maybe it is possible to use Piped-Streams to pass through to the webdav
server, but currently I do not have enough spare time to do this -
contributions are welcome :-)

As a workaround you could add "-Xmx512m" (or what ever fits your needs)
to your JVM startup to allow java to allocate more memory.


Ciao,
Mario


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VFS] OutOfMemory Exception on large files

Torsten Curdt
>> We are using HTTP protocol. We have code something like this (in a
>> simple junit test):-
>
> Neither "http" nor "url" supports output streams, so I guess your target
> URL is a "webdav://" resource, isnt it?

We can even reproduce it with the "file:" protocol!

 final String sourceURI = "file:/Users/tcurdt/90MB";
 final String targetURI = "file:/Users/tcurdt/test";

 final FileSystemManager fsManager = VFS.getManager();

 final FileObject toDownload = fsManager.resolveFile(sourceURI);
 final InputStream is = toDownload.getContent().getInputStream();

 final FileObject toUpload = fsManager.resolveFile(targetURI);
 final OutputStream os = toUpload.getContent().getOutputStream();

 org.apache.commons.io.IOUtils.copy(is, os);

 os.close();
 is.close();

> webdav buffers the output until close and send the whole stuff at once.

Uh!

> Maybe it is possible to use Piped-Streams to pass through to the webdav
> server, but currently I do not have enough spare time to do this -
> contributions are welcome :-)

Can you point me into the right direction ...haven't had
a closer look into the vfs codebase yet.

> As a workaround you could add "-Xmx512m" (or what ever fits your needs)
> to your JVM startup to allow java to allocate more memory.

...naaa ...that's not even a workaround
we are talking about biiiiiig files! :-)

cheers
--
Torsten

signature.asc (193 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [VFS] OutOfMemory Exception on large files

Mario Ivankovits
Torsten Curdt wrote:
>>>We are using HTTP protocol. We have code something like this (in a
>>>simple junit test):-
>>>      
>>Neither "http" nor "url" supports output streams, so I guess your target
>>URL is a "webdav://" resource, isnt it?
>>    
>
>We can even reproduce it with the "file:" protocol!
>  
What? I cant belief.

I tried:

        FileSystemManager fsm = VFS.getManager();
        FileObject foSrc  = fsm.resolveFile("/home/im/vmware/vfstest.tbz2");
        FileObject foDest =
fsm.resolveFile("/home/im/vmware/vfstest.tmp.tbz2");

        foDest.copyFrom(foSrc, Selectors.SELECT_SELF);

        FileObject foDest2 =
fsm.resolveFile("/home/im/vmware/vfstest.tmp2.tbz2");


        InputStream is = foSrc.getContent().getInputStream();
        OutputStream os = foDest2.getContent().getOutputStream();

        CopyUtils.copy(is, os);

        os.close();
        is.close();

where vfstest.tbz2 is a 159.726.161 bytes big file. - works like a charme.
As you might see, I tried VFS own copyFrom and IOUtils, but ...
> org.apache.commons.io.IOUtils.copy(is, os);
>  
... which version of IOUtils do you use?
I downloaded 1.0 but instead of finding a  IOUtils.copy I found
CopyUtils.copy ...

>>Maybe it is possible to use Piped-Streams to pass through to the webdav
>>server, but currently I do not have enough spare time to do this -
>>contributions are welcome :-)
>>    
>
>Can you point me into the right direction ...haven't had
>a closer look into the vfs codebase yet.
>  
Please find:
org.apache.commons.vfs.provider.webdav.WebdavFileObject and its inner
class WebdavOutputStream

Also the sftp FileProvider uses a byte buffer. Its easy to change this
to jsch own "getOutputStream()" method but jsch internally also uses a
pipe and it is dog slow then. Java uses a buffer of 1K for its pipe wich
seems to be a huge performance bottleneck.


---
Mario
Reply | Threaded
Open this post in threaded view
|

Re: [VFS] OutOfMemory Exception on large files

Torsten Curdt
>> We can even reproduce it with the "file:" protocol!
>>  
>
> What? I cant belief.

Doh! ...forget what I said - all cool with "file:"
(obviously my jetlag is kickin in) Sorry for the
noise and thanks for the crosscheck.

>> Can you point me into the right direction ...haven't had
>> a closer look into the vfs codebase yet.
>>  
>
> Please find:
> org.apache.commons.vfs.provider.webdav.WebdavFileObject and its inner
> class WebdavOutputStream
>
> Also the sftp FileProvider uses a byte buffer. Its easy to change this
> to jsch own "getOutputStream()" method but jsch internally also uses a
> pipe and it is dog slow then. Java uses a buffer of 1K for its pipe wich
> seems to be a huge performance bottleneck.
Ok ...thanks for the pointers - will see what I can do

cheers
--
Torsten

signature.asc (193 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [VFS] OutOfMemory Exception on large files

Mario Ivankovits
Hi Torsten!
>Doh! ...
No problem!
>Ok ...thanks for the pointers - will see what I can do
>  
Feel free to contact me if there is something else I can help with.

Ciao,
Mario


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]