[jira] [Commented] (COMPRESS-446) Resource Leak in ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

[jira] [Commented] (COMPRESS-446) Resource Leak in ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/COMPRESS-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679878#comment-16679878 ]

Melloware commented on COMPRESS-446:

We are still seeing this error on Commons Compress 1.18 intermittently leaving these TMP files for ParallelScatter.  Our system runs 24/7 and is zipping files hourly so over a 1 month period we have seen it leave say 4-6 of these files around.  The problem is they don't get cleaned up and fills our /TMP eventually to 100%.

Any thoughts?

> Resource Leak in ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)
> --------------------------------------------------------------------------
>                 Key: COMPRESS-446
>                 URL: https://issues.apache.org/jira/browse/COMPRESS-446
>             Project: Commons Compress
>          Issue Type: Bug
>          Components: Archivers
>    Affects Versions: 1.16.1
>         Environment: The application was running inside a Docker container, the JVM had about 1.7 GByte heap space.
>            Reporter: Christoph Ludwig
>            Priority: Major
>              Labels: zip
>             Fix For: 1.17
> Before it does anything else, {{ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)}} loops over all futures returned by the creator`s executor service and calls {{Future#get()}}. This will block until the future's computation is completed, respectively - i.e., until all entries have been written to the thread-local scatter streams.
> However, if the computation of a future fails, then {{Future#get()}} can also throw an exception. This exception escapes {{ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)}} before the executor service is shut down. The latter means that also the thread-local variables in the executor service's threads and all objects referenced by them continue to exist and cannot be reclaimed by the GC.
> I encountered this situation when - while processing an archive with 130,000 documents - the JVM threw an {{OutOfMemoryError}}. The application was not able to recover from this OOM error because most of the heap was occupied by objects reachable from the executor service's threads.
> Of course, the OOM is mostly the fault of my own code; I will be able to work around the "leaked" executor service because I supply it in the first place and can therefore shut it down if I detect an error situation.  
> The effect would be the same, though, if, say, {{Future#get()}} throws an {{InterruptedException}}. Therefore, {{ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)}}  should either shut down and release all resources if it cannot complete its task due to an Exception thrown by a future or it should offer a reasonable recovery strategy.

This message was sent by Atlassian JIRA