Proper use of Executors

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Proper use of Executors

Chris Gamache
Hi all,

I'm using org.apache.commons:commons-exec:1.3 on Java 8.

I'm having an issue where my Tomcat server is bleeding out hundreds of
threads and all of the memory in the form of Executors that I'm running but
don't seem to be closing down ... When the server finally grinds to a halt
I have to restart. When I do it looks like this at shutdown time:

<snip>

27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [ROOT##000252] appears to have started a thread named [Exec
Default Executor] but has failed to stop it. This is very likely to create
a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:502)
 java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
 org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
 org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
 org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
 java.lang.Thread.run(Thread.java:745)

27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [ROOT##000252] appears to have started a thread named [Exec
Stream Pumper] but has failed to stop it. This is very likely to create a
memory leak. Stack trace of thread:
 java.io.FileInputStream.readBytes(Native Method)
 java.io.FileInputStream.read(FileInputStream.java:255)
 java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
 java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
 java.io.BufferedInputStream.read(BufferedInputStream.java:345)
 java.io.FilterInputStream.read(FilterInputStream.java:107)
 org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
 java.lang.Thread.run(Thread.java:745)

</snip>

And my thread dump is a mile long.

I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
relevant code. It is called from a regular method in a regular class,
nothing fancy:

CommandLine cmdLine = CommandLine.parse(command.toString());
DefaultExecutor executor = new DefaultExecutor();
PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
executor.setStreamHandler(esh);
executor.execute(cmdLine);

`is` and `os` are passed in on the constructor. Their opens and closes are
managed well and cleaned up on the outside of this class...
Are there further steps I'm missing to ensure the threads I'm creating are
getting shut down properly and the resources they are using are being
returned?

Any help is much appreciated.
Reply | Threaded
Open this post in threaded view
|

Re: Proper use of Executors

Siegfried Göschl
Hi Chris,

there are couple of things to consider

* You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
* That could happen if the process being executed actually writes some error messages :-)
* Are you 100% sure that the processes will terminate? See ExecuteWatchdog
* You might habe a look at ProcessDestroyer to cleanup during shutdown

Thanks in advance,

Siegfried Goeschl


> On 27 May 2017, at 14:27, Chris Gamache <[hidden email]> wrote:
>
> Hi all,
>
> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>
> I'm having an issue where my Tomcat server is bleeding out hundreds of
> threads and all of the memory in the form of Executors that I'm running but
> don't seem to be closing down ... When the server finally grinds to a halt
> I have to restart. When I do it looks like this at shutdown time:
>
> <snip>
>
> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Default Executor] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> java.lang.Thread.run(Thread.java:745)
>
> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Stream Pumper] but has failed to stop it. This is very likely to create a
> memory leak. Stack trace of thread:
> java.io.FileInputStream.readBytes(Native Method)
> java.io.FileInputStream.read(FileInputStream.java:255)
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> java.io.FilterInputStream.read(FilterInputStream.java:107)
> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> java.lang.Thread.run(Thread.java:745)
>
> </snip>
>
> And my thread dump is a mile long.
>
> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> relevant code. It is called from a regular method in a regular class,
> nothing fancy:
>
> CommandLine cmdLine = CommandLine.parse(command.toString());
> DefaultExecutor executor = new DefaultExecutor();
> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> executor.setStreamHandler(esh);
> executor.execute(cmdLine);
>
> `is` and `os` are passed in on the constructor. Their opens and closes are
> managed well and cleaned up on the outside of this class...
> Are there further steps I'm missing to ensure the threads I'm creating are
> getting shut down properly and the resources they are using are being
> returned?
>
> Any help is much appreciated.

Reply | Threaded
Open this post in threaded view
|

Re: Proper use of Executors

mgainty

MG>below


________________________________
From: Siegfried Goeschl <[hidden email]>
Sent: Sunday, May 28, 2017 3:41 PM
To: Commons Users List
Cc: [hidden email]
Subject: Re: Proper use of Executors

Hi Chris,

there are couple of things to consider

* You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
* That could happen if the process being executed actually writes some error messages :-)
* Are you 100% sure that the processes will terminate? See ExecuteWatchdog
* You might habe a look at ProcessDestroyer to cleanup during shutdown
MG>correct ThreadMonitor needs a watchdog for timeout e.g.

ExecuteWatchdog watchDog = new ExecuteWatchdog(60L * 60L * 1000L);

MG>once timeout is achieved you will need ProcessDestroyer to kill the process

 ShutdownHookProcessDestroyer processDestroyer = new ShutdownHookProcessDestroyer();

MG>Executor needs to identify its watchdog and ProcessDestroyer

executor.setStreamHandler(streamHandler); //PumpStreamHandler
executor.setWorkingDirectory(new File(getExecutionDirectory())); //which directory to write to
executor.setProcessDestroyer(processDestroyer); //what is processDestroyer
executor.setWatchdog(watchDog); //identify watchdog


 // Async Executor  doesn't block.
    DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();

//Executor HashMap

// Inherit the current process's environment variables and add the user-defined ones
    @SuppressWarnings("unchecked")
    Map<String, String> newEnvironment = EnvironmentUtils.getProcEnvironment();
    newEnvironment.putAll(this.environment);


//place to stuff the results

DefaultExecuteResultHandler resultHandler = new DefaultExecuteResultHandler();

//use executor to execute commandLine, environment HashHap

executor.execute(command, newEnvironment, resultHandler);

//return results

return resultHandler;

http://www.programcreek.com/java-api-examples/index.php?api=org.apache.commons.exec.DefaultExecuteResultHandler
Java Code Example org.apache.commons.exec ...<http://www.programcreek.com/java-api-examples/index.php?api=org.apache.commons.exec.DefaultExecuteResultHandler>
www.programcreek.com
This page provides Java code examples for org.apache.commons.exec.DefaultExecuteResultHandler. The examples are extracted from open source Java projects from GitHub.




Thanks in advance,

Siegfried Goeschl


> On 27 May 2017, at 14:27, Chris Gamache <[hidden email]> wrote:
>
> Hi all,
>
> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>
> I'm having an issue where my Tomcat server is bleeding out hundreds of
> threads and all of the memory in the form of Executors that I'm running but
> don't seem to be closing down ... When the server finally grinds to a halt
> I have to restart. When I do it looks like this at shutdown time:
>
> <snip>
>
> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Default Executor] but has failed to stop it. This is very likely to create
> a memory leak. Stack trace of thread:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> java.lang.Thread.run(Thread.java:745)
>
> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> web application [ROOT##000252] appears to have started a thread named [Exec
> Stream Pumper] but has failed to stop it. This is very likely to create a
> memory leak. Stack trace of thread:
> java.io.FileInputStream.readBytes(Native Method)
> java.io.FileInputStream.read(FileInputStream.java:255)
> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> java.io.FilterInputStream.read(FilterInputStream.java:107)
> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> java.lang.Thread.run(Thread.java:745)
>
> </snip>
>
> And my thread dump is a mile long.
>
> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> relevant code. It is called from a regular method in a regular class,
> nothing fancy:
>
> CommandLine cmdLine = CommandLine.parse(command.toString());
> DefaultExecutor executor = new DefaultExecutor();
> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> executor.setStreamHandler(esh);
> executor.execute(cmdLine);
>
> `is` and `os` are passed in on the constructor. Their opens and closes are
> managed well and cleaned up on the outside of this class...
> Are there further steps I'm missing to ensure the threads I'm creating are
> getting shut down properly and the resources they are using are being
> returned?
>
> Any help is much appreciated.

Reply | Threaded
Open this post in threaded view
|

[Re] Proper use of Executors

Siegfried Göschl
In reply to this post by Siegfried Göschl
Hi Chris,

any new findings from your side?

Thanks in advance,

Siegfried Goeschl

> On 28 May 2017, at 21:41, Siegfried Goeschl <[hidden email]> wrote:
>
> Hi Chris,
>
> there are couple of things to consider
>
> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
> * That could happen if the process being executed actually writes some error messages :-)
> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
> * You might habe a look at ProcessDestroyer to cleanup during shutdown
>
> Thanks in advance,
>
> Siegfried Goeschl
>
>
>> On 27 May 2017, at 14:27, Chris Gamache <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>>
>> I'm having an issue where my Tomcat server is bleeding out hundreds of
>> threads and all of the memory in the form of Executors that I'm running but
>> don't seem to be closing down ... When the server finally grinds to a halt
>> I have to restart. When I do it looks like this at shutdown time:
>>
>> <snip>
>>
>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>> web application [ROOT##000252] appears to have started a thread named [Exec
>> Default Executor] but has failed to stop it. This is very likely to create
>> a memory leak. Stack trace of thread:
>> java.lang.Object.wait(Native Method)
>> java.lang.Object.wait(Object.java:502)
>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
>> java.lang.Thread.run(Thread.java:745)
>>
>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>> web application [ROOT##000252] appears to have started a thread named [Exec
>> Stream Pumper] but has failed to stop it. This is very likely to create a
>> memory leak. Stack trace of thread:
>> java.io.FileInputStream.readBytes(Native Method)
>> java.io.FileInputStream.read(FileInputStream.java:255)
>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
>> java.lang.Thread.run(Thread.java:745)
>>
>> </snip>
>>
>> And my thread dump is a mile long.
>>
>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
>> relevant code. It is called from a regular method in a regular class,
>> nothing fancy:
>>
>> CommandLine cmdLine = CommandLine.parse(command.toString());
>> DefaultExecutor executor = new DefaultExecutor();
>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
>> executor.setStreamHandler(esh);
>> executor.execute(cmdLine);
>>
>> `is` and `os` are passed in on the constructor. Their opens and closes are
>> managed well and cleaned up on the outside of this class...
>> Are there further steps I'm missing to ensure the threads I'm creating are
>> getting shut down properly and the resources they are using are being
>> returned?
>>
>> Any help is much appreciated.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Re] Proper use of Executors

Chris Gamache
Hi Siegfried,

I have implemented the executor watchdogs and shutdown hooks. I will know if our efforts have been fruitful when we make our first restart which I'm sure will be soon. I promise I'll write back and let you know how it goes.

Thank you vary much for checking in with me!

> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <[hidden email]> wrote:
>
> Hi Chris,
>
> any new findings from your side?
>
> Thanks in advance,
>
> Siegfried Goeschl
>
>> On 28 May 2017, at 21:41, Siegfried Goeschl <[hidden email]> wrote:
>>
>> Hi Chris,
>>
>> there are couple of things to consider
>>
>> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
>> * That could happen if the process being executed actually writes some error messages :-)
>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
>>
>> Thanks in advance,
>>
>> Siegfried Goeschl
>>
>>
>>> On 27 May 2017, at 14:27, Chris Gamache <[hidden email]> wrote:
>>>
>>> Hi all,
>>>
>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>>>
>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
>>> threads and all of the memory in the form of Executors that I'm running but
>>> don't seem to be closing down ... When the server finally grinds to a halt
>>> I have to restart. When I do it looks like this at shutdown time:
>>>
>>> <snip>
>>>
>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>> Default Executor] but has failed to stop it. This is very likely to create
>>> a memory leak. Stack trace of thread:
>>> java.lang.Object.wait(Native Method)
>>> java.lang.Object.wait(Object.java:502)
>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>> Stream Pumper] but has failed to stop it. This is very likely to create a
>>> memory leak. Stack trace of thread:
>>> java.io.FileInputStream.readBytes(Native Method)
>>> java.io.FileInputStream.read(FileInputStream.java:255)
>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> </snip>
>>>
>>> And my thread dump is a mile long.
>>>
>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
>>> relevant code. It is called from a regular method in a regular class,
>>> nothing fancy:
>>>
>>> CommandLine cmdLine = CommandLine.parse(command.toString());
>>> DefaultExecutor executor = new DefaultExecutor();
>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
>>> executor.setStreamHandler(esh);
>>> executor.execute(cmdLine);
>>>
>>> `is` and `os` are passed in on the constructor. Their opens and closes are
>>> managed well and cleaned up on the outside of this class...
>>> Are there further steps I'm missing to ensure the threads I'm creating are
>>> getting shut down properly and the resources they are using are being
>>> returned?
>>>
>>> Any help is much appreciated.
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Re] Proper use of Executors

Siegfried Göschl
Staying tuned :-)

> On 5 Jun 2017, at 13:59, Chris Gamache <[hidden email]> wrote:
>
> Hi Siegfried,
>
> I have implemented the executor watchdogs and shutdown hooks. I will know if our efforts have been fruitful when we make our first restart which I'm sure will be soon. I promise I'll write back and let you know how it goes.
>
> Thank you vary much for checking in with me!
>
>> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <[hidden email]> wrote:
>>
>> Hi Chris,
>>
>> any new findings from your side?
>>
>> Thanks in advance,
>>
>> Siegfried Goeschl
>>
>>> On 28 May 2017, at 21:41, Siegfried Goeschl <[hidden email]> wrote:
>>>
>>> Hi Chris,
>>>
>>> there are couple of things to consider
>>>
>>> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
>>> * That could happen if the process being executed actually writes some error messages :-)
>>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
>>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
>>>
>>> Thanks in advance,
>>>
>>> Siegfried Goeschl
>>>
>>>
>>>> On 27 May 2017, at 14:27, Chris Gamache <[hidden email]> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
>>>>
>>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
>>>> threads and all of the memory in the form of Executors that I'm running but
>>>> don't seem to be closing down ... When the server finally grinds to a halt
>>>> I have to restart. When I do it looks like this at shutdown time:
>>>>
>>>> <snip>
>>>>
>>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
>>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>>> Default Executor] but has failed to stop it. This is very likely to create
>>>> a memory leak. Stack trace of thread:
>>>> java.lang.Object.wait(Native Method)
>>>> java.lang.Object.wait(Object.java:502)
>>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
>>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
>>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
>>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
>>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
>>>> web application [ROOT##000252] appears to have started a thread named [Exec
>>>> Stream Pumper] but has failed to stop it. This is very likely to create a
>>>> memory leak. Stack trace of thread:
>>>> java.io.FileInputStream.readBytes(Native Method)
>>>> java.io.FileInputStream.read(FileInputStream.java:255)
>>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> </snip>
>>>>
>>>> And my thread dump is a mile long.
>>>>
>>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
>>>> relevant code. It is called from a regular method in a regular class,
>>>> nothing fancy:
>>>>
>>>> CommandLine cmdLine = CommandLine.parse(command.toString());
>>>> DefaultExecutor executor = new DefaultExecutor();
>>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
>>>> executor.setStreamHandler(esh);
>>>> executor.execute(cmdLine);
>>>>
>>>> `is` and `os` are passed in on the constructor. Their opens and closes are
>>>> managed well and cleaned up on the outside of this class...
>>>> Are there further steps I'm missing to ensure the threads I'm creating are
>>>> getting shut down properly and the resources they are using are being
>>>> returned?
>>>>
>>>> Any help is much appreciated.
>>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Re] Proper use of Executors

Chris Gamache
Ok so, it's hard to tell if this made much of a difference. There might be
a few less messages? Hard to be sure.

I'm still getting those messages, but I'm now suspecting Selenium is
generating the majority of these messages. It is behaving badly when it
comes to killing its spawned phantomjs processes, thereby leaving the
executors open. Seems like a recurring theme, eh Siegfried? ;)

Selenium is co-opting the Apache Commons Exec. It looks like it is using
ExecuteWatchdog.INFINITE_TIMEOUT, so no help there. I see they are making a
pretty good effort to kill the processes. I won't rule-out that I'm doing
it wrong, but I'm pretty sure

} finally {
  driver.quit();
}

is all I'm responsible for if I want Selenium to close phantomjs processes.
I do have some room to upgrade Selenium and PhantomJS which might help. I
didn't see any meaningful changes in Selenium that would lead me to believe
that an upgrade would fix the issue. But, it's a good practice and any
patches will have to be made at the head of the project anyway.

Thanks for all the help!

CG


On Mon, Jun 19, 2017 at 4:11 PM, Siegfried Goeschl <
[hidden email]> wrote:

> Staying tuned :-)
>
> > On 5 Jun 2017, at 13:59, Chris Gamache <[hidden email]> wrote:
> >
> > Hi Siegfried,
> >
> > I have implemented the executor watchdogs and shutdown hooks. I will
> know if our efforts have been fruitful when we make our first restart which
> I'm sure will be soon. I promise I'll write back and let you know how it
> goes.
> >
> > Thank you vary much for checking in with me!
> >
> >> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <
> [hidden email]> wrote:
> >>
> >> Hi Chris,
> >>
> >> any new findings from your side?
> >>
> >> Thanks in advance,
> >>
> >> Siegfried Goeschl
> >>
> >>> On 28 May 2017, at 21:41, Siegfried Goeschl <
> [hidden email]> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> there are couple of things to consider
> >>>
> >>> * You are using a PumpStreamHander but the STDERR is not consumed.
> Each process has an internal buffer (size depends on the OS) and when the
> buffer is full any write to STDERR is blocked
> >>> * That could happen if the process being executed actually writes some
> error messages :-)
> >>> * Are you 100% sure that the processes will terminate? See
> ExecuteWatchdog
> >>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
> >>>
> >>> Thanks in advance,
> >>>
> >>> Siegfried Goeschl
> >>>
> >>>
> >>>> On 27 May 2017, at 14:27, Chris Gamache <[hidden email]> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
> >>>>
> >>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
> >>>> threads and all of the memory in the form of Executors that I'm
> running but
> >>>> don't seem to be closing down ... When the server finally grinds to a
> halt
> >>>> I have to restart. When I do it looks like this at shutdown time:
> >>>>
> >>>> <snip>
> >>>>
> >>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> [Exec
> >>>> Default Executor] but has failed to stop it. This is very likely to
> create
> >>>> a memory leak. Stack trace of thread:
> >>>> java.lang.Object.wait(Native Method)
> >>>> java.lang.Object.wait(Object.java:502)
> >>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> >>>> org.apache.commons.exec.DefaultExecutor.executeInternal(
> DefaultExecutor.java:364)
> >>>> org.apache.commons.exec.DefaultExecutor.access$200(
> DefaultExecutor.java:48)
> >>>> org.apache.commons.exec.DefaultExecutor$1.run(
> DefaultExecutor.java:200)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The
> >>>> web application [ROOT##000252] appears to have started a thread named
> [Exec
> >>>> Stream Pumper] but has failed to stop it. This is very likely to
> create a
> >>>> memory leak. Stack trace of thread:
> >>>> java.io.FileInputStream.readBytes(Native Method)
> >>>> java.io.FileInputStream.read(FileInputStream.java:255)
> >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
> >>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> </snip>
> >>>>
> >>>> And my thread dump is a mile long.
> >>>>
> >>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's
> the
> >>>> relevant code. It is called from a regular method in a regular class,
> >>>> nothing fancy:
> >>>>
> >>>> CommandLine cmdLine = CommandLine.parse(command.toString());
> >>>> DefaultExecutor executor = new DefaultExecutor();
> >>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> >>>> executor.setStreamHandler(esh);
> >>>> executor.execute(cmdLine);
> >>>>
> >>>> `is` and `os` are passed in on the constructor. Their opens and
> closes are
> >>>> managed well and cleaned up on the outside of this class...
> >>>> Are there further steps I'm missing to ensure the threads I'm
> creating are
> >>>> getting shut down properly and the resources they are using are being
> >>>> returned?
> >>>>
> >>>> Any help is much appreciated.
> >>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Re] Proper use of Executors

Siegfried Göschl
Hi Chris,

the “recurring theme” does ring a bell - I guess you already know  https://www.mail-archive.com/user@.../msg10368.html <https://www.mail-archive.com/user@.../msg10368.html> :-)

Do you have some Git repo to re-produce the problem?

Thanks in advance,

Siegfried Goeschl


> On 21 Jun 2017, at 17:48, Chris Gamache <[hidden email]> wrote:
>
> Ok so, it's hard to tell if this made much of a difference. There might be a few less messages? Hard to be sure.
>
> I'm still getting those messages, but I'm now suspecting Selenium is generating the majority of these messages. It is behaving badly when it comes to killing its spawned phantomjs processes, thereby leaving the executors open. Seems like a recurring theme, eh Siegfried? ;)
>
> Selenium is co-opting the Apache Commons Exec. It looks like it is using ExecuteWatchdog.INFINITE_TIMEOUT, so no help there. I see they are making a pretty good effort to kill the processes. I won't rule-out that I'm doing it wrong, but I'm pretty sure
>
> } finally {
>   driver.quit();
> }
>
> is all I'm responsible for if I want Selenium to close phantomjs processes. I do have some room to upgrade Selenium and PhantomJS which might help. I didn't see any meaningful changes in Selenium that would lead me to believe that an upgrade would fix the issue. But, it's a good practice and any patches will have to be made at the head of the project anyway.
>
> Thanks for all the help!
>
> CG
>
>
> On Mon, Jun 19, 2017 at 4:11 PM, Siegfried Goeschl <[hidden email] <mailto:[hidden email]>> wrote:
> Staying tuned :-)
>
> > On 5 Jun 2017, at 13:59, Chris Gamache <[hidden email] <mailto:[hidden email]>> wrote:
> >
> > Hi Siegfried,
> >
> > I have implemented the executor watchdogs and shutdown hooks. I will know if our efforts have been fruitful when we make our first restart which I'm sure will be soon. I promise I'll write back and let you know how it goes.
> >
> > Thank you vary much for checking in with me!
> >
> >> On Jun 5, 2017, at 11:45 AM, Siegfried Goeschl <[hidden email] <mailto:[hidden email]>> wrote:
> >>
> >> Hi Chris,
> >>
> >> any new findings from your side?
> >>
> >> Thanks in advance,
> >>
> >> Siegfried Goeschl
> >>
> >>> On 28 May 2017, at 21:41, Siegfried Goeschl <[hidden email] <mailto:[hidden email]>> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> there are couple of things to consider
> >>>
> >>> * You are using a PumpStreamHander but the STDERR is not consumed. Each process has an internal buffer (size depends on the OS) and when the buffer is full any write to STDERR is blocked
> >>> * That could happen if the process being executed actually writes some error messages :-)
> >>> * Are you 100% sure that the processes will terminate? See ExecuteWatchdog
> >>> * You might habe a look at ProcessDestroyer to cleanup during shutdown
> >>>
> >>> Thanks in advance,
> >>>
> >>> Siegfried Goeschl
> >>>
> >>>
> >>>> On 27 May 2017, at 14:27, Chris Gamache <[hidden email] <mailto:[hidden email]>> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I'm using org.apache.commons:commons-exec:1.3 on Java 8.
> >>>>
> >>>> I'm having an issue where my Tomcat server is bleeding out hundreds of
> >>>> threads and all of the memory in the form of Executors that I'm running but
> >>>> don't seem to be closing down ... When the server finally grinds to a halt
> >>>> I have to restart. When I do it looks like this at shutdown time:
> >>>>
> >>>> <snip>
> >>>>
> >>>> 27-May-2017 07:56:21.631 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> >>>> web application [ROOT##000252] appears to have started a thread named [Exec
> >>>> Default Executor] but has failed to stop it. This is very likely to create
> >>>> a memory leak. Stack trace of thread:
> >>>> java.lang.Object.wait(Native Method)
> >>>> java.lang.Object.wait(Object.java:502)
> >>>> java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
> >>>> org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
> >>>> org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
> >>>> org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 27-May-2017 07:56:21.633 WARNING [localhost-startStop-11]
> >>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
> >>>> web application [ROOT##000252] appears to have started a thread named [Exec
> >>>> Stream Pumper] but has failed to stop it. This is very likely to create a
> >>>> memory leak. Stack trace of thread:
> >>>> java.io.FileInputStream.readBytes(Native Method)
> >>>> java.io.FileInputStream.read(FileInputStream.java:255)
> >>>> java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> >>>> java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> >>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
> >>>> org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
> >>>> java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> </snip>
> >>>>
> >>>> And my thread dump is a mile long.
> >>>>
> >>>> I am certainly willing to concede I'm Doing It Wrong(tm) ... Here's the
> >>>> relevant code. It is called from a regular method in a regular class,
> >>>> nothing fancy:
> >>>>
> >>>> CommandLine cmdLine = CommandLine.parse(command.toString());
> >>>> DefaultExecutor executor = new DefaultExecutor();
> >>>> PumpStreamHandler esh = new PumpStreamHandler(os,null,is);
> >>>> executor.setStreamHandler(esh);
> >>>> executor.execute(cmdLine);
> >>>>
> >>>> `is` and `os` are passed in on the constructor. Their opens and closes are
> >>>> managed well and cleaned up on the outside of this class...
> >>>> Are there further steps I'm missing to ensure the threads I'm creating are
> >>>> getting shut down properly and the resources they are using are being
> >>>> returned?
> >>>>
> >>>> Any help is much appreciated.
> >>>
> >>
>
>