[math] testing issue with Kolmogorov-Smirnov bootstrap methods

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[math] testing issue with Kolmogorov-Smirnov bootstrap methods

Luc Maisonobe-2
Hi all,

It seems the new bootstrap method introduced in fbc327e9 in order
to solve MATH-1246 creates some random test failures.

The reason is that an EnumeratedRealDistribution instance is
created without a random generator (there are no way to pass
a random generator in the bootstrap method). This
EnumeratedRealDistribution will therefore create a Well19937c
generator with the default constructor, which uses time to seed
the generator. Depending on the way this generator is seeded,
the testBootstrapSmallSamplesWithTies test fails quite often.

Would it be possible to pass a random generator somewhere and
to configure it with a fixed seed for the Junit tests?

best regards,
Luc

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] testing issue with Kolmogorov-Smirnov bootstrap methods

Gilles Sadowski
On Tue, 24 Nov 2015 11:40:23 +0100, Luc Maisonobe wrote:

> Hi all,
>
> It seems the new bootstrap method introduced in fbc327e9 in order
> to solve MATH-1246 creates some random test failures.
>
> The reason is that an EnumeratedRealDistribution instance is
> created without a random generator (there are no way to pass
> a random generator in the bootstrap method). This
> EnumeratedRealDistribution will therefore create a Well19937c
> generator with the default constructor, which uses time to seed
> the generator. Depending on the way this generator is seeded,
> the testBootstrapSmallSamplesWithTies test fails quite often.

If it fails often, doesn't it indicate a problem with the code or
a wrong assumption about the test?

Best,
Gilles

> Would it be possible to pass a random generator somewhere and
> to configure it with a fixed seed for the Junit tests?
>
> best regards,
> Luc


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] testing issue with Kolmogorov-Smirnov bootstrap methods

Phil Steitz


> On Nov 24, 2015, at 3:50 AM, Gilles <[hidden email]> wrote:
>
>> On Tue, 24 Nov 2015 11:40:23 +0100, Luc Maisonobe wrote:
>> Hi all,
>>
>> It seems the new bootstrap method introduced in fbc327e9 in order
>> to solve MATH-1246 creates some random test failures.
>>
>> The reason is that an EnumeratedRealDistribution instance is
>> created without a random generator (there are no way to pass
>> a random generator in the bootstrap method). This
>> EnumeratedRealDistribution will therefore create a Well19937c
>> generator with the default constructor, which uses time to seed
>> the generator. Depending on the way this generator is seeded,
>> the testBootstrapSmallSamplesWithTies test fails quite often.
>
> If it fails often, doesn't it indicate a problem with the code or
> a wrong assumption about the test?
>

Possibly, but not likely. The bootstrap estimates are not very stable.  The same instability is evident in the R tests of ks.boot.  I will find a way to supply a fixed seed.

I am still working on the full resolution of MAtH-1246 which requires adding jitter. The bootstrap is an optional method never activated by the default implementation.  I am close to having the jitter code ready to commit.

Phil

> Best,
> Gilles
>
>> Would it be possible to pass a random generator somewhere and
>> to configure it with a fixed seed for the Junit tests?
>>
>> best regards,
>> Luc
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] testing issue with Kolmogorov-Smirnov bootstrap methods

Phil Steitz
In reply to this post by Luc Maisonobe-2
On 11/24/15 3:40 AM, Luc Maisonobe wrote:

> Hi all,
>
> It seems the new bootstrap method introduced in fbc327e9 in order
> to solve MATH-1246 creates some random test failures.
>
> The reason is that an EnumeratedRealDistribution instance is
> created without a random generator (there are no way to pass
> a random generator in the bootstrap method). This
> EnumeratedRealDistribution will therefore create a Well19937c
> generator with the default constructor, which uses time to seed
> the generator. Depending on the way this generator is seeded,
> the testBootstrapSmallSamplesWithTies test fails quite often.
>
> Would it be possible to pass a random generator somewhere and
> to configure it with a fixed seed for the Junit tests?

I just changed the code (3_X: 3cfafe051 / master: 5f9cfa6eb) to have
the bootstrap method use the rng configured for the class.  The
tests supply a fixed seed generator to the KS instance, but that was
not being passed by the bootstrap to the EnumeratedDistribution.
Sorry I missed that.

Phil

>
> best regards,
> Luc
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
> .
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] testing issue with Kolmogorov-Smirnov bootstrap methods

Luc Maisonobe-2
Le 24/11/2015 14:34, Phil Steitz a écrit :

> On 11/24/15 3:40 AM, Luc Maisonobe wrote:
>> Hi all,
>>
>> It seems the new bootstrap method introduced in fbc327e9 in order
>> to solve MATH-1246 creates some random test failures.
>>
>> The reason is that an EnumeratedRealDistribution instance is
>> created without a random generator (there are no way to pass
>> a random generator in the bootstrap method). This
>> EnumeratedRealDistribution will therefore create a Well19937c
>> generator with the default constructor, which uses time to seed
>> the generator. Depending on the way this generator is seeded,
>> the testBootstrapSmallSamplesWithTies test fails quite often.
>>
>> Would it be possible to pass a random generator somewhere and
>> to configure it with a fixed seed for the Junit tests?
>
> I just changed the code (3_X: 3cfafe051 / master: 5f9cfa6eb) to have
> the bootstrap method use the rng configured for the class.  The
> tests supply a fixed seed generator to the KS instance, but that was
> not being passed by the bootstrap to the EnumeratedDistribution.
> Sorry I missed that.

Never mind. Thanks a lot for this fast fix.

best regards,
Luc

>
> Phil
>>
>> best regards,
>> Luc
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>> .
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]