[jira] [Created] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Created] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
------------------------------------------------------------------------------------------

                 Key: MATH-718
                 URL: https://issues.apache.org/jira/browse/MATH-718
             Project: Commons Math
          Issue Type: Bug
    Affects Versions: 2.2, 3.0
            Reporter: Yuji Uchiyama


The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.

{{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}

This returns 499525, though it should be 499999.

I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162326#comment-13162326 ]

Sébastien Brisard commented on MATH-718:
----------------------------------------

Hi Yuji,
thanks for reporting this. For version 3.0, we are currently reshaping the package distribution, and the next item on the list was inverseCumulativeProbability for discrete distributions (see MATH-699). So I'll gladly look into the reported bug.
As for version 2.2, I don't really know how to include the correction (if possible).
Sébastien
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Issue Comment Edited] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162326#comment-13162326 ]

Sébastien Brisard edited comment on MATH-718 at 12/5/11 6:53 AM:
-----------------------------------------------------------------

Hi Yuji,
thanks for reporting this. For version 3.0, we are currently reshaping the package distribution, and this will probably get resolved once we are over with MATH-692.
Best regards,
Sébastien
               
      was (Author: celestin):
    Hi Yuji,
thanks for reporting this. For version 3.0, we are currently reshaping the package distribution, and the next item on the list was inverseCumulativeProbability for discrete distributions (see MATH-699). So I'll gladly look into the reported bug.
As for version 2.2, I don't really know how to include the correction (if possible).
Sébastien
                 

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170599#comment-13170599 ]

Christian Winter commented on MATH-718:
---------------------------------------

There seem to be stability problems with Beta.regularizedBeta(...) when using extreme parameters. {{PascalDistribution.cumulativeProbability(Integer.MAX_VALUE)}} returns {{NaN}} instead of 1. We should look for a way to avoid both infinite values and NaNs in the implementation of the regularized beta function.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       
Reply | Threaded
Open this post in threaded view
|

[jira] [Resolved] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

     [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sébastien Brisard resolved MATH-718.
------------------------------------

       Resolution: Later
    Fix Version/s: 4.0
                   3.1
         Assignee:     (was: Sébastien Brisard)

As rightly pointed out by Christian, this issue is strongly related to MATH-738.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Reopened] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

     [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sébastien Brisard reopened MATH-718:
------------------------------------


Sorry, I only meant to postpone this issue.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252598#comment-13252598 ]

James Kaufman commented on MATH-718:
------------------------------------

I just wanted to let you know that our open source project (http://www.eclipse.org/stem/) needs this function and we are eagerly awaiting the update. We are experiencing the issue of wrong values for large trials.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252722#comment-13252722 ]

Sébastien Brisard commented on MATH-718:
----------------------------------------

Hi James,
thanks for your interest. STEM is a very interesting project!
I will try and find a fix for this issue as soon as possible. Any ideas are welcome!
Sébastien
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253360#comment-13253360 ]

Thomas Neidhart commented on MATH-718:
--------------------------------------

The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:

{noformat}
    public double cumulativeProbability(int x) {
        double ret;
        if (x < 0) {
            ret = 0.0;
        } else {
            ret = Beta.regularizedBeta(probabilityOfSuccess,
                    numberOfSuccesses, x + 1);
        }
        return ret;
    }
{noformat}

when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Issue Comment Edited] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253360#comment-13253360 ]

Thomas Neidhart edited comment on MATH-718 at 4/13/12 1:22 PM:
---------------------------------------------------------------

The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:

{noformat}
    public double cumulativeProbability(int x) {
        double ret;
        if (x < 0) {
            ret = 0.0;
        } else {
            ret = Beta.regularizedBeta(probabilityOfSuccess,
                    numberOfSuccesses, x + 1);
        }
        return ret;
    }
{noformat}

when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.

Edit: Similar things happen btw also in other Distribution implementations, so it should be fixed also there, e.g. BinomialDistribution
               
      was (Author: tn):
    The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:

{noformat}
    public double cumulativeProbability(int x) {
        double ret;
        if (x < 0) {
            ret = 0.0;
        } else {
            ret = Beta.regularizedBeta(probabilityOfSuccess,
                    numberOfSuccesses, x + 1);
        }
        return ret;
    }
{noformat}

when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.
                 

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Issue Comment Edited] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253360#comment-13253360 ]

Thomas Neidhart edited comment on MATH-718 at 4/13/12 1:25 PM:
---------------------------------------------------------------

The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:

{noformat}
    public double cumulativeProbability(int x) {
        double ret;
        if (x < 0) {
            ret = 0.0;
        } else {
            ret = Beta.regularizedBeta(probabilityOfSuccess,
                    numberOfSuccesses, x + 1);
        }
        return ret;
    }
{noformat}

when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, so it should be changed to something like "1L + x" to enforce a long addition.

Edit: Similar things happen btw also in other Distribution implementations, so it should be fixed also there, e.g. BinomialDistribution
               
      was (Author: tn):
    The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:

{noformat}
    public double cumulativeProbability(int x) {
        double ret;
        if (x < 0) {
            ret = 0.0;
        } else {
            ret = Beta.regularizedBeta(probabilityOfSuccess,
                    numberOfSuccesses, x + 1);
        }
        return ret;
    }
{noformat}

when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.

Edit: Similar things happen btw also in other Distribution implementations, so it should be fixed also there, e.g. BinomialDistribution
                 

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253381#comment-13253381 ]

Thomas Neidhart commented on MATH-718:
--------------------------------------

The problem is not only related to the Beta function, also the ContinuedFraction.evaluate is numerically unstable.

The reason the cumulativeProbability returns infinity instead of NaN is because the evaluate return 0.0 when called from Beta.regularizedBeta, which leads to a division by zero. The used default epsilon of 10e-15 seems also quite restrictive, when relaxing the epsilon I got much better results (e.g. with 10e-5 I got a result of 499997).
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279875#comment-13279875 ]

Thomas Neidhart commented on MATH-718:
--------------------------------------

I further looked into this with relation to MATH-785. First of all, in the original bug report, the reporter mentions that the expected result should be 499999 which is wrong, imho it should be 500000.

After implementing the modified Lentz-Thompson algorithm, the results for the BinomialDistribution of large trials show correct results.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

     [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Neidhart updated MATH-718:
---------------------------------

    Attachment: MATH-718.diff

The attached diff files shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.

Need to go through all failing tests and check the reason why they fail with this patch.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>         Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

     [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Neidhart updated MATH-718:
---------------------------------

    Attachment:     (was: MATH-718.diff)
   

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>         Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

     [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Neidhart updated MATH-718:
---------------------------------

    Attachment: MATH-718.diff
   

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>         Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Comment Edited] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279877#comment-13279877 ]

Thomas Neidhart edited comment on MATH-718 at 5/20/12 9:59 PM:
---------------------------------------------------------------

The attached diff file shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.

Need to go through all failing tests and check the reason why they fail with this patch.

Edit: re-uploaded the diff file as it was broken.
               
      was (Author: tn):
    The attached diff files shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.

Need to go through all failing tests and check the reason why they fail with this patch.
                 

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>         Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Comment Edited] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279877#comment-13279877 ]

Thomas Neidhart edited comment on MATH-718 at 5/20/12 10:05 PM:
----------------------------------------------------------------

The attached diff file shows the (preliminary) implementation of the modified Lentz-Thompson algorithm.

Edit: re-uploaded the diff file as it was broken.

Edit2: the failing unit tests I mentioned before were due to a wrong loop, the latest diff shows no unit test errors.
               
      was (Author: tn):
    The attached diff file shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.

Need to go through all failing tests and check the reason why they fail with this patch.

Edit: re-uploaded the diff file as it was broken.
                 

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>            Assignee: Sébastien Brisard
>             Fix For: 3.1, 4.0
>
>         Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply | Threaded
Open this post in threaded view
|

[jira] [Resolved] (MATH-718) inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.

Gary D. Gregory (Jira)
In reply to this post by Gary D. Gregory (Jira)

     [ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Neidhart resolved MATH-718.
----------------------------------

       Resolution: Fixed
    Fix Version/s:     (was: 4.0)

Fixed in r1341171.
               

> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
>                 Key: MATH-718
>                 URL: https://issues.apache.org/jira/browse/MATH-718
>             Project: Commons Math
>          Issue Type: Bug
>    Affects Versions: 2.2, 3.0
>            Reporter: Yuji Uchiyama
>             Fix For: 3.1
>
>         Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials.  Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN.  As the result the checkedCumulativeProbability method doesn't work as expected.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira