[math] Is overflow dealt with properly in Dfp?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[math] Is overflow dealt with properly in Dfp?

Sébastien Brisard
Hi,
I'm currently working on MATH-753, and the underlying accuracy issues.
I therefore implemented the Gamma function for Dfps, and came accross
a few strange things. For example, althoug the javadoc states that the
maximum exponent is 32768, there is no systematic check on this limit.
I suppose overflow should happen in the following case
{code}
    DfpField field = new DfpField(40);
    final Dfp x = field.getOne().power10K(Integer.MAX_VALUE);
    System.out.println(x);
    System.out.println(x.isInfinite());
{code}

Instead, what happens is
{console}
1.000000000000000000000000000000000000e-4
false
{console}

The surprising answer comes from the fact that power10K actually
increments its integer argument. I do not think the above printout is
the expected behavior, am I wrong? I think this is potentially a vast
issue, because a quick check seems to indicate that "this.nans ==
INFINITE" is often checked in class Dfp, but I don't think the
statement "this.nans = INFINITE" ever occurs, which would mean that
overflow is never dealt with properly. Again, I'm probably wrong, I'd
like to have the feeling of someone more used to this class than I am.

Sébastien


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] Is overflow dealt with properly in Dfp?

Luc Maisonobe-2
On Wed, 11 Apr 2012 06:23:56 +0200, Sébastien Brisard wrote:
> Hi,

Hi Sébastien,

> I'm currently working on MATH-753, and the underlying accuracy
> issues.
> I therefore implemented the Gamma function for Dfps, and came accross
> a few strange things. For example, althoug the javadoc states that
> the
> maximum exponent is 32768, there is no systematic check on this
> limit.
> I suppose overflow should happen in the following case
> {code}
>     DfpField field = new DfpField(40);
>     final Dfp x = field.getOne().power10K(Integer.MAX_VALUE);
>     System.out.println(x);
>     System.out.println(x.isInfinite());
> {code}
>
> Instead, what happens is
> {console}
> 1.000000000000000000000000000000000000e-4
> false
> {console}
>
> The surprising answer comes from the fact that power10K actually
> increments its integer argument. I do not think the above printout is
> the expected behavior, am I wrong? I think this is potentially a vast
> issue, because a quick check seems to indicate that "this.nans ==
> INFINITE" is often checked in class Dfp, but I don't think the
> statement "this.nans = INFINITE" ever occurs, which would mean that
> overflow is never dealt with properly. Again, I'm probably wrong, I'd
> like to have the feeling of someone more used to this class than I
> am.

I obviously agree this result is wrong ...

Is William (the creator of both Dfp and FastMath) still around to deal
with this issue ?

Luc

>
> Sébastien
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] Is overflow dealt with properly in Dfp?

Sébastien Brisard
Hello,

>
>> I'm currently working on MATH-753, and the underlying accuracy issues.
>> I therefore implemented the Gamma function for Dfps, and came accross
>> a few strange things. For example, althoug the javadoc states that the
>> maximum exponent is 32768, there is no systematic check on this limit.
>> I suppose overflow should happen in the following case
>> {code}
>>    DfpField field = new DfpField(40);
>>    final Dfp x = field.getOne().power10K(Integer.MAX_VALUE);
>>    System.out.println(x);
>>    System.out.println(x.isInfinite());
>> {code}
>>
>> Instead, what happens is
>> {console}
>> 1.000000000000000000000000000000000000e-4
>> false
>> {console}
>>
>> The surprising answer comes from the fact that power10K actually
>> increments its integer argument. I do not think the above printout is
>> the expected behavior, am I wrong? I think this is potentially a vast
>> issue, because a quick check seems to indicate that "this.nans ==
>> INFINITE" is often checked in class Dfp, but I don't think the
>> statement "this.nans = INFINITE" ever occurs, which would mean that
>> overflow is never dealt with properly. Again, I'm probably wrong, I'd
>> like to have the feeling of someone more used to this class than I am.
>
>
> I obviously agree this result is wrong ...
>
> Is William (the creator of both Dfp and FastMath) still around to deal with
> this issue ?
>
I'm afraid he probably is not... I feel like walking on eggs on this
issue, because I'm not yet fully familiar with the Dfp class. If
everyone agrees, I'll wait a little bit before I file a bug report.
Sébastien


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]