commons-math, matrix-toolkits-java and consolidation

classic Classic list List threaded Threaded
93 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: commons-math, matrix-toolkits-java and consolidation

Edward J. Yoon-3
Hmm. It's a really great idea.

I think It could be a top level math project of apache.

On Fri, May 22, 2009 at 7:37 PM, Sam Halliday <[hidden email]> wrote:

>
> Edward, Hama looks fantastic! I fully agree, 'distributed' isn't agreeable
> with commons-math. However, there might be parts of it that are relevant to
> Hama.
>
> We should absolutely ensure that the "ducks are aligned" between
> commons-math and Hama. It would be win-win if Hama were able to use
> commons-math's linear interfaces. commons-math will hopefully be moving to
> use a netlib-java style backend (not user facing), we should ensure that
> ScaLAPACK is handled in the same way.
>
> I have some ideas of people who might be interested... I've encountered them
> as maintainer of MTJ, but never worked with them. I'll dig through my e-mail
> archive and make the intros.
>
>
> Edward J. Yoon-2 wrote:
>>
>> That's really cool.
>>
>> BTW, Can I ask about the plan of data distribution strategies of your
>> 'distributed' package in the future? IMO, it seems, it doesn't sit
>> well with 'common-math' project.
>>
>> If if there is a developer who wants to implement 'distributed', pls
>> let us know, too. I'm working for the Hama
>> (http://incubator.apache.org/hama) with ScaLAPACK members.
>>
>> On Thu, May 14, 2009 at 7:18 PM, Sam Halliday <[hidden email]>
>> wrote:
>>>
>>> Dear all,
>>>
>>> I am a maintainer of the matrix-toolkits-java
>>>
>>>  http://code.google.com/p/matrix-toolkits-java/
>>>
>>> which is a comprehensive collection of matrix data structures, linear
>>> solvers, least squares methods, eigenvalue and singular value
>>> decompositions.
>>>
>>> This note is in regard to the commons-math library. It is clear that our
>>> projects dovetail, especially when I look at "linear" in version 2.0 of
>>> the
>>> API. It would be good if we could either complement or consolidate
>>> efforts,
>>> rather than reproduce.
>>>
>>> It would be excellent if all the functionality of matrix-toolkits-java
>>> were
>>> available as part of commons-math. There is already too much diversity
>>> and
>>> un-maintained maths code out there for Java!
>>>
>>> As a start, I'd like to discourage the use of a solid implementation for
>>> SparseReal{Vector, Matrix}... please prefer an interface approach,
>>> allowing
>>> implementations based on the Templates project:-
>>>
>>>  http://www.netlib.org/templates
>>>
>>> The reason is that the storage implementation should be related to the
>>> type
>>> of data being stored. For example, there are many well-known kinds of
>>> sparse
>>> matrix that are well suited to particular kinds of calculations...
>>> consider
>>> multiplying sparse matrices that you know to be diagonal!
>>>
>>> In general, the netlib.org folk (BLAS/LAPACK) have spent a *lot* of time
>>> thinking about linear algebra and have set up unrivalled standard APIs
>>> which
>>> have been implemented right down to the architecture level. It would be a
>>> major mistake if commons-math didn't build on their good work.
>>>
>>> I believe commons-math should move to a netlib-java backend (allowing the
>>> use of machine optimised BLAS/LAPACK).
>>>
>>>  http://code.google.com/p/netlib-java/
>>>
>>> The largest problems facing MTJ are support for Sparse BLAS/LAPACK and
>>> scalability to parallel architectures which use Parallel BLAS/LAPACK. The
>>> former should be possible with some work within the current API, but I
>>> fear
>>> major API changes would be needed for the latter. I do not want the
>>> commons-math API to walk into this trap without having first considered
>>> future architectures! MTJ has a distributed package, but I am not sure if
>>> this is something that is completely future proof either.
>>>
>>> What say ye'?
>>>
>>> --
>>> Sam
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/commons-math%2C-matrix-toolkits-java-and-consolidation-tp23537813p23537813.html
>>> Sent from the Commons - Dev mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon @ NHN, corp.
>> [hidden email]
>> http://blog.udanax.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/commons-math%2C-matrix-toolkits-java-and-consolidation-tp23537813p23668079.html
> Sent from the Commons - Dev mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>



--
Best Regards, Edward J. Yoon @ NHN, corp.
[hidden email]
http://blog.udanax.org

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: commons-math, matrix-toolkits-java and consolidation

Jin Mingjian
I like the idea about the "top level math project":) Is it possible or
interesting to host a sub-project for DSL to commons-math?(such as an
enhanced Object-supported matlab-compatible script)


2009/5/23 Edward J. Yoon <[hidden email]>

> Hmm. It's a really great idea.
>
> I think It could be a top level math project of apache.
>
> On Fri, May 22, 2009 at 7:37 PM, Sam Halliday <[hidden email]>
> wrote:
> >
> > Edward, Hama looks fantastic! I fully agree, 'distributed' isn't
> agreeable
> > with commons-math. However, there might be parts of it that are relevant
> to
> > Hama.
> >
> > We should absolutely ensure that the "ducks are aligned" between
> > commons-math and Hama. It would be win-win if Hama were able to use
> > commons-math's linear interfaces. commons-math will hopefully be moving
> to
> > use a netlib-java style backend (not user facing), we should ensure that
> > ScaLAPACK is handled in the same way.
> >
> > I have some ideas of people who might be interested... I've encountered
> them
> > as maintainer of MTJ, but never worked with them. I'll dig through my
> e-mail
> > archive and make the intros.
> >
> >
> > Edward J. Yoon-2 wrote:
> >>
> >> That's really cool.
> >>
> >> BTW, Can I ask about the plan of data distribution strategies of your
> >> 'distributed' package in the future? IMO, it seems, it doesn't sit
> >> well with 'common-math' project.
> >>
> >> If if there is a developer who wants to implement 'distributed', pls
> >> let us know, too. I'm working for the Hama
> >> (http://incubator.apache.org/hama) with ScaLAPACK members.
> >>
> >> On Thu, May 14, 2009 at 7:18 PM, Sam Halliday <[hidden email]>
> >> wrote:
> >>>
> >>> Dear all,
> >>>
> >>> I am a maintainer of the matrix-toolkits-java
> >>>
> >>>  http://code.google.com/p/matrix-toolkits-java/
> >>>
> >>> which is a comprehensive collection of matrix data structures, linear
> >>> solvers, least squares methods, eigenvalue and singular value
> >>> decompositions.
> >>>
> >>> This note is in regard to the commons-math library. It is clear that
> our
> >>> projects dovetail, especially when I look at "linear" in version 2.0 of
> >>> the
> >>> API. It would be good if we could either complement or consolidate
> >>> efforts,
> >>> rather than reproduce.
> >>>
> >>> It would be excellent if all the functionality of matrix-toolkits-java
> >>> were
> >>> available as part of commons-math. There is already too much diversity
> >>> and
> >>> un-maintained maths code out there for Java!
> >>>
> >>> As a start, I'd like to discourage the use of a solid implementation
> for
> >>> SparseReal{Vector, Matrix}... please prefer an interface approach,
> >>> allowing
> >>> implementations based on the Templates project:-
> >>>
> >>>  http://www.netlib.org/templates
> >>>
> >>> The reason is that the storage implementation should be related to the
> >>> type
> >>> of data being stored. For example, there are many well-known kinds of
> >>> sparse
> >>> matrix that are well suited to particular kinds of calculations...
> >>> consider
> >>> multiplying sparse matrices that you know to be diagonal!
> >>>
> >>> In general, the netlib.org folk (BLAS/LAPACK) have spent a *lot* of
> time
> >>> thinking about linear algebra and have set up unrivalled standard APIs
> >>> which
> >>> have been implemented right down to the architecture level. It would be
> a
> >>> major mistake if commons-math didn't build on their good work.
> >>>
> >>> I believe commons-math should move to a netlib-java backend (allowing
> the
> >>> use of machine optimised BLAS/LAPACK).
> >>>
> >>>  http://code.google.com/p/netlib-java/
> >>>
> >>> The largest problems facing MTJ are support for Sparse BLAS/LAPACK and
> >>> scalability to parallel architectures which use Parallel BLAS/LAPACK.
> The
> >>> former should be possible with some work within the current API, but I
> >>> fear
> >>> major API changes would be needed for the latter. I do not want the
> >>> commons-math API to walk into this trap without having first considered
> >>> future architectures! MTJ has a distributed package, but I am not sure
> if
> >>> this is something that is completely future proof either.
> >>>
> >>> What say ye'?
> >>>
> >>> --
> >>> Sam
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://www.nabble.com/commons-math%2C-matrix-toolkits-java-and-consolidation-tp23537813p23537813.html
> >>> Sent from the Commons - Dev mailing list archive at Nabble.com.
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: [hidden email]
> >>> For additional commands, e-mail: [hidden email]
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon @ NHN, corp.
> >> [hidden email]
> >> http://blog.udanax.org
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >>
> >
> > --
> > View this message in context:
> http://www.nabble.com/commons-math%2C-matrix-toolkits-java-and-consolidation-tp23537813p23668079.html
> > Sent from the Commons - Dev mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
>
> --
> Best Regards, Edward J. Yoon @ NHN, corp.
> [hidden email]
> http://blog.udanax.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: commons-math, matrix-toolkits-java and consolidation

Ted Dunning
-1 to a TLP at the current time.

I really agree that the current momentum may be a flash.

On Sat, May 23, 2009 at 6:28 AM, Jin Mingjian <[hidden email]> wrote:

> I like the idea about the "top level math project":) Is it possible or
> interesting to host a sub-project for DSL to commons-math?(such as an
> enhanced Object-supported matlab-compatible script)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Bill Barker
In reply to this post by Bill Barker
Ok, most of the rest look like transient classes (e.g. all of
linear.decomposition).  So asking for what needs to be Serialized, and what
doesn't.

----- Original Message -----
From: "Bill Barker" <[hidden email]>
To: "Commons Developers List" <[hidden email]>
Sent: Saturday, May 23, 2009 12:13 AM
Subject: Re: [math] Serialization


>
> ----- Original Message -----
> From: "Luc Maisonobe" <[hidden email]>
> To: "Commons Developers List" <[hidden email]>
> Sent: Friday, May 22, 2009 4:19 AM
> Subject: Re: [math] Serialization
>
>
>> [I changed the subject to help follow the thread]
>>
>> Phil Steitz a écrit :
>>> Luc Maisonobe wrote:
>>>> Ted Dunning a écrit :
>>>>
>>>>> In favor or not, Serializable shouldn't in in widely used interfaces.
>>>>>
>>>>> As an example, a Lucene index is a reasonable implementation of a
>>>>> sparse
>>>>> matrix.
>>>>>
>>>>> Would you require that I have to figure out how to make it
>>>>> serializable just
>>>>> because I declare it as a Matrix?
>>>>>
>>>>> Do you imagine that most developers will do more than just punt in
>>>>> such a
>>>>> situation if the interface absolutely requires that the object be
>>>>> serializable?
>>>>>
>>>>> Leave it to particular implementations to be serializable or not.
>>>>> Please,
>>>>> please, please don't force it into the contract for all
>>>>> implementations.
>>>>>
>>>>
>>>> So we have reached a consensus: remove Serializable from interfaces and
>>>> push it down to implementations only.
>>>>
>>> +1
>>
>> There is one interface at least for which I ask to retain Externalizable
>> (not really the same as Serializable, but in the same spirit). It is the
>> StepInterpolator interface in the ode.sampling package. Externalization
>> for this interface is a desired and important feature used for example
>> in the ContinuousOutputModel class. The interface is not intended to be
>> implemented by users, and in fact even the class implementing it are not
>> directly visible by users: instances are directly built by ODE
>> integrators (each integrator has its own interpolator).
>>
>>>> Any volunteer to do this rather boring work ?
>
> I can take a stab at it (but may have fewer spare cycles than sebb).
>
>>>>
>>> I wish I could say yes, but I am running out of buffer space atm ;)
>>>
>>> Phil
>>>>
>>>>> On Thu, May 21, 2009 at 8:20 PM, Bill Barker
>>>>> <[hidden email]>wrote:
>>>>>
>>>>>
>>>>>> - I *strongly* urge you to remove Serializable from everything!
>>>>>> Please, we
>>>>>>
>>>>>>> did this in MTJ and it turned out to be a major pain. A more
>>>>>>> appropriate
>>>>>>> approach is to define a class for reading/writing Matrix Market
>>>>>>> files.
>>>>>>> This
>>>>>>> can be a new feature in 2.1. If you're going to leave it, at least
>>>>>>> document
>>>>>>> that the Serializable form is not guaranteed to remain compatible
>>>>>>> across
>>>>>>> versions.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Like Luc, I'm generallly in favor of Serializable.  Since some of
>>>>>> the posts
>>>>>> on this thread have suggested problems with the current
>>>>>> implementation, I'll
>>>>>> try and run some tests over the (what is here, long) weekend.
>>>>>> Again, no
>>>>>> consensus so not doing anything immediately.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Ted Dunning
Bill,

Did you commit those changes?

I ask because I see similar issues with abstract and static classes.  Gamma
is an example of the static class issue.  This class has only static
methods, but it is marked as serializable (why?).  ContinuedFraction is a
good example of an abstract class that is marked as serializable.  This
latter case leads to code that is either strange or incorrect.  The strange
version has a version identifier even on anonymous classes as in this code
here:

         // create continued fraction
            ContinuedFraction cf = new ContinuedFraction() {

                private static final long serialVersionUID =
5378525034886164398L;

                @Override
                protected double getA(int n, double x) {
                    return ((2.0 * n) + 1.0) - a + x;
                }

                @Override
                protected double getB(int n, double x) {
                    return n * (a - n);
                }
            };

            ret = 1.0 / cf.evaluate(x, epsilon, maxIterations);
            ret = Math.exp(-x + (a * Math.log(x)) - logGamma(a)) * ret;

If the serialVersionUID is ommitted, the anonymous class would be
incorrectly serialized because it would inherit it's parent UID (I think).
More importantly, I can't imagine why an abstract class like
ContinuedFraction should be serializable.

On Sun, May 24, 2009 at 6:10 PM, Bill Barker <[hidden email]>wrote:

> Ok, most of the rest look like transient classes (e.g. all of
> linear.decomposition).  So asking for what needs to be Serialized, and what
> doesn't.
>



--
Ted Dunning, CTO
DeepDyve
Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Luc Maisonobe
In reply to this post by Bill Barker
Bill Barker a écrit :
> Ok, most of the rest look like transient classes (e.g. all of
> linear.decomposition).  So asking for what needs to be Serialized, and
> what doesn't.

As the main developer of the decomposition package, I can say the
reasons for having these classes Serializable are the same bad reasons I
had for all the other ones ... So as far as I am concerned you can
safely remove Serializable for all these interfaces and also for the
implementation classes.

Sorry for having caused such a mess.

Luc

>
> ----- Original Message ----- From: "Bill Barker" <[hidden email]>
> To: "Commons Developers List" <[hidden email]>
> Sent: Saturday, May 23, 2009 12:13 AM
> Subject: Re: [math] Serialization
>
>
>>
>> ----- Original Message ----- From: "Luc Maisonobe"
>> <[hidden email]>
>> To: "Commons Developers List" <[hidden email]>
>> Sent: Friday, May 22, 2009 4:19 AM
>> Subject: Re: [math] Serialization
>>
>>
>>> [I changed the subject to help follow the thread]
>>>
>>> Phil Steitz a écrit :
>>>> Luc Maisonobe wrote:
>>>>> Ted Dunning a écrit :
>>>>>
>>>>>> In favor or not, Serializable shouldn't in in widely used interfaces.
>>>>>>
>>>>>> As an example, a Lucene index is a reasonable implementation of a
>>>>>> sparse
>>>>>> matrix.
>>>>>>
>>>>>> Would you require that I have to figure out how to make it
>>>>>> serializable just
>>>>>> because I declare it as a Matrix?
>>>>>>
>>>>>> Do you imagine that most developers will do more than just punt in
>>>>>> such a
>>>>>> situation if the interface absolutely requires that the object be
>>>>>> serializable?
>>>>>>
>>>>>> Leave it to particular implementations to be serializable or not.
>>>>>> Please,
>>>>>> please, please don't force it into the contract for all
>>>>>> implementations.
>>>>>>
>>>>>
>>>>> So we have reached a consensus: remove Serializable from interfaces
>>>>> and
>>>>> push it down to implementations only.
>>>>>
>>>> +1
>>>
>>> There is one interface at least for which I ask to retain Externalizable
>>> (not really the same as Serializable, but in the same spirit). It is the
>>> StepInterpolator interface in the ode.sampling package. Externalization
>>> for this interface is a desired and important feature used for example
>>> in the ContinuousOutputModel class. The interface is not intended to be
>>> implemented by users, and in fact even the class implementing it are not
>>> directly visible by users: instances are directly built by ODE
>>> integrators (each integrator has its own interpolator).
>>>
>>>>> Any volunteer to do this rather boring work ?
>>
>> I can take a stab at it (but may have fewer spare cycles than sebb).
>>
>>>>>
>>>> I wish I could say yes, but I am running out of buffer space atm ;)
>>>>
>>>> Phil
>>>>>
>>>>>> On Thu, May 21, 2009 at 8:20 PM, Bill Barker
>>>>>> <[hidden email]>wrote:
>>>>>>
>>>>>>
>>>>>>> - I *strongly* urge you to remove Serializable from everything!
>>>>>>> Please, we
>>>>>>>
>>>>>>>> did this in MTJ and it turned out to be a major pain. A more
>>>>>>>> appropriate
>>>>>>>> approach is to define a class for reading/writing Matrix Market
>>>>>>>> files.
>>>>>>>> This
>>>>>>>> can be a new feature in 2.1. If you're going to leave it, at least
>>>>>>>> document
>>>>>>>> that the Serializable form is not guaranteed to remain compatible
>>>>>>>> across
>>>>>>>> versions.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Like Luc, I'm generallly in favor of Serializable.  Since some of
>>>>>>> the posts
>>>>>>> on this thread have suggested problems with the current
>>>>>>> implementation, I'll
>>>>>>> try and run some tests over the (what is here, long) weekend.
>>>>>>> Again, no
>>>>>>> consensus so not doing anything immediately.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Bill Barker
In reply to this post by Ted Dunning

----- Original Message -----
From: "Ted Dunning" <[hidden email]>
To: "Commons Developers List" <[hidden email]>
Sent: Sunday, May 24, 2009 11:47 PM
Subject: Re: [math] Serialization


> Bill,
>
> Did you commit those changes?
>

Yes, but just for o.a.c.m.linear.

> I ask because I see similar issues with abstract and static classes.
> Gamma
> is an example of the static class issue.  This class has only static
> methods, but it is marked as serializable (why?).  ContinuedFraction is a
> good example of an abstract class that is marked as serializable.  This
> latter case leads to code that is either strange or incorrect.  The
> strange
> version has a version identifier even on anonymous classes as in this code
> here:

I didn't say that I was finished ;)

>
>         // create continued fraction
>            ContinuedFraction cf = new ContinuedFraction() {
>
>                private static final long serialVersionUID =
> 5378525034886164398L;
>
>                @Override
>                protected double getA(int n, double x) {
>                    return ((2.0 * n) + 1.0) - a + x;
>                }
>
>                @Override
>                protected double getB(int n, double x) {
>                    return n * (a - n);
>                }
>            };
>
>            ret = 1.0 / cf.evaluate(x, epsilon, maxIterations);
>            ret = Math.exp(-x + (a * Math.log(x)) - logGamma(a)) * ret;
>
> If the serialVersionUID is ommitted, the anonymous class would be
> incorrectly serialized because it would inherit it's parent UID (I think).
> More importantly, I can't imagine why an abstract class like
> ContinuedFraction should be serializable.
>
> On Sun, May 24, 2009 at 6:10 PM, Bill Barker
> <[hidden email]>wrote:
>
>> Ok, most of the rest look like transient classes (e.g. all of
>> linear.decomposition).  So asking for what needs to be Serialized, and
>> what
>> doesn't.
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Bill Barker
In reply to this post by Luc Maisonobe

----- Original Message -----
From: "Luc Maisonobe" <[hidden email]>
To: "Commons Developers List" <[hidden email]>
Sent: Monday, May 25, 2009 12:39 PM
Subject: Re: [math] Serialization


> Bill Barker a écrit :
>> Ok, most of the rest look like transient classes (e.g. all of
>> linear.decomposition).  So asking for what needs to be Serialized, and
>> what doesn't.
>
> As the main developer of the decomposition package, I can say the
> reasons for having these classes Serializable are the same bad reasons I
> had for all the other ones ... So as far as I am concerned you can
> safely remove Serializable for all these interfaces and also for the
> implementation classes.
>
> Sorry for having caused such a mess.
>

You should see the mess Phil caused in stats ;).  Almost (or more likely
all) of the classes under stats and subpackages look transient to me (e.g.
why would anyone want to serialize Kurtosis?).  But again, want to ask
advice before going terminator on those packages.  IMHO, we'll need to write
junit tests for the ones that we leave Serializable, since I already found
(and fixed) two classes in linear that couldn't be de-serialized.

> Luc
>
>>
>> ----- Original Message ----- From: "Bill Barker"
>> <[hidden email]>
>> To: "Commons Developers List" <[hidden email]>
>> Sent: Saturday, May 23, 2009 12:13 AM
>> Subject: Re: [math] Serialization
>>
>>
>>>
>>> ----- Original Message ----- From: "Luc Maisonobe"
>>> <[hidden email]>
>>> To: "Commons Developers List" <[hidden email]>
>>> Sent: Friday, May 22, 2009 4:19 AM
>>> Subject: Re: [math] Serialization
>>>
>>>
>>>> [I changed the subject to help follow the thread]
>>>>
>>>> Phil Steitz a écrit :
>>>>> Luc Maisonobe wrote:
>>>>>> Ted Dunning a écrit :
>>>>>>
>>>>>>> In favor or not, Serializable shouldn't in in widely used
>>>>>>> interfaces.
>>>>>>>
>>>>>>> As an example, a Lucene index is a reasonable implementation of a
>>>>>>> sparse
>>>>>>> matrix.
>>>>>>>
>>>>>>> Would you require that I have to figure out how to make it
>>>>>>> serializable just
>>>>>>> because I declare it as a Matrix?
>>>>>>>
>>>>>>> Do you imagine that most developers will do more than just punt in
>>>>>>> such a
>>>>>>> situation if the interface absolutely requires that the object be
>>>>>>> serializable?
>>>>>>>
>>>>>>> Leave it to particular implementations to be serializable or not.
>>>>>>> Please,
>>>>>>> please, please don't force it into the contract for all
>>>>>>> implementations.
>>>>>>>
>>>>>>
>>>>>> So we have reached a consensus: remove Serializable from interfaces
>>>>>> and
>>>>>> push it down to implementations only.
>>>>>>
>>>>> +1
>>>>
>>>> There is one interface at least for which I ask to retain
>>>> Externalizable
>>>> (not really the same as Serializable, but in the same spirit). It is
>>>> the
>>>> StepInterpolator interface in the ode.sampling package. Externalization
>>>> for this interface is a desired and important feature used for example
>>>> in the ContinuousOutputModel class. The interface is not intended to be
>>>> implemented by users, and in fact even the class implementing it are
>>>> not
>>>> directly visible by users: instances are directly built by ODE
>>>> integrators (each integrator has its own interpolator).
>>>>
>>>>>> Any volunteer to do this rather boring work ?
>>>
>>> I can take a stab at it (but may have fewer spare cycles than sebb).
>>>
>>>>>>
>>>>> I wish I could say yes, but I am running out of buffer space atm ;)
>>>>>
>>>>> Phil
>>>>>>
>>>>>>> On Thu, May 21, 2009 at 8:20 PM, Bill Barker
>>>>>>> <[hidden email]>wrote:
>>>>>>>
>>>>>>>
>>>>>>>> - I *strongly* urge you to remove Serializable from everything!
>>>>>>>> Please, we
>>>>>>>>
>>>>>>>>> did this in MTJ and it turned out to be a major pain. A more
>>>>>>>>> appropriate
>>>>>>>>> approach is to define a class for reading/writing Matrix Market
>>>>>>>>> files.
>>>>>>>>> This
>>>>>>>>> can be a new feature in 2.1. If you're going to leave it, at least
>>>>>>>>> document
>>>>>>>>> that the Serializable form is not guaranteed to remain compatible
>>>>>>>>> across
>>>>>>>>> versions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Like Luc, I'm generallly in favor of Serializable.  Since some of
>>>>>>>> the posts
>>>>>>>> on this thread have suggested problems with the current
>>>>>>>> implementation, I'll
>>>>>>>> try and run some tests over the (what is here, long) weekend.
>>>>>>>> Again, no
>>>>>>>> consensus so not doing anything immediately.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: [hidden email]
>>>>>> For additional commands, e-mail: [hidden email]
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: [hidden email]
>>>>> For additional commands, e-mail: [hidden email]
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [hidden email]
>>>> For additional commands, e-mail: [hidden email]
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Ted Dunning
This sounds like the germ of a repair process.

Nuke all Serializable declarations without test cases.  Then, add them back
where people care enough to guarantee correctness by demonstrating a correct
round trip.

On Mon, May 25, 2009 at 4:32 PM, Bill Barker <[hidden email]>wrote:

> But again, want to ask advice before going terminator on those packages.
>  IMHO, we'll need to write junit tests for the ones that we leave
> Serializable, since I already found (and fixed) two classes in linear that
> couldn't be de-serialized.




--
Ted Dunning, CTO
DeepDyve
Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Bill Barker

----- Original Message -----
From: "Ted Dunning" <[hidden email]>
To: "Commons Developers List" <[hidden email]>
Sent: Monday, May 25, 2009 4:52 PM
Subject: Re: [math] Serialization


> This sounds like the germ of a repair process.
>
> Nuke all Serializable declarations without test cases.  Then, add them
> back
> where people care enough to guarantee correctness by demonstrating a
> correct
> round trip.
>

Would get my +1, but for the moment stats.** is such a mess (UID inherited
from an abstract base class 3-4 levels up) that I'd like a consensus before
putting hours into fixing it.

> On Mon, May 25, 2009 at 4:32 PM, Bill Barker
> <[hidden email]>wrote:
>
>> But again, want to ask advice before going terminator on those packages.
>>  IMHO, we'll need to write junit tests for the ones that we leave
>> Serializable, since I already found (and fixed) two classes in linear
>> that
>> couldn't be de-serialized.
>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: commons-math, matrix-toolkits-java and consolidation

Samuel Halliday
In reply to this post by Samuel Halliday
I discovered/remembered something that might make the inclusion of Java source translated BLAS/LAPACK code troublesome. The F2J translator does a two step process... it generates Java from the Fortran, but then has to do some jiggery-pokery to the class file to handle GOTO statements. That means any source code would not compile correctly without an additional post-compilation stage.

The obvious answer is to just depend on the translated binaries... which could of course be distributed with commons-math. The F2J translator allows the namespace to be specified, so there is no reason the translated binaries couldn't be "org.apache.commons.math.{blas, lapack}" etc.

I'm currently asking the author if he can create a Maven bundle of the resulting binaries. That leaves external dependency an option. I'd greatly appreciate anybody contacting me off-list to help with this if they have any Maven experience!

Sam Halliday wrote
Dear all,

I am a maintainer of the matrix-toolkits-java

  http://code.google.com/p/matrix-toolkits-java/

which is a comprehensive collection of matrix data structures, linear solvers, least squares methods, eigenvalue and singular value decompositions.

This note is in regard to the commons-math library. It is clear that our projects dovetail, especially when I look at "linear" in version 2.0 of the API. It would be good if we could either complement or consolidate efforts, rather than reproduce.

It would be excellent if all the functionality of matrix-toolkits-java were available as part of commons-math. There is already too much diversity and un-maintained maths code out there for Java!

As a start, I'd like to discourage the use of a solid implementation for SparseReal{Vector, Matrix}... please prefer an interface approach, allowing implementations based on the Templates project:-

  http://www.netlib.org/templates

The reason is that the storage implementation should be related to the type of data being stored. For example, there are many well-known kinds of sparse matrix that are well suited to particular kinds of calculations... consider multiplying sparse matrices that you know to be diagonal!

In general, the netlib.org folk (BLAS/LAPACK) have spent a *lot* of time thinking about linear algebra and have set up unrivalled standard APIs which have been implemented right down to the architecture level. It would be a major mistake if commons-math didn't build on their good work.

I believe commons-math should move to a netlib-java backend (allowing the use of machine optimised BLAS/LAPACK).

  http://code.google.com/p/netlib-java/

The largest problems facing MTJ are support for Sparse BLAS/LAPACK and scalability to parallel architectures which use Parallel BLAS/LAPACK. The former should be possible with some work within the current API, but I fear major API changes would be needed for the latter. I do not want the commons-math API to walk into this trap without having first considered future architectures! MTJ has a distributed package, but I am not sure if this is something that is completely future proof either.

What say ye'?

--
Sam
Reply | Threaded
Open this post in threaded view
|

Re: commons-math, matrix-toolkits-java and consolidation

Samuel Halliday
In reply to this post by Samuel Halliday
In preparation for the post-2.0 discussions, I've created three issues on the tracker to deal with the consolidation of F2J (aka JLAPACK), netlib-java and matrix-toolkits-java.

 http://issues.apache.org/jira/browse/MATH-269
 http://issues.apache.org/jira/browse/MATH-270
 http://issues.apache.org/jira/browse/MATH-271

Sam Halliday wrote
...
This note is in regard to the commons-math library. It is clear that our projects dovetail, especially when I look at "linear" in version 2.0 of the API. It would be good if we could either complement or consolidate efforts, rather than reproduce.
...
Reply | Threaded
Open this post in threaded view
|

Re: [math] Serialization

Phil Steitz
In reply to this post by Bill Barker
Bill Barker wrote:

>
> ----- Original Message ----- From: "Ted Dunning" <[hidden email]>
> To: "Commons Developers List" <[hidden email]>
> Sent: Monday, May 25, 2009 4:52 PM
> Subject: Re: [math] Serialization
>
>
>> This sounds like the germ of a repair process.
>>
>> Nuke all Serializable declarations without test cases.  Then, add
>> them back
>> where people care enough to guarantee correctness by demonstrating a
>> correct
>> round trip.
>>
>
> Would get my +1, but for the moment stats.** is such a mess (UID
> inherited from an abstract base class 3-4 levels up) that I'd like a
> consensus before putting hours into fixing it.
+1 to fix.  I was +1 in the early days when things were "overserialized"
(IMO)  [1] and am +1 today.

[1] http://markmail.org/thread/ui47gz75ol5euxj4

Phil

>
>> On Mon, May 25, 2009 at 4:32 PM, Bill Barker
>> <[hidden email]>wrote:
>>
>>> But again, want to ask advice before going terminator on those
>>> packages.
>>>  IMHO, we'll need to write junit tests for the ones that we leave
>>> Serializable, since I already found (and fixed) two classes in
>>> linear that
>>> couldn't be de-serialized.
>>
>>
>>
>>
>> --
>> Ted Dunning, CTO
>> DeepDyve
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12345