Commons-Collections enhanced with Java Generics Support

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Commons-Collections enhanced with Java Generics Support

John Watkinson
Dear Jakarta Commons developers,

We are pleased to announce that we have enhanced the Commons-Collections
with Java 1.5 generics support! We have the results hosted on SourceForge:

http://collections.sf.net

We have done our utmost to maintain the exact same functionality as
Collections 3.1. We have eliminated all deprecated classes that were
scheduled for removal in 4.0. Furthermore, we have deprecated all the
TypedXXX classes since they are made obsolete by generics.

The following classes were not converted, or were only partially converted:

- MultiMap and derivatives: MultiMap with generics breaks the
java.util.Map contract, so only partial conversion was possible.
- All TransformedXXX collections: These classes break the
java.util.Collection contract (or the java.util.Map contract, as
appropriate) once generics are added. They were not converted to support
generics at all.

We are considering adding MorphCollection, MorphMap, MorphSet, etc.
interfaces to support MultiMaps and TransformedXXX collections. These
interfaces would be the same as their non-morph counterparts, but
'getting' would be templated independently of 'setting/adding'. For
example, the MultiMap<K,V> interface would extend
MorphMap<K,V,Collection<V>>. However, this is a large step as it would
move these collections out from under the standard collection interfaces
defined in java.util. We welcome any suggestions regarding this.

The collections pass all the unit tests. We have released it as a beta,
but plan to move quickly towards a final release. We look forward to any
suggestions and feedback from the community.

Sincerely,

 Matt Hall and John Watkinson

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

James Carman
Why can't we host this project at ASF?  Couldn't we create a new branch for
JDK5 collections or something?

-----Original Message-----
From: John Watkinson [mailto:[hidden email]]
Sent: Monday, May 23, 2005 12:10 PM
To: [hidden email]
Subject: Commons-Collections enhanced with Java Generics Support

Dear Jakarta Commons developers,

We are pleased to announce that we have enhanced the Commons-Collections
with Java 1.5 generics support! We have the results hosted on SourceForge:

http://collections.sf.net

We have done our utmost to maintain the exact same functionality as
Collections 3.1. We have eliminated all deprecated classes that were
scheduled for removal in 4.0. Furthermore, we have deprecated all the
TypedXXX classes since they are made obsolete by generics.

The following classes were not converted, or were only partially converted:

- MultiMap and derivatives: MultiMap with generics breaks the
java.util.Map contract, so only partial conversion was possible.
- All TransformedXXX collections: These classes break the
java.util.Collection contract (or the java.util.Map contract, as
appropriate) once generics are added. They were not converted to support
generics at all.

We are considering adding MorphCollection, MorphMap, MorphSet, etc.
interfaces to support MultiMaps and TransformedXXX collections. These
interfaces would be the same as their non-morph counterparts, but
'getting' would be templated independently of 'setting/adding'. For
example, the MultiMap<K,V> interface would extend
MorphMap<K,V,Collection<V>>. However, this is a large step as it would
move these collections out from under the standard collection interfaces
defined in java.util. We welcome any suggestions regarding this.

The collections pass all the unit tests. We have released it as a beta,
but plan to move quickly towards a final release. We look forward to any
suggestions and feedback from the community.

Sincerely,

 Matt Hall and John Watkinson

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Thomas Dudziak
On 5/24/05, James Carman <[hidden email]> wrote:
> Why can't we host this project at ASF?  Couldn't we create a new branch for
> JDK5 collections or something?

+1, though I'd prefer the simple solution of two jars, one for pre-1.5
and one for 1.5 which contains the generics-support (either version or
add-on jar). This would prevent divergences in future versions as only
one codebase has to be supported.

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

James Carman
I don't think I understand your proposal.  Are you saying that we should
just put the classes with generics support into the HEAD branch of the
source repository?  

-----Original Message-----
From: Thomas Dudziak [mailto:[hidden email]]
Sent: Tuesday, May 24, 2005 10:06 AM
To: Jakarta Commons Developers List
Subject: Re: Commons-Collections enhanced with Java Generics Support

On 5/24/05, James Carman <[hidden email]> wrote:
> Why can't we host this project at ASF?  Couldn't we create a new branch
for
> JDK5 collections or something?

+1, though I'd prefer the simple solution of two jars, one for pre-1.5
and one for 1.5 which contains the generics-support (either version or
add-on jar). This would prevent divergences in future versions as only
one codebase has to be supported.

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Simon Kitching
In reply to this post by Thomas Dudziak
On Tue, 2005-05-24 at 16:05 +0200, Thomas Dudziak wrote:
> On 5/24/05, James Carman <[hidden email]> wrote:
> > Why can't we host this project at ASF?  Couldn't we create a new branch for
> > JDK5 collections or something?
>
> +1, though I'd prefer the simple solution of two jars, one for pre-1.5
> and one for 1.5 which contains the generics-support (either version or
> add-on jar). This would prevent divergences in future versions as only
> one codebase has to be supported.

The major reason the project was developed on sourceforge is that the
people who wanted to do the implementation were not commons committers,
and no commons committers had time to manage the project.

I don't know whether the developers (John/Matt) are even interested in
the project being merged back into apache at the moment.

But if they were, then in order for it to become a commons project
(including being a branch of the existing collections project) either
some existing commons committers would need to volunteer to commit
patches submitted (including being responsible for the quality,
licensing, and ensuring longterm support etc) or some/all of the
generics project developers would need to be elected as commons
committers.

But we can't just elect someone as a commons committer without knowing
the quality of their work or their ability to work well within commons
(esp. having plenty of patience ;-). I think it very likely that
Matt/John would be fine additions to the commons team, but we just don't
know them yet (at least I don't). If someone had the time to check the
collections.sf.net project mail list and review the existing code
thoroughly this might be enough to give a +1 on this issue. Or if they
have a track record with some other large open-source project. Otherwise
the project really needs a few commons committers to work with them for
a month or two until we can feel comfortable about electing them.

There is also the question of how large the generic collections
community will be. There aren't yet a whole lot of projects coding to
1.5 as far as I know. That would mean that it might be hard to ensure a
pool of interested developers for this project that would continue
maintenance. And that would be bad - commons doesn't need any more
zombie projects than it already has. Then again I may be wrong; there
might be huge demand for this. Or Matt/John may be enthusiastic enough
to provide maintenance over the next year or two until java 1.5 becomes
more prevalent. Checking the sf project forums should provide some
evidence of whether there is a solid user community for this project or
not. Certainly a pool of only two developers is a little fragile for
long-term support if there is only a small user pool to draw new
developers from.

Note that I'm not saying it's impossible to bring this project into
commons if Matt/John want to. And I certainly don't mean any offence to
Matt/John, who have clearly put a lot of effort into writing code that
is available for free and are therefore "good guys" by any definition.
But these are issues that need to be addressed before adopting this code
into commons.

In the meantime, though, I see no harm in making sure we have plenty of
references from commons to the collections.sf.net project (see, there's
one!) from the commons site, wiki, etc. We can make sure people who
might visit commons-collections are made aware of the generics version
and then those people can make up their own minds about which code base
they want to use. That's just being friendly and helpful.

And there's nothing *wrong* with projects on sourceforge anyway.

If you want to address some of these issues and push for generic
collections in commons then go ahead and put a case that addresses the
above issues.


Regards,

Simon


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Mattias Jiderhamn-2
At 2005-05-24 17:11, Simon Kitching wrote:
>There is also the question of how large the generic collections
>community will be. There aren't yet a whole lot of projects coding to
>1.5 as far as I know.

Assuming this statement is true (which I am not too sure about), the fact
that many Open Source project do not yet support 1.5 features - like
generics - might actually be the reason people have not yet upgraded. Put
in another way; good and useful Open Source projects (like Jakarta Commons)
supporting JDK 1.5 may motivate more people to upgrade to JDK 1.5.

 From the other perspective, for those of us that *do* use JDK 1.5 (and
have done so for a while now...), the fact that Commons Collections does
not support generics may motivate us to use the standard java.util.*
collections, or some proprietary ones, instead.

In summary, I don't think the number of people interested in 1.5 features
will increase while Commons Collections developers just sit around waiting...

  /Mattias Jiderhamn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

James Carman
In reply to this post by Simon Kitching
Simon,
Suppose we do want to further pursue this (I think we should).  How would
you recommend we set up the project?  Should we branch commons-collections
off and start doing releases off of the JDK5 branch along side the main
branch?  Or, would you recommend creating an entirely new commons project?
Or, would you recommend creating a
org.apache.jakarta.commons.collections.jdk5 package with the new classes in
them and set up the build to conditionally compile them only if using JDK5?


James

-----Original Message-----
From: Simon Kitching [mailto:[hidden email]]
Sent: Tuesday, May 24, 2005 11:12 AM
To: Jakarta Commons Developers List
Subject: Re: Commons-Collections enhanced with Java Generics Support

On Tue, 2005-05-24 at 16:05 +0200, Thomas Dudziak wrote:
> On 5/24/05, James Carman <[hidden email]> wrote:
> > Why can't we host this project at ASF?  Couldn't we create a new branch
for
> > JDK5 collections or something?
>
> +1, though I'd prefer the simple solution of two jars, one for pre-1.5
> and one for 1.5 which contains the generics-support (either version or
> add-on jar). This would prevent divergences in future versions as only
> one codebase has to be supported.

The major reason the project was developed on sourceforge is that the
people who wanted to do the implementation were not commons committers,
and no commons committers had time to manage the project.

I don't know whether the developers (John/Matt) are even interested in
the project being merged back into apache at the moment.

But if they were, then in order for it to become a commons project
(including being a branch of the existing collections project) either
some existing commons committers would need to volunteer to commit
patches submitted (including being responsible for the quality,
licensing, and ensuring longterm support etc) or some/all of the
generics project developers would need to be elected as commons
committers.

But we can't just elect someone as a commons committer without knowing
the quality of their work or their ability to work well within commons
(esp. having plenty of patience ;-). I think it very likely that
Matt/John would be fine additions to the commons team, but we just don't
know them yet (at least I don't). If someone had the time to check the
collections.sf.net project mail list and review the existing code
thoroughly this might be enough to give a +1 on this issue. Or if they
have a track record with some other large open-source project. Otherwise
the project really needs a few commons committers to work with them for
a month or two until we can feel comfortable about electing them.

There is also the question of how large the generic collections
community will be. There aren't yet a whole lot of projects coding to
1.5 as far as I know. That would mean that it might be hard to ensure a
pool of interested developers for this project that would continue
maintenance. And that would be bad - commons doesn't need any more
zombie projects than it already has. Then again I may be wrong; there
might be huge demand for this. Or Matt/John may be enthusiastic enough
to provide maintenance over the next year or two until java 1.5 becomes
more prevalent. Checking the sf project forums should provide some
evidence of whether there is a solid user community for this project or
not. Certainly a pool of only two developers is a little fragile for
long-term support if there is only a small user pool to draw new
developers from.

Note that I'm not saying it's impossible to bring this project into
commons if Matt/John want to. And I certainly don't mean any offence to
Matt/John, who have clearly put a lot of effort into writing code that
is available for free and are therefore "good guys" by any definition.
But these are issues that need to be addressed before adopting this code
into commons.

In the meantime, though, I see no harm in making sure we have plenty of
references from commons to the collections.sf.net project (see, there's
one!) from the commons site, wiki, etc. We can make sure people who
might visit commons-collections are made aware of the generics version
and then those people can make up their own minds about which code base
they want to use. That's just being friendly and helpful.

And there's nothing *wrong* with projects on sourceforge anyway.

If you want to address some of these issues and push for generic
collections in commons then go ahead and put a case that addresses the
above issues.


Regards,

Simon


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Mario Ivankovits
In reply to this post by Mattias Jiderhamn-2
Mattias J wrote:
>> There is also the question of how large the generic collections
>> community will be. There aren't yet a whole lot of projects coding to
>> 1.5 as far as I know.
We use JDK 1.5 since the 1.5.0_01 is out and already use it in
productive environments.
Dont know how large the community is, but I cant await to get more and
more libraries supporting 1.5.
> In summary, I don't think the number of people interested in 1.5
> features will increase while Commons Collections developers just sit
> around waiting...
Thats not fair.
We do what we can and mostly in our free time. And its also your
(contributor) job to push a project forward - for sure what you have
done now.


But I wonder if we could let the jdk 1.3 behind us and move on to jdk
1.5 by allowing backward compatiblity with retroweaver.
At least with new to create x.0 branches.
For sure that forbid us to use any new api feature, but we can use
generics and some other new programming style stuff.

---
Mario


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Mattias Jiderhamn-2

>>In summary, I don't think the number of people interested in 1.5 features
>>will increase while Commons Collections developers just sit around waiting...
>Thats not fair.
>We do what we can and mostly in our free time. And its also your
>(contributor) job to push a project forward - for sure what you have done now.

Surely I didn't mean "...doing nothing", I just meant not supporting JDK
1.5 features until enough people are likely to want them.
My apologies if anybody was offended.

>But I wonder if we could let the jdk 1.3 behind us and move on to jdk 1.5
>by allowing backward compatiblity with retroweaver.

AFAIK generics is only a compile time feature, no type checking is done
during runtime, so a generics supporting JAR should be possible to use with
older JDKs.
I have not looked at the Collections API over at sf.net, but possibly other
things have changed to be able to add the generics support, which would
make this library behave differently than the original one, on an older JDK.

   /Mattias Jiderhamn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Thomas Dudziak
In reply to this post by Mario Ivankovits
> But I wonder if we could let the jdk 1.3 behind us and move on to jdk
> 1.5 by allowing backward compatiblity with retroweaver.
> At least with new to create x.0 branches.
> For sure that forbid us to use any new api feature, but we can use
> generics and some other new programming style stuff.

Especially for components like commons-collections, IMO it is quite
important to support JDKs as far back as 1.2 (and additional ones like
J2ME), if not even 1.1.

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

James Carman
In reply to this post by Mario Ivankovits
Retroweaver's documentation specifically says that it allows you to develop
on 1.5 and deploy on 1.4.  I don't think it supports 1.3 and below.  Anyone
know for sure?

-----Original Message-----
From: Mario Ivankovits [mailto:[hidden email]]
Sent: Tuesday, May 24, 2005 11:43 AM
To: Jakarta Commons Developers List
Subject: Re: Commons-Collections enhanced with Java Generics Support

Mattias J wrote:
>> There is also the question of how large the generic collections
>> community will be. There aren't yet a whole lot of projects coding to
>> 1.5 as far as I know.
We use JDK 1.5 since the 1.5.0_01 is out and already use it in
productive environments.
Dont know how large the community is, but I cant await to get more and
more libraries supporting 1.5.
> In summary, I don't think the number of people interested in 1.5
> features will increase while Commons Collections developers just sit
> around waiting...
Thats not fair.
We do what we can and mostly in our free time. And its also your
(contributor) job to push a project forward - for sure what you have
done now.


But I wonder if we could let the jdk 1.3 behind us and move on to jdk
1.5 by allowing backward compatiblity with retroweaver.
At least with new to create x.0 branches.
For sure that forbid us to use any new api feature, but we can use
generics and some other new programming style stuff.

---
Mario


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

Michael Heuer-2
In reply to this post by James Carman

James Carman wrote:

> Suppose we do want to further pursue this (I think we should).  How would
> you recommend we set up the project?  Should we branch commons-collections
> off and start doing releases off of the JDK5 branch along side the main
> branch?

With a nod to "Rules for Revolutionaries" [1] I think it makes the most
sense to branch commons-collections.  This should be a lot easier process
now that the repository is in subversion.

The biggest obstacle to overcome is the lack of interested commons
committers to coordinate non-Apache developers, to merge different
implementations (there is a second 1.5 collections implementation at
collections15.sourceforge.net), and address some of the problem areas in
the generic conversion process, e.g. MultiMap.

   michael

[1]  http://incubator.apache.org/learn/rules-for-revolutionaries.html


> -----Original Message-----
> From: Simon Kitching [mailto:[hidden email]]
> Sent: Tuesday, May 24, 2005 11:12 AM
> To: Jakarta Commons Developers List
> Subject: Re: Commons-Collections enhanced with Java Generics Support
>
> On Tue, 2005-05-24 at 16:05 +0200, Thomas Dudziak wrote:
> > On 5/24/05, James Carman <[hidden email]> wrote:
> > > Why can't we host this project at ASF?  Couldn't we create a new branch
> for
> > > JDK5 collections or something?
> >
> > +1, though I'd prefer the simple solution of two jars, one for pre-1.5
> > and one for 1.5 which contains the generics-support (either version or
> > add-on jar). This would prevent divergences in future versions as only
> > one codebase has to be supported.
>
> The major reason the project was developed on sourceforge is that the
> people who wanted to do the implementation were not commons committers,
> and no commons committers had time to manage the project.
>
> I don't know whether the developers (John/Matt) are even interested in
> the project being merged back into apache at the moment.
>
> But if they were, then in order for it to become a commons project
> (including being a branch of the existing collections project) either
> some existing commons committers would need to volunteer to commit
> patches submitted (including being responsible for the quality,
> licensing, and ensuring longterm support etc) or some/all of the
> generics project developers would need to be elected as commons
> committers.
>
> But we can't just elect someone as a commons committer without knowing
> the quality of their work or their ability to work well within commons
> (esp. having plenty of patience ;-). I think it very likely that
> Matt/John would be fine additions to the commons team, but we just don't
> know them yet (at least I don't). If someone had the time to check the
> collections.sf.net project mail list and review the existing code
> thoroughly this might be enough to give a +1 on this issue. Or if they
> have a track record with some other large open-source project. Otherwise
> the project really needs a few commons committers to work with them for
> a month or two until we can feel comfortable about electing them.
>
> There is also the question of how large the generic collections
> community will be. There aren't yet a whole lot of projects coding to
> 1.5 as far as I know. That would mean that it might be hard to ensure a
> pool of interested developers for this project that would continue
> maintenance. And that would be bad - commons doesn't need any more
> zombie projects than it already has. Then again I may be wrong; there
> might be huge demand for this. Or Matt/John may be enthusiastic enough
> to provide maintenance over the next year or two until java 1.5 becomes
> more prevalent. Checking the sf project forums should provide some
> evidence of whether there is a solid user community for this project or
> not. Certainly a pool of only two developers is a little fragile for
> long-term support if there is only a small user pool to draw new
> developers from.
>
> Note that I'm not saying it's impossible to bring this project into
> commons if Matt/John want to. And I certainly don't mean any offence to
> Matt/John, who have clearly put a lot of effort into writing code that
> is available for free and are therefore "good guys" by any definition.
> But these are issues that need to be addressed before adopting this code
> into commons.
>
> In the meantime, though, I see no harm in making sure we have plenty of
> references from commons to the collections.sf.net project (see, there's
> one!) from the commons site, wiki, etc. We can make sure people who
> might visit commons-collections are made aware of the generics version
> and then those people can make up their own minds about which code base
> they want to use. That's just being friendly and helpful.
>
> And there's nothing *wrong* with projects on sourceforge anyway.
>
> If you want to address some of these issues and push for generic
> collections in commons then go ahead and put a case that addresses the
> above issues.
>
>
> Regards,
>
> Simon
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

John Watkinson
In reply to this post by John Watkinson
Hi guys,

Thanks for your interest in this. We are definitely ready and willing to
bring the code back in to the Jakarta Commons when the time is right.
Our goal was to get it done so that developers (us included) could start
using and improving it, but we hope that we can eventually contribute it
all back to the Commons.

Let me address the points raised as best as I can:

1) Should it be a separate project or a branch of Commons-Collections?

It is the case that Java 1.5 classes are NOT binary compatible with
previous runtime environments (1.4 and earlier). So, the original
Collections have a very long life ahead of them. Also, an enormous
number of third-party libraries have the Commons-Collections as a
dependency and it will not be possible to ensure binary compatibility
between the generic-enabled Commons-Collections and the original
library, even if the code is run on the 1.5 platform. This could result
in some misery if a programmer requires libraries that depend on both
versions. So, this suggests to me that the generics-enabled
Commons-Collections should be a new project with a different package
structure than the original.

2) Is there enough support for the 1.5 Collections to warrant a new project?

I agree with the points raised earlier on this topic. Developers may not
yet be clamoring for this, but as more of them adopt 1.5, they may begin
doing so. Also, a high-profile project like the Commons-Collections
could help attract developers to 1.5.

3) What are the open-source credentials of Matt Hall and John Watkinson?

We are the authors of Streamsicle (http://www.streamsicle.com). I have
also written SwarmCache (http://swarmcache.sourceforge.net) and MegaMap
(http://megamap.sourceforge.net). We are the authors of Starfish, a
project that we are in the process of opening
(http://www.larvalabs.com/starfish).

Sincerely,

 John Watkinson

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Thomas Dudziak
> 1) Should it be a separate project or a branch of Commons-Collections?
>
> It is the case that Java 1.5 classes are NOT binary compatible with
> previous runtime environments (1.4 and earlier). So, the original
> Collections have a very long life ahead of them. Also, an enormous
> number of third-party libraries have the Commons-Collections as a
> dependency and it will not be possible to ensure binary compatibility
> between the generic-enabled Commons-Collections and the original
> library, even if the code is run on the 1.5 platform. This could result
> in some misery if a programmer requires libraries that depend on both
> versions. So, this suggests to me that the generics-enabled
> Commons-Collections should be a new project with a different package
> structure than the original.

Mhmm, I'm not sure why missing binary compatibility makes it necessary
to create a different package structure let alone a different project
? Shouldn't it suffice to create two jars, one for pre-1.5 and one for
1.5 and above ? Depending on which Java version the user employs, he
then chooses one of these jars. This even ensures drop-in
compatibility for people that already use commons-collections with
Java 5.
And branching or a different project means maintaining two APIs in parallel ...

> 2) Is there enough support for the 1.5 Collections to warrant a new project?
>
> I agree with the points raised earlier on this topic. Developers may not
> yet be clamoring for this, but as more of them adopt 1.5, they may begin
> doing so. Also, a high-profile project like the Commons-Collections
> could help attract developers to 1.5.

+1, especially if enough gaps are filled (like with the 'old'
commons-collections).

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Stephen Colebourne
In reply to this post by John Watkinson
John Watkinson wrote:
> We are definitely ready and willing to
> bring the code back in to the Jakarta Commons when the time is right.
> Our goal was to get it done so that developers (us included) could start
> using and improving it, but we hope that we can eventually contribute it
> all back to the Commons.
+1, This sounds like the right approach to me.

> 1) Should it be a separate project or a branch of Commons-Collections?
>
> It is the case that Java 1.5 classes are NOT binary compatible with
> previous runtime environments (1.4 and earlier). So, the original
> Collections have a very long life ahead of them. Also, an enormous
> number of third-party libraries have the Commons-Collections as a
> dependency and it will not be possible to ensure binary compatibility
> between the generic-enabled Commons-Collections and the original
> library, even if the code is run on the 1.5 platform.
There are numerous good points here. [collections] got a bad name with
the 3.0 release for breaking binary compatability. (In fact this was 99%
FUD spread by Hani and TSS, because only a couple of constants were
incompatible).

However, what it does serve to emphasise is that [collections] must not
only be 'clean', but also be perceived to be clean. To me, this means
that clever tools like Retroweaver would not be an option.

> 2) Is there enough support for the 1.5 Collections to warrant a new
> project?
>
> I agree with the points raised earlier on this topic. Developers may not
> yet be clamoring for this, but as more of them adopt 1.5, they may begin
> doing so. Also, a high-profile project like the Commons-Collections
> could help attract developers to 1.5.
The main problem is keeping bug fixes in sync between the two codebases.
However, a separate package structure is the only practical solution to
achieve 1.5 collections. Whether it is a separate project is a moot
point, and I don't have a strong view either way.

I would expect the package name for collections JDK1.5 to be
  org.apache.commons.collections15

> 3) What are the open-source credentials of Matt Hall and John Watkinson?
>
> We are the authors of Streamsicle (http://www.streamsicle.com). I have
> also written SwarmCache (http://swarmcache.sourceforge.net) and MegaMap
> (http://megamap.sourceforge.net). We are the authors of Starfish, a
> project that we are in the process of opening
> (http://www.larvalabs.com/starfish).
Having previous OSS experience will be helpful.

Finally, any ideas from http://collections15.sourceforge.net should also
be taken into account.

Stephen

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

James Carman
Stephen,

Maybe we could call it Commons Generics?  Just a thought.  But, that might
leave a door open to put non-collection classes in there, as generics can be
any type of parameterized class.  Anyway, how should we proceed from here
(regardless of what we call it)?  Maybe a new project in Jakarta Commons
would be in order.  Wouldn't this situation be analogous to the Xerces Java
2 vs. Xerces Java 1 situation?  They created a new project for their "next
generation."  Shouldn't we?  They kept the same package structure, from what
I can tell (org.apache.xerces).  Do you think we could do the same and let
folks upgrade to the next generation without having to change their code?  I
don't know much about the binary compatibility issue.  Would they just have
to recompile their classes using a JDK5 compiler against the new classes?
Of course, they'd get those new warning messages, but at least the code
would work.  Right?

James

-----Original Message-----
From: Stephen Colebourne [mailto:[hidden email]]
Sent: Tuesday, May 24, 2005 6:56 PM
To: Jakarta Commons Developers List
Subject: Re: Commons-Collections enhanced with Java Generics Support

John Watkinson wrote:
> We are definitely ready and willing to
> bring the code back in to the Jakarta Commons when the time is right.
> Our goal was to get it done so that developers (us included) could start
> using and improving it, but we hope that we can eventually contribute it
> all back to the Commons.
+1, This sounds like the right approach to me.

> 1) Should it be a separate project or a branch of Commons-Collections?
>
> It is the case that Java 1.5 classes are NOT binary compatible with
> previous runtime environments (1.4 and earlier). So, the original
> Collections have a very long life ahead of them. Also, an enormous
> number of third-party libraries have the Commons-Collections as a
> dependency and it will not be possible to ensure binary compatibility
> between the generic-enabled Commons-Collections and the original
> library, even if the code is run on the 1.5 platform.
There are numerous good points here. [collections] got a bad name with
the 3.0 release for breaking binary compatability. (In fact this was 99%
FUD spread by Hani and TSS, because only a couple of constants were
incompatible).

However, what it does serve to emphasise is that [collections] must not
only be 'clean', but also be perceived to be clean. To me, this means
that clever tools like Retroweaver would not be an option.

> 2) Is there enough support for the 1.5 Collections to warrant a new
> project?
>
> I agree with the points raised earlier on this topic. Developers may not
> yet be clamoring for this, but as more of them adopt 1.5, they may begin
> doing so. Also, a high-profile project like the Commons-Collections
> could help attract developers to 1.5.
The main problem is keeping bug fixes in sync between the two codebases.
However, a separate package structure is the only practical solution to
achieve 1.5 collections. Whether it is a separate project is a moot
point, and I don't have a strong view either way.

I would expect the package name for collections JDK1.5 to be
  org.apache.commons.collections15

> 3) What are the open-source credentials of Matt Hall and John Watkinson?
>
> We are the authors of Streamsicle (http://www.streamsicle.com). I have
> also written SwarmCache (http://swarmcache.sourceforge.net) and MegaMap
> (http://megamap.sourceforge.net). We are the authors of Starfish, a
> project that we are in the process of opening
> (http://www.larvalabs.com/starfish).
Having previous OSS experience will be helpful.

Finally, any ideas from http://collections15.sourceforge.net should also
be taken into account.

Stephen

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

Chris Lambrou
In reply to this post by Stephen Colebourne
Hi all,

Sometime last summer, a there was a discussion about providing a generic
port of collections. The upshot was that no Apache commiters could
commit to providing time to handle this at Apache, and so I started the
collections15.sf.net project over on SourceForge. A lot of the issues
raised in recent messages, such as maintaining two separate codebases,
the use of Retroweaver, etc.) were discussed back then. The general
consensus back then was that a generic port of collections should free
itself from the constraints of pre-1.5 code and should concentrate on
providing broadly the same functionality as collections, but with a
focus on the new language features, and a number of other changes as
appropriate - e.g. a rearranged package structure to avoid the clutter
that was then present in the root package of collections; either ignore
or cut back the the Buffer classes and package in light of the new Queue
API in Java 5.0; change APIs where the current collections API was
inappropriate. As a result of these issues and a discussion of them last
summer, it was felt that a simple generification of the existing
collections API was not unsuitable. And so work on collections15.sf.net
began. I started with an empty source tree, and began by copying over
all of the interfaces defined in collections, and then generifying them.
After that, I started to copy packages over one by one, giving each
class a thorough work over to fully generify both its public API, its
internal implementation, its javadoc and its associated unit tests.
Although work was initially quite fast, it should be apparent what I
large task this is. A couple of other developers joined the effort along
the way, which improved progress, but to counter that there were some
significant health problems that have all but brought my efforts to a
halt since early this year. Basically, the code that's been written is
pretty solid, and passes a thorough set of unit test (well, apart from
some of the classes that are still work in progress), but there's plenty
more to be done.

In contrast to this, Matt and John have made their recent announcement
of a generified port of collections a couple of weeks ago -
collections.sf.net. They've obviously put a lot of time and effort into
this, which is to be commended. I've checked out the collections.sf.nef
project and had a bit of a look round, and have come up with a number of
issues that I'd like to raise, some of which need to be addressed.

1. The port appears to me to be a direct attempt to take the existing
collections codebase and generify its API. It's an approach I initially
took but abandoned after a while when I realised that much of the
existing codebase was inappropriate for generifying. To clarify this,
much of the current collections API is not typesafe, and raises problems
when trying to generify it. For example, the ChainedTransformer class
has a constructor that accepts a Collection. The javadoc indicates that
this should be a collection of Transformer instances, and the resulting
ChainedTransformer's transform method takes the input object and
transforms it using each Transformer in the chain, returning the result.
When generifying the class to ChainedTransformer<I, O>, it's not
possible to use the following constructor

    public ChainedTransformer(List<Transformer<I, O>> transformers)

because it's generally not possible to take the output of each of the
chained Transformers and pass it into the next Transformer in the chain.
After much consideration, it was decided that to maintain compile-time
type safety, the behaviour of ChainedTransformer had to fundamentally
change. This e-mail is already long enough, so I don't want to elaborate
any further. The point I wish to make is that the collections.sf.net
project addresses such issues by compromising type safety - it's
constructor to ChainedTransformer<I, O> is as follows

    public ChainedTransformer(Transformer[] transformers)

In this instance, the difficulties that generification raises have been
skirted by sacrificing type-safety, and it's an approach that is taken
throughout the collections.sf.net port of collections. I think this is
an important point to consider, as probably the most important point of
generics is to provide compile-time type-safety.

2. Whilst the public API of the collections.sf.net port has been
generified, the internal implementation is largely untouched. It's still
the non-generic code that is present in the current  commons-collections
codebase. From a black-box approach, this isn't especially important
provided that the implementation honours the documented API. As I've
mentioned earlier, this isn't the approach I've taken in
collections15.sf.net, where all of the code has been fully generified,
rather than just the API. This isn't a particular criticism of
collections.sf.net - Sun's own implementation of ArrayList<E> takes the
same pragmatic approach - but it's just a difference I wanted to point
out. I must say, however, that in the process of generifying all of the
code in collections15.sf.net, a number of subtle improvements to the
generified API became apparent that would not have been so had I only
generified the public API. That this level of attention hasn't been paid
to the implementation code in collections.sf.net, leads me to worry that
the generification of the API isn't optimal, though I admit that this
may be because my first stab at generifying the interfaces was not the
best and so had to be changed a lot as I generified the implementing
classes and the problems in the API became more apparent.

3. Here's my biggest worry. The unit tests in collections.sf.net don't
appear to have been modified to reflect the generification of the APIs.
The 100% success rate of the unit tests is therefore misleading, as it's
more of an indication that the original commons-collections code on
which the collections.sf.net port was based doesn't fail any of its unit
tests. What's missing in the unit tests is an attempt to exercise the
generic modifications made to the APIs. Whilst updating the unit tests
in collections15.sf.net, a fair number of minor errors where uncovered.
They were typically problems whereby it became apparent when writing the
unit tests that the generic arguments of various methods weren't
sufficiently flexible. I'm worried that since the unit tests in
collections.sf.net don't exercise the generic modification that have
been made, the modifications may not have been exercised at all.

4. The javadoc in collections.sf.net doesn't appear to have been updated
to reflect the generification of the API.

Assuming that both Matt and John have been monitoring this mailing list
since their recent announcement, I hope these issues are taken as the
constructive criticism that I intend them to be. They've clearly put in
a significant amount of effort in creating collections.sf.net, which I
applaud. I just think that their project needs more work. I can't speak
for the other two developers on collections15.sf.net, but my personal
feeling is that it's wasteful to continue two separate attempts to
create a generics port of collections, and that we should be
collaborating to bring together the good work and good ideas present in
both projects. I'd really like to hear what Matt and John have to say
regarding the difference in approaches that have been taken by the two
projects, particularly regarding the the 1.5 specific tailoring of
collections15.sf.net compared to the more direct port of collections.sf.net.

Regarding the idea of pulling the collections.sf.net code into the
commons SVN repository, I think that the original reasons for developing
collections15.sf.net at SourceForge still apply. The regular
contributors to the project are not commons committers, and until they
become so, or an existing commons committer is prepared to take on the
burden of acting as a broker between the active developers and SVN, the
project has a better chance of thriving at SourceForge.

Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Commons-Collections enhanced with Java Generics Support

John Watkinson
In reply to this post by John Watkinson
Hi Chris, you've made excellent points, which I'll address below:

---
1. The port appears to me to be a direct attempt to take the existing
collections codebase and generify its API. It's an approach I initially
took but abandoned after a while when I realised that much of the
existing codebase was inappropriate for generifying. To clarify this,
much of the current collections API is not typesafe, and raises problems
when trying to generify it. For example, the ChainedTransformer class
has a constructor that accepts a Collection. The javadoc indicates that
this should be a collection of Transformer instances, and the resulting
ChainedTransformer's transform method takes the input object and
transforms it using each Transformer in the chain, returning the result.
When generifying the class to ChainedTransformer<I, O>, it's not
possible to use the following constructor

   public ChainedTransformer(List<Transformer<I, O>> transformers)

because it's generally not possible to take the output of each of the
chained Transformers and pass it into the next Transformer in the chain.
After much consideration, it was decided that to maintain compile-time
type safety, the behaviour of ChainedTransformer had to fundamentally
change. This e-mail is already long enough, so I don't want to elaborate
any further. The point I wish to make is that the collections.sf.net
project addresses such issues by compromising type safety - it's
constructor to ChainedTransformer<I, O> is as follows

   public ChainedTransformer(Transformer[] transformers)

In this instance, the difficulties that generification raises have been
skirted by sacrificing type-safety, and it's an approach that is taken
throughout the collections.sf.net port of collections. I think this is
an important point to consider, as probably the most important point of
generics is to provide compile-time type-safety.
---

There are definitely some collections that don't take very well to
generics. ChainedTransformer is one of the worst offenders, as it is not
clear how it should be changed so as to make it support generics. Other
collections that have problems are MultiMap and the TransforedXXX
collections. However, there are some good solutions available for those,
and we are working on them actively.

Our approach was to shoot for supporting generics in the majority of the
classes. This was because we wanted to begin using the collections in
1.5 in our own projects immediately, and there was no existing solution
for us. Those classes that did not lend themselves well to generics were
either not converted or were only partially converted. This way, the
software worked right away and was a clear improvement over the
non-generic collections. We agree that it has some short-comings in the
partially-/non-converted areas and we look forward to working with
everybody to resolve those in the best way possible.

It is our philosophy to not be bogged down by the more obscure cases
(such as ChainedTransformer). Those few classes that just don't make
sense at all from a generics point of view should perhaps just be
labelled as such in the documentation.

---
2. Whilst the public API of the collections.sf.net port has been
generified, the internal implementation is largely untouched. It's still
the non-generic code that is present in the current  commons-collections
codebase. From a black-box approach, this isn't especially important
provided that the implementation honours the documented API. As I've
mentioned earlier, this isn't the approach I've taken in
collections15.sf.net, where all of the code has been fully generified,
rather than just the API. This isn't a particular criticism of
collections.sf.net - Sun's own implementation of ArrayList<E> takes the
same pragmatic approach - but it's just a difference I wanted to point
out. I must say, however, that in the process of generifying all of the
code in collections15.sf.net, a number of subtle improvements to the
generified API became apparent that would not have been so had I only
generified the public API. That this level of attention hasn't been paid
to the implementation code in collections.sf.net, leads me to worry that
the generification of the API isn't optimal, though I admit that this
may be because my first stab at generifying the interfaces was not the
best and so had to be changed a lot as I generified the implementing
classes and the problems in the API became more apparent.
---

Yes, we feel that the some of the internal implementations are
well-converted, but not all. We look forward to improving the use of
generics in those classes and throughout the source. These refactors are
important, but will not change the experience for the users of the library.

---
3. Here's my biggest worry. The unit tests in collections.sf.net don't
appear to have been modified to reflect the generification of the APIs.
The 100% success rate of the unit tests is therefore misleading, as it's
more of an indication that the original commons-collections code on
which the collections.sf.net port was based doesn't fail any of its unit
tests. What's missing in the unit tests is an attempt to exercise the
generic modifications made to the APIs. Whilst updating the unit tests
in collections15.sf.net, a fair number of minor errors where uncovered.
They were typically problems whereby it became apparent when writing the
unit tests that the generic arguments of various methods weren't
sufficiently flexible. I'm worried that since the unit tests in
collections.sf.net don't exercise the generic modification that have
been made, the modifications may not have been exercised at all.
---

On this point, I don't entirely agree. The behavior of a class is the
same at runtime regardless of whether the user used generics in the
source code or not. Any errors in our addition of generics to the
collections will manifest themselves at compile-time, not runtime. Now,
such issues are important, and we should make sure that each method of
each collection is defined consistently from a generics point of view.
However, as the vast majority of collections implement the java.util
collection interfaces, a lot of the parameterized types are already
strictly enforced in the overriding methods at compile-time. This
trickles down to the helper/private methods, and makes things pretty tight.

However, I agree that some test code should be written to ensure that
generics are properly employed in those places where issues could have
slipped between the cracks. Just having that code compile will ensure
that the generics are properly employed.

Incidentally, there were classes that underwent fairly severe
refactoring in order to support generics externally and internally.
Probably the best example is AbstractReferenceMap. For such classes, the
test suite was invaluable in ensuring that the proper behavior was retained.

---
4. The javadoc in collections.sf.net doesn't appear to have been updated
to reflect the generification of the API.
---

True, the docs have only been updated to indicate those classes that
support partial or no type safety under Java 1.5.

Chris, thanks for these points. We agree that there should not be
duplicated effort on this project, and we look forward to open
collaboration on this. More work is certainly required to bring those
straggling non-generic collections in to the fold, but we emphasize an
approach that retains the full, working funtionality of the collections
during these early releases. That way, developers can use the package,
get used to the generic versions of the collections, and provide us with
valuable feedback.

 John Watkinson

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

Laurent Brucher
Hi John,

First off, thanks for the time and effort you guys have put into the port.
Having joined Chris recently on the Collections15 port, I understand the
work that
is required to get this done.

Some comments and thoughts below.


> Our approach was to shoot for supporting generics in the
> majority of the
> classes. This was because we wanted to begin using the collections in
> 1.5 in our own projects immediately, and there was no
> existing solution
> for us. Those classes that did not lend themselves well to
> generics were
> either not converted or were only partially converted. This way, the
> software worked right away and was a clear improvement over the
> non-generic collections. We agree that it has some
> short-comings in the
> partially-/non-converted areas and we look forward to working with
> everybody to resolve those in the best way possible.
>
I understand your needs to have something quick that fits your particular
needs.
But I feel like the ported [collections] should contain all of the current
features and
classes at the time it is released to the public, as you mentioned at the
end of your email.

>
> Yes, we feel that the some of the internal implementations are
> well-converted, but not all. We look forward to improving the use of
> generics in those classes and throughout the source. These
> refactors are
> important, but will not change the experience for the users
> of the library.

Well, I'd say that if the plan is to rework the internal implementation
anyway,
then I think this should be done first before releasing to public, saving
them
from potential bug fixes, etc. that may appear during the rework.


>
> ---
> 3. Here's my biggest worry. The unit tests in
> collections.sf.net don't
> appear to have been modified to reflect the generification of
> the APIs.
> The 100% success rate of the unit tests is therefore
> misleading, as it's
> more of an indication that the original commons-collections code on
> which the collections.sf.net port was based doesn't fail any
> of its unit
> tests. What's missing in the unit tests is an attempt to exercise the
> generic modifications made to the APIs. Whilst updating the
> unit tests
> in collections15.sf.net, a fair number of minor errors where
> uncovered.
> They were typically problems whereby it became apparent when
> writing the
> unit tests that the generic arguments of various methods weren't
> sufficiently flexible. I'm worried that since the unit tests in
> collections.sf.net don't exercise the generic modification that have
> been made, the modifications may not have been exercised at all.
> ---
>
> On this point, I don't entirely agree. The behavior of a class is the
> same at runtime regardless of whether the user used generics in the
> source code or not. Any errors in our addition of generics to the
> collections will manifest themselves at compile-time, not
> runtime. Now,
> such issues are important, and we should make sure that each
> method of
> each collection is defined consistently from a generics point
> of view.

Indeed, things will be the same at runtime. But I also feel that it is
essential to have unit tests written specifically for generics to ensure
the API is valid at compile time. This is what users will first run into
when moving to the new [collections].
Having said that and having gone through a bunch of them already, converting
those unit tests to use generics is far from being a straightforward task.
But gotta be done.


> Chris, thanks for these points. We agree that there should not be
> duplicated effort on this project, and we look forward to open
> collaboration on this. More work is certainly required to bring those
> straggling non-generic collections in to the fold, but we
> emphasize an
> approach that retains the full, working funtionality of the
> collections
> during these early releases. That way, developers can use the
> package,
> get used to the generic versions of the collections, and
> provide us with
> valuable feedback.

I agree with that but I also remember Stephen's word recently saying that
the [collections] must not only be 'clean' but also be perceived to be
clean.
I do think it is important that early adopters feel comfortable with the
API once they start using them with generics and not only using the JAR with
their
previous, non-generic, code.

Finally, I do agree with both Chris and yourself that combining efforts is
certainly a better idea than duplicating them.
Looking forward to this then :o)

Laurent Brucher.


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Commons-Collections enhanced with Java Generics Support

Matt Hall
In reply to this post by John Watkinson
Hi everyone, I'm the other half of the team that did the generifying (if
that's even a word) of collections. Sorry I've been silent so far during
the discussion of what to do with the new collections, I was out of town
and just got caught up with the debate.

So after reading the discussion and seeing it stall out a bit, I propose
that we start by creating a new project in apache which will contain the
current source code from our collections.sf.net repository. To this new
project we can all then add things like new MultiMap and
ChainedTransformer interfaces and implementations that will break
compatibility with existing Collections releases. We can then also
accept new generics specific test cases to add to the already existing
set. The specifics of how all this would work, i.e. if John and I would
become apache committers or not, I'm not sure what the best route here
is. If this is to become a semi-autonomous project then I think having
us as the main committers probably does make the most sense.

Throughout the process I think it's important to maintain a beta or
alpha release of the project so early adopters can start using it and
contributing feedback. There is no advantage in my mind to holding off
on releases until we have the last few controversial classes converted,
better to get feedback from the entire community on the bulk of the work
up front.

I think with our working implementation of the majority of classes,
along with the collection15 team's input and contributions on the
remainders we should be in great shape overall.

Matt

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

12