BCEL 6 API breakage

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

BCEL 6 API breakage

Andrey Loskutov
Hi all,

this is a follow up on https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2016-June/004282.html.

I'm cross-posting this to [hidden email] because the discussion on FindBugs mailing list is related to the BCEL 6 API future, and because I would like to know the opinions from the BCEL community on the upcoming BCEL 6 release compatibility story.

Please see my answers inline.

On Monday 06 June 2016 17:30 sebb wrote:

> On 6 June 2016 at 16:23, Andrey Loskutov <[hidden email]> wrote:
> > Hi all,
> >
> > here is the current state of FindBugs adoption to Java 9.
> >
> > 1) FindBugs standalone/Eclipse plugin run fine now on/with Java 9, the latest code is on java9 branch (not on master yet!), see [0, 1]. If there is interest, I can provide binary snapshots.
> >
> > 2)  I have difficulties to use BCEL6 trunk, see [2]. Looks like even after fixing all compile errors due the various API breakages in BCEL 6 (see [3]), the current BCEL 6 head can't be used directly as a replacement for our old BCEL5.2 fork, see [4]. If anyone from FB and/or BCEL gurus could check this, this would be really helpful. Either our BCEL 5.2 patches were not fully propagated upstream to BCEL or BCEL 6 trunk has few regressions, or I missed something during update? I have no idea, because of the next point below. The experimental BCEL 6 port is on an extra branch on top of Java 9 related changes, see commits prefixed with BCEL6 on java9_bcel6 branch at [5].
> >
> > 3) I would be very happy if someone (Bill?) would explain how the *current* BCEL5.2 fork used by FindBugs was built? It was added in commit [6] but I miss instructions how it differs from the original BCEL code and so unable to re-build it.
> >
> > 4) Assuming BCEL6 bugs/FB errors would be fixed (see [4]), transition to the current BCEL6 head would break each and every FindBugs client, because BCEL6 at current state uses different namespace and also added some other API breaking changes. If we chose this path, none of the 3rd party detectors will work anymore and therefore we must bump FindBugs version to 4.0.
>
> This is useful to know.
> So do the 3rd party detectors depend on the BCEL namespace?

Yes

> Or is it because of the BCEL API changes?

Also yes.

> If so, which ones?

The biggest one is the package namespace change, because this affect each existing BCEL class/interface.
See commit https://github.com/findbugsproject/findbugs/commit/917b692d9a12e048921cd1216102b851866ac3e4 which affects ~400 files in FindBugs.

Much smaller (but still breaking API) changes were class name changes Constants -> Const, StackMapTable[Entry] -> StackMap[Entry] and the move of constants defined in Constants from the interface to the class, thus breaking everyone who implemented the interface and now miss the constants. The rename of StackMapTable/Entry broke also additionally every detector implemented on top of PreorderVisitor. StackMapTableEntry not only changed its name, but also changed signature: getByteCodeOffsetDelta -> getByteCodeOffset which gives you an additional piece of happiness.

Finally, VisitorSupportsInvokeDynamic interface was removed, which broke all FB visitors based on it via AbstractFrameModelingVisitor and 8 methods were added to the Visitor interface.

That's all I saw in our FB code (where we have lot of detectors), probably others will report additional API breakage too, I can't say for sure.

But main issue is the namespace change - it is really unnecessary and surprising. I've read through the commons mailing list and I was surprised that I saw no real request for it or any discussion about it (I haven't read through all the years but around the namespace change last summer). The only thing I saw was the Jira request https://issues.apache.org/jira/browse/BCEL-222, out of nowhere, and few commits later BCEL 6 API was made incompatible to every existing client. :-(

> I'm a bit suprised that the BCEL API should affect the detectors, but
> perhaps there's a good reason for that.

BF analyses bytecode, and although we have also few recently added ASM based detectors (which are mostly BCEL free), most of the detectors (and unfortunately many places in the FB API) use BCEL data structures. It was a natural choice 15 years ago, where BCEL was the only one bytecode framework...

One way to "fix" the current FindBugs misery  is to replace BCEL with ASM (asm.tree package &Co) but this requires lot of effort because API and design in ASM do not match BCEL 1:1 - and it will also break every FB client in much harder way BCEL 6 API breakage does it  today. Doing this will effectively mean a complete fork/rewrite of FindBugs code, and no one is willing to spend time for it.

> > Question is: should we go this way? Alternative is: undo BCEL package renaming and revert few API changes were made. This sounds complicated, but is doable, see BCEL 6 fork where I renamed all packages back to the old namespace in few minutes [7]. Fixing other "smaller" breaking BCEL API changes is not that complicated either. However, it is also possible that BCEL 6 will be released without breaking API's, if I understood right the discussion on the apache commons-dev mailing list [8]. If anyone from BCEL is listening to this mailing list, it would be nice to get some feedback on BCEL 6 plans.
>
> I have done quite a bit of work trying to eliminate the API breaks
> without compromising the BCEL 6 updates.

I really appreciate your effort. Please keep it going.

> Though I have yet to revert the Java and Maven namespace changes as I
> wanted agreement with the approach first.
>
> From my point of view I would be happy to see a compatible version of
> BCEL using the original namespaces.
> I'm not sure what other Commons devs think.

I hope to see a binary compatible BCEL 6 release, which is might be not 1:1 drop-in replacement of BCEL 5 but at least 99%. Some changes must happen, API must evolve, this is natural and no one can keep on the old API forever.
But! After walking over the all BCEL renamings etc I do not really see a real, functional reason to break *everything*, and a behavior change with annotations parsing is something one can live with for a major release. Not all detectors rely on annotations and BCEL behavior change can probably be fixed in FB core code (so hidden from 3rd party libraries).

> There are still some Java8/Java9 features that are not fully supported.
> This is true regardless of the namespace issue.

That's absolutely fine and understandable.

My main goal it to get rid of private BCEL forks which cannot be rebuilt/updated anymore as we see it today, so that we can compile FB on BCEL head, catching all the fixes you will provide in  future BCEL versions. ...And in my ideal world, the new FindBugs release based on BCEL 6 will not break any existing 3rd party FindBugs detectors library, or eventually only require a few trivial changes.

Thanks!

> > [0] https://github.com/findbugsproject/findbugs/commits/java9
> > [1] https://github.com/findbugsproject/findbugs/issues/105
> > [2] https://github.com/findbugsproject/findbugs/issues/106
> > [3] https://issues.apache.org/jira/browse/BCEL-222
> > [4] https://issues.apache.org/jira/browse/BCEL-273
> > [5] https://github.com/findbugsproject/findbugs/commits/java9_bcel6
> > [6] https://github.com/findbugsproject/findbugs/commit/f9f46bf97c47f011e3757bf9904249faf1039239
> > [7] https://github.com/iloveeclipse/commons-bcel/commits/old_structure
> > [8] http://mail-archives.apache.org/mod_mbox/commons-dev/201606.mbox/thread
> >
> > On Sunday 05 June 2016 12:55 Andrey Loskutov wrote:
> >> Hi all,
> >>
> >> I've got some free time and now working on Java 9 support for FindBugs,
> >> the first draft works already, but need some more polishing.
> >>
> >> The main goal is to support FB running on Java 9 JRE, to support reading
> >> Java 9 class files, and to support FB running on Java 8 but analyzing
> >> Java 9 code. Nice to have (but not in my scope right now) is to support
> >> any new Java 8/9 constructs like lambdas, type annotations etc.
> >>
> >> I've documented briefly tasks coming to my mind via [1].
> >>
> >> I plan to push my changes on new java9 branch ASAP.
> >>
> >> Main discussion points I see so far:
> >>
> >> 1) We must bump the required JRE for FB to 1.8. I see no reason trying
> >> to support obsoleted 1.7 JRE. If someone wants run FB on 1.7, the old FB
> >> 3.0.1 should be used. Objections?
> >>
> >> 2) Since there are no official releases from ASM/BCEL with Java 9
> >> support yet, we can release first version based on our own FB private
> >> snapshot versions. The maven folks will cry but this is a chicken and
> >> egg problem, so I don't care about maven support for the first round (of
> >> course any help is welcome). Objections?
> >>
> >> 3) Due the JRE version bump I would propose to bump FB version to 3.1.0.
> >> Objections?
> >>
> >> Please give your feedback either on the lists or on the github task [1].
> >>
> >> [1] https://github.com/findbugsproject/findbugs/issues/105
> >>
> >
> > --
> > Kind regards,
> > google.com/+AndreyLoskutov
> > _______________________________________________
> > Findbugs-discuss mailing list
> > [hidden email]
> > https://mailman.cs.umd.edu/mailman/listinfo/findbugs-discuss

--
Kind regards,
google.com/+AndreyLoskutov

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

Benedikt Ritter-4
Hello Andrey,

the current plan is:

- trunk had breaking changes which were mostly reverted. We did this to
push out one last release of the 5.2 line with all the fixes that had gone
into BCEL.
- We're going to do one last 5.x release soon. For this we're going to
branch trunk and do a rename of the packages so that that release will be
binary compatible with the 5.x line. AFAICT that release won't cover all
the feature needed for Java 8 and 9
- After the 5.x release, we plan to start implementing the missing stuff to
make it work with Java 8.
- We haven't thought about Java 9 support but we welcome any input.

Thank you!
Benedikt

Andrey Loskutov <[hidden email]> schrieb am Mo., 6. Juni 2016 um 20:31 Uhr:

> Hi all,
>
> this is a follow up on
> https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2016-June/004282.html
> .
>
> I'm cross-posting this to [hidden email] because the discussion
> on FindBugs mailing list is related to the BCEL 6 API future, and because I
> would like to know the opinions from the BCEL community on the upcoming
> BCEL 6 release compatibility story.
>
> Please see my answers inline.
>
> On Monday 06 June 2016 17:30 sebb wrote:
> > On 6 June 2016 at 16:23, Andrey Loskutov <[hidden email]> wrote:
> > > Hi all,
> > >
> > > here is the current state of FindBugs adoption to Java 9.
> > >
> > > 1) FindBugs standalone/Eclipse plugin run fine now on/with Java 9, the
> latest code is on java9 branch (not on master yet!), see [0, 1]. If there
> is interest, I can provide binary snapshots.
> > >
> > > 2)  I have difficulties to use BCEL6 trunk, see [2]. Looks like even
> after fixing all compile errors due the various API breakages in BCEL 6
> (see [3]), the current BCEL 6 head can't be used directly as a replacement
> for our old BCEL5.2 fork, see [4]. If anyone from FB and/or BCEL gurus
> could check this, this would be really helpful. Either our BCEL 5.2 patches
> were not fully propagated upstream to BCEL or BCEL 6 trunk has few
> regressions, or I missed something during update? I have no idea, because
> of the next point below. The experimental BCEL 6 port is on an extra branch
> on top of Java 9 related changes, see commits prefixed with BCEL6 on
> java9_bcel6 branch at [5].
> > >
> > > 3) I would be very happy if someone (Bill?) would explain how the
> *current* BCEL5.2 fork used by FindBugs was built? It was added in commit
> [6] but I miss instructions how it differs from the original BCEL code and
> so unable to re-build it.
> > >
> > > 4) Assuming BCEL6 bugs/FB errors would be fixed (see [4]), transition
> to the current BCEL6 head would break each and every FindBugs client,
> because BCEL6 at current state uses different namespace and also added some
> other API breaking changes. If we chose this path, none of the 3rd party
> detectors will work anymore and therefore we must bump FindBugs version to
> 4.0.
> >
> > This is useful to know.
> > So do the 3rd party detectors depend on the BCEL namespace?
>
> Yes
>
> > Or is it because of the BCEL API changes?
>
> Also yes.
>
> > If so, which ones?
>
> The biggest one is the package namespace change, because this affect each
> existing BCEL class/interface.
> See commit
> https://github.com/findbugsproject/findbugs/commit/917b692d9a12e048921cd1216102b851866ac3e4
> which affects ~400 files in FindBugs.
>
> Much smaller (but still breaking API) changes were class name changes
> Constants -> Const, StackMapTable[Entry] -> StackMap[Entry] and the move of
> constants defined in Constants from the interface to the class, thus
> breaking everyone who implemented the interface and now miss the constants.
> The rename of StackMapTable/Entry broke also additionally every detector
> implemented on top of PreorderVisitor. StackMapTableEntry not only changed
> its name, but also changed signature: getByteCodeOffsetDelta ->
> getByteCodeOffset which gives you an additional piece of happiness.
>
> Finally, VisitorSupportsInvokeDynamic interface was removed, which broke
> all FB visitors based on it via AbstractFrameModelingVisitor and 8 methods
> were added to the Visitor interface.
>
> That's all I saw in our FB code (where we have lot of detectors), probably
> others will report additional API breakage too, I can't say for sure.
>
> But main issue is the namespace change - it is really unnecessary and
> surprising. I've read through the commons mailing list and I was surprised
> that I saw no real request for it or any discussion about it (I haven't
> read through all the years but around the namespace change last summer).
> The only thing I saw was the Jira request
> https://issues.apache.org/jira/browse/BCEL-222, out of nowhere, and few
> commits later BCEL 6 API was made incompatible to every existing client. :-(
>
> > I'm a bit suprised that the BCEL API should affect the detectors, but
> > perhaps there's a good reason for that.
>
> BF analyses bytecode, and although we have also few recently added ASM
> based detectors (which are mostly BCEL free), most of the detectors (and
> unfortunately many places in the FB API) use BCEL data structures. It was a
> natural choice 15 years ago, where BCEL was the only one bytecode
> framework...
>
> One way to "fix" the current FindBugs misery  is to replace BCEL with ASM
> (asm.tree package &Co) but this requires lot of effort because API and
> design in ASM do not match BCEL 1:1 - and it will also break every FB
> client in much harder way BCEL 6 API breakage does it  today. Doing this
> will effectively mean a complete fork/rewrite of FindBugs code, and no one
> is willing to spend time for it.
>
> > > Question is: should we go this way? Alternative is: undo BCEL package
> renaming and revert few API changes were made. This sounds complicated, but
> is doable, see BCEL 6 fork where I renamed all packages back to the old
> namespace in few minutes [7]. Fixing other "smaller" breaking BCEL API
> changes is not that complicated either. However, it is also possible that
> BCEL 6 will be released without breaking API's, if I understood right the
> discussion on the apache commons-dev mailing list [8]. If anyone from BCEL
> is listening to this mailing list, it would be nice to get some feedback on
> BCEL 6 plans.
> >
> > I have done quite a bit of work trying to eliminate the API breaks
> > without compromising the BCEL 6 updates.
>
> I really appreciate your effort. Please keep it going.
>
> > Though I have yet to revert the Java and Maven namespace changes as I
> > wanted agreement with the approach first.
> >
> > From my point of view I would be happy to see a compatible version of
> > BCEL using the original namespaces.
> > I'm not sure what other Commons devs think.
>
> I hope to see a binary compatible BCEL 6 release, which is might be not
> 1:1 drop-in replacement of BCEL 5 but at least 99%. Some changes must
> happen, API must evolve, this is natural and no one can keep on the old API
> forever.
> But! After walking over the all BCEL renamings etc I do not really see a
> real, functional reason to break *everything*, and a behavior change with
> annotations parsing is something one can live with for a major release. Not
> all detectors rely on annotations and BCEL behavior change can probably be
> fixed in FB core code (so hidden from 3rd party libraries).
>
> > There are still some Java8/Java9 features that are not fully supported.
> > This is true regardless of the namespace issue.
>
> That's absolutely fine and understandable.
>
> My main goal it to get rid of private BCEL forks which cannot be
> rebuilt/updated anymore as we see it today, so that we can compile FB on
> BCEL head, catching all the fixes you will provide in  future BCEL
> versions. ...And in my ideal world, the new FindBugs release based on BCEL
> 6 will not break any existing 3rd party FindBugs detectors library, or
> eventually only require a few trivial changes.
>
> Thanks!
>
> > > [0] https://github.com/findbugsproject/findbugs/commits/java9
> > > [1] https://github.com/findbugsproject/findbugs/issues/105
> > > [2] https://github.com/findbugsproject/findbugs/issues/106
> > > [3] https://issues.apache.org/jira/browse/BCEL-222
> > > [4] https://issues.apache.org/jira/browse/BCEL-273
> > > [5] https://github.com/findbugsproject/findbugs/commits/java9_bcel6
> > > [6]
> https://github.com/findbugsproject/findbugs/commit/f9f46bf97c47f011e3757bf9904249faf1039239
> > > [7] https://github.com/iloveeclipse/commons-bcel/commits/old_structure
> > > [8]
> http://mail-archives.apache.org/mod_mbox/commons-dev/201606.mbox/thread
> > >
> > > On Sunday 05 June 2016 12:55 Andrey Loskutov wrote:
> > >> Hi all,
> > >>
> > >> I've got some free time and now working on Java 9 support for
> FindBugs,
> > >> the first draft works already, but need some more polishing.
> > >>
> > >> The main goal is to support FB running on Java 9 JRE, to support
> reading
> > >> Java 9 class files, and to support FB running on Java 8 but analyzing
> > >> Java 9 code. Nice to have (but not in my scope right now) is to
> support
> > >> any new Java 8/9 constructs like lambdas, type annotations etc.
> > >>
> > >> I've documented briefly tasks coming to my mind via [1].
> > >>
> > >> I plan to push my changes on new java9 branch ASAP.
> > >>
> > >> Main discussion points I see so far:
> > >>
> > >> 1) We must bump the required JRE for FB to 1.8. I see no reason trying
> > >> to support obsoleted 1.7 JRE. If someone wants run FB on 1.7, the old
> FB
> > >> 3.0.1 should be used. Objections?
> > >>
> > >> 2) Since there are no official releases from ASM/BCEL with Java 9
> > >> support yet, we can release first version based on our own FB private
> > >> snapshot versions. The maven folks will cry but this is a chicken and
> > >> egg problem, so I don't care about maven support for the first round
> (of
> > >> course any help is welcome). Objections?
> > >>
> > >> 3) Due the JRE version bump I would propose to bump FB version to
> 3.1.0.
> > >> Objections?
> > >>
> > >> Please give your feedback either on the lists or on the github task
> [1].
> > >>
> > >> [1] https://github.com/findbugsproject/findbugs/issues/105
> > >>
> > >
> > > --
> > > Kind regards,
> > > google.com/+AndreyLoskutov
> > > _______________________________________________
> > > Findbugs-discuss mailing list
> > > [hidden email]
> > > https://mailman.cs.umd.edu/mailman/listinfo/findbugs-discuss
>
> --
> Kind regards,
> google.com/+AndreyLoskutov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

Charles Honton
In reply to this post by Andrey Loskutov
How Apache Commons BCEL got to where it is currently.

1. I wanted to release a version of BCEL which would support Java 6 and 7.
2. I updated several classes that handled the new instructions and new code attributes.
3. This required new methods on several interfaces.
4. These new methods broke binary compatibility.
5. Whenever binary compatibility is broken, Apache Commons policy is to update the Maven GAV to prevent jar hell.
6. Part of updating GAV is to also update the package names.
7. I created a release candidate which was deemed unsuitable for several reasons; mostly due to FindBugs warnings.
8. Multiple refactorings were completed (including moving Interface to Class) to handle FindBugs warnings.
9. Refactoring died out after no response from users as to Apache direction.
10. Recent new interest has us revisiting these changes.

At this point, we’re somewhat stuck.
1. Do we break Apache Commons Policy regarding binary compatibility GAV and package names?
2. Do we ignore the FindBugs warnings?

Personally, I am against either of those.  I also believe that to fix BCEL correctly, we’ll end up with an api sufficiently different that users will have a non-trivial porting task.  It might be saner if Apache Commons moves BCEL into the attic and suggest that our clients to migrate to either ASM or JavaAssist.

regards,
chas

> On Jun 6, 2016, at 11:27 AM, Andrey Loskutov <[hidden email]> wrote:
>
> Hi all,
>
> this is a follow up on https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2016-June/004282.html.
>
> I'm cross-posting this to [hidden email] because the discussion on FindBugs mailing list is related to the BCEL 6 API future, and because I would like to know the opinions from the BCEL community on the upcoming BCEL 6 release compatibility story.
>
> Please see my answers inline.
>
> On Monday 06 June 2016 17:30 sebb wrote:
>> On 6 June 2016 at 16:23, Andrey Loskutov <[hidden email]> wrote:
>>> Hi all,
>>>
>>> here is the current state of FindBugs adoption to Java 9.
>>>
>>> 1) FindBugs standalone/Eclipse plugin run fine now on/with Java 9, the latest code is on java9 branch (not on master yet!), see [0, 1]. If there is interest, I can provide binary snapshots.
>>>
>>> 2)  I have difficulties to use BCEL6 trunk, see [2]. Looks like even after fixing all compile errors due the various API breakages in BCEL 6 (see [3]), the current BCEL 6 head can't be used directly as a replacement for our old BCEL5.2 fork, see [4]. If anyone from FB and/or BCEL gurus could check this, this would be really helpful. Either our BCEL 5.2 patches were not fully propagated upstream to BCEL or BCEL 6 trunk has few regressions, or I missed something during update? I have no idea, because of the next point below. The experimental BCEL 6 port is on an extra branch on top of Java 9 related changes, see commits prefixed with BCEL6 on java9_bcel6 branch at [5].
>>>
>>> 3) I would be very happy if someone (Bill?) would explain how the *current* BCEL5.2 fork used by FindBugs was built? It was added in commit [6] but I miss instructions how it differs from the original BCEL code and so unable to re-build it.
>>>
>>> 4) Assuming BCEL6 bugs/FB errors would be fixed (see [4]), transition to the current BCEL6 head would break each and every FindBugs client, because BCEL6 at current state uses different namespace and also added some other API breaking changes. If we chose this path, none of the 3rd party detectors will work anymore and therefore we must bump FindBugs version to 4.0.
>>
>> This is useful to know.
>> So do the 3rd party detectors depend on the BCEL namespace?
>
> Yes
>
>> Or is it because of the BCEL API changes?
>
> Also yes.
>
>> If so, which ones?
>
> The biggest one is the package namespace change, because this affect each existing BCEL class/interface.
> See commit https://github.com/findbugsproject/findbugs/commit/917b692d9a12e048921cd1216102b851866ac3e4 which affects ~400 files in FindBugs.
>
> Much smaller (but still breaking API) changes were class name changes Constants -> Const, StackMapTable[Entry] -> StackMap[Entry] and the move of constants defined in Constants from the interface to the class, thus breaking everyone who implemented the interface and now miss the constants. The rename of StackMapTable/Entry broke also additionally every detector implemented on top of PreorderVisitor. StackMapTableEntry not only changed its name, but also changed signature: getByteCodeOffsetDelta -> getByteCodeOffset which gives you an additional piece of happiness.
>
> Finally, VisitorSupportsInvokeDynamic interface was removed, which broke all FB visitors based on it via AbstractFrameModelingVisitor and 8 methods were added to the Visitor interface.
>
> That's all I saw in our FB code (where we have lot of detectors), probably others will report additional API breakage too, I can't say for sure.
>
> But main issue is the namespace change - it is really unnecessary and surprising. I've read through the commons mailing list and I was surprised that I saw no real request for it or any discussion about it (I haven't read through all the years but around the namespace change last summer). The only thing I saw was the Jira request https://issues.apache.org/jira/browse/BCEL-222, out of nowhere, and few commits later BCEL 6 API was made incompatible to every existing client. :-(
>
>> I'm a bit suprised that the BCEL API should affect the detectors, but
>> perhaps there's a good reason for that.
>
> BF analyses bytecode, and although we have also few recently added ASM based detectors (which are mostly BCEL free), most of the detectors (and unfortunately many places in the FB API) use BCEL data structures. It was a natural choice 15 years ago, where BCEL was the only one bytecode framework...
>
> One way to "fix" the current FindBugs misery  is to replace BCEL with ASM (asm.tree package &Co) but this requires lot of effort because API and design in ASM do not match BCEL 1:1 - and it will also break every FB client in much harder way BCEL 6 API breakage does it  today. Doing this will effectively mean a complete fork/rewrite of FindBugs code, and no one is willing to spend time for it.
>
>>> Question is: should we go this way? Alternative is: undo BCEL package renaming and revert few API changes were made. This sounds complicated, but is doable, see BCEL 6 fork where I renamed all packages back to the old namespace in few minutes [7]. Fixing other "smaller" breaking BCEL API changes is not that complicated either. However, it is also possible that BCEL 6 will be released without breaking API's, if I understood right the discussion on the apache commons-dev mailing list [8]. If anyone from BCEL is listening to this mailing list, it would be nice to get some feedback on BCEL 6 plans.
>>
>> I have done quite a bit of work trying to eliminate the API breaks
>> without compromising the BCEL 6 updates.
>
> I really appreciate your effort. Please keep it going.
>
>> Though I have yet to revert the Java and Maven namespace changes as I
>> wanted agreement with the approach first.
>>
>> From my point of view I would be happy to see a compatible version of
>> BCEL using the original namespaces.
>> I'm not sure what other Commons devs think.
>
> I hope to see a binary compatible BCEL 6 release, which is might be not 1:1 drop-in replacement of BCEL 5 but at least 99%. Some changes must happen, API must evolve, this is natural and no one can keep on the old API forever.
> But! After walking over the all BCEL renamings etc I do not really see a real, functional reason to break *everything*, and a behavior change with annotations parsing is something one can live with for a major release. Not all detectors rely on annotations and BCEL behavior change can probably be fixed in FB core code (so hidden from 3rd party libraries).
>
>> There are still some Java8/Java9 features that are not fully supported.
>> This is true regardless of the namespace issue.
>
> That's absolutely fine and understandable.
>
> My main goal it to get rid of private BCEL forks which cannot be rebuilt/updated anymore as we see it today, so that we can compile FB on BCEL head, catching all the fixes you will provide in  future BCEL versions. ...And in my ideal world, the new FindBugs release based on BCEL 6 will not break any existing 3rd party FindBugs detectors library, or eventually only require a few trivial changes.
>
> Thanks!
>
>>> [0] https://github.com/findbugsproject/findbugs/commits/java9
>>> [1] https://github.com/findbugsproject/findbugs/issues/105
>>> [2] https://github.com/findbugsproject/findbugs/issues/106
>>> [3] https://issues.apache.org/jira/browse/BCEL-222
>>> [4] https://issues.apache.org/jira/browse/BCEL-273
>>> [5] https://github.com/findbugsproject/findbugs/commits/java9_bcel6
>>> [6] https://github.com/findbugsproject/findbugs/commit/f9f46bf97c47f011e3757bf9904249faf1039239
>>> [7] https://github.com/iloveeclipse/commons-bcel/commits/old_structure
>>> [8] http://mail-archives.apache.org/mod_mbox/commons-dev/201606.mbox/thread
>>>
>>> On Sunday 05 June 2016 12:55 Andrey Loskutov wrote:
>>>> Hi all,
>>>>
>>>> I've got some free time and now working on Java 9 support for FindBugs,
>>>> the first draft works already, but need some more polishing.
>>>>
>>>> The main goal is to support FB running on Java 9 JRE, to support reading
>>>> Java 9 class files, and to support FB running on Java 8 but analyzing
>>>> Java 9 code. Nice to have (but not in my scope right now) is to support
>>>> any new Java 8/9 constructs like lambdas, type annotations etc.
>>>>
>>>> I've documented briefly tasks coming to my mind via [1].
>>>>
>>>> I plan to push my changes on new java9 branch ASAP.
>>>>
>>>> Main discussion points I see so far:
>>>>
>>>> 1) We must bump the required JRE for FB to 1.8. I see no reason trying
>>>> to support obsoleted 1.7 JRE. If someone wants run FB on 1.7, the old FB
>>>> 3.0.1 should be used. Objections?
>>>>
>>>> 2) Since there are no official releases from ASM/BCEL with Java 9
>>>> support yet, we can release first version based on our own FB private
>>>> snapshot versions. The maven folks will cry but this is a chicken and
>>>> egg problem, so I don't care about maven support for the first round (of
>>>> course any help is welcome). Objections?
>>>>
>>>> 3) Due the JRE version bump I would propose to bump FB version to 3.1.0.
>>>> Objections?
>>>>
>>>> Please give your feedback either on the lists or on the github task [1].
>>>>
>>>> [1] https://github.com/findbugsproject/findbugs/issues/105
>>>>
>>>
>>> --
>>> Kind regards,
>>> google.com/+AndreyLoskutov
>>> _______________________________________________
>>> Findbugs-discuss mailing list
>>> [hidden email]
>>> https://mailman.cs.umd.edu/mailman/listinfo/findbugs-discuss
>
> --
> Kind regards,
> google.com/+AndreyLoskutov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

garydgregory
On Mon, Jun 6, 2016 at 7:50 PM, Charles Honton <[hidden email]> wrote:

> How Apache Commons BCEL got to where it is currently.
>
> 1. I wanted to release a version of BCEL which would support Java 6 and 7.
> 2. I updated several classes that handled the new instructions and new
> code attributes.
> 3. This required new methods on several interfaces.
> 4. These new methods broke binary compatibility.
> 5. Whenever binary compatibility is broken, Apache Commons policy is to
> update the Maven GAV to prevent jar hell.
> 6. Part of updating GAV is to also update the package names.
> 7. I created a release candidate which was deemed unsuitable for several
> reasons; mostly due to FindBugs warnings.
> 8. Multiple refactorings were completed (including moving Interface to
> Class) to handle FindBugs warnings.
> 9. Refactoring died out after no response from users as to Apache
> direction.
> 10. Recent new interest has us revisiting these changes.
>
> At this point, we’re somewhat stuck.
> 1. Do we break Apache Commons Policy regarding binary compatibility GAV
> and package names?
> 2. Do we ignore the FindBugs warnings?
>
> Personally, I am against either of those.  I also believe that to fix BCEL
> correctly, we’ll end up with an api sufficiently different that users will
> have a non-trivial porting task.  It might be saner if Apache Commons moves
> BCEL into the attic and suggest that our clients to migrate to either ASM
> or JavaAssist.
>

FB makes is pretty clear that porting to another f/w is not going to
happen. See my other msg WRT a suggested path.

Gary



>
> regards,
> chas
>
> > On Jun 6, 2016, at 11:27 AM, Andrey Loskutov <[hidden email]> wrote:
> >
> > Hi all,
> >
> > this is a follow up on
> https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2016-June/004282.html
> .
> >
> > I'm cross-posting this to [hidden email] because the discussion
> on FindBugs mailing list is related to the BCEL 6 API future, and because I
> would like to know the opinions from the BCEL community on the upcoming
> BCEL 6 release compatibility story.
> >
> > Please see my answers inline.
> >
> > On Monday 06 June 2016 17:30 sebb wrote:
> >> On 6 June 2016 at 16:23, Andrey Loskutov <[hidden email]> wrote:
> >>> Hi all,
> >>>
> >>> here is the current state of FindBugs adoption to Java 9.
> >>>
> >>> 1) FindBugs standalone/Eclipse plugin run fine now on/with Java 9, the
> latest code is on java9 branch (not on master yet!), see [0, 1]. If there
> is interest, I can provide binary snapshots.
> >>>
> >>> 2)  I have difficulties to use BCEL6 trunk, see [2]. Looks like even
> after fixing all compile errors due the various API breakages in BCEL 6
> (see [3]), the current BCEL 6 head can't be used directly as a replacement
> for our old BCEL5.2 fork, see [4]. If anyone from FB and/or BCEL gurus
> could check this, this would be really helpful. Either our BCEL 5.2 patches
> were not fully propagated upstream to BCEL or BCEL 6 trunk has few
> regressions, or I missed something during update? I have no idea, because
> of the next point below. The experimental BCEL 6 port is on an extra branch
> on top of Java 9 related changes, see commits prefixed with BCEL6 on
> java9_bcel6 branch at [5].
> >>>
> >>> 3) I would be very happy if someone (Bill?) would explain how the
> *current* BCEL5.2 fork used by FindBugs was built? It was added in commit
> [6] but I miss instructions how it differs from the original BCEL code and
> so unable to re-build it.
> >>>
> >>> 4) Assuming BCEL6 bugs/FB errors would be fixed (see [4]), transition
> to the current BCEL6 head would break each and every FindBugs client,
> because BCEL6 at current state uses different namespace and also added some
> other API breaking changes. If we chose this path, none of the 3rd party
> detectors will work anymore and therefore we must bump FindBugs version to
> 4.0.
> >>
> >> This is useful to know.
> >> So do the 3rd party detectors depend on the BCEL namespace?
> >
> > Yes
> >
> >> Or is it because of the BCEL API changes?
> >
> > Also yes.
> >
> >> If so, which ones?
> >
> > The biggest one is the package namespace change, because this affect
> each existing BCEL class/interface.
> > See commit
> https://github.com/findbugsproject/findbugs/commit/917b692d9a12e048921cd1216102b851866ac3e4
> which affects ~400 files in FindBugs.
> >
> > Much smaller (but still breaking API) changes were class name changes
> Constants -> Const, StackMapTable[Entry] -> StackMap[Entry] and the move of
> constants defined in Constants from the interface to the class, thus
> breaking everyone who implemented the interface and now miss the constants.
> The rename of StackMapTable/Entry broke also additionally every detector
> implemented on top of PreorderVisitor. StackMapTableEntry not only changed
> its name, but also changed signature: getByteCodeOffsetDelta ->
> getByteCodeOffset which gives you an additional piece of happiness.
> >
> > Finally, VisitorSupportsInvokeDynamic interface was removed, which broke
> all FB visitors based on it via AbstractFrameModelingVisitor and 8 methods
> were added to the Visitor interface.
> >
> > That's all I saw in our FB code (where we have lot of detectors),
> probably others will report additional API breakage too, I can't say for
> sure.
> >
> > But main issue is the namespace change - it is really unnecessary and
> surprising. I've read through the commons mailing list and I was surprised
> that I saw no real request for it or any discussion about it (I haven't
> read through all the years but around the namespace change last summer).
> The only thing I saw was the Jira request
> https://issues.apache.org/jira/browse/BCEL-222, out of nowhere, and few
> commits later BCEL 6 API was made incompatible to every existing client. :-(
> >
> >> I'm a bit suprised that the BCEL API should affect the detectors, but
> >> perhaps there's a good reason for that.
> >
> > BF analyses bytecode, and although we have also few recently added ASM
> based detectors (which are mostly BCEL free), most of the detectors (and
> unfortunately many places in the FB API) use BCEL data structures. It was a
> natural choice 15 years ago, where BCEL was the only one bytecode
> framework...
> >
> > One way to "fix" the current FindBugs misery  is to replace BCEL with
> ASM (asm.tree package &Co) but this requires lot of effort because API and
> design in ASM do not match BCEL 1:1 - and it will also break every FB
> client in much harder way BCEL 6 API breakage does it  today. Doing this
> will effectively mean a complete fork/rewrite of FindBugs code, and no one
> is willing to spend time for it.
> >
> >>> Question is: should we go this way? Alternative is: undo BCEL package
> renaming and revert few API changes were made. This sounds complicated, but
> is doable, see BCEL 6 fork where I renamed all packages back to the old
> namespace in few minutes [7]. Fixing other "smaller" breaking BCEL API
> changes is not that complicated either. However, it is also possible that
> BCEL 6 will be released without breaking API's, if I understood right the
> discussion on the apache commons-dev mailing list [8]. If anyone from BCEL
> is listening to this mailing list, it would be nice to get some feedback on
> BCEL 6 plans.
> >>
> >> I have done quite a bit of work trying to eliminate the API breaks
> >> without compromising the BCEL 6 updates.
> >
> > I really appreciate your effort. Please keep it going.
> >
> >> Though I have yet to revert the Java and Maven namespace changes as I
> >> wanted agreement with the approach first.
> >>
> >> From my point of view I would be happy to see a compatible version of
> >> BCEL using the original namespaces.
> >> I'm not sure what other Commons devs think.
> >
> > I hope to see a binary compatible BCEL 6 release, which is might be not
> 1:1 drop-in replacement of BCEL 5 but at least 99%. Some changes must
> happen, API must evolve, this is natural and no one can keep on the old API
> forever.
> > But! After walking over the all BCEL renamings etc I do not really see a
> real, functional reason to break *everything*, and a behavior change with
> annotations parsing is something one can live with for a major release. Not
> all detectors rely on annotations and BCEL behavior change can probably be
> fixed in FB core code (so hidden from 3rd party libraries).
> >
> >> There are still some Java8/Java9 features that are not fully supported.
> >> This is true regardless of the namespace issue.
> >
> > That's absolutely fine and understandable.
> >
> > My main goal it to get rid of private BCEL forks which cannot be
> rebuilt/updated anymore as we see it today, so that we can compile FB on
> BCEL head, catching all the fixes you will provide in  future BCEL
> versions. ...And in my ideal world, the new FindBugs release based on BCEL
> 6 will not break any existing 3rd party FindBugs detectors library, or
> eventually only require a few trivial changes.
> >
> > Thanks!
> >
> >>> [0] https://github.com/findbugsproject/findbugs/commits/java9
> >>> [1] https://github.com/findbugsproject/findbugs/issues/105
> >>> [2] https://github.com/findbugsproject/findbugs/issues/106
> >>> [3] https://issues.apache.org/jira/browse/BCEL-222
> >>> [4] https://issues.apache.org/jira/browse/BCEL-273
> >>> [5] https://github.com/findbugsproject/findbugs/commits/java9_bcel6
> >>> [6]
> https://github.com/findbugsproject/findbugs/commit/f9f46bf97c47f011e3757bf9904249faf1039239
> >>> [7] https://github.com/iloveeclipse/commons-bcel/commits/old_structure
> >>> [8]
> http://mail-archives.apache.org/mod_mbox/commons-dev/201606.mbox/thread
> >>>
> >>> On Sunday 05 June 2016 12:55 Andrey Loskutov wrote:
> >>>> Hi all,
> >>>>
> >>>> I've got some free time and now working on Java 9 support for
> FindBugs,
> >>>> the first draft works already, but need some more polishing.
> >>>>
> >>>> The main goal is to support FB running on Java 9 JRE, to support
> reading
> >>>> Java 9 class files, and to support FB running on Java 8 but analyzing
> >>>> Java 9 code. Nice to have (but not in my scope right now) is to
> support
> >>>> any new Java 8/9 constructs like lambdas, type annotations etc.
> >>>>
> >>>> I've documented briefly tasks coming to my mind via [1].
> >>>>
> >>>> I plan to push my changes on new java9 branch ASAP.
> >>>>
> >>>> Main discussion points I see so far:
> >>>>
> >>>> 1) We must bump the required JRE for FB to 1.8. I see no reason trying
> >>>> to support obsoleted 1.7 JRE. If someone wants run FB on 1.7, the old
> FB
> >>>> 3.0.1 should be used. Objections?
> >>>>
> >>>> 2) Since there are no official releases from ASM/BCEL with Java 9
> >>>> support yet, we can release first version based on our own FB private
> >>>> snapshot versions. The maven folks will cry but this is a chicken and
> >>>> egg problem, so I don't care about maven support for the first round
> (of
> >>>> course any help is welcome). Objections?
> >>>>
> >>>> 3) Due the JRE version bump I would propose to bump FB version to
> 3.1.0.
> >>>> Objections?
> >>>>
> >>>> Please give your feedback either on the lists or on the github task
> [1].
> >>>>
> >>>> [1] https://github.com/findbugsproject/findbugs/issues/105
> >>>>
> >>>
> >>> --
> >>> Kind regards,
> >>> google.com/+AndreyLoskutov
> >>> _______________________________________________
> >>> Findbugs-discuss mailing list
> >>> [hidden email]
> >>> https://mailman.cs.umd.edu/mailman/listinfo/findbugs-discuss
> >
> > --
> > Kind regards,
> > google.com/+AndreyLoskutov
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
E-Mail: [hidden email] | [hidden email]
Java Persistence with Hibernate, Second Edition
<http://www.manning.com/bauer3/>
JUnit in Action, Second Edition <http://www.manning.com/tahchiev/>
Spring Batch in Action <http://www.manning.com/templier/>
Blog: http://garygregory.wordpress.com
Home: http://garygregory.com/
Tweet! http://twitter.com/GaryGregory
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

sebb-2-2
In reply to this post by Charles Honton
On 7 June 2016 at 03:50, Charles Honton <[hidden email]> wrote:
> How Apache Commons BCEL got to where it is currently.
>
> 1. I wanted to release a version of BCEL which would support Java 6 and 7.
> 2. I updated several classes that handled the new instructions and new code attributes.
> 3. This required new methods on several interfaces.
> 4. These new methods broke binary compatibility.

However, there are ways around this.
1) assume code will use the abstract implementations
2) use Java 8 default implementations in the interface
3) use a sub-interface
4) handle the exceptions at run-time

> 5. Whenever binary compatibility is broken, Apache Commons policy is to update the Maven GAV to prevent jar hell.

Not exactly, see below.

> 6. Part of updating GAV is to also update the package names.

Avoiding Jar hell for non-Maven users requires package name changes.
For Maven users the GA coords must correspond 1-1 with Java packages
otherwise Maven cannot determine which jars can co-exist safely on the
classpath.

> 7. I created a release candidate which was deemed unsuitable for several reasons; mostly due to FindBugs warnings.
> 8. Multiple refactorings were completed (including moving Interface to Class) to handle FindBugs warnings.
> 9. Refactoring died out after no response from users as to Apache direction.
> 10. Recent new interest has us revisiting these changes.
>
> At this point, we’re somewhat stuck.
> 1. Do we break Apache Commons Policy regarding binary compatibility GAV and package names?

Please, no.

> 2. Do we ignore the FindBugs warnings?
>
> Personally, I am against either of those.  I also believe that to fix BCEL correctly, we’ll end up with an api sufficiently different that users will have a non-trivial porting task.  It might be saner if Apache Commons moves BCEL into the attic and suggest that our clients to migrate to either ASM or JavaAssist.

As I point out above, there are ways to avoid breaking compatibility.
This requires extra effort, but makes the update much simpler for end users.

i.e. it's a trade-off between Commons developer time and everyone else's time.

> regards,
> chas
>
>> On Jun 6, 2016, at 11:27 AM, Andrey Loskutov <[hidden email]> wrote:
>>
>> Hi all,
>>
>> this is a follow up on https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2016-June/004282.html.
>>
>> I'm cross-posting this to [hidden email] because the discussion on FindBugs mailing list is related to the BCEL 6 API future, and because I would like to know the opinions from the BCEL community on the upcoming BCEL 6 release compatibility story.
>>
>> Please see my answers inline.
>>
>> On Monday 06 June 2016 17:30 sebb wrote:
>>> On 6 June 2016 at 16:23, Andrey Loskutov <[hidden email]> wrote:
>>>> Hi all,
>>>>
>>>> here is the current state of FindBugs adoption to Java 9.
>>>>
>>>> 1) FindBugs standalone/Eclipse plugin run fine now on/with Java 9, the latest code is on java9 branch (not on master yet!), see [0, 1]. If there is interest, I can provide binary snapshots.
>>>>
>>>> 2)  I have difficulties to use BCEL6 trunk, see [2]. Looks like even after fixing all compile errors due the various API breakages in BCEL 6 (see [3]), the current BCEL 6 head can't be used directly as a replacement for our old BCEL5.2 fork, see [4]. If anyone from FB and/or BCEL gurus could check this, this would be really helpful. Either our BCEL 5.2 patches were not fully propagated upstream to BCEL or BCEL 6 trunk has few regressions, or I missed something during update? I have no idea, because of the next point below. The experimental BCEL 6 port is on an extra branch on top of Java 9 related changes, see commits prefixed with BCEL6 on java9_bcel6 branch at [5].
>>>>
>>>> 3) I would be very happy if someone (Bill?) would explain how the *current* BCEL5.2 fork used by FindBugs was built? It was added in commit [6] but I miss instructions how it differs from the original BCEL code and so unable to re-build it.
>>>>
>>>> 4) Assuming BCEL6 bugs/FB errors would be fixed (see [4]), transition to the current BCEL6 head would break each and every FindBugs client, because BCEL6 at current state uses different namespace and also added some other API breaking changes. If we chose this path, none of the 3rd party detectors will work anymore and therefore we must bump FindBugs version to 4.0.
>>>
>>> This is useful to know.
>>> So do the 3rd party detectors depend on the BCEL namespace?
>>
>> Yes
>>
>>> Or is it because of the BCEL API changes?
>>
>> Also yes.
>>
>>> If so, which ones?
>>
>> The biggest one is the package namespace change, because this affect each existing BCEL class/interface.
>> See commit https://github.com/findbugsproject/findbugs/commit/917b692d9a12e048921cd1216102b851866ac3e4 which affects ~400 files in FindBugs.
>>
>> Much smaller (but still breaking API) changes were class name changes Constants -> Const, StackMapTable[Entry] -> StackMap[Entry] and the move of constants defined in Constants from the interface to the class, thus breaking everyone who implemented the interface and now miss the constants. The rename of StackMapTable/Entry broke also additionally every detector implemented on top of PreorderVisitor. StackMapTableEntry not only changed its name, but also changed signature: getByteCodeOffsetDelta -> getByteCodeOffset which gives you an additional piece of happiness.
>>
>> Finally, VisitorSupportsInvokeDynamic interface was removed, which broke all FB visitors based on it via AbstractFrameModelingVisitor and 8 methods were added to the Visitor interface.
>>
>> That's all I saw in our FB code (where we have lot of detectors), probably others will report additional API breakage too, I can't say for sure.
>>
>> But main issue is the namespace change - it is really unnecessary and surprising. I've read through the commons mailing list and I was surprised that I saw no real request for it or any discussion about it (I haven't read through all the years but around the namespace change last summer). The only thing I saw was the Jira request https://issues.apache.org/jira/browse/BCEL-222, out of nowhere, and few commits later BCEL 6 API was made incompatible to every existing client. :-(
>>
>>> I'm a bit suprised that the BCEL API should affect the detectors, but
>>> perhaps there's a good reason for that.
>>
>> BF analyses bytecode, and although we have also few recently added ASM based detectors (which are mostly BCEL free), most of the detectors (and unfortunately many places in the FB API) use BCEL data structures. It was a natural choice 15 years ago, where BCEL was the only one bytecode framework...
>>
>> One way to "fix" the current FindBugs misery  is to replace BCEL with ASM (asm.tree package &Co) but this requires lot of effort because API and design in ASM do not match BCEL 1:1 - and it will also break every FB client in much harder way BCEL 6 API breakage does it  today. Doing this will effectively mean a complete fork/rewrite of FindBugs code, and no one is willing to spend time for it.
>>
>>>> Question is: should we go this way? Alternative is: undo BCEL package renaming and revert few API changes were made. This sounds complicated, but is doable, see BCEL 6 fork where I renamed all packages back to the old namespace in few minutes [7]. Fixing other "smaller" breaking BCEL API changes is not that complicated either. However, it is also possible that BCEL 6 will be released without breaking API's, if I understood right the discussion on the apache commons-dev mailing list [8]. If anyone from BCEL is listening to this mailing list, it would be nice to get some feedback on BCEL 6 plans.
>>>
>>> I have done quite a bit of work trying to eliminate the API breaks
>>> without compromising the BCEL 6 updates.
>>
>> I really appreciate your effort. Please keep it going.
>>
>>> Though I have yet to revert the Java and Maven namespace changes as I
>>> wanted agreement with the approach first.
>>>
>>> From my point of view I would be happy to see a compatible version of
>>> BCEL using the original namespaces.
>>> I'm not sure what other Commons devs think.
>>
>> I hope to see a binary compatible BCEL 6 release, which is might be not 1:1 drop-in replacement of BCEL 5 but at least 99%. Some changes must happen, API must evolve, this is natural and no one can keep on the old API forever.
>> But! After walking over the all BCEL renamings etc I do not really see a real, functional reason to break *everything*, and a behavior change with annotations parsing is something one can live with for a major release. Not all detectors rely on annotations and BCEL behavior change can probably be fixed in FB core code (so hidden from 3rd party libraries).
>>
>>> There are still some Java8/Java9 features that are not fully supported.
>>> This is true regardless of the namespace issue.
>>
>> That's absolutely fine and understandable.
>>
>> My main goal it to get rid of private BCEL forks which cannot be rebuilt/updated anymore as we see it today, so that we can compile FB on BCEL head, catching all the fixes you will provide in  future BCEL versions. ...And in my ideal world, the new FindBugs release based on BCEL 6 will not break any existing 3rd party FindBugs detectors library, or eventually only require a few trivial changes.
>>
>> Thanks!
>>
>>>> [0] https://github.com/findbugsproject/findbugs/commits/java9
>>>> [1] https://github.com/findbugsproject/findbugs/issues/105
>>>> [2] https://github.com/findbugsproject/findbugs/issues/106
>>>> [3] https://issues.apache.org/jira/browse/BCEL-222
>>>> [4] https://issues.apache.org/jira/browse/BCEL-273
>>>> [5] https://github.com/findbugsproject/findbugs/commits/java9_bcel6
>>>> [6] https://github.com/findbugsproject/findbugs/commit/f9f46bf97c47f011e3757bf9904249faf1039239
>>>> [7] https://github.com/iloveeclipse/commons-bcel/commits/old_structure
>>>> [8] http://mail-archives.apache.org/mod_mbox/commons-dev/201606.mbox/thread
>>>>
>>>> On Sunday 05 June 2016 12:55 Andrey Loskutov wrote:
>>>>> Hi all,
>>>>>
>>>>> I've got some free time and now working on Java 9 support for FindBugs,
>>>>> the first draft works already, but need some more polishing.
>>>>>
>>>>> The main goal is to support FB running on Java 9 JRE, to support reading
>>>>> Java 9 class files, and to support FB running on Java 8 but analyzing
>>>>> Java 9 code. Nice to have (but not in my scope right now) is to support
>>>>> any new Java 8/9 constructs like lambdas, type annotations etc.
>>>>>
>>>>> I've documented briefly tasks coming to my mind via [1].
>>>>>
>>>>> I plan to push my changes on new java9 branch ASAP.
>>>>>
>>>>> Main discussion points I see so far:
>>>>>
>>>>> 1) We must bump the required JRE for FB to 1.8. I see no reason trying
>>>>> to support obsoleted 1.7 JRE. If someone wants run FB on 1.7, the old FB
>>>>> 3.0.1 should be used. Objections?
>>>>>
>>>>> 2) Since there are no official releases from ASM/BCEL with Java 9
>>>>> support yet, we can release first version based on our own FB private
>>>>> snapshot versions. The maven folks will cry but this is a chicken and
>>>>> egg problem, so I don't care about maven support for the first round (of
>>>>> course any help is welcome). Objections?
>>>>>
>>>>> 3) Due the JRE version bump I would propose to bump FB version to 3.1.0.
>>>>> Objections?
>>>>>
>>>>> Please give your feedback either on the lists or on the github task [1].
>>>>>
>>>>> [1] https://github.com/findbugsproject/findbugs/issues/105
>>>>>
>>>>
>>>> --
>>>> Kind regards,
>>>> google.com/+AndreyLoskutov
>>>> _______________________________________________
>>>> Findbugs-discuss mailing list
>>>> [hidden email]
>>>> https://mailman.cs.umd.edu/mailman/listinfo/findbugs-discuss
>>
>> --
>> Kind regards,
>> google.com/+AndreyLoskutov
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

James Carman
We have to be willing to reevaluate the BC stringency we have had. Is it
working for our users? Is it worth the trouble it causes (people have left
this community over it)? Are there better options? Is it too strict and
could just be relaxed?

Our BC policies sound good on paper and do address things like "jar hell"
very well, but we definitely are out on the fringe. We are the fanatics
when it comes to BC.

On Tue, Jun 7, 2016 at 5:40 AM sebb <[hidden email]> wrote:

> On 7 June 2016 at 03:50, Charles Honton <[hidden email]> wrote:
> > How Apache Commons BCEL got to where it is currently.
> >
> > 1. I wanted to release a version of BCEL which would support Java 6 and
> 7.
> > 2. I updated several classes that handled the new instructions and new
> code attributes.
> > 3. This required new methods on several interfaces.
> > 4. These new methods broke binary compatibility.
>
> However, there are ways around this.
> 1) assume code will use the abstract implementations
> 2) use Java 8 default implementations in the interface
> 3) use a sub-interface
> 4) handle the exceptions at run-time
>
> > 5. Whenever binary compatibility is broken, Apache Commons policy is to
> update the Maven GAV to prevent jar hell.
>
> Not exactly, see below.
>
> > 6. Part of updating GAV is to also update the package names.
>
> Avoiding Jar hell for non-Maven users requires package name changes.
> For Maven users the GA coords must correspond 1-1 with Java packages
> otherwise Maven cannot determine which jars can co-exist safely on the
> classpath.
>
> > 7. I created a release candidate which was deemed unsuitable for several
> reasons; mostly due to FindBugs warnings.
> > 8. Multiple refactorings were completed (including moving Interface to
> Class) to handle FindBugs warnings.
> > 9. Refactoring died out after no response from users as to Apache
> direction.
> > 10. Recent new interest has us revisiting these changes.
> >
> > At this point, we’re somewhat stuck.
> > 1. Do we break Apache Commons Policy regarding binary compatibility GAV
> and package names?
>
> Please, no.
>
> > 2. Do we ignore the FindBugs warnings?
> >
> > Personally, I am against either of those.  I also believe that to fix
> BCEL correctly, we’ll end up with an api sufficiently different that users
> will have a non-trivial porting task.  It might be saner if Apache Commons
> moves BCEL into the attic and suggest that our clients to migrate to either
> ASM or JavaAssist.
>
> As I point out above, there are ways to avoid breaking compatibility.
> This requires extra effort, but makes the update much simpler for end
> users.
>
> i.e. it's a trade-off between Commons developer time and everyone else's
> time.
>
> > regards,
> > chas
> >
> >> On Jun 6, 2016, at 11:27 AM, Andrey Loskutov <[hidden email]> wrote:
> >>
> >> Hi all,
> >>
> >> this is a follow up on
> https://mailman.cs.umd.edu/pipermail/findbugs-discuss/2016-June/004282.html
> .
> >>
> >> I'm cross-posting this to [hidden email] because the
> discussion on FindBugs mailing list is related to the BCEL 6 API future,
> and because I would like to know the opinions from the BCEL community on
> the upcoming BCEL 6 release compatibility story.
> >>
> >> Please see my answers inline.
> >>
> >> On Monday 06 June 2016 17:30 sebb wrote:
> >>> On 6 June 2016 at 16:23, Andrey Loskutov <[hidden email]> wrote:
> >>>> Hi all,
> >>>>
> >>>> here is the current state of FindBugs adoption to Java 9.
> >>>>
> >>>> 1) FindBugs standalone/Eclipse plugin run fine now on/with Java 9,
> the latest code is on java9 branch (not on master yet!), see [0, 1]. If
> there is interest, I can provide binary snapshots.
> >>>>
> >>>> 2)  I have difficulties to use BCEL6 trunk, see [2]. Looks like even
> after fixing all compile errors due the various API breakages in BCEL 6
> (see [3]), the current BCEL 6 head can't be used directly as a replacement
> for our old BCEL5.2 fork, see [4]. If anyone from FB and/or BCEL gurus
> could check this, this would be really helpful. Either our BCEL 5.2 patches
> were not fully propagated upstream to BCEL or BCEL 6 trunk has few
> regressions, or I missed something during update? I have no idea, because
> of the next point below. The experimental BCEL 6 port is on an extra branch
> on top of Java 9 related changes, see commits prefixed with BCEL6 on
> java9_bcel6 branch at [5].
> >>>>
> >>>> 3) I would be very happy if someone (Bill?) would explain how the
> *current* BCEL5.2 fork used by FindBugs was built? It was added in commit
> [6] but I miss instructions how it differs from the original BCEL code and
> so unable to re-build it.
> >>>>
> >>>> 4) Assuming BCEL6 bugs/FB errors would be fixed (see [4]), transition
> to the current BCEL6 head would break each and every FindBugs client,
> because BCEL6 at current state uses different namespace and also added some
> other API breaking changes. If we chose this path, none of the 3rd party
> detectors will work anymore and therefore we must bump FindBugs version to
> 4.0.
> >>>
> >>> This is useful to know.
> >>> So do the 3rd party detectors depend on the BCEL namespace?
> >>
> >> Yes
> >>
> >>> Or is it because of the BCEL API changes?
> >>
> >> Also yes.
> >>
> >>> If so, which ones?
> >>
> >> The biggest one is the package namespace change, because this affect
> each existing BCEL class/interface.
> >> See commit
> https://github.com/findbugsproject/findbugs/commit/917b692d9a12e048921cd1216102b851866ac3e4
> which affects ~400 files in FindBugs.
> >>
> >> Much smaller (but still breaking API) changes were class name changes
> Constants -> Const, StackMapTable[Entry] -> StackMap[Entry] and the move of
> constants defined in Constants from the interface to the class, thus
> breaking everyone who implemented the interface and now miss the constants.
> The rename of StackMapTable/Entry broke also additionally every detector
> implemented on top of PreorderVisitor. StackMapTableEntry not only changed
> its name, but also changed signature: getByteCodeOffsetDelta ->
> getByteCodeOffset which gives you an additional piece of happiness.
> >>
> >> Finally, VisitorSupportsInvokeDynamic interface was removed, which
> broke all FB visitors based on it via AbstractFrameModelingVisitor and 8
> methods were added to the Visitor interface.
> >>
> >> That's all I saw in our FB code (where we have lot of detectors),
> probably others will report additional API breakage too, I can't say for
> sure.
> >>
> >> But main issue is the namespace change - it is really unnecessary and
> surprising. I've read through the commons mailing list and I was surprised
> that I saw no real request for it or any discussion about it (I haven't
> read through all the years but around the namespace change last summer).
> The only thing I saw was the Jira request
> https://issues.apache.org/jira/browse/BCEL-222, out of nowhere, and few
> commits later BCEL 6 API was made incompatible to every existing client. :-(
> >>
> >>> I'm a bit suprised that the BCEL API should affect the detectors, but
> >>> perhaps there's a good reason for that.
> >>
> >> BF analyses bytecode, and although we have also few recently added ASM
> based detectors (which are mostly BCEL free), most of the detectors (and
> unfortunately many places in the FB API) use BCEL data structures. It was a
> natural choice 15 years ago, where BCEL was the only one bytecode
> framework...
> >>
> >> One way to "fix" the current FindBugs misery  is to replace BCEL with
> ASM (asm.tree package &Co) but this requires lot of effort because API and
> design in ASM do not match BCEL 1:1 - and it will also break every FB
> client in much harder way BCEL 6 API breakage does it  today. Doing this
> will effectively mean a complete fork/rewrite of FindBugs code, and no one
> is willing to spend time for it.
> >>
> >>>> Question is: should we go this way? Alternative is: undo BCEL package
> renaming and revert few API changes were made. This sounds complicated, but
> is doable, see BCEL 6 fork where I renamed all packages back to the old
> namespace in few minutes [7]. Fixing other "smaller" breaking BCEL API
> changes is not that complicated either. However, it is also possible that
> BCEL 6 will be released without breaking API's, if I understood right the
> discussion on the apache commons-dev mailing list [8]. If anyone from BCEL
> is listening to this mailing list, it would be nice to get some feedback on
> BCEL 6 plans.
> >>>
> >>> I have done quite a bit of work trying to eliminate the API breaks
> >>> without compromising the BCEL 6 updates.
> >>
> >> I really appreciate your effort. Please keep it going.
> >>
> >>> Though I have yet to revert the Java and Maven namespace changes as I
> >>> wanted agreement with the approach first.
> >>>
> >>> From my point of view I would be happy to see a compatible version of
> >>> BCEL using the original namespaces.
> >>> I'm not sure what other Commons devs think.
> >>
> >> I hope to see a binary compatible BCEL 6 release, which is might be not
> 1:1 drop-in replacement of BCEL 5 but at least 99%. Some changes must
> happen, API must evolve, this is natural and no one can keep on the old API
> forever.
> >> But! After walking over the all BCEL renamings etc I do not really see
> a real, functional reason to break *everything*, and a behavior change with
> annotations parsing is something one can live with for a major release. Not
> all detectors rely on annotations and BCEL behavior change can probably be
> fixed in FB core code (so hidden from 3rd party libraries).
> >>
> >>> There are still some Java8/Java9 features that are not fully supported.
> >>> This is true regardless of the namespace issue.
> >>
> >> That's absolutely fine and understandable.
> >>
> >> My main goal it to get rid of private BCEL forks which cannot be
> rebuilt/updated anymore as we see it today, so that we can compile FB on
> BCEL head, catching all the fixes you will provide in  future BCEL
> versions. ...And in my ideal world, the new FindBugs release based on BCEL
> 6 will not break any existing 3rd party FindBugs detectors library, or
> eventually only require a few trivial changes.
> >>
> >> Thanks!
> >>
> >>>> [0] https://github.com/findbugsproject/findbugs/commits/java9
> >>>> [1] https://github.com/findbugsproject/findbugs/issues/105
> >>>> [2] https://github.com/findbugsproject/findbugs/issues/106
> >>>> [3] https://issues.apache.org/jira/browse/BCEL-222
> >>>> [4] https://issues.apache.org/jira/browse/BCEL-273
> >>>> [5] https://github.com/findbugsproject/findbugs/commits/java9_bcel6
> >>>> [6]
> https://github.com/findbugsproject/findbugs/commit/f9f46bf97c47f011e3757bf9904249faf1039239
> >>>> [7]
> https://github.com/iloveeclipse/commons-bcel/commits/old_structure
> >>>> [8]
> http://mail-archives.apache.org/mod_mbox/commons-dev/201606.mbox/thread
> >>>>
> >>>> On Sunday 05 June 2016 12:55 Andrey Loskutov wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> I've got some free time and now working on Java 9 support for
> FindBugs,
> >>>>> the first draft works already, but need some more polishing.
> >>>>>
> >>>>> The main goal is to support FB running on Java 9 JRE, to support
> reading
> >>>>> Java 9 class files, and to support FB running on Java 8 but analyzing
> >>>>> Java 9 code. Nice to have (but not in my scope right now) is to
> support
> >>>>> any new Java 8/9 constructs like lambdas, type annotations etc.
> >>>>>
> >>>>> I've documented briefly tasks coming to my mind via [1].
> >>>>>
> >>>>> I plan to push my changes on new java9 branch ASAP.
> >>>>>
> >>>>> Main discussion points I see so far:
> >>>>>
> >>>>> 1) We must bump the required JRE for FB to 1.8. I see no reason
> trying
> >>>>> to support obsoleted 1.7 JRE. If someone wants run FB on 1.7, the
> old FB
> >>>>> 3.0.1 should be used. Objections?
> >>>>>
> >>>>> 2) Since there are no official releases from ASM/BCEL with Java 9
> >>>>> support yet, we can release first version based on our own FB private
> >>>>> snapshot versions. The maven folks will cry but this is a chicken and
> >>>>> egg problem, so I don't care about maven support for the first round
> (of
> >>>>> course any help is welcome). Objections?
> >>>>>
> >>>>> 3) Due the JRE version bump I would propose to bump FB version to
> 3.1.0.
> >>>>> Objections?
> >>>>>
> >>>>> Please give your feedback either on the lists or on the github task
> [1].
> >>>>>
> >>>>> [1] https://github.com/findbugsproject/findbugs/issues/105
> >>>>>
> >>>>
> >>>> --
> >>>> Kind regards,
> >>>> google.com/+AndreyLoskutov
> >>>> _______________________________________________
> >>>> Findbugs-discuss mailing list
> >>>> [hidden email]
> >>>> https://mailman.cs.umd.edu/mailman/listinfo/findbugs-discuss
> >>
> >> --
> >> Kind regards,
> >> google.com/+AndreyLoskutov
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

jochen-2
On Tue, Jun 7, 2016 at 12:57 PM, James Carman
<[hidden email]> wrote:
> We have to be willing to reevaluate the BC stringency we have had. Is it
> working for our users? Is it worth the trouble it causes (people have left
> this community over it)? Are there better options? Is it too strict and
> could just be relaxed?
>
> Our BC policies sound good on paper and do address things like "jar hell"
> very well, but we definitely are out on the fringe. We are the fanatics
> when it comes to BC.

In the light of the current discussions, you may be right.

However, what I still don't understand: Why is BC such an issue for people?

I think, it is perfectly reasonable to do, what other projects do:

- Maintain several release branches.
- Depending on the branch, limit yourself to binary compatible changes, or not.
- Whenever binary incompatibilities are desirable: Create a new
branch, and start to
  publish releases out of that branch.

So, what is the big deal?

Jochen

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

James Carman
On Tue, Jun 7, 2016 at 8:15 AM Jochen Wiedmann <[hidden email]>
wrote:

> In the light of the current discussions, you may be right.
>
> However, what I still don't understand: Why is BC such an issue for people?
>
> I think, it is perfectly reasonable to do, what other projects do:
>
> - Maintain several release branches.
> - Depending on the branch, limit yourself to binary compatible changes, or
> not.
> - Whenever binary incompatibilities are desirable: Create a new
> branch, and start to
>   publish releases out of that branch.
>
> So, what is the big deal?
>
>
I totally agree with you, but it is as if this community has a seemingly
"maintain backward compatibility at all costs" mentality which can be
tremendously stifling to innovation.  We should be able to maintain (and we
have to for security patches) multiple release streams (2 or 3 seems
reasonable).  Obviously there's a balance we have to maintain, but I think
we have been way too far on one side of the spectrum for far too long.
People just aren't having much fun any more.  For a completely
volunteer-based work force, fun is important.
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

Benedikt Ritter-4
James Carman <[hidden email]> schrieb am Di., 7. Juni 2016 um
14:36 Uhr:

> On Tue, Jun 7, 2016 at 8:15 AM Jochen Wiedmann <[hidden email]>
> wrote:
>
> > In the light of the current discussions, you may be right.
> >
> > However, what I still don't understand: Why is BC such an issue for
> people?
> >
> > I think, it is perfectly reasonable to do, what other projects do:
> >
> > - Maintain several release branches.
> > - Depending on the branch, limit yourself to binary compatible changes,
> or
> > not.
> > - Whenever binary incompatibilities are desirable: Create a new
> > branch, and start to
> >   publish releases out of that branch.
> >
> > So, what is the big deal?
> >
> >
> I totally agree with you, but it is as if this community has a seemingly
> "maintain backward compatibility at all costs" mentality which can be
> tremendously stifling to innovation.  We should be able to maintain (and we
> have to for security patches) multiple release streams (2 or 3 seems
> reasonable).  Obviously there's a balance we have to maintain, but I think
> we have been way too far on one side of the spectrum for far too long.
> People just aren't having much fun any more.  For a completely
> volunteer-based work force, fun is important.
>

I think Andrey has made a point in the ohter thread that the backwards
compatibility is exactly what users like the findbugs project are looking
for...

Benedikt
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

Andrey Loskutov
In reply to this post by James Carman
Hi all,

following on the recent discussion and on the recent mails regarding the keeping the "old" BCEL namespace for 6.0 I just wanted to share my view on the BCEL API compatibility story, both from the API contributor/consumer sides, since I'm playing both roles in projects below.

###
TL;DR
Thanks for keeping the old namespace in the next BCEL release!
Please try to stay compatible in future releases - and your users will stay with the project.
Break the API often and your users will go away and the project will die.
###

Eclipse platform project has incredibly long API support and very clear rules of the API compatibility, see for example https://wiki.eclipse.org/Evolving_Java-based_APIs.
They main rule is: "API Prime Directive: When evolving the Component API from release to release, do not break existing Clients."

They managed to evolve API's over decade without changing the namespace. It is really hard (or almost impossible) to break API in Eclipse, and as a committer you are always under pressure NOT to release any API because otherwise you must write really good code, because you know, it will stay almost forever. So it is a big burden for the committer, but is is a blessing for a plugin provider or system integrator, because you know your investment will not be wasted with the next release. As a client of the platform I'm pretty sure that my plugins will most likely work even with the next major release - and if not, the solution is not far away. I know for sure that all my time I spent years in different Eclipse projects was well invested. Sure committers aren't fully free in their decisions, but they who spent time for this project can be really proud to work on something which will be used by millions of other people. At same time they also know how big the responsibility is if they decide to change API.

Another, completely different project is ASM. It does not have that number of committers, that power and that strict rules as Eclipse, but it also aims to keep the compatibility as a very important feature.
The purpose of the tool is same as BCEL, and it has a very good compatibility history, despite the fact that if timely follows on changes in Java bytecode (see  http://asm.ow2.org/history.html).
Although the ASM API compatibility was broken few times, the transition to the next major release was and is really easy for clients! Since 4.x release transition to new ASM API's is a no-brainer.

ASM also managed to evolve API's over decade without changing the namespace. And if you ever used ASM 4.x API, you are pretty sure it would need just a really small change to use 6.x.
I've switched FindBugs from ASM 5 to ASM 6 in few seconds (the most work was to replace library name in build files), the actual change was to replace a single character in the line below:
-    public static final int ASM_VERSION = Opcodes.ASM5;
+    public static final int ASM_VERSION = Opcodes.ASM6;
That is! This one character transition gives ASM clients Java 9 support "almost for free". Isn't cool? This is the real power of compatibility! I was done in a second and *everything* still worked!

Why I'm talking about Eclipse and ASM? Both Eclipse and ASM are very different in they nature, size and governance.
But!
Both projects have exact same resource problems with most committers doing the job in their free time.
They are also very similar because they have huge success, and one significant part of this success is because they are very cautious regarding the API changes, because they care a lot about API clients!
Also one can't really argue Eclipse or ASM aren't innovative because they care about compatibility. They just trying to deliver both, because they recognized the value of innovation *and* compatibility. Sure the committers life could be easier without keeping the API compatibility, but nothing is for free and success has proved them right.

Let's now take FindBugs. Assume we would release and innovative but fully incompatible 4.0 version based on the original BCEL6 trunk state before undoing package renaming.
ALL existing FB plugins will immediately stop working, because we expose BCEL API to clients. We will get a shit storm on the mailing list and few plugin providers will simply give up. Then suppose there will be Java 9 and BCEL7. Assume that this will cause another BCEL API breakage, and assume we will follow with fully incompatible FB 5.0. The small number of 3rd party providers who managed to stay and migrate to FB 4.0 will see they investment wasted again. And this time they probably will simply stop investing into FB plugins forever and switch to ASM/whatever based tools. Why should they regularly waste they time and money and rewrite they software just to get it working on a newer JVM? So at the end we will have no users and a project without users is effectively dead.

Now back to BCEL. Historically this was one of the first and only Java bytecode frameworks. Honestly speaking, the reason why it is still used today in 3rd party software (like FindBugs) is because of the investment costs. We simply cannot afford full switch to ASM because it requires a huge amount of time (and also because it will break every single client). If BCEL still have users today, than only those who want stable API's, all others are already ASM users or users of gazillions other bytecode manipulation frameworks we have today (mostly ASM based).

API's you have in BCEL *is* the main reason why the project is still used, and API compatibility (or easy of the transition to the new API) is one of the biggest features (if not THE feature) you can offer to the existing BCEL clients. It has  *very* big value!

Sure API can and should evolve, there is no doubt, but it should not be broken just because "project rule requires it" and "volunteers should have more fun". If the API must change, then please only because for the real, important *functional* reasons and in a most gentle way for the API consumers.

BTW the *real* fun for volunteer is to implement features by keeping the compatibility as far as one can manage it. Everyone can write some new fun code out of nowhere , but the hardest possible challenge for real hacker is to implement a new feature without breaking existing API clients! This is the real black art of programming!

On Tuesday 07 June 2016 12:36 James Carman wrote:

> On Tue, Jun 7, 2016 at 8:15 AM Jochen Wiedmann <[hidden email]>
> wrote:
>
> > In the light of the current discussions, you may be right.
> >
> > However, what I still don't understand: Why is BC such an issue for people?
> >
> > I think, it is perfectly reasonable to do, what other projects do:
> >
> > - Maintain several release branches.
> > - Depending on the branch, limit yourself to binary compatible changes, or
> > not.
> > - Whenever binary incompatibilities are desirable: Create a new
> > branch, and start to
> >   publish releases out of that branch.
> >
> > So, what is the big deal?
> >
> >
> I totally agree with you, but it is as if this community has a seemingly
> "maintain backward compatibility at all costs" mentality which can be
> tremendously stifling to innovation.  We should be able to maintain (and we
> have to for security patches) multiple release streams (2 or 3 seems
> reasonable).  Obviously there's a balance we have to maintain, but I think
> we have been way too far on one side of the spectrum for far too long.
> People just aren't having much fun any more.  For a completely
> volunteer-based work force, fun is important.

--
Kind regards,
google.com/+AndreyLoskutov

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

Matt Benson-2
AFAIK ASM already put its users through the pain of BC breakage when they
went from an interface-based design to one based on abstract classes (3 to
4 or 4 to 5) but mixing versions across that boundary is the very
definition of jar hell and is why ASM recommends that their classes be
shaded into another namespace.

Java 8 interface default methods would help here, but a library using them,
while preserving API compatibility, would break strict BC simply by
requiring Java 8.
On Jun 7, 2016 2:06 PM, "Andrey Loskutov" <[hidden email]> wrote:

Hi all,

following on the recent discussion and on the recent mails regarding the
keeping the "old" BCEL namespace for 6.0 I just wanted to share my view on
the BCEL API compatibility story, both from the API contributor/consumer
sides, since I'm playing both roles in projects below.

###
TL;DR
Thanks for keeping the old namespace in the next BCEL release!
Please try to stay compatible in future releases - and your users will stay
with the project.
Break the API often and your users will go away and the project will die.
###

Eclipse platform project has incredibly long API support and very clear
rules of the API compatibility, see for example
https://wiki.eclipse.org/Evolving_Java-based_APIs.
They main rule is: "API Prime Directive: When evolving the Component API
from release to release, do not break existing Clients."

They managed to evolve API's over decade without changing the namespace. It
is really hard (or almost impossible) to break API in Eclipse, and as a
committer you are always under pressure NOT to release any API because
otherwise you must write really good code, because you know, it will stay
almost forever. So it is a big burden for the committer, but is is a
blessing for a plugin provider or system integrator, because you know your
investment will not be wasted with the next release. As a client of the
platform I'm pretty sure that my plugins will most likely work even with
the next major release - and if not, the solution is not far away. I know
for sure that all my time I spent years in different Eclipse projects was
well invested. Sure committers aren't fully free in their decisions, but
they who spent time for this project can be really proud to work on
something which will be used by millions of other people. At same time they
also know how big the responsibility is if they decide to change API.

Another, completely different project is ASM. It does not have that number
of committers, that power and that strict rules as Eclipse, but it also
aims to keep the compatibility as a very important feature.
The purpose of the tool is same as BCEL, and it has a very good
compatibility history, despite the fact that if timely follows on changes
in Java bytecode (see  http://asm.ow2.org/history.html).
Although the ASM API compatibility was broken few times, the transition to
the next major release was and is really easy for clients! Since 4.x
release transition to new ASM API's is a no-brainer.

ASM also managed to evolve API's over decade without changing the
namespace. And if you ever used ASM 4.x API, you are pretty sure it would
need just a really small change to use 6.x.
I've switched FindBugs from ASM 5 to ASM 6 in few seconds (the most work
was to replace library name in build files), the actual change was to
replace a single character in the line below:
-    public static final int ASM_VERSION = Opcodes.ASM5;
+    public static final int ASM_VERSION = Opcodes.ASM6;
That is! This one character transition gives ASM clients Java 9 support
"almost for free". Isn't cool? This is the real power of compatibility! I
was done in a second and *everything* still worked!

Why I'm talking about Eclipse and ASM? Both Eclipse and ASM are very
different in they nature, size and governance.
But!
Both projects have exact same resource problems with most committers doing
the job in their free time.
They are also very similar because they have huge success, and one
significant part of this success is because they are very cautious
regarding the API changes, because they care a lot about API clients!
Also one can't really argue Eclipse or ASM aren't innovative because they
care about compatibility. They just trying to deliver both, because they
recognized the value of innovation *and* compatibility. Sure the committers
life could be easier without keeping the API compatibility, but nothing is
for free and success has proved them right.

Let's now take FindBugs. Assume we would release and innovative but fully
incompatible 4.0 version based on the original BCEL6 trunk state before
undoing package renaming.
ALL existing FB plugins will immediately stop working, because we expose
BCEL API to clients. We will get a shit storm on the mailing list and few
plugin providers will simply give up. Then suppose there will be Java 9 and
BCEL7. Assume that this will cause another BCEL API breakage, and assume we
will follow with fully incompatible FB 5.0. The small number of 3rd party
providers who managed to stay and migrate to FB 4.0 will see they
investment wasted again. And this time they probably will simply stop
investing into FB plugins forever and switch to ASM/whatever based tools.
Why should they regularly waste they time and money and rewrite they
software just to get it working on a newer JVM? So at the end we will have
no users and a project without users is effectively dead.

Now back to BCEL. Historically this was one of the first and only Java
bytecode frameworks. Honestly speaking, the reason why it is still used
today in 3rd party software (like FindBugs) is because of the investment
costs. We simply cannot afford full switch to ASM because it requires a
huge amount of time (and also because it will break every single client).
If BCEL still have users today, than only those who want stable API's, all
others are already ASM users or users of gazillions other bytecode
manipulation frameworks we have today (mostly ASM based).

API's you have in BCEL *is* the main reason why the project is still used,
and API compatibility (or easy of the transition to the new API) is one of
the biggest features (if not THE feature) you can offer to the existing
BCEL clients. It has  *very* big value!

Sure API can and should evolve, there is no doubt, but it should not be
broken just because "project rule requires it" and "volunteers should have
more fun". If the API must change, then please only because for the real,
important *functional* reasons and in a most gentle way for the API
consumers.

BTW the *real* fun for volunteer is to implement features by keeping the
compatibility as far as one can manage it. Everyone can write some new fun
code out of nowhere , but the hardest possible challenge for real hacker is
to implement a new feature without breaking existing API clients! This is
the real black art of programming!

On Tuesday 07 June 2016 12:36 James Carman wrote:
> On Tue, Jun 7, 2016 at 8:15 AM Jochen Wiedmann <[hidden email]>
> wrote:
>
> > In the light of the current discussions, you may be right.
> >
> > However, what I still don't understand: Why is BC such an issue for
people?
> >
> > I think, it is perfectly reasonable to do, what other projects do:
> >
> > - Maintain several release branches.
> > - Depending on the branch, limit yourself to binary compatible changes,
or

> > not.
> > - Whenever binary incompatibilities are desirable: Create a new
> > branch, and start to
> >   publish releases out of that branch.
> >
> > So, what is the big deal?
> >
> >
> I totally agree with you, but it is as if this community has a seemingly
> "maintain backward compatibility at all costs" mentality which can be
> tremendously stifling to innovation.  We should be able to maintain (and
we
> have to for security patches) multiple release streams (2 or 3 seems
> reasonable).  Obviously there's a balance we have to maintain, but I think
> we have been way too far on one side of the spectrum for far too long.
> People just aren't having much fun any more.  For a completely
> volunteer-based work force, fun is important.

--
Kind regards,
google.com/+AndreyLoskutov

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

Torsten Curdt-3
While I am a big fan of ASM it feels a bit strange to put it forth as a
great example in this regard.

Indeed some ASM API changes were simple - some weren't so much. And many
required source level changes. Some changes are often just a quick refactor
away. If we'd allow just that we'd be a good step further.

BCEL has committed some horrible (and I mean horrible!) API sins (one word:
"static"). There is just no way those can be solved without breaking
anything (if there ever is going to be enough progress forward).

Of course there are investment costs - but that doesn't mean you can/should
expect upgrades at cost zero. It's not like we have enough dev resources to
actually turn BCEL around like that for that major changes anyway (I
assume).

ALL existing FB plugins will immediately stop working, because we expose
> BCEL API to clients. We will get a shit storm on the mailing list and few
> plugin providers will simply give up.


I guess that should raise a couple of questions for the FB dev team:
- Do you really need to expose BCEL through the plugin API? Was that a wise
choice?
- Why will people not just update the plugins? Why will there be a
"shitstorm"?
- Why not help the plugin providers?
- Will those changes be really so hard to apply?

It's not about "volunteers should have more fun" but finding people to do
the job.
Living in the past just does not help with that.

cheers,
Torsten
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

dbrosIus
In reply to this post by Andrey Loskutov
I have almost certainly the largest FindBugs plug in.  I expect it will take less than 15 minutes to update it to use a breaking package/api version of BCEL if it ever was to go out. 
I've got this fancy thing called a text editor with search/replace. I pity those who haven't adopted this new tech. 

-------- Original message --------
From: Torsten Curdt <[hidden email]>
Date: 06/07/2016  9:18 PM  (GMT-05:00)
To: Commons Developers List <[hidden email]>, [hidden email]
Cc: James Carman <[hidden email]>, Matt Benson <[hidden email]>
Subject: Re: BCEL 6 API breakage

While I am a big fan of ASM it feels a bit strange to put it forth as a
great example in this regard.

Indeed some ASM API changes were simple - some weren't so much. And many
required source level changes. Some changes are often just a quick refactor
away. If we'd allow just that we'd be a good step further.

BCEL has committed some horrible (and I mean horrible!) API sins (one word:
"static"). There is just no way those can be solved without breaking
anything (if there ever is going to be enough progress forward).

Of course there are investment costs - but that doesn't mean you can/should
expect upgrades at cost zero. It's not like we have enough dev resources to
actually turn BCEL around like that for that major changes anyway (I
assume).

ALL existing FB plugins will immediately stop working, because we expose
> BCEL API to clients. We will get a shit storm on the mailing list and few
> plugin providers will simply give up.


I guess that should raise a couple of questions for the FB dev team:
- Do you really need to expose BCEL through the plugin API? Was that a wise
choice?
- Why will people not just update the plugins? Why will there be a
"shitstorm"?
- Why not help the plugin providers?
- Will those changes be really so hard to apply?

It's not about "volunteers should have more fun" but finding people to do
the job.
Living in the past just does not help with that.

cheers,
Torsten
Reply | Threaded
Open this post in threaded view
|

Re: BCEL 6 API breakage

sebb-2-2
On 8 June 2016 at 02:32, dbrosIus <[hidden email]> wrote:
> I have almost certainly the largest FindBugs plug in.  I expect it will take less than 15 minutes to update it to use a breaking package/api version of BCEL if it ever was to go out.
> I've got this fancy thing called a text editor with search/replace. I pity those who haven't adopted this new tech.

That's not all that needs to be done to produce a new release.

The changes need to be tested, built, and published.
And if the plugin is downloaded separately from Findbugs, users need
to download it again when they update Findbugs.

IMO it's important to take what could be called a "holistic approach" to change.
i.e. consider the effect on downstream users as well.

> -------- Original message --------
> From: Torsten Curdt <[hidden email]>
> Date: 06/07/2016  9:18 PM  (GMT-05:00)
> To: Commons Developers List <[hidden email]>, [hidden email]
> Cc: James Carman <[hidden email]>, Matt Benson <[hidden email]>
> Subject: Re: BCEL 6 API breakage
>
> While I am a big fan of ASM it feels a bit strange to put it forth as a
> great example in this regard.
>
> Indeed some ASM API changes were simple - some weren't so much. And many
> required source level changes. Some changes are often just a quick refactor
> away. If we'd allow just that we'd be a good step further.
>
> BCEL has committed some horrible (and I mean horrible!) API sins (one word:
> "static"). There is just no way those can be solved without breaking
> anything (if there ever is going to be enough progress forward).
>
> Of course there are investment costs - but that doesn't mean you can/should
> expect upgrades at cost zero. It's not like we have enough dev resources to
> actually turn BCEL around like that for that major changes anyway (I
> assume).
>
> ALL existing FB plugins will immediately stop working, because we expose
>> BCEL API to clients. We will get a shit storm on the mailing list and few
>> plugin providers will simply give up.
>
>
> I guess that should raise a couple of questions for the FB dev team:
> - Do you really need to expose BCEL through the plugin API? Was that a wise
> choice?
> - Why will people not just update the plugins? Why will there be a
> "shitstorm"?
> - Why not help the plugin providers?
> - Will those changes be really so hard to apply?
>
> It's not about "volunteers should have more fun" but finding people to do
> the job.
> Living in the past just does not help with that.
>
> cheers,
> Torsten

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]