[compress] What to do with Pack200?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[compress] What to do with Pack200?

Stefan Bodewig
Hi

our travis build also uses the combination

    - dist: trusty
      jdk: openjdk-ea

which has started to use OpenJDK 14 a while ago and our build has been
failing there ever since. The reason is
https://openjdk.java.net/jeps/367 - the whole Pack200 stuff has been
removed without any replacement.

AFAICT there is no (official) alternative implementation. There has been
some discussion on the Eclipse lists eighteen months ago but it looks as
if it never came to fruition - mostly because of legal reasons. The same
legal reasons would probably apply if we resurrected the pack200 code
from the atticed Apache Harmony codebase.

If there was an alternative that used the same packages and APIs we
could add a new dependency conditional on JDK being 1.14 or
above.

Without that it looks as if we had to remove the pack200 code from
commons-compress and create a separate artifact for that which simply
would not be built on Java 14+.

Any other ideas?

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Bruno P. Kinoshita-3
I think if we are going to release it soon, the only option to support java 14+ the would be to remove it. Tell users it may be added back later either as-was or as a separate artefact.
Just my 0.02 cents
CheersBruno

Sent from Yahoo Mail on Android
 
  On Thu, 2 Jan 2020 at 23:06, Stefan Bodewig<[hidden email]> wrote:   Hi

our travis build also uses the combination

    - dist: trusty
      jdk: openjdk-ea

which has started to use OpenJDK 14 a while ago and our build has been
failing there ever since. The reason is
https://openjdk.java.net/jeps/367 - the whole Pack200 stuff has been
removed without any replacement.

AFAICT there is no (official) alternative implementation. There has been
some discussion on the Eclipse lists eighteen months ago but it looks as
if it never came to fruition - mostly because of legal reasons. The same
legal reasons would probably apply if we resurrected the pack200 code
from the atticed Apache Harmony codebase.

If there was an alternative that used the same packages and APIs we
could add a new dependency conditional on JDK being 1.14 or
above.

Without that it looks as if we had to remove the pack200 code from
commons-compress and create a separate artifact for that which simply
would not be built on Java 14+.

Any other ideas?

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

 
Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Stefan Bodewig
On 2020-01-02, Bruno P. Kinoshita wrote:

> I think if we are going to release it soon, the only option to support java 14+ the would be to remove it. Tell users it may be added back later either as-was or as a separate artefact.

If we want to cut a new release soon, we can just tell people to not use
JDK 14+ when building Compress ;-)

Actually I hope to release 1.20 soonish and it will still target Java 7
- as there is not reason to require more than that right now.  Compress
as it is can be built with any released version of Java (14 is still
early access) newer than Java 6, this is good enough for now.

This question - unlike the question about the OSGi bundle name is more
about what happens after the next release. We could even mention our
plans about pack200 with the 1.20 release notes.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Emmanuel Bourg-3
In reply to this post by Stefan Bodewig
Le 02/01/2020 à 11:06, Stefan Bodewig a écrit :

> Any other ideas?

The removal of pack200 isn't an issue for the upcoming release yet since
we still target Java 7. For now this is just an issue with the CI using
Java 14+. Maybe we can deal with this by using a Maven profile excluding
the pack200 classes from the compilation when Java 14+ is used.

This leaves some time to find an alternative implementation, I don't
think Commons Compress will require Java 14+ in the next 5 years.

FYI I plan to fork the JDK pack200 implementation into a standalone
project. I've reserved the 'pack200' group on GitHub for this. The code
isn't published yet but I've already Mavenized the project and converted
the tests to JUnit. I'm struggling a bit to get the native code to build.

The fork will inherit the GPL2 with classpath exception license from
OpenJDK. I'm not sure Apache projects are allowed to depend on such
libraries, but that would be odd not to be able to use it since it's
exactly the same code that we used in the JDK.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Stefan Bodewig
On 2020-01-04, Emmanuel Bourg wrote:

> Le 02/01/2020 à 11:06, Stefan Bodewig a écrit :

>> Any other ideas?

> The removal of pack200 isn't an issue for the upcoming release yet since
> we still target Java 7. For now this is just an issue with the CI using
> Java 14+. Maybe we can deal with this by using a Maven profile excluding
> the pack200 classes from the compilation when Java 14+ is used.

This won't work as CompressorStreamFactory depends on the package.

> This leaves some time to find an alternative implementation, I don't
> think Commons Compress will require Java 14+ in the next 5 years.

Me neither.

> FYI I plan to fork the JDK pack200 implementation into a standalone
> project. I've reserved the 'pack200' group on GitHub for this. The code
> isn't published yet

https://github.com/pack200/pack200 looks pretty public to me :-)

> but I've already Mavenized the project and converted the tests to
> JUnit. I'm struggling a bit to get the native code to build.

I vaguely recall there've been some legal strings attached that were
raised when Eclipse discussed ther alternatives for p2 but its been a
while and I haven't re-read the threads, yet.

> The fork will inherit the GPL2 with classpath exception license from
> OpenJDK. I'm not sure Apache projects are allowed to depend on such
> libraries, but that would be odd not to be able to use it since it's
> exactly the same code that we used in the JDK.

I'd say it is fair to call pack200 support an optional feature of
Commons Compress https://apache.org/legal/resolved.html#optional

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

sebb-2-2
On Sat, 4 Jan 2020 at 11:19, Stefan Bodewig <[hidden email]> wrote:

> On 2020-01-04, Emmanuel Bourg wrote:
>
> > Le 02/01/2020 à 11:06, Stefan Bodewig a écrit :
>
> >> Any other ideas?
>
> > The removal of pack200 isn't an issue for the upcoming release yet since
> > we still target Java 7. For now this is just an issue with the CI using
> > Java 14+. Maybe we can deal with this by using a Maven profile excluding
> > the pack200 classes from the compilation when Java 14+ is used.
>
> This won't work as CompressorStreamFactory depends on the package.
>
> > This leaves some time to find an alternative implementation, I don't
> > think Commons Compress will require Java 14+ in the next 5 years.
>
> Me neither.
>
> > FYI I plan to fork the JDK pack200 implementation into a standalone
> > project. I've reserved the 'pack200' group on GitHub for this. The code
> > isn't published yet
>
> https://github.com/pack200/pack200 looks pretty public to me :-)
>
> > but I've already Mavenized the project and converted the tests to
> > JUnit. I'm struggling a bit to get the native code to build.
>
> I vaguely recall there've been some legal strings attached that were
> raised when Eclipse discussed ther alternatives for p2 but its been a
> while and I haven't re-read the threads, yet.
>
> > The fork will inherit the GPL2 with classpath exception license from
> > OpenJDK. I'm not sure Apache projects are allowed to depend on such
> > libraries, but that would be odd not to be able to use it since it's
> > exactly the same code that we used in the JDK.
>
> I'd say it is fair to call pack200 support an optional feature of
> Commons Compress https://apache.org/legal/resolved.html#optional
>
>
That only applies if Commons Compress still works without the optional
component.


> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Emmanuel Bourg-3
In reply to this post by Stefan Bodewig
Le 04/01/2020 à 12:19, Stefan Bodewig a écrit :

> This won't work as CompressorStreamFactory depends on the package.

Maybe some reflection could address that?


> https://github.com/pack200/pack200 looks pretty public to me :-)

I've just pushed a raw extraction of the OpenJDK code with its history.
I'll have to redo it to include all the C files required to compile the
native library.


> I'd say it is fair to call pack200 support an optional feature of
> Commons Compress https://apache.org/legal/resolved.html#optional

Good point, I wasn't aware of this rule.

Alternatively I'm considering adding an independent Apache-2.0 licensed
interface, and having the GPLed code provide an implementation, maybe
using a META-INF service mechanism.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

garydgregory
Java has a service leader mechanism for that. No sense in duplicating it.
If that means updating to Java 8, then so be it.

Gary

On Sat, Jan 4, 2020, 07:58 Emmanuel Bourg <[hidden email]> wrote:

> Le 04/01/2020 à 12:19, Stefan Bodewig a écrit :
>
> > This won't work as CompressorStreamFactory depends on the package.
>
> Maybe some reflection could address that?
>
>
> > https://github.com/pack200/pack200 looks pretty public to me :-)
>
> I've just pushed a raw extraction of the OpenJDK code with its history.
> I'll have to redo it to include all the C files required to compile the
> native library.
>
>
> > I'd say it is fair to call pack200 support an optional feature of
> > Commons Compress https://apache.org/legal/resolved.html#optional
>
> Good point, I wasn't aware of this rule.
>
> Alternatively I'm considering adding an independent Apache-2.0 licensed
> interface, and having the GPLed code provide an implementation, maybe
> using a META-INF service mechanism.
>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Stefan Bodewig
On 2020-01-04, Gary Gregory wrote:

> Java has a service leader mechanism for that. No sense in duplicating it.

Compress already supports that, and IIRC you've added that yourself :-)

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

garydgregory
Is Pack200 100% Java? Even if not, I think it would be neat to bring it in
from Apache Harmony as a new module. I think we should consider converting
compress into a multi-module project and avoid hard dependencies. For folks
who want everything, we can create a BOM POM.

For the test, the simplest might be to use a JUnit Assume to skip running
that test on Java > 14.

Gary

On Sat, Jan 4, 2020 at 9:52 AM Stefan Bodewig <[hidden email]> wrote:

> On 2020-01-04, Gary Gregory wrote:
>
> > Java has a service leader mechanism for that. No sense in duplicating it.
>
> Compress already supports that, and IIRC you've added that yourself :-)
>
> Stefan
>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Emmanuel Bourg-3
Le 05/01/2020 à 14:09, Gary Gregory a écrit :

> Is Pack200 100% Java?

In OpenJDK the pack200 encoder is 100% Java, and there are two decoder
implementations in Java and in C. The native decoder is 5 times faster
than the Java one.


> I think it would be neat to bring it in from Apache Harmony

The pack200 implementation in Harmony is a bit old now, it doesn't
support the new bytecode instructions introduced since Java 7.


> I think we should consider converting
> compress into a multi-module project and avoid hard dependencies.

As compress grows and supports more formats by using external
dependencies, I also think some form of modularization will be needed.
Otherwise it'll become a behemoth like Tika.


> For the test, the simplest might be to use a JUnit Assume to skip running
> that test on Java > 14.

Is it possible to build with Java 8 and test with Java 14 in the same CI
run?

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

garydgregory
Phone, top post, sorry. Yes you can build on J8 and test on J14 using a
Maven toochain. But it might be easier to use GitHub actions and Travis to
do it all on one at a time and all Java versions.

Gary

On Sun, Jan 5, 2020, 09:04 Emmanuel Bourg <[hidden email]> wrote:

> Le 05/01/2020 à 14:09, Gary Gregory a écrit :
>
> > Is Pack200 100% Java?
>
> In OpenJDK the pack200 encoder is 100% Java, and there are two decoder
> implementations in Java and in C. The native decoder is 5 times faster
> than the Java one.
>
>
> > I think it would be neat to bring it in from Apache Harmony
>
> The pack200 implementation in Harmony is a bit old now, it doesn't
> support the new bytecode instructions introduced since Java 7.
>
>
> > I think we should consider converting
> > compress into a multi-module project and avoid hard dependencies.
>
> As compress grows and supports more formats by using external
> dependencies, I also think some form of modularization will be needed.
> Otherwise it'll become a behemoth like Tika.
>
>
> > For the test, the simplest might be to use a JUnit Assume to skip running
> > that test on Java > 14.
>
> Is it possible to build with Java 8 and test with Java 14 in the same CI
> run?
>
> Emmanuel Bourg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Stefan Bodewig
In reply to this post by garydgregory
On 2020-01-05, Gary Gregory wrote:

> Is Pack200 100% Java? Even if not, I think it would be neat to bring it in
> from Apache Harmony as a new module.

What Emmanuel says.

> I think we should consider converting compress into a multi-module
> project and avoid hard dependencies. For folks who want everything, we
> can create a BOM POM.

Time to give the compress-2.0 branch another chance?

Our current ServiceLoader approach does not include auto-detection of
formats, something that is supported by Pack200. So if we want to move
pack200 - or any of the other formats requiring an external lib that
would support auto-detection - out of the main code base we need to
extend the CompressorStreamProvider interface anyway (likely create a
new one). compress-2.0 is completely built around the abstraction of a
(service-loadable) format.

> For the test, the simplest might be to use a JUnit Assume to skip running
> that test on Java > 14.

Yes, we already do so for the tests with optional dependencies.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] What to do with Pack200?

Emmanuel Bourg-3
In reply to this post by Stefan Bodewig
Le 04/01/2020 à 12:19, Stefan Bodewig a écrit :

>> FYI I plan to fork the JDK pack200 implementation into a standalone
>> project. I've reserved the 'pack200' group on GitHub for this. The code
>> isn't published yet
>
> https://github.com/pack200/pack200 looks pretty public to me :-)

I've published my work in progress at the aforementioned URL. It
consists in the Pack200 code with full history extracted from OpenJDK
before the removal, the project has been mavenized and the jtreg tests
were converted to JUnit.

The API is the same but the package was changed from java.util.jar to
io.pack200.

It's still missing the native library to speed up unpacking.

Emmanuel Bourg

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]