[compress] High Level API for Archives

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[compress] High Level API for Archives

Stefan Bodewig
Hi all

I've started to work on COMPRESS-118 and added Archiver and Expander
classes - without any tests for now. As I'm trying to cover a bunch of
possible use cases there are lots of inputs or outputs that can
represent archives. When expanding archives you may want to use
auto-detect the format or specify it explicitly. You may want to filter
the files to add/entries to expand. All this leads to an explosion of
overloads that I'm not comfortable with.

One idea that I came up with is using a fluent interface like

Expander.forFile(someFile).filtering(someFilter).expandTo(someDirectory);

or similar. But before I delve deeper into it, I'd like to collect
feedback.

And then I'd like to collect additional features that might be
needed. What I can easily imaging is

* When expanding, don't overwrite existing files when expanding an
  archive (or only if the archive entries are newer.

* Add an API for updating archives - of course this boils down to
  writing a completely new archive with entries taken from two sources.

* we may want to provide a utility class that makes dealing with the
  subclasses of ArchiveEntry easier. Many of them provide a userid, but
  there is no common interface. I'm thinking about something like
  https://github.com/apache/ant-antlibs-compress/blob/master/src/main/org/apache/ant/compress/util/EntryHelper.java

Any feedback is very much appreciated.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Torsten Curdt-3
TBH I am not such super big fan of adding and maintaining a high level
API at this stage.
You will never find the right abstraction that everyone is happy with.
If you would - well, then that should be the real API afterall.

Honestly - I would just add example code for now. Maybe that can turn
into a helper class in the long run.
But for now we would not introduce something we may no longer break.

Maybe we need to look at the various users of the API and take that as
the basis for the next step.

My 2 cents.
Torsten

On Mon, Apr 23, 2018 at 8:57 PM, Stefan Bodewig <[hidden email]> wrote:

> Hi all
>
> I've started to work on COMPRESS-118 and added Archiver and Expander
> classes - without any tests for now. As I'm trying to cover a bunch of
> possible use cases there are lots of inputs or outputs that can
> represent archives. When expanding archives you may want to use
> auto-detect the format or specify it explicitly. You may want to filter
> the files to add/entries to expand. All this leads to an explosion of
> overloads that I'm not comfortable with.
>
> One idea that I came up with is using a fluent interface like
>
> Expander.forFile(someFile).filtering(someFilter).expandTo(someDirectory);
>
> or similar. But before I delve deeper into it, I'd like to collect
> feedback.
>
> And then I'd like to collect additional features that might be
> needed. What I can easily imaging is
>
> * When expanding, don't overwrite existing files when expanding an
>   archive (or only if the archive entries are newer.
>
> * Add an API for updating archives - of course this boils down to
>   writing a completely new archive with entries taken from two sources.
>
> * we may want to provide a utility class that makes dealing with the
>   subclasses of ArchiveEntry easier. Many of them provide a userid, but
>   there is no common interface. I'm thinking about something like
>   https://github.com/apache/ant-antlibs-compress/blob/master/src/main/org/apache/ant/compress/util/EntryHelper.java
>
> Any feedback is very much appreciated.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Matt Sicker
I went through a few iterations of my own over the weekend and wasn't
satisfied with the direction in any of them. This approach looks
interesting, though I need to take a closer look.

On 23 April 2018 at 14:48, Torsten Curdt <[hidden email]> wrote:

> TBH I am not such super big fan of adding and maintaining a high level
> API at this stage.
> You will never find the right abstraction that everyone is happy with.
> If you would - well, then that should be the real API afterall.
>
> Honestly - I would just add example code for now. Maybe that can turn
> into a helper class in the long run.
> But for now we would not introduce something we may no longer break.
>
> Maybe we need to look at the various users of the API and take that as
> the basis for the next step.
>
> My 2 cents.
> Torsten
>
> On Mon, Apr 23, 2018 at 8:57 PM, Stefan Bodewig <[hidden email]>
> wrote:
> > Hi all
> >
> > I've started to work on COMPRESS-118 and added Archiver and Expander
> > classes - without any tests for now. As I'm trying to cover a bunch of
> > possible use cases there are lots of inputs or outputs that can
> > represent archives. When expanding archives you may want to use
> > auto-detect the format or specify it explicitly. You may want to filter
> > the files to add/entries to expand. All this leads to an explosion of
> > overloads that I'm not comfortable with.
> >
> > One idea that I came up with is using a fluent interface like
> >
> > Expander.forFile(someFile).filtering(someFilter).
> expandTo(someDirectory);
> >
> > or similar. But before I delve deeper into it, I'd like to collect
> > feedback.
> >
> > And then I'd like to collect additional features that might be
> > needed. What I can easily imaging is
> >
> > * When expanding, don't overwrite existing files when expanding an
> >   archive (or only if the archive entries are newer.
> >
> > * Add an API for updating archives - of course this boils down to
> >   writing a completely new archive with entries taken from two sources.
> >
> > * we may want to provide a utility class that makes dealing with the
> >   subclasses of ArchiveEntry easier. Many of them provide a userid, but
> >   there is no common interface. I'm thinking about something like
> >   https://github.com/apache/ant-antlibs-compress/blob/master/
> src/main/org/apache/ant/compress/util/EntryHelper.java
> >
> > Any feedback is very much appreciated.
> >
> > Stefan
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Matt Sicker <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
In reply to this post by Torsten Curdt-3
On 2018-04-23, Torsten Curdt wrote:

> TBH I am not such super big fan of adding and maintaining a high level
> API at this stage.  You will never find the right abstraction that
> everyone is happy with.

TIt's not my intention to make everybody happy, there will still be use
cases that won't be covered by the high level API where you stil need to
go to the lower level. The parallel creation of zip archives is
something that is far out of scope for the high level IMHO.

> Maybe we need to look at the various users of the API and take that as
> the basis for the next step.

Yes, this is soemthing I intend to do.

Thanks

        Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

sebb-2-2
In reply to this post by Torsten Curdt-3
On 23 April 2018 at 20:48, Torsten Curdt <[hidden email]> wrote:
> TBH I am not such super big fan of adding and maintaining a high level
> API at this stage.
> You will never find the right abstraction that everyone is happy with.
> If you would - well, then that should be the real API afterall.
>
> Honestly - I would just add example code for now. Maybe that can turn
> into a helper class in the long run.
> But for now we would not introduce something we may no longer break.

I like the idea of introducing it as example code.
That makes it obvious that it will change.

If not, maybe put the code in an experimental package.

> Maybe we need to look at the various users of the API and take that as
> the basis for the next step.
>
> My 2 cents.
> Torsten
>
> On Mon, Apr 23, 2018 at 8:57 PM, Stefan Bodewig <[hidden email]> wrote:
>> Hi all
>>
>> I've started to work on COMPRESS-118 and added Archiver and Expander
>> classes - without any tests for now. As I'm trying to cover a bunch of
>> possible use cases there are lots of inputs or outputs that can
>> represent archives. When expanding archives you may want to use
>> auto-detect the format or specify it explicitly. You may want to filter
>> the files to add/entries to expand. All this leads to an explosion of
>> overloads that I'm not comfortable with.
>>
>> One idea that I came up with is using a fluent interface like
>>
>> Expander.forFile(someFile).filtering(someFilter).expandTo(someDirectory);
>>
>> or similar. But before I delve deeper into it, I'd like to collect
>> feedback.
>>
>> And then I'd like to collect additional features that might be
>> needed. What I can easily imaging is
>>
>> * When expanding, don't overwrite existing files when expanding an
>>   archive (or only if the archive entries are newer.
>>
>> * Add an API for updating archives - of course this boils down to
>>   writing a completely new archive with entries taken from two sources.
>>
>> * we may want to provide a utility class that makes dealing with the
>>   subclasses of ArchiveEntry easier. Many of them provide a userid, but
>>   there is no common interface. I'm thinking about something like
>>   https://github.com/apache/ant-antlibs-compress/blob/master/src/main/org/apache/ant/compress/util/EntryHelper.java
>>
>> Any feedback is very much appreciated.
>>
>> Stefan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
On 2018-04-24, sebb wrote:

> On 23 April 2018 at 20:48, Torsten Curdt <[hidden email]> wrote:
>> TBH I am not such super big fan of adding and maintaining a high level
>> API at this stage.
>> You will never find the right abstraction that everyone is happy with.
>> If you would - well, then that should be the real API afterall.

>> Honestly - I would just add example code for now. Maybe that can turn
>> into a helper class in the long run.
>> But for now we would not introduce something we may no longer break.

> I like the idea of introducing it as example code.
> That makes it obvious that it will change.

As the only people who spoke up prefer to not provide the API as an
officially supported one, I'm fine with moving the stuff to an examples
package, add a warning and add unit tests so we now it keeps on working.

Unless anybody yells and says "no we really need this high level API",
that is.

I'd still like to spend some time and gather feedback on a nicer API
than many overloads - is fluent the way to go?

> If not, maybe put the code in an experimental package.

The changeset package is in this state and has been like this for years
now, I doubt we'd ever get anything out of the experimental phase :-)

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Matt Sicker
If it helps in design, the most promising approach I found was to integrate
with the java.nio.file API. However, that turns into a sort of hybrid
library between VFS and Compress. However, they are rather strongly related
(if the two libraries were modular, then it wouldn't be such a problem to
combine them).

On 26 April 2018 at 10:04, Stefan Bodewig <[hidden email]> wrote:

> On 2018-04-24, sebb wrote:
>
> > On 23 April 2018 at 20:48, Torsten Curdt <[hidden email]> wrote:
> >> TBH I am not such super big fan of adding and maintaining a high level
> >> API at this stage.
> >> You will never find the right abstraction that everyone is happy with.
> >> If you would - well, then that should be the real API afterall.
>
> >> Honestly - I would just add example code for now. Maybe that can turn
> >> into a helper class in the long run.
> >> But for now we would not introduce something we may no longer break.
>
> > I like the idea of introducing it as example code.
> > That makes it obvious that it will change.
>
> As the only people who spoke up prefer to not provide the API as an
> officially supported one, I'm fine with moving the stuff to an examples
> package, add a warning and add unit tests so we now it keeps on working.
>
> Unless anybody yells and says "no we really need this high level API",
> that is.
>
> I'd still like to spend some time and gather feedback on a nicer API
> than many overloads - is fluent the way to go?
>
> > If not, maybe put the code in an experimental package.
>
> The changeset package is in this state and has been like this for years
> now, I doubt we'd ever get anything out of the experimental phase :-)
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Matt Sicker <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Oliver Heger-3


Am 26.04.2018 um 17:37 schrieb Matt Sicker:

> If it helps in design, the most promising approach I found was to integrate
> with the java.nio.file API. However, that turns into a sort of hybrid
> library between VFS and Compress. However, they are rather strongly related
> (if the two libraries were modular, then it wouldn't be such a problem to
> combine them).
>
> On 26 April 2018 at 10:04, Stefan Bodewig <[hidden email]> wrote:
>
>> On 2018-04-24, sebb wrote:
>>
>>> On 23 April 2018 at 20:48, Torsten Curdt <[hidden email]> wrote:
>>>> TBH I am not such super big fan of adding and maintaining a high level
>>>> API at this stage.
>>>> You will never find the right abstraction that everyone is happy with.
>>>> If you would - well, then that should be the real API afterall.
>>
>>>> Honestly - I would just add example code for now. Maybe that can turn
>>>> into a helper class in the long run.
>>>> But for now we would not introduce something we may no longer break.
>>
>>> I like the idea of introducing it as example code.
>>> That makes it obvious that it will change.
>>
>> As the only people who spoke up prefer to not provide the API as an
>> officially supported one, I'm fine with moving the stuff to an examples
>> package, add a warning and add unit tests so we now it keeps on working.
>>
>> Unless anybody yells and says "no we really need this high level API",
>> that is.
>>
>> I'd still like to spend some time and gather feedback on a nicer API
>> than many overloads - is fluent the way to go?

Recently I had a closer look at streaming libraries like Akka Streams.
So I had the idea to model the problem in a similar way:

An archive or deflate operation could be modeled by a flow from a source
via some filtering or modifying stages to a sink. The source and the
sink could both either refer to a directory or an archive file. In order
to create a new archive, the source would point to a directory and the
sink would represent the archive file to be created. To extract an
archive file, it would be the other way around. When both the source and
the sink point to an archive file, you have an update operation. So the
basic concepts can be combined in a natural way.

There could be stages that filter for files or archive entries to select
the content of an archive file to be created or the files to be
extracted. Maybe it makes also sense to map on entries to manipulate
them somehow (rename?).

The API could be extensible by allowing clients to create their own
implementations of sources, sinks, and flow stages.

Just some high-level thoughts without having thought that through.

Oliver

>>
>>> If not, maybe put the code in an experimental package.
>>
>> The changeset package is in this state and has been like this for years
>> now, I doubt we'd ever get anything out of the experimental phase :-)
>>
>> Stefan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
In reply to this post by Matt Sicker
On 2018-04-26, Matt Sicker wrote:

> If it helps in design, the most promising approach I found was to integrate
> with the java.nio.file API. However, that turns into a sort of hybrid
> library between VFS and Compress.

The current compress-2.0 branch - which is something that I haven't fully
given up on but is dormant right now - at least uses the file views for
permissions and stuff.

I must admit I was underwhelmed by the java.nio.file implementation when
I tried it for real in Ant's setpermissions task (even though Windows
file system can provide POSIX attributes for everything it won't provide
a PosixFileAttributes view and you are restricted to set a DOS read-only
flag). But losing focus here. :-)

> However, they are rather strongly related (if the two libraries were
> modular, then it wouldn't be such a problem to combine them).

I'm not sure what you mean by modular in this context.

Splitting Compress into an API-only library and separate libs for the
different formats? This might be doable in a 2.0 version because
everything else would not be backwards compatible. But we'd end up with
a double digit number of jars which I'm not sure will be easy to
maintain. You likely would want to release them independently (why would
I release the commons-compress-pack200 component if there only is a
change in commons-compress-tar?) and end up with a strange mix of
versions inside of a single reactor build. I'm a bit scared by the idea,
I must admit :-)

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
In reply to this post by Oliver Heger-3
On 2018-04-26, Oliver Heger wrote:

> Recently I had a closer look at streaming libraries like Akka Streams.
> So I had the idea to model the problem in a similar way:

> An archive or deflate operation could be modeled by a flow from a source
> via some filtering or modifying stages to a sink. The source and the
> sink could both either refer to a directory or an archive file. In order
> to create a new archive, the source would point to a directory and the
> sink would represent the archive file to be created. To extract an
> archive file, it would be the other way around. When both the source and
> the sink point to an archive file, you have an update operation. So the
> basic concepts can be combined in a natural way.

> There could be stages that filter for files or archive entries to select
> the content of an archive file to be created or the files to be
> extracted. Maybe it makes also sense to map on entries to manipulate
> them somehow (rename?).

> The API could be extensible by allowing clients to create their own
> implementations of sources, sinks, and flow stages.

> Just some high-level thoughts without having thought that through.

This resonates with some ideas I've had, but I'm afraid this is a bit
too much if the target goal is an examples package :-)

In a way the current "lots of overloaded methods" implementation already
does something similar internally - while keeping files and archive
entries strictly separate in two different classes. Trying to find a
common abstraction for Files read from/written to a directory and
ArchiveEntries really adds another layer of complexity.

I'll see how far I can get by turning the internal work of my current
implementation into APIs. From your high level description sources/sinks
for all the different ways archives can be represented and the filter
operations are already there. The current implementation lacks
projection/transformation, I'll think about that separately.

Finally the sink could be something that just discards the contents and
just lists the files/entries and we could re-implement the Lister class
that way.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
Hi all

I've introduced a whole lot of infrastructure code with

https://github.com/apache/commons-compress/commit/f62c523154dfedcf49a87a865db545bb8c55e795

but the interface now looks nicer to me, in particual see

        try (Sink<File> sink = FileToArchiveSink.forFile(args[1], new File(args[2]))) {
            Archive.directory(new File(args[0]))
                .to(sink);
        }

in ArchiveCli - all of this in a new examples package. No unit tests,
but I've tried the new cli with all formats we support writing for. Only
archive creation is there, yet, but the opposite direction will reuse a
lot of the infrastructure.

If this is something that looks acceptable I'll add expansion and remove
the other two classes. If not, we'll need to think about an acceptable
example API a bit longer. :-)

Cheers

        Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Matt Sicker
In reply to this post by Stefan Bodewig
On 29 April 2018 at 06:24, Stefan Bodewig <[hidden email]> wrote:

> The current compress-2.0 branch - which is something that I haven't fully
> given up on but is dormant right now - at least uses the file views for
> permissions and stuff.
>

Neat! I've experimented a bit locally in the past to see how big an effort
it would be to migrate vfs to this API, and I got frustrated by having to
implement Path in a completely different way than the existing path
abstraction in vfs (which only has absolute URLs as far as I could tell,
but maybe I'm wrong).


> I must admit I was underwhelmed by the java.nio.file implementation when
> I tried it for real in Ant's setpermissions task (even though Windows
> file system can provide POSIX attributes for everything it won't provide
> a PosixFileAttributes view and you are restricted to set a DOS read-only
> flag). But losing focus here. :-)
>

Yes, maybe that's why I've barely seen anyone implement this API. Perhaps
Commons needs to be that starting point for everyone else? :)


> I'm not sure what you mean by modular in this context.
>

As in one module/jar per implementation. One module for a zipfs, one for an
sshfs, etc., instead of just "commons-vfs" as a monolith with optional
dependencies.

Splitting Compress into an API-only library and separate libs for the
> different formats? This might be doable in a 2.0 version because
> everything else would not be backwards compatible. But we'd end up with
> a double digit number of jars which I'm not sure will be easy to
> maintain. You likely would want to release them independently (why would
> I release the commons-compress-pack200 component if there only is a
> change in commons-compress-tar?) and end up with a strange mix of
> versions inside of a single reactor build. I'm a bit scared by the idea,
> I must admit :-)
>

Depending if the modules are separated by git repo (like how maven handles
its individual plugins they maintain) or all in a single repo, that would
certainly determine how often each FS gets released. Based on the release
process, I'd suggest the all-in-one repo unless each FS has a couple people
to vote on releases individually.


--
Matt Sicker <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Matt Sicker
In reply to this post by Oliver Heger-3
On 26 April 2018 at 14:38, Oliver Heger <[hidden email]>
wrote:

> Recently I had a closer look at streaming libraries like Akka Streams.
> So I had the idea to model the problem in a similar way:
>

I've used Akka Streams a bit in the past, and while it's pretty awesome, I
feel as though taking this approach would at least require collaboration
with an existing reactive streams library or a new Commons one. In that
case, I wonder if it's worth competing with existing RS APIs like RxJava,
Reactor, Vert.x, and Akka Streams. I'm not even sure if any RS people are
active here at Commons as it is.


> An archive or deflate operation could be modeled by a flow from a source
> via some filtering or modifying stages to a sink. The source and the
> sink could both either refer to a directory or an archive file. In order
> to create a new archive, the source would point to a directory and the
> sink would represent the archive file to be created. To extract an
> archive file, it would be the other way around. When both the source and
> the sink point to an archive file, you have an update operation. So the
> basic concepts can be combined in a natural way.
>

This approach is interesting and could potentially be done without reactive
streams. It would essentially be a similar looking API, but it just would
be implemented differently.


> There could be stages that filter for files or archive entries to select
> the content of an archive file to be created or the files to be
> extracted. Maybe it makes also sense to map on entries to manipulate
> them somehow (rename?).
>

Now you're really starting to describe a high level framework. While this
sounds really neat, I feel like it might be beyond the scope of Commons and
would warrant its own project at that point.


--
Matt Sicker <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Oliver Heger-3
In reply to this post by Stefan Bodewig


Am 29.04.2018 um 19:08 schrieb Stefan Bodewig:

> Hi all
>
> I've introduced a whole lot of infrastructure code with
>
> https://github.com/apache/commons-compress/commit/f62c523154dfedcf49a87a865db545bb8c55e795
>
> but the interface now looks nicer to me, in particual see
>
>         try (Sink<File> sink = FileToArchiveSink.forFile(args[1], new File(args[2]))) {
>             Archive.directory(new File(args[0]))
>                 .to(sink);
>         }
>
> in ArchiveCli - all of this in a new examples package. No unit tests,
> but I've tried the new cli with all formats we support writing for. Only
> archive creation is there, yet, but the opposite direction will reuse a
> lot of the infrastructure.
>
> If this is something that looks acceptable I'll add expansion and remove
> the other two classes. If not, we'll need to think about an acceptable
> example API a bit longer. :-)

Big +1! Looks very good to me.

Oliver

>
> Cheers
>
>         Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Oliver Heger-3
In reply to this post by Matt Sicker
Hi Matt,

Am 29.04.2018 um 20:30 schrieb Matt Sicker:

> On 26 April 2018 at 14:38, Oliver Heger <[hidden email]>
> wrote:
>
>> Recently I had a closer look at streaming libraries like Akka Streams.
>> So I had the idea to model the problem in a similar way:
>>
>
> I've used Akka Streams a bit in the past, and while it's pretty awesome, I
> feel as though taking this approach would at least require collaboration
> with an existing reactive streams library or a new Commons one. In that
> case, I wonder if it's worth competing with existing RS APIs like RxJava,
> Reactor, Vert.x, and Akka Streams. I'm not even sure if any RS people are
> active here at Commons as it is.

This is similar to my own line of thoughts. Developing an own streaming
lib is way too ambitious and also not needed as there are already some
really powerful libraries around. So there are the following options:
- Provide an API that resembles the core concepts of such libraries
without going too deep or providing similar flexibility. That would be
my first take and is probably also the route Stefan is following.
- Integrate with one of the existing libraries, e.g. by implementing
special transformers, sources, and/or sinks that handle compression or
extraction. Not sure whether this fits into commons as we currently do
not implement extensions for specific other libraries. It would probably
already be a longish discussion to choose one or more libraries to support.

>
>
>> An archive or deflate operation could be modeled by a flow from a source
>> via some filtering or modifying stages to a sink. The source and the
>> sink could both either refer to a directory or an archive file. In order
>> to create a new archive, the source would point to a directory and the
>> sink would represent the archive file to be created. To extract an
>> archive file, it would be the other way around. When both the source and
>> the sink point to an archive file, you have an update operation. So the
>> basic concepts can be combined in a natural way.
>>
>
> This approach is interesting and could potentially be done without reactive
> streams. It would essentially be a similar looking API, but it just would
> be implemented differently.
Yes, my first option above.

>
>
>> There could be stages that filter for files or archive entries to select
>> the content of an archive file to be created or the files to be
>> extracted. Maybe it makes also sense to map on entries to manipulate
>> them somehow (rename?).
>>
>
> Now you're really starting to describe a high level framework. While this
> sounds really neat, I feel like it might be beyond the scope of Commons and
> would warrant its own project at that point.
>
>
Perhaps. Maybe a stream-like API to move or copy files could fit into
[io]? Then [compress] could integrate with this. But I am aware that
this is also an ambitious project and goes beyond what [compress]
currently needs.

Oliver

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
In reply to this post by Stefan Bodewig
On 2018-04-29, Stefan Bodewig wrote:

> If this is something that looks acceptable I'll add expansion and remove
> the other two classes.

Just went ahead and added expansion as well, which doesn't mean it has
to stay.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Matt Sicker
Not sure if it helps much, but based on my experience in writing a totally
unrelated library from scratch in Java 8, I feel as though the use of
default interface methods would help tremendously in providing a nice API
while maintaining BC. I had considered proposing an update to Java 8, but
that wouldn't really help the users who improperly use compress in Java 7
already.

On 29 April 2018 at 23:34, Stefan Bodewig <[hidden email]> wrote:

> On 2018-04-29, Stefan Bodewig wrote:
>
> > If this is something that looks acceptable I'll add expansion and remove
> > the other two classes.
>
> Just went ahead and added expansion as well, which doesn't mean it has
> to stay.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


--
Matt Sicker <[hidden email]>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
In reply to this post by Matt Sicker
On 2018-04-29, Matt Sicker wrote:

> On 29 April 2018 at 06:24, Stefan Bodewig <[hidden email]> wrote:

>> I'm not sure what you mean by modular in this context.

> As in one module/jar per implementation. One module for a zipfs, one for an
> sshfs, etc., instead of just "commons-vfs" as a monolith with optional
> dependencies.

Oh, you've been talking about VFS here.

If we tried the same with Compress we'd end up with one or two API jars
(compressors could be separate from archivers if we wanted to go to
extremes), an SPI jar with the utils package, eight archiver and
thirteen compressor implementations. Together with some
interdependencies as zip and sevenz would depend on several
compressors. While this might be good from a modularization POV it also
doesn't sound attractive from a maintentance POV to me :-)

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Stefan Bodewig
In reply to this post by Matt Sicker
On 2018-04-30, Matt Sicker wrote:

> Not sure if it helps much, but based on my experience in writing a totally
> unrelated library from scratch in Java 8, I feel as though the use of
> default interface methods would help tremendously in providing a nice API
> while maintaining BC.

I may have been able to avoid two or three abstract classes with default
methods, that's true. Static methods on interfaces would have helped
avoid some of the factories but all in all I feel Java7 hasn't been an
obstacle.

> I had considered proposing an update to Java 8,

That would be a separate thread, I guess. I'd not oppose such a move but
would probably prefer to get Compress 1.17 out of the door with Java7 as
baseline first (Tika is likely waiting for COMPRESS-445) .

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] High Level API for Archives

Torsten Curdt-3
In reply to this post by Stefan Bodewig
>
> > As in one module/jar per implementation. One module for a zipfs, one for
> an
> > sshfs, etc., instead of just "commons-vfs" as a monolith with optional
> > dependencies.
>

Just one question: Why?

The implementation jars for compress would be tiny and would have close to
none dependencies.
12