[compress] uncompress a file containing lz4 stripes

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[compress] uncompress a file containing lz4 stripes

Simo Chiegang, Boris Arthur
Hello,

So, we develop a special kind of image file with the following structure:

-          Header file [offset 0 to 4096 bytes]

-          Stripe 1 [4096 - 20000 bytes] (LZ4 block compressed)

-          Empty bloc [20000 - 22000 bytes]

-          Stripe 2 [22000 - 35000] (LZ4 block compressed)

-          ...

Something like that. So now I want to read this file with the class BlockLZ4CompressorInputStream. It is possible?

I tried to write an input stream that read this file and only stream the stripes and I give this input stream to the BlockLZ4CompressorInputStream.
It this logic correct?, or the BlockLZ4CompressorInputStream call only take a whole LZ4 compressed file without another data?.

Thanks


________________________________

Confidentiality note:
The information in this email and any attachment may contain confidential and proprietary information of Heidelberger Druckmaschinen AG and/or its affiliates and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any review, reliance or distribution by others or forwarding without express permission is strictly prohibited and may cause liability. In case you have received this message due to an error in transmission, we kindly ask you to notify the sender immediately and to delete this email and any attachment from your system.
Reply | Threaded
Open this post in threaded view
|

Re: [compress] uncompress a file containing lz4 stripes

Stefan Bodewig
On 2017-09-15, Simo Chiegang, Boris Arthur wrote:

> So, we develop a special kind of image file with the following structure:

> -          Header file [offset 0 to 4096 bytes]

> -          Stripe 1 [4096 - 20000 bytes] (LZ4 block compressed)

> -          Empty bloc [20000 - 22000 bytes]

> -          Stripe 2 [22000 - 35000] (LZ4 block compressed)

> -          ...

> Something like that. So now I want to read this file with the class
> BlockLZ4CompressorInputStream. It is possible?

In general yes, but probbaly not with a single instance of it (and you
may need to adapt something).

BlockLZ4CompressorInputStream should be able to read a single block, but
it wont be able to detect the end of the LZ4 block unless the underlying
stream is exhausted. There is no way to identify the last sequence other
than via EOF.

If your header contains information about the compressed size of your
block(s) then you can wrap the current input stream into a
BoundedInputStream that returns EOF once the block has been read
completely. You then create a BlockLZ4CompressorInputStream wrapped
around the BoundedInputStream - and do this for each stripe.

This is pretty much is what FramedLZ4CompressorInputStream does as well.

Things get a bit more complex if match offsets of stripe 2 are allowed
to point into the contents of stripe 1. FramedLZ4CompressorInputStream
handles this in the case where "block dependency" is expected.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: [compress] uncompress a file containing lz4 stripes

Simo Chiegang, Boris Arthur
Thanks, it works!

-----Original Message-----
From: Stefan Bodewig [mailto:[hidden email]]
Sent: Saturday, September 16, 2017 6:39 PM
To: [hidden email]
Subject: Re: [compress] uncompress a file containing lz4 stripes

On 2017-09-15, Simo Chiegang, Boris Arthur wrote:

> So, we develop a special kind of image file with the following structure:

> -          Header file [offset 0 to 4096 bytes]

> -          Stripe 1 [4096 - 20000 bytes] (LZ4 block compressed)

> -          Empty bloc [20000 - 22000 bytes]

> -          Stripe 2 [22000 - 35000] (LZ4 block compressed)

> -          ...

> Something like that. So now I want to read this file with the class
> BlockLZ4CompressorInputStream. It is possible?

In general yes, but probbaly not with a single instance of it (and you may need to adapt something).

BlockLZ4CompressorInputStream should be able to read a single block, but it wont be able to detect the end of the LZ4 block unless the underlying stream is exhausted. There is no way to identify the last sequence other than via EOF.

If your header contains information about the compressed size of your
block(s) then you can wrap the current input stream into a BoundedInputStream that returns EOF once the block has been read completely. You then create a BlockLZ4CompressorInputStream wrapped around the BoundedInputStream - and do this for each stripe.

This is pretty much is what FramedLZ4CompressorInputStream does as well.

Things get a bit more complex if match offsets of stripe 2 are allowed to point into the contents of stripe 1. FramedLZ4CompressorInputStream handles this in the case where "block dependency" is expected.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


________________________________

Confidentiality note:
The information in this email and any attachment may contain confidential and proprietary information of Heidelberger Druckmaschinen AG and/or its affiliates and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any review, reliance or distribution by others or forwarding without express permission is strictly prohibited and may cause liability. In case you have received this message due to an error in transmission, we kindly ask you to notify the sender immediately and to delete this email and any attachment from your system.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]