[compress] LZ4 compress time to slow

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[compress] LZ4 compress time to slow

Simo Chiegang, Boris Arthur
Hello,

I tried simple to compress a byte array using the LZ4 compression:

int numberReaded = tifFile.readEncodedStrip( 49, pointer, -1 );
byte[] byteResult = pointer.getByteArray( 0, numberReaded);   // The array has a length of 4194048 so 4Mb
ByteArrayOutputStream outStr = new ByteArrayOutputStream();
BlockLZ4CompressorOutputStream outputStream = new BlockLZ4CompressorOutputStream( outStr );
outputStream.write( byteResult );

So , the execution time of the write method is more than 1 minutes. It's not acceptable, may be it is a bug or I made somethings wrong?

P.S: I tried another library and  I get the time under 10ms!

Best Regards
Boris Simo


________________________________

Confidentiality note:
The information in this email and any attachment may contain confidential and proprietary information of Heidelberger Druckmaschinen AG and/or its affiliates and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you are hereby notified that any review, reliance or distribution by others or forwarding without express permission is strictly prohibited and may cause liability. In case you have received this message due to an error in transmission, we kindly ask you to notify the sender immediately and to delete this email and any attachment from your system.
Reply | Threaded
Open this post in threaded view
|

Re: [compress] LZ4 compress time to slow

Stefan Bodewig
On 2017-10-05, Simo Chiegang, Boris Arthur wrote:

> I tried simple to compress a byte array using the LZ4 compression:

> int numberReaded = tifFile.readEncodedStrip( 49, pointer, -1 );
> byte[] byteResult = pointer.getByteArray( 0, numberReaded);   // The array has a length of 4194048 so 4Mb
> ByteArrayOutputStream outStr = new ByteArrayOutputStream();
> BlockLZ4CompressorOutputStream outputStream = new BlockLZ4CompressorOutputStream( outStr );
> outputStream.write( byteResult );

> So , the execution time of the write method is more than 1
> minutes. It's not acceptable, may be it is a bug or I made somethings
> wrong?

By default the LZ4 implementation tries very hard to create the optimal
compression result. You can tweak it by using the two-arg constructor
and tweaking the parameters. Something like

BlockLZ4CompressorOutputStream.createParameterBuilder().tunedForSpeed().build()

> P.S: I tried another library and  I get the time under 10ms!

Unfortunately I doubt you'll get there with tunedForSpeed for biggish
arrays. You could try tweaking parameters further. You will probably
gain most by reducing maxOffset and maxBackReferenceLength.

While developing the LZ4 code I created a few benchmarks

https://github.com/bodewig/commons-compress-benchmarks/wiki/Comparison-of-Compression-Formats

as you can see LZ4 is quite a but slower than Snappy and the main
difference (they use the same compression code) is that Snappy is
restricts the back reference length to 64 bytes while it is almost
unlimited for LZ4. So when Snappy finds a match of 64 bytes it is done
while LZ4 keeps searching for longer matches.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]