[compress] How to implement zip-bomb protection with Java 10

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[compress] How to implement zip-bomb protection with Java 10

Dominik Stadler-2

Hi,

Apache POI is opening zip-files on a regular basis because Microsoft Excel/Word/... files are zip-files in their newer format. In order to prevent some types of denial-of-service-attacks, we have added functionality when opening Zip-files to not read files which expand a lot and thus could be used to overwhelm the main memory by providing small malicious file which explodes when uncompressed into memory. We call this zip-bomb protection.

Up to Java 9 we could use some workaround via reflection to inject a counting-InputStream into ZipFile/ZipEntry to detect an explosion in expanded data and this way prevent zip-bombs.

However in Java 10 this is not possible any more because the implementation of ZipFile was changed in a way that prevents this (hard cast to ZipFile$ZipFileInputStream in ZipFile).

So we are looking for a different way to count the number of extracted bytes while extracting to be able to stop as soon as the compression ratio reaches a certain limit.

Could we do this with Commons Compress? I.e. wrap the InputStreams with some counting so that we can stop during extracting.

Or does anybody know of a way to do zip-bomb-detection differently without resorting to reflection?

Thanks... Dominik.
Apache POI PMC/Comitter



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] How to implement zip-bomb protection with Java 10

Stefan Bodewig
On 2018-03-31, Dominik Stadler wrote:

> Apache POI is opening zip-files on a regular basis because Microsoft
> Excel/Word/... files are zip-files in their newer format. In order to
> prevent some types of denial-of-service-attacks, we have added
> functionality when opening Zip-files to not read files which expand a
> lot and thus could be used to overwhelm the main memory by providing
> small malicious file which explodes when uncompressed into memory. We
> call this zip-bomb protection.

> Up to Java 9 we could use some workaround via reflection to inject a
> counting-InputStream into ZipFile/ZipEntry to detect an explosion in
> expanded data and this way prevent zip-bombs.

This is using java.util.zip.ZipFile, not Commons Compress, right? I just
want to double check.

> So we are looking for a different way to count the number of extracted
> bytes while extracting to be able to stop as soon as the compression
> ratio reaches a certain limit.

At least in Commons Compress, if you use ZipFile - as opposed to
ZipArchiveInputStream then the compressed and uncompressed sizes of each
ZipArchiveEntry are known before you try to read th stream. Can't you
simply reject reading entries who's uncompressed size is too big?

If you want to handle it on the stream level, there is a
BoundedInputStream class in Commons Compress that you could wrap around
the stream returned by ZipFile#getInputStream in order to enforce an
upper limit of bytes being read. This stream class expects to be used in
a certain context, though, and won't close the wrapped stream when close
is called - if you want to go down this route, you'll have to close the
underlying stream yourself.

Hope this helps

     Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] How to implement zip-bomb protection with Java 10

kiwiwings
Stefan Bodewig wrote
> This is using java.util.zip.ZipFile, not Commons Compress, right? I just
> want to double check.

Correct


Stefan Bodewig wrote
> ... then the compressed and uncompressed sizes of each
> ZipArchiveEntry are known before you try to read the stream. Can't you
> simply reject reading entries who's uncompressed size is too big?

Are those sizes taken from the central directory?
Isn't it possible to tamper around those entries?
if so, would you rely on those?


Stefan Bodewig wrote
> If you want to handle it on the stream level, there is a
> BoundedInputStream class in Commons Compress that you could wrap around
> the stream returned by ZipFile#getInputStream in order to enforce an
> upper limit of bytes being read.

To prevent false positives with people who put an insane amount of data into
Excel sheets, I've chosen the ratio way, i.e. comparing how many bytes of
the compressed stream are read vs. the uncompressed size.
There's another limit check which does exactly what you've explained, but
this defaults to around 2GB.
Although POI has anyway a problem with the normal processing mode to deal
with such big files, there's also a SAX streaming mode which can take
benefit of this.

I haven't dived into the compress code yet, but my hope is, that the ratio
logic can be also applied there.

For a reference I've looked through the archives to find related discussions
[1], but maybe missed the private ones. So the discussion were similar to
this thread, i.e. about taking the metadata/sizes from the central
directory.

So my approach would be to try to move to commons-compress, check if the
ratio logic can be added and if this is possible, we might also need to look
at the performance trade-off (if any?) and decide if we want a switchable
implementation ...

Andi

[1]
https://issues.apache.org/jira/browse/COMPRESS-445
https://issues.apache.org/jira/browse/COMPRESS-386
http://apache-commons.680414.n4.nabble.com/compress-Security-considerations-bomb-links-absolute-paths-tp4698822.html



--
Sent from: http://apache-commons.680414.n4.nabble.com/Commons-User-f735979.html

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] How to implement zip-bomb protection with Java 10

Stefan Bodewig
On 2018-03-31, kiwiwings wrote:

> Stefan Bodewig wrote
>> ... then the compressed and uncompressed sizes of each
>> ZipArchiveEntry are known before you try to read the stream. Can't you
>> simply reject reading entries who's uncompressed size is too big?

> Are those sizes taken from the central directory?
> Isn't it possible to tamper around those entries?
> if so, would you rely on those?

You are certainly correct. In that case wrapping the InputStream into a
BoundedInputStream capped at the claimed uncompressed size would
work. If the uncompressed size hasn't been tampered, then all is well,
otherwise the stream will simply stop reading any further.

> To prevent false positives with people who put an insane amount of
> data into Excel sheets, I've chosen the ratio way, i.e. comparing how
> many bytes of the compressed stream are read vs. the uncompressed
> size.  There's another limit check which does exactly what you've
> explained, but this defaults to around 2GB.  Although POI has anyway a
> problem with the normal processing mode to deal with such big files,
> there's also a SAX streaming mode which can take benefit of this.

> I haven't dived into the compress code yet, but my hope is, that the ratio
> logic can be also applied there.

The CompressorInputStreams provide counts of the compressed bytes read,
but at least the InputStream provided by ZipFile#getInputStream does
not. So I'm afraid there currently is no easy way to tell how many
compressed bytes have been processed at any point in time.

> For a reference I've looked through the archives to find related discussions
> [1], but maybe missed the private ones.

I'm not aware of any private discussion.

In this thread as well as COMPRESS-445 we are only talking about ZIP,
but all the other compressing archive formats (and compression formats)
can certainly also be used for "bombs". Compress' test resources contain
tar.gz files of about 8MB size that expand to 8+GB - these are the tests
for really big tars.

What I'd really want to do is provide a solution that works for all our
formats and not just for ZIP.

> So the discussion were similar to this thread, i.e. about taking the
> metadata/sizes from the central directory.

True, and you are absolutely correct that this is short-sightened when
somebody who crafts a malicious input may just put the wrong values in
there.

> So my approach would be to try to move to commons-compress, check if the
> ratio logic can be added and if this is possible, we might also need to look
> at the performance trade-off (if any?) and decide if we want a switchable
> implementation ...

The first thing you'd need is to get the information of how much of the
uncompressed size has been read so far. Given an API of

InputStream getInputStream(ZipArchiveEntry e)

there isn't much room for providing that information apart from
documenting "InputStream will be a FooInputStream" and have
FooInputStream provide the information.

An alternative may be a progress listener approach which would
frequently provide information about the number of bytes processed. This
has been discussed before (on this list a few days ago and in
COMPRESS-207).

I'm absolutely open to providing the means to implement the
protection. The dev list might be the better place. Right now I'm not
convinced it should be enabled by default, but we can discuss that once
we get there.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [compress] How to implement zip-bomb protection with Java 10

kiwiwings
Just a short update on this - I've provided a patch for POI to use commons compress [1]
So now we can focus on how the zip bomb handling can be provided by commons compress,
i.e. as you already have mentioned with "InputStream will be a FooInputStream",
some kind of interface which the InputStream can be cast to,
to request further compression ratio stats, is what I have in mind.

I think this pull mechanism is easier from user perspective than registering
a progress handler or getting the meta data pushed by a callback.

Andi.

[1] https://bz.apache.org/bugzilla/show_bug.cgi?id=62187



signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [compress] How to implement zip-bomb protection with Java 10

Robert Paasche
In reply to this post by Dominik Stadler-2
Hi,

sorry it's a litte bit late, but have you tried to open an bug-report for
the hard cast on https://bugreport.java.com/ ?

Best
Rob

Robert Paasche
Senior Developer

pripares GmbH
Altheimer Eck 2
80331 München

Tel  +49 (0)89 45 22 808 - 30
Fax  +49 (0)89 45 22 808 - 58
Mail [hidden email]
Web  www.pripares.com

Handelsregister: Registergericht München HRB 138701
Sitz der Gesellschaft: München
Geschäftsführer: Aßmann Christoph, Ertl Andreas

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
löschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail und der darin enthaltenen Informationen sind nicht
gestattet.

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and delete this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

2018-03-31 10:26 GMT+02:00 Dominik Stadler <[hidden email]>:

>
> Hi,
>
> Apache POI is opening zip-files on a regular basis because Microsoft
> Excel/Word/... files are zip-files in their newer format. In order to
> prevent some types of denial-of-service-attacks, we have added
> functionality when opening Zip-files to not read files which expand a lot
> and thus could be used to overwhelm the main memory by providing small
> malicious file which explodes when uncompressed into memory. We call this
> zip-bomb protection.
>
> Up to Java 9 we could use some workaround via reflection to inject a
> counting-InputStream into ZipFile/ZipEntry to detect an explosion in
> expanded data and this way prevent zip-bombs.
>
> However in Java 10 this is not possible any more because the
> implementation of ZipFile was changed in a way that prevents this (hard
> cast to ZipFile$ZipFileInputStream in ZipFile).
>
> So we are looking for a different way to count the number of extracted
> bytes while extracting to be able to stop as soon as the compression ratio
> reaches a certain limit.
>
> Could we do this with Commons Compress? I.e. wrap the InputStreams with
> some counting so that we can stop during extracting.
>
> Or does anybody know of a way to do zip-bomb-detection differently without
> resorting to reflection?
>
> Thanks... Dominik.
> Apache POI PMC/Comitter
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [compress] How to implement zip-bomb protection with Java 10

Stefan Bodewig
In reply to this post by kiwiwings
On 2018-04-09, Andreas Beeker wrote:

> Just a short update on this - I've provided a patch for POI to use
> commons compress [1] So now we can focus on how the zip bomb handling
> can be provided by commons compress, i.e. as you already have
> mentioned with "InputStream will be a FooInputStream", some kind of
> interface which the InputStream can be cast to, to request further
> compression ratio stats, is what I have in mind.

> I think this pull mechanism is easier from user perspective than
> registering a progress handler or getting the meta data pushed by a
> callback.

Sounds good to me.

If you want to discuss implementation details, we probably better move
over to the dev list.

Thanks

        Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]