[jira] [Commented] (IO-650) Improve IOUtils performance by increasing DEFAULT_BUFFER_SIZE

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (IO-650) Improve IOUtils performance by increasing DEFAULT_BUFFER_SIZE

Gary D. Gregory (Jira)

    [ https://issues.apache.org/jira/browse/IO-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015081#comment-17015081 ]

Michael Osipov commented on IO-650:
-----------------------------------

I can confirm the first test on the following machine:
{noformat}
$ sysctl -a | grep model
hw.model: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
$ uname -a
FreeBSD deblndw011x.ad001.siemens.net 12.1-STABLE FreeBSD 12.1-STABLE #4 r356437: Tue Jan  7 14:17:23 CET 2020     [hidden email]:/usr/obj/usr/src/amd64.amd64/sys/DEBLNDW011X  amd64
{noformat}
 

with this output:
 
{noformat}
42726365        65077381
41092613        56539813
41340653        57771915
41260906        57100291
43628662        56817779
41679902        56969041
39059266        53804538
39170168        53087482
38555712        52766964
38928121        55344044
38839020        52901258
39127235        52106839
38814114        52770678
38421275        53190439
38819153        53332602
39053075        52326426
38799316        53072601
38743910        52785995
38653971        53132402
39314262        53287530
38647202        52809558
38805353        53193302
 {noformat}

> Improve IOUtils performance by increasing DEFAULT_BUFFER_SIZE
> -------------------------------------------------------------
>
>                 Key: IO-650
>                 URL: https://issues.apache.org/jira/browse/IO-650
>             Project: Commons IO
>          Issue Type: Improvement
>    Affects Versions: 1.0
>            Reporter: Brett Lounsbury
>            Priority: Major
>             Fix For: 2.6
>
>
> IOUtils has a 4096B default buffer size that is used by copy() methods (and will be used for contentEquals methods when IO-649 is pulled).  This number should be updated to 8192 for a few reasons:
>  # It has a big improvement in performance in my micro-benchmark.  I tested both copy() and contentEquals() with 4K and 8K buffers.  This was done on a Late 2019 Macbook Pro (2.8GHz i7) with a 128MB file loaded into the OS buffer cache.  See below for the test harness used.  Past 8K performance does improve but it begins to experience some diminishing returns and could lead to excessive memory allocation.
>  # It mirrors the default buffer size of java.io.Buffered* classes.  This makes buffer sizing consistent regardless of if it is being done internally in the method or externally via a Buffered*.  These classes are used internally in IOUtils as well so the buffer size is not unreasonable.
>  
> For copy():
> |*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
> |*AVG*|44.2853417|64.2600679|0.31084197|
> |*P50*|42.692406|62.371984|0.31551951|
> |*P90*|49.5538826|68.4303876|0.27584975|
> |*P99*|62.8831473|89.759114|0.29942326|
> |*P100*|102.563615|177.143364|0.42101351|
>  
> For contentEquals() with IO-649:
> |*Metric*|*8K Buffer Millisecond*|*4K Buffer Millisecond*|*Performance Improvement*|
> |*AVG*|81.5009567|128.497828|0.36574059|
> |*P50*|78.517749|124.191476|0.36776861|
> |*P90*|89.9172708|136.779763|0.34261276|
> |*P99*|125.814333|183.881989|0.31578762|
> |*P100*|308.936585|559.611217|0.44794426|
> {color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
>       NullOutputStream nos = NullOutputStream.{color:#660e7a}NULL_OUTPUT_STREAM{color};
>       {color:#000080}for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
>           InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
>           {color:#000080}long {color}start = System.nanoTime();
>           IOUtils.copy(fis, nos);
>           {color:#000080}long {color}defaultCopyTime = System.nanoTime() - start;
>          fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
>          start = System.nanoTime();
>           IOUtils.copy(fis, nos, {color:#0000ff}8192{color});
>           {color:#000080}long {color}bufferSizeSpecifiedCopyTime = System.nanoTime() - start;
>          System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedCopyTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultCopyTime);
>       }
>  }
> {color:#000080}public static void {color}main(String[] args) {color:#000080}throws {color}Exception {
>  {color:#000080}    for {color}({color:#000080}int {color}i = {color:#0000ff}0{color}; i < {color:#0000ff}1000{color}; i++) {
>          InputStream fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
>          InputStream fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
>         {color:#000080}long {color}start = System.nanoTime();
>          IOUtils.contentEquals(fis, fis2);
>          {color:#000080}long {color}defaultContentEqualsTime = System.nanoTime() - start;
>         fis = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
>          fis2 = {color:#000080}new {color}FileInputStream({color:#000080}new {color}File({color:#008000}"/tmp/random_data"{color}));
>          start = System.nanoTime();
>           IOUtils.contentEquals(fis, fis2, {color:#0000ff}8192{color});
>           {color:#000080}long {color}bufferSizeSpecifiedContentEqualsTime = System.nanoTime() - start;
>          System.{color:#660e7a}out{color}.println(bufferSizeSpecifiedContentEqualsTime + {color:#008000}"{color}{color:#000080}\t{color}{color:#008000}" {color}+ defaultContentEqualsTime);
>     }
> }



--
This message was sent by Atlassian Jira
(v8.3.4#803005)