[jira] Created: (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

Gilles (Jira)
FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)
---------------------------------------------------------------------------------------------------

                 Key: IO-266
                 URL: https://issues.apache.org/jira/browse/IO-266
             Project: Commons IO
          Issue Type: Bug
            Reporter: Igor Smereka



java.io.IOException: Insufficient system resources exist to complete the requested service
        at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
        at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
        at sun.nio.ch.IOUtil.write(Unknown Source)
        at sun.nio.ch.FileChannelImpl.write(Unknown Source)
        at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
        at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
        at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
        at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
        at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)

NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).

Here is a test program to reproduce the issue:

package org.test.igor;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;

public class TestFileUtils {

    public static void main(String[] args){
       
       
        File src = new File("D:\\2011.1-dev\\test\\test");
        File dest = new File("\\\\ismerek1\\Shared");
       
        String filename = "jdk-6u19-windows-x64.exe";
       
        File file = new File(src, filename);
        File toFile = new File(dest, filename);
       
        try {
            FileUtils.copyFile(file, toFile, true);
            System.out.println("Successful copy");
        }
        catch (IOException e1) {
            e1.printStackTrace();
        }
    }
}








--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

Gilles (Jira)

    [ https://issues.apache.org/jira/browse/IO-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017556#comment-13017556 ]

Sebb commented on IO-266:
-------------------------

50MB does seem rather large as a default; I would have thought 16MB would be plenty?
Perhaps we should make the default smaller.

It would be possible to add new methods which allowed the buffer size to be provided; however this would mean adding about 7 new methods.

Unfortunately changing the buffersize to a variable would be thread-hostile: if two threads set different values, there is a timing window between setting the value and starting the copy method. This cannot be avoided.

It's a pity the class methods are all static ...

One other possibility would be to allow the default to be set by using a System Property; this could be thread-safe, but the value could not be changed once set.

> FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)
> ---------------------------------------------------------------------------------------------------
>
>                 Key: IO-266
>                 URL: https://issues.apache.org/jira/browse/IO-266
>             Project: Commons IO
>          Issue Type: Bug
>    Affects Versions: 2.0.1
>         Environment: Windows 2003 Server 64-bit
>            Reporter: Igor Smereka
>
> java.io.IOException: Insufficient system resources exist to complete the requested service
> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
> at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)
> NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
> Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).
> Here is a test program to reproduce the issue:
> package org.test.igor;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.nio.channels.FileChannel;
> import org.apache.commons.io.FileUtils;
> import org.apache.commons.io.IOUtils;
> public class TestFileUtils {
>     public static void main(String[] args){
>        
>        
>         File src = new File("D:\\2011.1-dev\\test\\test");
>         File dest = new File("\\\\ismerek1\\Shared");
>        
>         String filename = "jdk-6u19-windows-x64.exe";
>        
>         File file = new File(src, filename);
>         File toFile = new File(dest, filename);
>        
>         try {
>             FileUtils.copyFile(file, toFile, true);
>             System.out.println("Successful copy");
>         }
>         catch (IOException e1) {
>             e1.printStackTrace();
>         }
>     }
> }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

Gilles (Jira)
In reply to this post by Gilles (Jira)

    [ https://issues.apache.org/jira/browse/IO-266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018285#comment-13018285 ]

Niall Pemberton commented on IO-266:
------------------------------------

I think we should just reduce the size to fix the problem Igor found - so 30M is the nearest (rounded)

> FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)
> ---------------------------------------------------------------------------------------------------
>
>                 Key: IO-266
>                 URL: https://issues.apache.org/jira/browse/IO-266
>             Project: Commons IO
>          Issue Type: Bug
>    Affects Versions: 2.0.1
>         Environment: Windows 2003 Server 64-bit
>            Reporter: Igor Smereka
>
> java.io.IOException: Insufficient system resources exist to complete the requested service
> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
> at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)
> NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
> Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).
> Here is a test program to reproduce the issue:
> package org.test.igor;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.nio.channels.FileChannel;
> import org.apache.commons.io.FileUtils;
> import org.apache.commons.io.IOUtils;
> public class TestFileUtils {
>     public static void main(String[] args){
>        
>        
>         File src = new File("D:\\2011.1-dev\\test\\test");
>         File dest = new File("\\\\ismerek1\\Shared");
>        
>         String filename = "jdk-6u19-windows-x64.exe";
>        
>         File file = new File(src, filename);
>         File toFile = new File(dest, filename);
>        
>         try {
>             FileUtils.copyFile(file, toFile, true);
>             System.out.println("Successful copy");
>         }
>         catch (IOException e1) {
>             e1.printStackTrace();
>         }
>     }
> }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Reply | Threaded
Open this post in threaded view
|

[jira] [Updated] (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

Gilles (Jira)
In reply to this post by Gilles (Jira)

     [ https://issues.apache.org/jira/browse/IO-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arnaud BRAND updated IO-266:
----------------------------

    Attachment: FileUtils.java.patch

I attached a patch setting the chunk at 30MB since I'm also affected by this issue.

Thanks.

> FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)
> ---------------------------------------------------------------------------------------------------
>
>                 Key: IO-266
>                 URL: https://issues.apache.org/jira/browse/IO-266
>             Project: Commons IO
>          Issue Type: Bug
>    Affects Versions: 2.0.1
>         Environment: Windows 2003 Server 64-bit
>            Reporter: Igor Smereka
>         Attachments: FileUtils.java.patch
>
>
> java.io.IOException: Insufficient system resources exist to complete the requested service
> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
> at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)
> NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
> Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).
> Here is a test program to reproduce the issue:
> package org.test.igor;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.nio.channels.FileChannel;
> import org.apache.commons.io.FileUtils;
> import org.apache.commons.io.IOUtils;
> public class TestFileUtils {
>     public static void main(String[] args){
>        
>        
>         File src = new File("D:\\2011.1-dev\\test\\test");
>         File dest = new File("\\\\ismerek1\\Shared");
>        
>         String filename = "jdk-6u19-windows-x64.exe";
>        
>         File file = new File(src, filename);
>         File toFile = new File(dest, filename);
>        
>         try {
>             FileUtils.copyFile(file, toFile, true);
>             System.out.println("Successful copy");
>         }
>         catch (IOException e1) {
>             e1.printStackTrace();
>         }
>     }
> }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Reply | Threaded
Open this post in threaded view
|

[jira] [Resolved] (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

Gilles (Jira)
In reply to this post by Gilles (Jira)

     [ https://issues.apache.org/jira/browse/IO-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sebb resolved IO-266.
---------------------

       Resolution: Fixed
    Fix Version/s: 2.1

Changed to 30MB

> FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)
> ---------------------------------------------------------------------------------------------------
>
>                 Key: IO-266
>                 URL: https://issues.apache.org/jira/browse/IO-266
>             Project: Commons IO
>          Issue Type: Bug
>    Affects Versions: 2.0.1
>         Environment: Windows 2003 Server 64-bit
>            Reporter: Igor Smereka
>             Fix For: 2.1
>
>         Attachments: FileUtils.java.patch
>
>
> java.io.IOException: Insufficient system resources exist to complete the requested service
> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
> at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)
> NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
> Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).
> Here is a test program to reproduce the issue:
> package org.test.igor;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.nio.channels.FileChannel;
> import org.apache.commons.io.FileUtils;
> import org.apache.commons.io.IOUtils;
> public class TestFileUtils {
>     public static void main(String[] args){
>        
>        
>         File src = new File("D:\\2011.1-dev\\test\\test");
>         File dest = new File("\\\\ismerek1\\Shared");
>        
>         String filename = "jdk-6u19-windows-x64.exe";
>        
>         File file = new File(src, filename);
>         File toFile = new File(dest, filename);
>        
>         try {
>             FileUtils.copyFile(file, toFile, true);
>             System.out.println("Successful copy");
>         }
>         catch (IOException e1) {
>             e1.printStackTrace();
>         }
>     }
> }

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Reply | Threaded
Open this post in threaded view
|

[jira] [Closed] (IO-266) FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)

Gilles (Jira)
In reply to this post by Gilles (Jira)

     [ https://issues.apache.org/jira/browse/IO-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gary D. Gregory closed IO-266.
------------------------------


Closing, we released version 2.1.
               

> FileUtils.copyFile() throws IOException when copying large files to a shared directory (on Windows)
> ---------------------------------------------------------------------------------------------------
>
>                 Key: IO-266
>                 URL: https://issues.apache.org/jira/browse/IO-266
>             Project: Commons IO
>          Issue Type: Bug
>    Affects Versions: 2.0.1
>         Environment: Windows 2003 Server 64-bit
>            Reporter: Igor Smereka
>             Fix For: 2.1
>
>         Attachments: FileUtils.java.patch
>
>
> java.io.IOException: Insufficient system resources exist to complete the requested service
> at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
> at sun.nio.ch.FileDispatcher.pwrite(Unknown Source)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
> at sun.nio.ch.IOUtil.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.write(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFromFileChannel(Unknown Source)
> at sun.nio.ch.FileChannelImpl.transferFrom(Unknown Source)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:813)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:783)
> at org.test.igor.TestFileUtils.main(TestFileUtils.java:55)
> NOTE: the issue is cased by the function doCopyFile(File srcFile, File destFile, boolean preserveFileDate) using hardcoded data chunks of FIFTY_MB in the transferFrom() call.
> Reducing this chunk from 50M to 31M solves the issue for my situation (32M still fails).
> Here is a test program to reproduce the issue:
> package org.test.igor;
> import java.io.File;
> import java.io.FileInputStream;
> import java.io.FileOutputStream;
> import java.io.IOException;
> import java.nio.channels.FileChannel;
> import org.apache.commons.io.FileUtils;
> import org.apache.commons.io.IOUtils;
> public class TestFileUtils {
>     public static void main(String[] args){
>        
>        
>         File src = new File("D:\\2011.1-dev\\test\\test");
>         File dest = new File("\\\\ismerek1\\Shared");
>        
>         String filename = "jdk-6u19-windows-x64.exe";
>        
>         File file = new File(src, filename);
>         File toFile = new File(dest, filename);
>        
>         try {
>             FileUtils.copyFile(file, toFile, true);
>             System.out.println("Successful copy");
>         }
>         catch (IOException e1) {
>             e1.printStackTrace();
>         }
>     }
> }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira