[net] Problem with completePendingCommand and large files > 600MB

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[net] Problem with completePendingCommand and large files > 600MB

SeungWook Kim
Hi,
  I am having a problem with commons-net version* 3.6* and with large files
> 600MB and offset and length that does not include the whole file.

I have included the test code below that appears to fail with large files >
600 MB and an offset and length that does not include the whole file. If I
choose a smaller file like 90KB with offset and length, it works
successfully. If I choose a large file > 600MB with offset and length that
INCLUDES the entire file, it also works successfully. The
completePendingCommand replies back with "426 Failure writing network
stream" when the file is large > 600MB AND the offset and length is not the
entire file.

  I believe this is a bug but not sure. I did a quick look in the postings
and did not see anything that is exactly like it. Any feedback is greatly
appreciated.



*** Need to specify serverName, userName,...absOutputFile below***
***Start of Test code***

public class FtpTest {
    private static Logger log = LoggerFactory.getLogger(FtpTest.class);

    private static final int DEFAULT_TIMEOUT = 10*1000;

    @Before
    public void setup() {

    }

    @Test
    public void test() {
        String serverName = "***";
        String userName = "***";
        String password = "***";
        String absRemoteFileName = "***";
        final long offset = ***;
        final long length = ***;
        String absOutputFile = "***";

        FTPClient ftpClient = new FTPClient();
        try {
            ftpClient.connect(serverName);
            int reply = ftpClient.getReplyCode();
            if (!FTPReply.isPositiveCompletion(reply)) {
                String lastReply = ftpClient.getReplyString();
                fail(lastReply);
            }
            ftpClient.addProtocolCommandListener(new
PrintCommandListener(new PrintWriter(System.out),
                    true));
            ftpClient.setKeepAlive(true);
            ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
            //turn on SO_LINGER w/ timeout of 0 to cause connection to be
aborted immediately and to discard any
            // pending data.
            ftpClient.setSoLinger(true, 0);
            if(ftpClient.login(userName, password)) {
                ftpClient.enterLocalPassiveMode();
                if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
                    log.info("Successfully connected in Passive Mode and
logged into {}", serverName);

                    ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
                    ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
                    ftpClient.setRestartOffset(offset);
                    File writeToFile = new File(absOutputFile);

                    try (InputStream inputStream =
ftpClient.retrieveFileStream(absRemoteFileName);
                         OutputStream outputStream =
FileUtils.openOutputStream(writeToFile)) {
                        if(inputStream == null || outputStream == null) {
                            fail("One of the input/output stream is null");
                        }
                        final int BUFFER_SIZE = 4096;
                        byte[] buffer = new byte[BUFFER_SIZE];
                        int bytesRead;
                        int len = (int)length;
                        int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
: len;
                        while ((bytesRead = inputStream.read(buffer, 0,
bufferSize)) != -1) {
                            len -= bytesRead;
                            bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
len;

                            outputStream.write(buffer, 0, bytesRead);

                            if(len <= 0) {
                                break;
                            }
                        }


                    } finally {
                        if(ftpClient.completePendingCommand()) {
                            log.info("Successful complete pending command");
                        } else {
                            fail("Unsuccessful complete pending command");
                        }
                    }

                } else {
                    fail("Problem entering passive mode. All modern ftp
server supports passive mode, but " +
                            serverName + " doesn't??");

                }
            } else {
                fail("Bad credentials to server " + serverName);
            }

        } catch (Exception e) {
            fail(e.toString() + "\n" + e.getStackTrace().toString());
        }
    }
}

***End of Test code***

Thank you
-Seung
Reply | Threaded
Open this post in threaded view
|

Re: [net] Problem with completePendingCommand and large files > 600MB

Bernd Eckenfels
426 sounds like a server error, did you try to use a different client with your server?

Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
From: SeungWook Kim <[hidden email]>
Sent: Wednesday, October 25, 2017 6:34:02 PM
To: [hidden email]
Subject: [net] Problem with completePendingCommand and large files > 600MB

Hi,
  I am having a problem with commons-net version* 3.6* and with large files
> 600MB and offset and length that does not include the whole file.

I have included the test code below that appears to fail with large files >
600 MB and an offset and length that does not include the whole file. If I
choose a smaller file like 90KB with offset and length, it works
successfully. If I choose a large file > 600MB with offset and length that
INCLUDES the entire file, it also works successfully. The
completePendingCommand replies back with "426 Failure writing network
stream" when the file is large > 600MB AND the offset and length is not the
entire file.

  I believe this is a bug but not sure. I did a quick look in the postings
and did not see anything that is exactly like it. Any feedback is greatly
appreciated.



*** Need to specify serverName, userName,...absOutputFile below***
***Start of Test code***

public class FtpTest {
    private static Logger log = LoggerFactory.getLogger(FtpTest.class);

    private static final int DEFAULT_TIMEOUT = 10*1000;

    @Before
    public void setup() {

    }

    @Test
    public void test() {
        String serverName = "***";
        String userName = "***";
        String password = "***";
        String absRemoteFileName = "***";
        final long offset = ***;
        final long length = ***;
        String absOutputFile = "***";

        FTPClient ftpClient = new FTPClient();
        try {
            ftpClient.connect(serverName);
            int reply = ftpClient.getReplyCode();
            if (!FTPReply.isPositiveCompletion(reply)) {
                String lastReply = ftpClient.getReplyString();
                fail(lastReply);
            }
            ftpClient.addProtocolCommandListener(new
PrintCommandListener(new PrintWriter(System.out),
                    true));
            ftpClient.setKeepAlive(true);
            ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
            //turn on SO_LINGER w/ timeout of 0 to cause connection to be
aborted immediately and to discard any
            // pending data.
            ftpClient.setSoLinger(true, 0);
            if(ftpClient.login(userName, password)) {
                ftpClient.enterLocalPassiveMode();
                if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
                    log.info("Successfully connected in Passive Mode and
logged into {}", serverName);

                    ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
                    ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
                    ftpClient.setRestartOffset(offset);
                    File writeToFile = new File(absOutputFile);

                    try (InputStream inputStream =
ftpClient.retrieveFileStream(absRemoteFileName);
                         OutputStream outputStream =
FileUtils.openOutputStream(writeToFile)) {
                        if(inputStream == null || outputStream == null) {
                            fail("One of the input/output stream is null");
                        }
                        final int BUFFER_SIZE = 4096;
                        byte[] buffer = new byte[BUFFER_SIZE];
                        int bytesRead;
                        int len = (int)length;
                        int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
: len;
                        while ((bytesRead = inputStream.read(buffer, 0,
bufferSize)) != -1) {
                            len -= bytesRead;
                            bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
len;

                            outputStream.write(buffer, 0, bytesRead);

                            if(len <= 0) {
                                break;
                            }
                        }


                    } finally {
                        if(ftpClient.completePendingCommand()) {
                            log.info("Successful complete pending command");
                        } else {
                            fail("Unsuccessful complete pending command");
                        }
                    }

                } else {
                    fail("Problem entering passive mode. All modern ftp
server supports passive mode, but " +
                            serverName + " doesn't??");

                }
            } else {
                fail("Bad credentials to server " + serverName);
            }

        } catch (Exception e) {
            fail(e.toString() + "\n" + e.getStackTrace().toString());
        }
    }
}

***End of Test code***

Thank you
-Seung
Reply | Threaded
Open this post in threaded view
|

Re: [net] Problem with completePendingCommand and large files > 600MB

sebb-2-2
What you are doing is stopping reading from the server before the file
has been fully sent.
Unless the server is expecting that, it may complain (the fact that it
works for smaller files may be because the server has already sent the
data).

AFAICT there is no standard way to tell the server to only send a
portion of the file.

A work-round would be to fetch all the data but stop writing it to
disk when you have got what you want.

Another way might be to cancel the transfer after enough data has been received.
However that relies on the server being able to process control
channel requests whilst sending data.

Or just live with the error message.

It's not a bug in NET.


On 25 October 2017 at 23:30, Bernd Eckenfels <[hidden email]> wrote:

> 426 sounds like a server error, did you try to use a different client with your server?
>
> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
> ________________________________
> From: SeungWook Kim <[hidden email]>
> Sent: Wednesday, October 25, 2017 6:34:02 PM
> To: [hidden email]
> Subject: [net] Problem with completePendingCommand and large files > 600MB
>
> Hi,
>   I am having a problem with commons-net version* 3.6* and with large files
>> 600MB and offset and length that does not include the whole file.
>
> I have included the test code below that appears to fail with large files >
> 600 MB and an offset and length that does not include the whole file. If I
> choose a smaller file like 90KB with offset and length, it works
> successfully. If I choose a large file > 600MB with offset and length that
> INCLUDES the entire file, it also works successfully. The
> completePendingCommand replies back with "426 Failure writing network
> stream" when the file is large > 600MB AND the offset and length is not the
> entire file.
>
>   I believe this is a bug but not sure. I did a quick look in the postings
> and did not see anything that is exactly like it. Any feedback is greatly
> appreciated.
>
>
>
> *** Need to specify serverName, userName,...absOutputFile below***
> ***Start of Test code***
>
> public class FtpTest {
>     private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>
>     private static final int DEFAULT_TIMEOUT = 10*1000;
>
>     @Before
>     public void setup() {
>
>     }
>
>     @Test
>     public void test() {
>         String serverName = "***";
>         String userName = "***";
>         String password = "***";
>         String absRemoteFileName = "***";
>         final long offset = ***;
>         final long length = ***;
>         String absOutputFile = "***";
>
>         FTPClient ftpClient = new FTPClient();
>         try {
>             ftpClient.connect(serverName);
>             int reply = ftpClient.getReplyCode();
>             if (!FTPReply.isPositiveCompletion(reply)) {
>                 String lastReply = ftpClient.getReplyString();
>                 fail(lastReply);
>             }
>             ftpClient.addProtocolCommandListener(new
> PrintCommandListener(new PrintWriter(System.out),
>                     true));
>             ftpClient.setKeepAlive(true);
>             ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
>             //turn on SO_LINGER w/ timeout of 0 to cause connection to be
> aborted immediately and to discard any
>             // pending data.
>             ftpClient.setSoLinger(true, 0);
>             if(ftpClient.login(userName, password)) {
>                 ftpClient.enterLocalPassiveMode();
>                 if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
> {
>                     log.info("Successfully connected in Passive Mode and
> logged into {}", serverName);
>
>                     ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
>                     ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
>                     ftpClient.setRestartOffset(offset);
>                     File writeToFile = new File(absOutputFile);
>
>                     try (InputStream inputStream =
> ftpClient.retrieveFileStream(absRemoteFileName);
>                          OutputStream outputStream =
> FileUtils.openOutputStream(writeToFile)) {
>                         if(inputStream == null || outputStream == null) {
>                             fail("One of the input/output stream is null");
>                         }
>                         final int BUFFER_SIZE = 4096;
>                         byte[] buffer = new byte[BUFFER_SIZE];
>                         int bytesRead;
>                         int len = (int)length;
>                         int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
> : len;
>                         while ((bytesRead = inputStream.read(buffer, 0,
> bufferSize)) != -1) {
>                             len -= bytesRead;
>                             bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
> len;
>
>                             outputStream.write(buffer, 0, bytesRead);
>
>                             if(len <= 0) {
>                                 break;
>                             }
>                         }
>
>
>                     } finally {
>                         if(ftpClient.completePendingCommand()) {
>                             log.info("Successful complete pending command");
>                         } else {
>                             fail("Unsuccessful complete pending command");
>                         }
>                     }
>
>                 } else {
>                     fail("Problem entering passive mode. All modern ftp
> server supports passive mode, but " +
>                             serverName + " doesn't??");
>
>                 }
>             } else {
>                 fail("Bad credentials to server " + serverName);
>             }
>
>         } catch (Exception e) {
>             fail(e.toString() + "\n" + e.getStackTrace().toString());
>         }
>     }
> }
>
> ***End of Test code***
>
> Thank you
> -Seung

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [net] Problem with completePendingCommand and large files > 600MB

sebb-2-2
Note: curl supports an FTP range parameter.
It seems it does this by sending ABOR after the required data has arrived.

However I just tried and it does not work with the server I used.

You could try using it with your server and see if it works:

$ curl -v -r nnn-mmm -o file URL

If so you may be able to use ABOR in your app for that server.


On 25 October 2017 at 23:57, sebb <[hidden email]> wrote:

> What you are doing is stopping reading from the server before the file
> has been fully sent.
> Unless the server is expecting that, it may complain (the fact that it
> works for smaller files may be because the server has already sent the
> data).
>
> AFAICT there is no standard way to tell the server to only send a
> portion of the file.
>
> A work-round would be to fetch all the data but stop writing it to
> disk when you have got what you want.
>
> Another way might be to cancel the transfer after enough data has been received.
> However that relies on the server being able to process control
> channel requests whilst sending data.
>
> Or just live with the error message.
>
> It's not a bug in NET.
>
>
> On 25 October 2017 at 23:30, Bernd Eckenfels <[hidden email]> wrote:
>> 426 sounds like a server error, did you try to use a different client with your server?
>>
>> Gruss
>> Bernd
>> --
>> http://bernd.eckenfels.net
>> ________________________________
>> From: SeungWook Kim <[hidden email]>
>> Sent: Wednesday, October 25, 2017 6:34:02 PM
>> To: [hidden email]
>> Subject: [net] Problem with completePendingCommand and large files > 600MB
>>
>> Hi,
>>   I am having a problem with commons-net version* 3.6* and with large files
>>> 600MB and offset and length that does not include the whole file.
>>
>> I have included the test code below that appears to fail with large files >
>> 600 MB and an offset and length that does not include the whole file. If I
>> choose a smaller file like 90KB with offset and length, it works
>> successfully. If I choose a large file > 600MB with offset and length that
>> INCLUDES the entire file, it also works successfully. The
>> completePendingCommand replies back with "426 Failure writing network
>> stream" when the file is large > 600MB AND the offset and length is not the
>> entire file.
>>
>>   I believe this is a bug but not sure. I did a quick look in the postings
>> and did not see anything that is exactly like it. Any feedback is greatly
>> appreciated.
>>
>>
>>
>> *** Need to specify serverName, userName,...absOutputFile below***
>> ***Start of Test code***
>>
>> public class FtpTest {
>>     private static Logger log = LoggerFactory.getLogger(FtpTest.class);
>>
>>     private static final int DEFAULT_TIMEOUT = 10*1000;
>>
>>     @Before
>>     public void setup() {
>>
>>     }
>>
>>     @Test
>>     public void test() {
>>         String serverName = "***";
>>         String userName = "***";
>>         String password = "***";
>>         String absRemoteFileName = "***";
>>         final long offset = ***;
>>         final long length = ***;
>>         String absOutputFile = "***";
>>
>>         FTPClient ftpClient = new FTPClient();
>>         try {
>>             ftpClient.connect(serverName);
>>             int reply = ftpClient.getReplyCode();
>>             if (!FTPReply.isPositiveCompletion(reply)) {
>>                 String lastReply = ftpClient.getReplyString();
>>                 fail(lastReply);
>>             }
>>             ftpClient.addProtocolCommandListener(new
>> PrintCommandListener(new PrintWriter(System.out),
>>                     true));
>>             ftpClient.setKeepAlive(true);
>>             ftpClient.setSoTimeout(DEFAULT_TIMEOUT); //set to 10 seconds
>>             //turn on SO_LINGER w/ timeout of 0 to cause connection to be
>> aborted immediately and to discard any
>>             // pending data.
>>             ftpClient.setSoLinger(true, 0);
>>             if(ftpClient.login(userName, password)) {
>>                 ftpClient.enterLocalPassiveMode();
>>                 if(FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
>> {
>>                     log.info("Successfully connected in Passive Mode and
>> logged into {}", serverName);
>>
>>                     ftpClient.setFileTransferMode(FTP.STREAM_TRANSFER_MODE);
>>                     ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
>>                     ftpClient.setRestartOffset(offset);
>>                     File writeToFile = new File(absOutputFile);
>>
>>                     try (InputStream inputStream =
>> ftpClient.retrieveFileStream(absRemoteFileName);
>>                          OutputStream outputStream =
>> FileUtils.openOutputStream(writeToFile)) {
>>                         if(inputStream == null || outputStream == null) {
>>                             fail("One of the input/output stream is null");
>>                         }
>>                         final int BUFFER_SIZE = 4096;
>>                         byte[] buffer = new byte[BUFFER_SIZE];
>>                         int bytesRead;
>>                         int len = (int)length;
>>                         int bufferSize = length > BUFFER_SIZE ? BUFFER_SIZE
>> : len;
>>                         while ((bytesRead = inputStream.read(buffer, 0,
>> bufferSize)) != -1) {
>>                             len -= bytesRead;
>>                             bufferSize = len > BUFFER_SIZE ? BUFFER_SIZE :
>> len;
>>
>>                             outputStream.write(buffer, 0, bytesRead);
>>
>>                             if(len <= 0) {
>>                                 break;
>>                             }
>>                         }
>>
>>
>>                     } finally {
>>                         if(ftpClient.completePendingCommand()) {
>>                             log.info("Successful complete pending command");
>>                         } else {
>>                             fail("Unsuccessful complete pending command");
>>                         }
>>                     }
>>
>>                 } else {
>>                     fail("Problem entering passive mode. All modern ftp
>> server supports passive mode, but " +
>>                             serverName + " doesn't??");
>>
>>                 }
>>             } else {
>>                 fail("Bad credentials to server " + serverName);
>>             }
>>
>>         } catch (Exception e) {
>>             fail(e.toString() + "\n" + e.getStackTrace().toString());
>>         }
>>     }
>> }
>>
>> ***End of Test code***
>>
>> Thank you
>> -Seung

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]