[net] FTPClient doesn't download full images

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[net] FTPClient doesn't download full images

Julia Ruzicka
Hi everyone!

 

This is my first time using a mailing list, so sorry in advance if I mess
something up!

 

I'm using version 3.6 of the "Net" Library, in combination with Java 8,
Android Studio 3.6.1 (min SDK=24, target SDK=27) and I'm experiencing a
weird issue with file download:

 

Downloading small images (jpg, around 12kb each) or text files from my FTP
server always succeeds but I've also got bigger images (png, 2000x2000,
around 8-9mb each) and they only download partially: I pulled them from the
emulator/device and they end up with a black, horizontal bar at the bottom
and they're smaller than they should be (see below for examples).

 

I tested my code with 'ftpClient.retrieveFile()' (version 1) and
'ftpClient.retrieveFileStream()' (version 2, InputStream or
BufferedInputStream) but I get the same result with both:

 

public void downloadFiles(String remoteFolder, String localFolder,
ArrayList<String> filenames) { //This runs inside an 'AsyncTask'!

    //login here

    ftpClient.setConnectTimeout(10000);

    ftpClient.setDefaultTimeout(30000);

    OutputStream out = null;

 

    try {

        ftpClient.enterLocalPassiveMode();

        ftpClient.setFileType(FTP.BINARY_FILE_TYPE);

 

        if(ftpClient.changeWorkingDirectory(remoteFolder)) {

            for(String filename : filenames) {

                FTPFile[] singleFile = ftpClient.listFiles(filename);

 

                if(singleFile.length > 0) { //check if file exists

                    String localPath = localFolder + File.separator +
filename;

                    out = new FileOutputStream(localPath);

                   

                    //Version 1:

                    /*if(!ftpClient.retrieveFile(filename, out)) {

                        //Set error message here

                        out.close();

                        break;

                    }

                    out.close();*/

 

 

                    //Version 2:

                    InputStream is = ftpClient.retrieveFileStream(filename);


                    BufferedInputStream bis = new BufferedInputStream(is);

                    byte[] buf = new byte[10000000];

                    int len;

 

                    while ((len = bis.read(buf)) > 0) { //Same result using
only 'is'

                        out.write(buf, 0, len);

                    }

                   

                    if(ftpClient.completePendingCommand()) {

                        Log.d(TAG,"DONE"); //This is always printed, even
when the files aren't complete

                    } else {

                        Log.d(TAG,"NOT DONE");

                    }

 

                    out.close();

                    bis.close();

                } else {

                    //Another error message here

                    break;

                }

            }

        }

    } catch (IOException e) {

        //And another error message here

    } finally {

        try {

            if(out!=null) { out.close(); }

        } catch(IOException e) {

            //Doesn't matter

        }

    }

 

    //logout here

}

 

I printed the full length (added up 'len') and only very rarely it manages
to download everything, usually there are a couple of bytes missing (always
less than 10 so far) - example:

 

File 1 is 8965167 bytes (according to Windows' file explorer) but the added
up 'len' is only 8965165 or 8965161 or .

File 2 is 8972047 bytes but it only downloads 8972044 or 8972038 or .

 

Did I miss anything, some type of setting for bigger files? How do I make it
download the full images?

 

Thanks in advance for any tip!

Reply | Threaded
Open this post in threaded view
|

AW: [net] FTPClient doesn't download full images

Julia Ruzicka
Hi everyone!

I created a small test app that uses my "download" class on a button press
(but doesn't do anything else) and did some more testing with it (with
"FTP.BINARY_FILE_TYPE"):

- Downloading a 14mb txt file (and its copies) always finishes with a
complete file (even with a long file name)
- Downloading a 12kb jpg (and its copies) always finishes with a complete
file
- Downloading the 8mb (png) or 6mb (jpg version) images sometimes completes
but most of the time it doesn't - the file size is now always the same but
there's still a black bar at the bottom of the images (height differs)
- I uploaded a couple of random google images to my FTP server and most of
them are affected by this problem too.

You can find all the tested google images (including the sources) and the
results in this imgur album:

https://imgur.com/a/Mzngp11

I uploaded a couple of the images to a public FTP test server
(https://dlptest.com/ftp-test/) and I get the same results with it.

Has anyone got an idea what's wrong and how do fix this?




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

Robert Paasche
The following line is wrong:
     while ((len = bis.read(buf)) > 0) { //Same result using only 'is'

read == -1 marks the end of the stream. 0 shows only that the stream is
still waiting for the next chunk of data.

Please read the JavaDocs:

> Returns:
> the number of bytes read, or -1 if the end of the stream has been reached.


  while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
if (len >0)
       out.write(buf, 0, len);
else
        // prevent high CPU load
        LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
 }
Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

Julia Ruzicka
In reply to this post by Julia Ruzicka
Thanks for your reply!

I changed the code and did the usual AndroidStudio restart/clean/build but most images are still downloaded incompletely.
I don't know if my other answer (sent about an hour ago) came through (I did some more testing today), so here's an imgur album with test images and their results:
https://imgur.com/a/Mzngp11

The last image (the satellite one) was now also affected for the first time.

-----Ursprüngliche Nachricht-----
Von: Robert Paasche <[hidden email]>
Gesendet: Donnerstag, 4. Juni 2020 12:18
An: Commons Users List <[hidden email]>
Betreff: Re: [net] FTPClient doesn't download full images

The following line is wrong:
     while ((len = bis.read(buf)) > 0) { //Same result using only 'is'

read == -1 marks the end of the stream. 0 shows only that the stream is still waiting for the next chunk of data.

Please read the JavaDocs:

> Returns:
> the number of bytes read, or -1 if the end of the stream has been reached.


  while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
if (len >0)
       out.write(buf, 0, len);
else
        // prevent high CPU load
        LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
 }



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

Robert Paasche
Hmm maybe it is the bufferdStream.

That's working code I'm using in a project for years now:

private FTPClient createClient()
{
    FTPClient ftp = new FTPClient();
    FTPClientConfig config = new FTPClientConfig();

    config.setServerTimeZoneId("UTC");

    ftp.setControlKeepAliveTimeout(1000 * 60 * 5l);
    ftp.setControlKeepAliveReplyTimeout(1000 * 60 * 5);
    ftp.setDataTimeout(1000 * 60 * 5);
    ftp.setRemoteVerificationEnabled(false);
    return ftp;
}

if (FetchMode.BYTE_DATA.equals(fetchMode))
{
    LOGGER.info("Using BINARY_FILE_TYPE.");
    ftp.setFileType(FTP.BINARY_FILE_TYPE);
    InputStream inStream = ftp.retrieveFileStream(ftpFile.getName());
    ByteArrayOutputStream buffer = new ByteArrayOutputStream();

    int nRead;
    byte[] data = new byte[(int) (ftpFile.getSize() * 2)];

    while ((nRead = inStream.read(data, 0, data.length)) != -1)
    {
        buffer.write(data, 0, nRead);
    }

    buffer.flush();

    content.setByteData(buffer.toByteArray());

}
Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

garydgregory
In reply to this post by Julia Ruzicka
If you want to read a stream without dealing with low-level headaches, you
can use
https://commons.apache.org/proper/commons-io/javadocs/api-release/org/apache/commons/io/IOUtils.html

Gary

On Thu, Jun 4, 2020 at 6:53 AM Julia Ruzicka <[hidden email]>
wrote:

> Thanks for your reply!
>
> I changed the code and did the usual AndroidStudio restart/clean/build but
> most images are still downloaded incompletely.
> I don't know if my other answer (sent about an hour ago) came through (I
> did some more testing today), so here's an imgur album with test images and
> their results:
> https://imgur.com/a/Mzngp11
>
> The last image (the satellite one) was now also affected for the first
> time.
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Paasche <[hidden email]>
> Gesendet: Donnerstag, 4. Juni 2020 12:18
> An: Commons Users List <[hidden email]>
> Betreff: Re: [net] FTPClient doesn't download full images
>
> The following line is wrong:
>      while ((len = bis.read(buf)) > 0) { //Same result using only 'is'
>
> read == -1 marks the end of the stream. 0 shows only that the stream is
> still waiting for the next chunk of data.
>
> Please read the JavaDocs:
>
> > Returns:
> > the number of bytes read, or -1 if the end of the stream has been
> reached.
>
>
>   while ((len = bis.read(buf)) >= 0) { //Same result using only 'is'
> if (len >0)
>        out.write(buf, 0, len);
> else
>         // prevent high CPU load
>         LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(500));
>  }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Re: [net] FTPClient doesn't download full images

Julia Ruzicka
In reply to this post by Julia Ruzicka
@Robert Paasche

At first I tried to do it with a `FileOutputStream` and `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the problem. I then switched to using ` InputStream is = ftpClient.retrieveFileStream(filename)` and when that didn't work properly either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`.

What type of files are you downloading?

What's your `content` and how do you write the data? With a DataOutputStream?



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

Christoph Läubrich
Don't know if the code you have showed is your original one but in
general its not a good idea to "catch and ignore" exceptions, yo either:

- simply buble up to the caller
- wrap into other exceptions if required by the API
- explicitly handle/document

I'm using comons-ftp for large files and small files and it always has
worked quite well.

Am 04.06.20 um 14:10 schrieb Julia Ruzicka:

> @Robert Paasche
>
> At first I tried to do it with a `FileOutputStream` and `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the problem. I then switched to using ` InputStream is = ftpClient.retrieveFileStream(filename)` and when that didn't work properly either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`.
>
> What type of files are you downloading?
>
> What's your `content` and how do you write the data? With a DataOutputStream?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Re: [net] FTPClient doesn't download full images

Robert Paasche
In reply to this post by Julia Ruzicka
@ julia.ruzicka
content  is just a simple POJO holding the byte and meta data (like
timestamps). In some situations I'm using a fileoutputstream after
disconnecting from ftp.  I'm downloading large (>300 mb) zip-Files.

Am Do., 4. Juni 2020 um 14:10 Uhr schrieb Julia Ruzicka <
[hidden email]>:

> @Robert Paasche
>
> At first I tried to do it with a `FileOutputStream` and
> `ftpClient.retrieveFile(fileName,fos)` but that's how I noticed the
> problem. I then switched to using ` InputStream is =
> ftpClient.retrieveFileStream(filename)` and when that didn't work properly
> either, I added ` BufferedInputStream bis = new BufferedInputStream(is)`.
>
> What type of files are you downloading?
>
> What's your `content` and how do you write the data? With a
> DataOutputStream?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Re: [net] FTPClient doesn't download full images

Julia Ruzicka
In reply to this post by Julia Ruzicka
@Robert Paasche

The code is used in an AsyncTask and there's more error handling to it (including proper error messages,...), I just simplified it. ;)

Have you ever tested it with images too? Text files always seem to download properly (no matter what size), it's just the images that are messed up.

Thanks for the code snippet, I just ran my app with it multiple times and the files are all the right size, even the images, but no program is able to actually open them, so not sure what's going on with that now:

if(singleFile.length > 0) { //check if file exists
    String localPath = localFolder + File.separator + filename;
    FTPFile single = singleFile[0];
    InputStream inStream = ftpClient.retrieveFileStream(filename);
    ByteArrayOutputStream buffer = new ByteArrayOutputStream();

    int nRead;
    byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but why?

    while ((nRead = inStream.read(data, 0, data.length)) != -1) {
        buffer.write(data, 0, nRead);
    }

    buffer.flush();
    inStream.close();

    FileOutputStream fos = new FileOutputStream(localPath);
    fos.write(data);
    fos.flush();
    fos.close();

    if(ftpClient.completePendingCommand()) {
        Log.d(TAG,"DONE");
    } else {
        Log.d(TAG,"NOT DONE");
    }
}



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Re: [net] FTPClient doesn't download full images

Robert Paasche
No never used with images.

 //length was "*2" but why?
That is for an buggy ftp-server, which mishandled multibyte encodings. On
UTF-16 Files he has reported the number of chars instead of bytes, so the
file size was just the half erverytime.

fos.write(data);
this should be something like:
  fos.write(buffer.toByteArray());


Am Do., 4. Juni 2020 um 15:29 Uhr schrieb Julia Ruzicka <
[hidden email]>:

> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
> (including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to
> download properly (no matter what size), it's just the images that are
> messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and
> the files are all the right size, even the images, but no program is able
> to actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
>     String localPath = localFolder + File.separator + filename;
>     FTPFile single = singleFile[0];
>     InputStream inStream = ftpClient.retrieveFileStream(filename);
>     ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
>     int nRead;
>     byte[] data = new byte[(int) (single.getSize())]; //length was "*2"
> but why?
>
>     while ((nRead = inStream.read(data, 0, data.length)) != -1) {
>         buffer.write(data, 0, nRead);
>     }
>
>     buffer.flush();
>     inStream.close();
>
>     FileOutputStream fos = new FileOutputStream(localPath);
>     fos.write(data);
>     fos.flush();
>     fos.close();
>
>     if(ftpClient.completePendingCommand()) {
>         Log.d(TAG,"DONE");
>     } else {
>         Log.d(TAG,"NOT DONE");
>     }
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

Erwin Hogeweg-3
In reply to this post by Julia Ruzicka
Hi,

I have no idea if this is related but I seem to remember from a life long, long ago that you can set up and FTP connection in two modes ASCII and binary. The default is (used to be) ASCII and when you feed  binary data through that connection then (some) non-ascii characters were translated.

Again, I may be talking complete nonsense here but it may be worth to check it out.

Cheers,

Erwin


> On Jun 4, 2020, at 09:29, Julia Ruzicka <[hidden email]> wrote:
>
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it (including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to download properly (no matter what size), it's just the images that are messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and the files are all the right size, even the images, but no program is able to actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
>    String localPath = localFolder + File.separator + filename;
>    FTPFile single = singleFile[0];
>    InputStream inStream = ftpClient.retrieveFileStream(filename);
>    ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
>    int nRead;
>    byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but why?
>
>    while ((nRead = inStream.read(data, 0, data.length)) != -1) {
>        buffer.write(data, 0, nRead);
>    }
>
>    buffer.flush();
>    inStream.close();
>
>    FileOutputStream fos = new FileOutputStream(localPath);
>    fos.write(data);
>    fos.flush();
>    fos.close();
>
>    if(ftpClient.completePendingCommand()) {
>        Log.d(TAG,"DONE");
>    } else {
>        Log.d(TAG,"NOT DONE");
>    }
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [net] FTPClient doesn't download full images

Julia Ruzicka
In reply to this post by Julia Ruzicka
@Erwin
Hi!

You can find the full code in the very first e-mail (May). I'm setting the
file type to `FTP.BINARY_FILE_TYPE` before I open the stream but when I'm
already connected to the server (as stated in the javadoc). I also tested it
with ASCII once, just to be sure that nothing was mixed up, but it obviously
didn't work.

-----Ursprüngliche Nachricht-----
Von: Erwin Hogeweg <[hidden email]>
Gesendet: Donnerstag, 4. Juni 2020 17:25
An: Commons Users List <[hidden email]>
Betreff: Re: [net] FTPClient doesn't download full images

Hi,

I have no idea if this is related but I seem to remember from a life long,
long ago that you can set up and FTP connection in two modes ASCII and
binary. The default is (used to be) ASCII and when you feed  binary data
through that connection then (some) non-ascii characters were translated.

Again, I may be talking complete nonsense here but it may be worth to check
it out.

Cheers,

Erwin


> On Jun 4, 2020, at 09:29, Julia Ruzicka <[hidden email]> wrote:
>
> @Robert Paasche
>
> The code is used in an AsyncTask and there's more error handling to it
(including proper error messages,...), I just simplified it. ;)
>
> Have you ever tested it with images too? Text files always seem to
download properly (no matter what size), it's just the images that are
messed up.
>
> Thanks for the code snippet, I just ran my app with it multiple times and
the files are all the right size, even the images, but no program is able to
actually open them, so not sure what's going on with that now:
>
> if(singleFile.length > 0) { //check if file exists
>    String localPath = localFolder + File.separator + filename;
>    FTPFile single = singleFile[0];
>    InputStream inStream = ftpClient.retrieveFileStream(filename);
>    ByteArrayOutputStream buffer = new ByteArrayOutputStream();
>
>    int nRead;
>    byte[] data = new byte[(int) (single.getSize())]; //length was "*2" but
why?

>
>    while ((nRead = inStream.read(data, 0, data.length)) != -1) {
>        buffer.write(data, 0, nRead);
>    }
>
>    buffer.flush();
>    inStream.close();
>
>    FileOutputStream fos = new FileOutputStream(localPath);
>    fos.write(data);
>    fos.flush();
>    fos.close();
>
>    if(ftpClient.completePendingCommand()) {
>        Log.d(TAG,"DONE");
>    } else {
>        Log.d(TAG,"NOT DONE");
>    }
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Re: [net] FTPClient doesn't download full images

Julia Ruzicka
@Robert Paasche
I see.
Does your FTP server use Linux or Windows?

Something must have been stuck in RAM because after a PC reboot the success rate (still in binary mode) with your very first tip (">=0" instead of ">0") is a lot higher. It's still not even close to acceptable though:

With a buffer size (byte[]) of 1000 about 1-2 out of 20 downloaded files are missing 1-2 bytes.
With a buffer size of 100k about 5-10 in 20 downloaded files are missing 1-4 bytes (sometimes more). I get images with a black bar/image artifacts again and text files are also affected (missing 1-5 bytes).
With a buffer size of 100 the success rate is about has high as with 1000 and the download obviously takes a lot longer.

The files with 1 missing bytes will still open fine and it doesn't look like there's a problem (won't find any problems with that until I try to display the images in my actual app) but it's nothing I can just ignore.

Thanks for correcting the bit, I tested it with `buffer.toByteArray()` instead:
The files will open now but about 2/3 of them are missing bytes (txt and images) and there are image artifacts.

After receiving Erwin's e-mail I tested ASCII mode again: It looks like the success rate with only txt files is 100% and it properly converts the original files' "\r\n" (created with Windows) to Android's/Unix' "\n".

Binary mode is set properly, at least according to the return value of `setFileType()`, but after all this testing the only explanation I can come up with for this problem is that the library simply doesn't like FTP servers with Windows in binary mode.

The only thing left to test now is using FTP commands instead of `retrieve` and streams.



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]