Very Large File Uploads

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Very Large File Uploads

serff
Hello.  I have seen a few threads about Large file uploads on the
mailing list, but most people are saying Large means ~10MB.  I'm trying
to upload files larger than 2 GB using FileUpload and HttpClient.  I
know some people are going to say "why aren't you using ftp for that?"
but I have my reasons, so please try to help me with my problems with
FileUpload.  It seems to work ok for files up to about 1GB, but once I'm
beyond that, it just hangs forever.  Has anyone had luck doing this with
FileUpload?  Also, the weird thing is that it seems like the whole file
gets there, but parseRequest never returns.  And yes, I've let it sit
for hours...

This is basically what I have in my doPost method of my servlet:
public void doPost(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
        String tempStore = System.getProperty("java.iot.tmpdir");
        int maxMemorySize = 10000;
        ServletFileUpload sfu = new ServletFileUpload(new
DiskFileItemFactory(maxMemorySize, new File(tempStore)));
        sfu.setSizeMax(-1);
        List fileItems = sfu.parseRequest(request);
       
        Iterator itr = fileItems.iterator();
       
        while(itr.hasNext()) {
            FileItem fi = (FileItem)itr.next();
            if (!fi.isFormField()) {
                File file = new File(tempStore + fi.getName());
                fi.write(file);
            }
        }
    }

Is there another why I should be doing this?

I have one other question too.  Is there a way to parse the request with
out accepting the full multipart content?  I'm using the HttpClient to
do this upload, and I'm setting a few StringParts in the request so that
I can authorize the user to upload.  Doing it the way listed above, I
have to accept the entire request (i.e. they upload the whole file)
because I get those StringParts.  Should I be putting those parameters
somewhere other than the PostMethod I'm sending in the executeMethod of
HttpClient?

Any help you all can give would be great!  Thanks!
Andrew


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Very Large File Uploads

Martin Cooper-3
On 8/25/06, Andrew <[hidden email]> wrote:
>
> Hello.  I have seen a few threads about Large file uploads on the
> mailing list, but most people are saying Large means ~10MB.  I'm trying
> to upload files larger than 2 GB using FileUpload and HttpClient.  I
> know some people are going to say "why aren't you using ftp for that?"
> but I have my reasons, so please try to help me with my problems with
> FileUpload.  It seems to work ok for files up to about 1GB, but once I'm
> beyond that, it just hangs forever.  Has anyone had luck doing this with
> FileUpload?


I've done uploads of between 1GB and 2GB before without problems. One thing
to note, though, is that the Servlet API uses an int for content length, so
problems can show up once the size is bigger than an int.

  Also, the weird thing is that it seems like the whole file
> gets there, but parseRequest never returns.  And yes, I've let it sit
> for hours...
>
> This is basically what I have in my doPost method of my servlet:
> public void doPost(HttpServletRequest request, HttpServletResponse
> response) throws ServletException, IOException {
>        String tempStore = System.getProperty("java.iot.tmpdir");


Typo in the mail message, or typo in your code? I assume you mean '
java.io.tmpdir'.

       int maxMemorySize = 10000;

>        ServletFileUpload sfu = new ServletFileUpload(new
> DiskFileItemFactory(maxMemorySize, new File(tempStore)));
>        sfu.setSizeMax(-1);
>        List fileItems = sfu.parseRequest(request);
>
>        Iterator itr = fileItems.iterator();
>
>        while(itr.hasNext()) {
>            FileItem fi = (FileItem)itr.next();
>            if (!fi.isFormField()) {
>                File file = new File(tempStore + fi.getName());
>                fi.write(file);
>            }
>        }
>    }
>
> Is there another why I should be doing this?


It looks OK, for the most part. However, you shouldn't use getName()
blindly, as you are, since some browsers will include the path to the
original file name in that value.

I have one other question too.  Is there a way to parse the request with
> out accepting the full multipart content?  I'm using the HttpClient to
> do this upload, and I'm setting a few StringParts in the request so that
> I can authorize the user to upload.  Doing it the way listed above, I
> have to accept the entire request (i.e. they upload the whole file)
> because I get those StringParts.  Should I be putting those parameters
> somewhere other than the PostMethod I'm sending in the executeMethod of
> HttpClient?


I'm not sure I understand what you want to do. Are you saying you want to
drop the uploaded file on the floor, but keep the other parameters? If so,
you might be able to do that with the new streaming API that Jochen added.
I'm not familiar with it, though, so perhaps he can chime in here.

--
Martin Cooper


Any help you all can give would be great!  Thanks!
> Andrew
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Very Large File Uploads

serff
Thanks for the info.  The int for content length is definitely something
I haven't thought of.  And yes, it was a type. I didn't actually have my
code with me when I wrote that email, so I was writing it on the fly in
the email.  :)

So for getName, you recommend I parse the contents for "/" or "\"?  Or
is there some other way to get the name from the filepart?

As for the last question, I would like to be able to check on every post
that the user is allowed to do the upload.  So first thing I would do it
check their credentials and then process the upload.  If they aren't
allowed to, I would like to stop them before they even begin to do the
upload.

I'd be interested to hear about this streaming API.

Thanks again for the help.
Andrew

Martin Cooper wrote:

> On 8/25/06, Andrew <[hidden email]> wrote:
>>
>> Hello.  I have seen a few threads about Large file uploads on the
>> mailing list, but most people are saying Large means ~10MB.  I'm trying
>> to upload files larger than 2 GB using FileUpload and HttpClient.  I
>> know some people are going to say "why aren't you using ftp for that?"
>> but I have my reasons, so please try to help me with my problems with
>> FileUpload.  It seems to work ok for files up to about 1GB, but once I'm
>> beyond that, it just hangs forever.  Has anyone had luck doing this with
>> FileUpload?
>
>
> I've done uploads of between 1GB and 2GB before without problems. One
> thing
> to note, though, is that the Servlet API uses an int for content
> length, so
> problems can show up once the size is bigger than an int.
>
>  Also, the weird thing is that it seems like the whole file
>> gets there, but parseRequest never returns.  And yes, I've let it sit
>> for hours...
>>
>> This is basically what I have in my doPost method of my servlet:
>> public void doPost(HttpServletRequest request, HttpServletResponse
>> response) throws ServletException, IOException {
>>        String tempStore = System.getProperty("java.iot.tmpdir");
>
>
> Typo in the mail message, or typo in your code? I assume you mean '
> java.io.tmpdir'.
>
>       int maxMemorySize = 10000;
>>        ServletFileUpload sfu = new ServletFileUpload(new
>> DiskFileItemFactory(maxMemorySize, new File(tempStore)));
>>        sfu.setSizeMax(-1);
>>        List fileItems = sfu.parseRequest(request);
>>
>>        Iterator itr = fileItems.iterator();
>>
>>        while(itr.hasNext()) {
>>            FileItem fi = (FileItem)itr.next();
>>            if (!fi.isFormField()) {
>>                File file = new File(tempStore + fi.getName());
>>                fi.write(file);
>>            }
>>        }
>>    }
>>
>> Is there another why I should be doing this?
>
>
> It looks OK, for the most part. However, you shouldn't use getName()
> blindly, as you are, since some browsers will include the path to the
> original file name in that value.
>
> I have one other question too.  Is there a way to parse the request with
>> out accepting the full multipart content?  I'm using the HttpClient to
>> do this upload, and I'm setting a few StringParts in the request so that
>> I can authorize the user to upload.  Doing it the way listed above, I
>> have to accept the entire request (i.e. they upload the whole file)
>> because I get those StringParts.  Should I be putting those parameters
>> somewhere other than the PostMethod I'm sending in the executeMethod of
>> HttpClient?
>
>
> I'm not sure I understand what you want to do. Are you saying you want to
> drop the uploaded file on the floor, but keep the other parameters? If
> so,
> you might be able to do that with the new streaming API that Jochen
> added.
> I'm not familiar with it, though, so perhaps he can chime in here.
>
> --
> Martin Cooper
>
>
> Any help you all can give would be great!  Thanks!
>> Andrew
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Very Large File Uploads

Martin Cooper-3
On 8/26/06, Andrew Serff <[hidden email]> wrote:
>
> Thanks for the info.  The int for content length is definitely something
> I haven't thought of.  And yes, it was a type. I didn't actually have my
> code with me when I wrote that email, so I was writing it on the fly in
> the email.  :)
>
> So for getName, you recommend I parse the contents for "/" or "\"?  Or
> is there some other way to get the name from the filepart?


There's a FAQ entry that shows a few lines of code:

http://jakarta.apache.org/commons/fileupload/faq.html

As for the last question, I would like to be able to check on every post
> that the user is allowed to do the upload.  So first thing I would do it
> check their credentials and then process the upload.  If they aren't
> allowed to, I would like to stop them before they even begin to do the
> upload.


Ah, OK. Other than the possibility of the streaming API, you could "cheat"
and use the URL for additional parameters. The FileUpload API deals only
with the actual form content. If you add regular request parameters to the
URL you post to, you can access those using the plain old servlet API (i.e.
request.getParameter() and friends). If you don't want to allow the upload,
you can then just skip calling FileUpload at all.

--
Martin Cooper


I'd be interested to hear about this streaming API.

>
> Thanks again for the help.
> Andrew
>
> Martin Cooper wrote:
> > On 8/25/06, Andrew <[hidden email]> wrote:
> >>
> >> Hello.  I have seen a few threads about Large file uploads on the
> >> mailing list, but most people are saying Large means ~10MB.  I'm trying
> >> to upload files larger than 2 GB using FileUpload and HttpClient.  I
> >> know some people are going to say "why aren't you using ftp for that?"
> >> but I have my reasons, so please try to help me with my problems with
> >> FileUpload.  It seems to work ok for files up to about 1GB, but once
> I'm
> >> beyond that, it just hangs forever.  Has anyone had luck doing this
> with
> >> FileUpload?
> >
> >
> > I've done uploads of between 1GB and 2GB before without problems. One
> > thing
> > to note, though, is that the Servlet API uses an int for content
> > length, so
> > problems can show up once the size is bigger than an int.
> >
> >  Also, the weird thing is that it seems like the whole file
> >> gets there, but parseRequest never returns.  And yes, I've let it sit
> >> for hours...
> >>
> >> This is basically what I have in my doPost method of my servlet:
> >> public void doPost(HttpServletRequest request, HttpServletResponse
> >> response) throws ServletException, IOException {
> >>        String tempStore = System.getProperty("java.iot.tmpdir");
> >
> >
> > Typo in the mail message, or typo in your code? I assume you mean '
> > java.io.tmpdir'.
> >
> >       int maxMemorySize = 10000;
> >>        ServletFileUpload sfu = new ServletFileUpload(new
> >> DiskFileItemFactory(maxMemorySize, new File(tempStore)));
> >>        sfu.setSizeMax(-1);
> >>        List fileItems = sfu.parseRequest(request);
> >>
> >>        Iterator itr = fileItems.iterator();
> >>
> >>        while(itr.hasNext()) {
> >>            FileItem fi = (FileItem)itr.next();
> >>            if (!fi.isFormField()) {
> >>                File file = new File(tempStore + fi.getName());
> >>                fi.write(file);
> >>            }
> >>        }
> >>    }
> >>
> >> Is there another why I should be doing this?
> >
> >
> > It looks OK, for the most part. However, you shouldn't use getName()
> > blindly, as you are, since some browsers will include the path to the
> > original file name in that value.
> >
> > I have one other question too.  Is there a way to parse the request with
> >> out accepting the full multipart content?  I'm using the HttpClient to
> >> do this upload, and I'm setting a few StringParts in the request so
> that
> >> I can authorize the user to upload.  Doing it the way listed above, I
> >> have to accept the entire request (i.e. they upload the whole file)
> >> because I get those StringParts.  Should I be putting those parameters
> >> somewhere other than the PostMethod I'm sending in the executeMethod of
> >> HttpClient?
> >
> >
> > I'm not sure I understand what you want to do. Are you saying you want
> to
> > drop the uploaded file on the floor, but keep the other parameters? If
> > so,
> > you might be able to do that with the new streaming API that Jochen
> > added.
> > I'm not familiar with it, though, so perhaps he can chime in here.
> >
> > --
> > Martin Cooper
> >
> >
> > Any help you all can give would be great!  Thanks!
> >> Andrew
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>