[jira] [Commented] (CODEC-265) java.lang.NegativeArraySizeException

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (CODEC-265) java.lang.NegativeArraySizeException

ASF GitHub Bot (Jira)

    [ https://issues.apache.org/jira/browse/CODEC-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986436#comment-16986436 ]

Alex Herbert commented on CODEC-265:
------------------------------------

Smarter memory allocation has been implemented for the Base32/Base64 codecs. They can now allocate up to the system limit for arrays.

Please check against master. [Snapshots|https://repository.apache.org/content/repositories/snapshots/commons-codec/commons-codec/1.14-SNAPSHOT/] are now deployed so you can include the 1.14 snapshot dependency for testing.

In the source there is a new test Base64Test.testCodec265 that demonstrates encoding a 1 gigabyte array. The test is memory hungry.

In a real world use case it may be worth looking at the streaming API using Base64OutputStream to write chunks of byte[] data to your target output. Even if this is a ByteArrayOutputStream you will have lower memory usage as you will not have to read the entire allocated random file to memory and you can convert the ByteArrayOutputStream to a String directly. Memory usage will still peak at around 4GiB as the String will have 2 bytes for each character:
{code:java}
String filePath = "/tmp/1gb.zip";
Path path = Paths.get(filePath);
String contentToBeSaved;
try (InputStream in = Files.newInputStream(path)) {
    // Allocate 1.5 GiB to avoid buffer reallocation
    final ByteArrayOutputStream out = new ByteArrayOutputStream(3 * (1 << 29));
    // Encode with no line separator
    final Base64OutputStream base64 = new Base64OutputStream(out, true, 0, null);
    final byte[] buffer = new byte[8192];
    int read = in.read(buffer);
    while (read >= 0) {
        base64.write(buffer, 0, read);
        read = in.read(buffer);
    }
    base64.close();
    // This will require 2.67GiB ~ 2 * 1GiB * 4/3
    contentToBeSaved = out.toString();
}
{code}
There will be a fair bit more memory usage for the JSONObject in your example. Hopefully codec is not a blocker any more and you can continue your testing.

 

>   java.lang.NegativeArraySizeException
> --------------------------------------
>
>                 Key: CODEC-265
>                 URL: https://issues.apache.org/jira/browse/CODEC-265
>             Project: Commons Codec
>          Issue Type: Bug
>    Affects Versions: 1.13
>         Environment: Linux = Ubuntu 18.04.3 LTS
> JDK = 1.8
>  
>            Reporter: Ingimar
>            Assignee: Alex Herbert
>            Priority: Critical
>         Attachments: NewClientEncodePost.java, Util.java, pom.xml
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hi,
> trying to encode a file that is 1GB of size.
> ( linux :
> {code:java}
> fallocate -l 1GB 1gb.zip{code}
> )
> I want to post that file to a RESTful-service, package in JSON.
> *here is the code*
>  
>  
> {code:java}
> String filePath = "/tmp/1gb.zip";
> System.out.println("\t Post to  : ".concat(URL));
>  System.out.println("\t file : ".concat(filePath));
> Path path = Paths.get(filePath);
>  byte[] bArray = Files.readAllBytes(path);
> // testing commons codec 1.16 (2019-11-05)
>  byte[] encodeBase64 = Base64.encodeBase64(bArray);
> final String contentToBeSaved = new String(encodeBase64);
> HttpClient client = HttpClientBuilder.create().build();
>  HttpResponse response = null;
> JSONObject metadata = new JSONObject();
>  metadata.put("owner", "Ingo");
>  metadata.put("access", "public");
>  metadata.put("licenseType", "CC BY");
>  metadata.put("fileName", "fileName");
>  metadata.put("fileDataBase64", contentToBeSaved);
> String metadataFormatted = StringEscapeUtils.unescapeJavaScript(metadata.toString());
> StringEntity entity = new StringEntity(metadataFormatted, ContentType.APPLICATION_JSON);
> HttpPost post = new HttpPost(URL);
>  post.setEntity(entity);
>  response = client.execute(post);
>  HttpEntity responseEntity = response.getEntity();
> String responseFromMediaserver = EntityUtils.toString(responseEntity, "UTF-8");
>  System.out.println("\n****");
>  System.out.println("Response is : " + responseFromMediaserver);
> JSONObject json = new JSONObject(responseFromMediaserver);
>  String uuid = json.getString("uuid");
>  System.out.println("UUID is " + uuid);
> {code}
>  
>  
>  # mvn clean package
>  #   java -Xms512m -Xmx20480m -jar target/mediaClient.jar 
> The crasch is in
>  
> {code:java}
> byte[] encodeBase64 = Base64.encodeBase64(bArray);{code}
>  
> the stacktrace is :
> {code:java}
>  
> Starting NewClientEncodePost
>  Post to : http://127.0.0.1:8080/MediaServerResteasy/media
>  file : /tmp/1gb.zip
> Exception in thread "main" java.lang.NegativeArraySizeException
>  at org.apache.commons.codec.binary.BaseNCodec.resizeBuffer(BaseNCodec.java:253)
>  at org.apache.commons.codec.binary.BaseNCodec.ensureBufferSize(BaseNCodec.java:269)
>  at org.apache.commons.codec.binary.Base64.encode(Base64.java:380)
>  at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:451)
>  at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:430)
>  at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:679)
>  at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:642)
>  at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:623)
>  at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:556)
>  at se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.posting(NewClientEncodePost.java:55)
>  at se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.main(NewClientEncodePost.java:38)
>  
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)