1
\$\begingroup\$

I'm uploading videos to Vimeo using the VimeoAPI. I want to show a progress bar to the user. The only way to do this with the VimeoAPI is to call a method to the server which verifies the chunk you've just sent. I can send a video up in chunks of 8 bytes or 2 gigabytes. For the progress bar, the smaller the better, but there's a huge performance hit it keeps verifying the bytes every couple bytes.

So I am wondering, is there an optimal buffer size in which you think would be the best of both worlds? Currently I'm using a size of 10 MB. The files I upload will normally be around 750 MB or so... But some are 30 MB or so... I'm including my code just in case anyone finds it helpful.

/**
* Send the video data
*
* @return whether the video successfully sent
*/
private boolean sendVideo(String endpoint, File file) throws FileNotFoundException, IOException {
  // Setup File
  long contentLength = file.length();
  String contentLengthString = Long.toString(contentLength);
  FileInputStream is = new FileInputStream(file);
  int bufferSize = 10485760; // 10 MB = 10485760 bytes
  byte[] bytesPortion = new byte[bufferSize];
  int maxAttempts = 5; //This is the maximum attempts that will be given to resend data if the vimeo server doesn't have the right number of bytes for the given portion of the video
  long lastByteOnServer = 0;
  boolean addContentRange = false;
  while (is.read(bytesPortion, 0, bufferSize) != -1 && getStatus() != TransferObject.STOPPED) {
    //Check that this isn't the last chunk. If it is, we want to send just the last bytesPortion, not a whole other full bufferSized one
    int remainingBytes = (int) (contentLength - lastByteOnServer);
    PrinterHelper.print(StringHelper.newline + getVideoTitle() + " has " + remainingBytes + " bytes remaining.");
    if (remainingBytes < bufferSize) {
      bytesPortion = Arrays.copyOf(bytesPortion, remainingBytes);
      bufferSize = remainingBytes; // Just so it doesn't complain on the next iteration (which should break the while loop)
    }
    lastByteOnServer = prepareAndSendByteChunk(endpoint, contentLengthString, lastByteOnServer, bytesPortion, addContentRange, 0, maxAttempts);
    if (lastByteOnServer == -1) {
      return false;
    }
    addContentRange = true;
    getProgressBar().setValue(NumberHelper.getPercentFromTotal((int) lastByteOnServer, (int) getTransfer().getFileSize()));
  }
  return true;
}

/**
* Prepares the given bytes to be sent to Vimeo
*
* @param endpoint
* @param contentLengthString
* @param lastByteOnServer
* @param byteChunk
* @param addContentRange
* @param attempt
* @param maxAttempts
* @return number of bytes currently on the server
* @throws FileNotFoundException
* @throws IOException
*/
private long prepareAndSendByteChunk(String endpoint, String contentLengthString, long lastByteOnServer, byte[] byteChunk, boolean addContentRange, int attempt, int maxAttempts) throws FileNotFoundException, IOException {
  if (attempt > maxAttempts) {
    return -1;
  } else if (attempt > 0) {
    PrinterHelper.printErr("Attempt number " + attempt + " for video " + getVideoTitle());
  }
  long totalBytesShouldBeOnServer = lastByteOnServer + byteChunk.length;
  String contentRange = lastByteOnServer + "-" + totalBytesShouldBeOnServer;
  long bytesOnServer = sendVideoBytes(endpoint, contentLengthString, "video/mp4", contentRange, byteChunk, addContentRange);
  addContentRange = true;
  if (bytesOnServer != totalBytesShouldBeOnServer) {
    PrinterHelper.printErr(bytesOnServer + " (bytesOnServer)" + " != " + totalBytesShouldBeOnServer + " (totalBytesShouldBeOnServer) for " + getVideoTitle());
    long remainingBytes = totalBytesShouldBeOnServer - bytesOnServer;
    int beginning = (int) (byteChunk.length - remainingBytes);
    int ending = (int) byteChunk.length;
    byte[] newByteChunk = Arrays.copyOfRange(byteChunk, beginning, ending);
    return prepareAndSendByteChunk(endpoint, contentLengthString, bytesOnServer, newByteChunk, addContentRange, attempt + 1, maxAttempts);
  } else {
    return bytesOnServer;
  }
}

/**
* Sends the given bytes to the given endpoint
*
* @return the last byte on the server (from verifyUpload(endpoint))
*/
private long sendVideoBytes(String endpoint, String contentLength, String fileType, String contentRange, byte[] fileBytes, boolean addContentRange) throws FileNotFoundException, IOException {
  OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint);
  request.addHeader("Content-Length", contentLength);
  request.addHeader("Content-Type", fileType);
  if (addContentRange) {
    request.addHeader("Content-Range", "bytes " + contentRange + "/" + contentLength);
  }
  request.addPayload(fileBytes);
  try {
    Response response = VimeoController.signAndSend(vimeoService, vimeoAccount.getVimeoToken(), request, "sendVideo on " + getVideoTitle(), false);
    if (response.getCode() != 200 && !response.isSuccessful()) {
      return -1;
    }
  } catch (Exception ex) {
    PrinterHelper.printErr("There was an exception for video " + getVideoTitle() + " in sendVideoBytes: "
            + StringHelper.newline + ex);
    Logger.getLogger(VimeoUpload.class.getName()).log(Level.SEVERE, null, ex);
    return -1;
  }
  return verifyUpload(endpoint);
}
\$\endgroup\$
1
\$\begingroup\$

An idea: measure how fast is the user's internet connection, how many bytes they can send per ten second, for example, then use this as chunk size. It should have a minimum value which is bigger than 8 bytes because of the TCP/IP overhead. Check the overhead with a packet sniffer (Ethereal, for example).

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.