I am trying to optimize my code when downloading around 50 pdf files from a website. All of the pdf are big in size (average 25mb). Is there any way to optimize further to download the pdf faster?

for (String pdfUrl : pdfUrls) {
URLConnection request;
request = new URL(pdfUrl).openConnection();
InputStream in = request.getInputStream();
byte[] buffer = new byte[10240];
int len = 0;
int bytesBuffered = 0;
while ((len = in.read(buffer)) != -1) {
out.write(buffer, 0, len);
bytesBuffered += len;
if (bytesBuffered > 1024 * 1024) {
out.flush();
}
}
in.close();
out.close();
}

• @DebmalyaBiswas, In general it's considered bad to edit the code after answers has been given, as it might invalidate questions. We're gonna let it slide this time, as that doesn't seem to be the main issue with your code. – holroy Jan 13 '16 at 4:39

## Correctness

Files over 10240 bytes will fail to download because the streams are closed after one iteration of the while loop(Edit: Was paste error by OP). As of java 7, it is highly preferable to use a try-with-resources block when working with AutoCloseable objects.

## Implementation

Wrapping your FileOutputStream in a BufferedOutputStream would be preferable to manually flushing.

## Performance

You may see some performance gain by running each file download on its own thread. You'd need to take your existing code and move much of it into a Runnable or Callable. Use an Executor to perform some or all of the downloads concurrently.

If you made these modifications, the code might look more like:

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
import java.util.Objects;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

private final ExecutorService executor = Executors.newCachedThreadPool();

super();
}

}

private static final int BUFFER_SIZE = 1024 * 1024;

private final URL targetUrl;
private final File destination;

this.targetUrl = targetUrl;
Objects.requireNonNull(targetUrl);

this.destination = destination;
Objects.requireNonNull(destination);
}

@Override
public File call() throws IOException {
final URLConnection request = this.targetUrl.openConnection();
try (final InputStream inputStream = request.getInputStream();
final FileOutputStream fileStream = new FileOutputStream(this.destination);
final BufferedOutputStream outputStream = new BufferedOutputStream(fileStream, BUFFER_SIZE);) {

final byte[] data = new byte[10240];
}
}
return this.destination;
}
}
}

• Keep in mind that System.currentTimeMillis() might not be unique enough when requesting multiple downloads at once. – Marvin Jan 12 '16 at 20:52
• @Marvin Yes, that's a very good point. A simple counter in FileDownloader would probably be a better idea. – Eric Stein Jan 13 '16 at 1:27