I'm trying to extract specific files and visualise them in a list/grid. For that purpose I'm starting a Service to do the background job, and when it finishes it fires an Intent which will be captured by a Fragment/Activity. The files are extracted in an singletone container which feeds the adapter when the broadcast is received.

I want to do this in the fastest way possible, so I'm using many threads.

The service:

import android.app.Service;
import android.content.Intent;
import android.os.Environment;
import android.os.IBinder;
import android.support.v4.content.LocalBroadcastManager;

import java.io.File;
import java.io.FileFilter;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;

public class ExtractFilesService extends Service {

    public static String FILE_SCAN_COMPLETE = "ExtractFilesService#FILE_SCAN_COMPLETE";

    private CopyOnWriteArrayList<File> extractedFilesList;

    private ExecutorService threadPool;

    private AtomicInteger threadCounter = new AtomicInteger(0);

    private FileFilter mFilenameFilter = new FileFilter() {
        public boolean accept(File pathname) {
            return pathname.isDirectory()
                    || pathname.getName().endsWith(".pdf")
                    || pathname.getName().endsWith(".epub");

    public IBinder onBind(Intent intent) {
        return null;

    public int onStartCommand(Intent intent, int flags, int startId) {

        extractedFilesList = new CopyOnWriteArrayList<>();

        // What is the optimal pool size ?
        // It should differ on various devices, so
        // how to adjust the value based on the device?
        // Are there any other best practices for executing unknown number of threads simultaneously?
        threadPool = Executors.newFixedThreadPool(10);

        // getting the root directory and start to iterate recursively

        return super.onStartCommand(intent, flags, startId);

    private void extractFiles(final File dir) {

        // Counting the threads so we know when the last finishes

        threadPool.execute(new Runnable() {
            public void run() {

                File[] files = dir.listFiles(mFilenameFilter);

                for (File file : files) {
                    if (file.isDirectory()) {
                        extractFiles(file); // recursive call
                    } else {
                        extractedFilesList.add(file); // add to collection

                if (threadCounter.decrementAndGet() == 0) {

    private void onScanEnd() {
        LocalBroadcastManager.getInstance(this).sendBroadcast(new Intent(FILE_SCAN_COMPLETE));


And here is the singletone container:

import java.io.File;
import java.util.ArrayList;
import java.util.concurrent.CopyOnWriteArrayList;

 * Created by tochkov.
public class ExtractedFilesManager {

    private ArrayList<File> fileList;

    private ExtractedFilesManager() {
        fileList = new ArrayList<>();

    private static ExtractedFilesManager instance;

    public static ExtractedFilesManager getInstance() {
        if (instance == null) {
            instance = new ExtractedFilesManager();
        return instance;

    public void initAll(CopyOnWriteArrayList<File> newFiles) {

    public ArrayList<File> getFiles() {
        return fileList;

    public void sortByCriteria() {

    public void deleteFileFromDevice() {

    public void getInfoForFile() {


  • What can go wrong?
  • Is there a static optimal threadPool size for Android? How to adjust it on different devices?
  • Am I using the executor properly?
  • What are the best practices for executing unknown number of threads simultaneously with some kind of a callback when they are all done?
  • Is there a way to make it even faster?


I tested this on a low-end phone and found a small problem. My onScanEnd() method is actually called several times. Everything is working fine, but apparently my threadCounter logic is wrong. Am I using AtomicInteger wrong? How can I implement something like a callback to know that all threads have finished?

  • \$\begingroup\$ Have you benchmarked this? How does it compare to, say, finding files on a single thread? \$\endgroup\$
    – janos
    Commented Sep 25, 2015 at 16:00
  • \$\begingroup\$ Yes. Single thread is more than 80% slower than this solution. Tested on Nexus 5 with relatively crowded file-system. \$\endgroup\$
    – tochkov
    Commented Sep 26, 2015 at 14:32

2 Answers 2


I was dubious multi-threading would speed up anything since you are just accessing the file system and using multiple threads on the file system is usually useless since there is only one file system and it is very slow. However I realized that you are only requesting a tiny amount of data at each call (the names of files or sub-directories in the current directory) and what is costly is the time you have to wait for an answer.

I see that multi-threading is useful because you can ask questions about many directories "at the same time" rather than having to wait for the answer for one directory before proceeding to another directory. I would say you can use many threads (much more than the number of processor cores), since those threads will just be waiting for an answer 99.9999% of the time (especially if you don't use CopyOnWriteArrayList, as pointed out by @janos). Note that multi-threading would be useful here even if the cpu has a single core. You should try benchmarking with various number of threads.

As for your bug where you see onScanEnd called many times, you should move threadCounter.incrementAndGet() at the very start of extractFiles instead of calling it in the Runnable. The problem is that it takes a long time for the Runnable to get started on another thread, so you can have situations where threadCounter.decrementAndGet() == 0, but the program has not really finished: There can be a few Runnables that have been queued up but have not started and have therefore not incremented threadCounter yet.

Also, you should make sure you never follow symbolic links because you will end up scanning the same directories/files many times. You should try to see if this can happen on the Android file system.

  • \$\begingroup\$ Great answer man, TYVM! As for the number of threads - when they are too much the code becomes slower (for example without using a thread pool), so there is definately an optimal value. I presume that this happens because managment of the threads is more complex than the work inside a single thread. Do you think something like CORES * 10 would be a good solution? \$\endgroup\$
    – tochkov
    Commented Oct 1, 2015 at 11:37
  • 1
    \$\begingroup\$ As I said, you should do some benchmark. I really don't know when it becomes too many threads. Also benchmarking is very hard on Android since there is a near infinity of devices. \$\endgroup\$
    – toto2
    Commented Oct 1, 2015 at 23:01

Have you read the docs of CopyOnWriteArrayList?

As the name suggest CopyOnWriteArrayList creates copy of underlying ArrayList with every mutation operation e.g. add or set. Normally CopyOnWriteArrayList is very expensive because it involves costly Array copy with every write operation but its very efficient if you have a List where Iteration outnumber mutation e.g. you mostly need to iterate the ArrayList and don't modify it too often.

That's really not a good fit for your use case. How about a ConcurrentLinkedQueue instead?

As for optimal number of threads, logically it's hard to do better than the number of CPU cores. If I remember correctly the recommended count is usually the number of cores + 1. I don't know specifically for Android, but I think the same logic should hold true. See also this related post: https://stackoverflow.com/a/13958877/641955

  • \$\begingroup\$ Thank you for your recommendations! I would be very grateful if you could see my edit and help me figure it out. \$\endgroup\$
    – tochkov
    Commented Sep 26, 2015 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.