6
\$\begingroup\$

I'm trying to extract specific files and visualise them in a list/grid. For that purpose I'm starting a Service to do the background job, and when it finishes it fires an Intent which will be captured by a Fragment/Activity. The files are extracted in an singletone container which feeds the adapter when the broadcast is received.

I want to do this in the fastest way possible, so I'm using many threads.

The service:

import android.app.Service;
import android.content.Intent;
import android.os.Environment;
import android.os.IBinder;
import android.support.v4.content.LocalBroadcastManager;

import java.io.File;
import java.io.FileFilter;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicInteger;

public class ExtractFilesService extends Service {

    public static String FILE_SCAN_COMPLETE = "ExtractFilesService#FILE_SCAN_COMPLETE";

    private CopyOnWriteArrayList<File> extractedFilesList;

    private ExecutorService threadPool;

    private AtomicInteger threadCounter = new AtomicInteger(0);

    private FileFilter mFilenameFilter = new FileFilter() {
        @Override
        public boolean accept(File pathname) {
            return pathname.isDirectory()
                    || pathname.getName().endsWith(".pdf")
                    || pathname.getName().endsWith(".epub");
        }
    };

    @Override
    public IBinder onBind(Intent intent) {
        return null;
    }

    @Override
    public int onStartCommand(Intent intent, int flags, int startId) {

        extractedFilesList = new CopyOnWriteArrayList<>();

        // What is the optimal pool size ?
        // It should differ on various devices, so
        // how to adjust the value based on the device?
        // Are there any other best practices for executing unknown number of threads simultaneously?
        threadPool = Executors.newFixedThreadPool(10);

        // getting the root directory and start to iterate recursively
        extractFiles(Environment.getExternalStorageDirectory());

        return super.onStartCommand(intent, flags, startId);
    }

    private void extractFiles(final File dir) {

        // Counting the threads so we know when the last finishes
        threadCounter.incrementAndGet();

        threadPool.execute(new Runnable() {
            @Override
            public void run() {

                File[] files = dir.listFiles(mFilenameFilter);

                for (File file : files) {
                    if (file.isDirectory()) {
                        extractFiles(file); // recursive call
                    } else {
                        extractedFilesList.add(file); // add to collection
                    }
                }

                if (threadCounter.decrementAndGet() == 0) {
                    onScanEnd();
                }
            }
        });
    }

    private void onScanEnd() {
        ExtractedFilesManager.getInstance().initAll(extractedFilesList);
        LocalBroadcastManager.getInstance(this).sendBroadcast(new Intent(FILE_SCAN_COMPLETE));
        stopSelf();
    }

}

And here is the singletone container:

import java.io.File;
import java.util.ArrayList;
import java.util.concurrent.CopyOnWriteArrayList;

/**
 * Created by tochkov.
 */
public class ExtractedFilesManager {

    private ArrayList<File> fileList;

    private ExtractedFilesManager() {
        fileList = new ArrayList<>();
    }

    private static ExtractedFilesManager instance;

    public static ExtractedFilesManager getInstance() {
        if (instance == null) {
            instance = new ExtractedFilesManager();
        }
        return instance;
    }

    public void initAll(CopyOnWriteArrayList<File> newFiles) {
        fileList.clear();
        fileList.addAll(newFiles);
    }

    public ArrayList<File> getFiles() {
        return fileList;
    }

    public void sortByCriteria() {
        //TODO
    }

    public void deleteFileFromDevice() {
        //TODO
    }

    public void getInfoForFile() {
        //TODO
    }

}

  • What can go wrong?
  • Is there a static optimal threadPool size for Android? How to adjust it on different devices?
  • Am I using the executor properly?
  • What are the best practices for executing unknown number of threads simultaneously with some kind of a callback when they are all done?
  • Is there a way to make it even faster?

EDIT:

I tested this on a low-end phone and found a small problem. My onScanEnd() method is actually called several times. Everything is working fine, but apparently my threadCounter logic is wrong. Am I using AtomicInteger wrong? How can I implement something like a callback to know that all threads have finished?

\$\endgroup\$
  • \$\begingroup\$ Have you benchmarked this? How does it compare to, say, finding files on a single thread? \$\endgroup\$ – Stop ongoing harm to Monica Sep 25 '15 at 16:00
  • \$\begingroup\$ Yes. Single thread is more than 80% slower than this solution. Tested on Nexus 5 with relatively crowded file-system. \$\endgroup\$ – tochkov Sep 26 '15 at 14:32
4
\$\begingroup\$

I was dubious multi-threading would speed up anything since you are just accessing the file system and using multiple threads on the file system is usually useless since there is only one file system and it is very slow. However I realized that you are only requesting a tiny amount of data at each call (the names of files or sub-directories in the current directory) and what is costly is the time you have to wait for an answer.

I see that multi-threading is useful because you can ask questions about many directories "at the same time" rather than having to wait for the answer for one directory before proceeding to another directory. I would say you can use many threads (much more than the number of processor cores), since those threads will just be waiting for an answer 99.9999% of the time (especially if you don't use CopyOnWriteArrayList, as pointed out by @janos). Note that multi-threading would be useful here even if the cpu has a single core. You should try benchmarking with various number of threads.

As for your bug where you see onScanEnd called many times, you should move threadCounter.incrementAndGet() at the very start of extractFiles instead of calling it in the Runnable. The problem is that it takes a long time for the Runnable to get started on another thread, so you can have situations where threadCounter.decrementAndGet() == 0, but the program has not really finished: There can be a few Runnables that have been queued up but have not started and have therefore not incremented threadCounter yet.

Also, you should make sure you never follow symbolic links because you will end up scanning the same directories/files many times. You should try to see if this can happen on the Android file system.

\$\endgroup\$
  • \$\begingroup\$ Great answer man, TYVM! As for the number of threads - when they are too much the code becomes slower (for example without using a thread pool), so there is definately an optimal value. I presume that this happens because managment of the threads is more complex than the work inside a single thread. Do you think something like CORES * 10 would be a good solution? \$\endgroup\$ – tochkov Oct 1 '15 at 11:37
  • 1
    \$\begingroup\$ As I said, you should do some benchmark. I really don't know when it becomes too many threads. Also benchmarking is very hard on Android since there is a near infinity of devices. \$\endgroup\$ – toto2 Oct 1 '15 at 23:01
5
\$\begingroup\$

Have you read the docs of CopyOnWriteArrayList?

As the name suggest CopyOnWriteArrayList creates copy of underlying ArrayList with every mutation operation e.g. add or set. Normally CopyOnWriteArrayList is very expensive because it involves costly Array copy with every write operation but its very efficient if you have a List where Iteration outnumber mutation e.g. you mostly need to iterate the ArrayList and don't modify it too often.

That's really not a good fit for your use case. How about a ConcurrentLinkedQueue instead?

As for optimal number of threads, logically it's hard to do better than the number of CPU cores. If I remember correctly the recommended count is usually the number of cores + 1. I don't know specifically for Android, but I think the same logic should hold true. See also this related post: https://stackoverflow.com/a/13958877/641955

\$\endgroup\$
  • \$\begingroup\$ Thank you for your recommendations! I would be very grateful if you could see my edit and help me figure it out. \$\endgroup\$ – tochkov Sep 26 '15 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.