Skip to main content
deleted 103 characters in body; edited title
Source Link
Jamal
  • 34.9k
  • 13
  • 133
  • 237

What are major flaws of my multi-threading implementations? Processing files in directories with multithreading

I moved this question to here from StackOverFlow for comments and suggestions. I am processing files in directories. I want to use multi-threading in two ways:

  1. 10 threads to process files/folders concurrently.
  2. 10 threads to process all lines in each file concurrently

In this code below, FolderProcessorFolderProcessor implements (1), and DocProcessorDocProcessor implements (2). Is there any flaw in this implementation? My testing seems to show that (2) works and reduces time a lot, but (1) doesn't really reduce the processing time. Please comment on this.

What are major flaws of my multi-threading implementations?

I moved this question to here from StackOverFlow for comments and suggestions. I am processing files in directories. I want to use multi-threading in two ways:

  1. 10 threads to process files/folders concurrently.
  2. 10 threads to process all lines in each file concurrently

In code below, FolderProcessor implements (1), and DocProcessor implements (2). Is there any flaw in this implementation? My testing seems to show that (2) works and reduces time a lot, but (1) doesn't really reduce the processing time. Please comment on this.

Processing files in directories with multithreading

I am processing files in directories. I want to use multi-threading in two ways:

  1. 10 threads to process files/folders concurrently
  2. 10 threads to process all lines in each file concurrently

In this code, FolderProcessor implements (1), and DocProcessor implements (2). Is there any flaw in this implementation? My testing seems to show that (2) works and reduces time a lot, but (1) doesn't really reduce the processing time.

edited body; edited tags
Source Link
alecxe
  • 17.3k
  • 8
  • 51
  • 93
Source Link
user697911
  • 173
  • 1
  • 1
  • 5

What are major flaws of my multi-threading implementations?

I moved this question to here from StackOverFlow for comments and suggestions. I am processing files in directories. I want to use multi-threading in two ways:

  1. 10 threads to process files/folders concurrently.
  2. 10 threads to process all lines in each file concurrently

In code below, FolderProcessor implements (1), and DocProcessor implements (2). Is there any flaw in this implementation? My testing seems to show that (2) works and reduces time a lot, but (1) doesn't really reduce the processing time. Please comment on this.

class FolderProcessor{   
      ...
 void processFolder(String inputPath, String outputPath){

            File inputFolder = new File(inputPath);
            String[] filenames = inputFolder.list();

            ExecutorService pool = Executors.newFixedThreadPool(10);

            for (String filename : filenames) {
                String filePath = inputPath + filename;
                File inputfile = new File(filePath);
                if (inputfile.isDirectory()) {
                    processFolder(filePath, outputPath + filename);
                } else {
                    pool.execute(new Runnable() {
                        public void run() {
                            log.info("Start processing " + filePath);
                            Writer.write(filePath, outputPath);
                        }
                    });
                }
            }
            pool.shutdown();
            try {
                pool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
}

class DocumentWriter{
    ...
 public static void write(String inputFile, String outputFile) {
        try {

            File test_output = new File(outputFile);
            test_output.getParentFile().mkdirs();
            test_output.createNewFile();
            FileWriter write_test_output = new FileWriter(outputFile);

            List<Document> docs = DocPrecessor.processDocs(inputFile);
            for (Document doc : docs) {
               ...
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

class DocPrecessor{
    ......
    public static List<Document> processDocs(String filePath) {

    BufferedReader br = null;
    ExecutorService pool = Executors.newFixedThreadPool(10);
    List<Document> processedDocs = new ArrayList<>();
    try {
        ...
        String line = null;
        int docNo = 0;
        List<Future<Document>> tasks = new ArrayList<>();
        while ((line = br.readLine()) != null) {
            Callable<Document> callable = new TextThread(line, ++docNo);
            tasks.add(pool.submit(callable));
        }
        for (Future<Document> task : tasks) {
            try {
                processedDocs.add(task.get());
            } catch (InterruptedException e) {
                log.error("InterruptedException Failure: " + line);
            } catch (ExecutionException e) {
                log.error("Thread ExecutionException e: " + line);
                e.printStackTrace();
            }
        }
        pool.shutdown();
        try {
            pool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            if (br != null)
                br.close();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }
    return processedDocs;
}