Word counter using a word list and some text files

Question

My program works with two types of files. File 1 contains 500,000 distinct words. File set 2 contains 173 text files, each containing 500 paragraphs, that I scraped from Wikipedia. The program counts how many times each word from the first file appears in the second set of files.

The main problem I have is that it's taking around 4 seconds per word to process so it will take around 24 days to complete all 500k words in a core5 7th gen 8gb ram laptop. Is it possible to make it more process efficient?

I am still learning Java so my knowledge is not that vast. I am using Java 8, with IntelliJ as my IDE.

import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.*;

public class Main {

public static void main(String[] args) {

    //This is the map that will contain each word
    Map<String, Integer> map = new HashMap<>();
    //int that will count how manny times the word is in the File set 2
    int wordCounter = 0;
    //List that contain arround 500k unrepeted words
    List<String> list = new ArrayList<>();
    //List that contains the current file words
    List<String> list1 = new ArrayList<>();


    try {
        //scans the file that contains the 500k unrepeted words
        Scanner s = new Scanner(new File("C:\\Users\\filepath"));
        //while loop that add the words to a list so it can manipulate it latter on
        while (s.hasNext()) {
            list.add(s.next());
        }
        //random output to see the Set size
        System.out.println(list.size());


        //main loop that will cheek each word in the 500k file
        for (int i = 0; i < list.size(); i++) {
            //loop to se each file of words
            for (int j = 0; j < 100; j++) {
                try {
                    //read each file
                    Scanner d = new Scanner(new File("C:\\Users\\filepath" + j));
                    //add the information of each file
                    while (d.hasNext()) {
                        list1.add(d.next());
                    }
                    d.close();
                    //this code counts the number of words in all the files a
                    wordCounter = wordCounter + Collections.frequency(list1, list.get(i).toLowerCase());
                    //clears the list so it has more space and not run out of it
                    list1.clear();

                } catch (IOException k) {
                    k.printStackTrace();
                }
            }
            //adds the information to the map
            map.put(list.get(i), wordCounter);
            //this sorts the information and discard the words that only has 1 or less matches
            if (wordCounter > 1) {
                try {
                    FileWriter fw = new FileWriter("C:\\Users\\filePath", true);
                    PrintWriter pw = new PrintWriter(fw);
                    pw.append("\n");
                    pw.append(map.toString());
                    pw.close();

                } catch (IOException f) {
                    f.printStackTrace();
                }
            }
            //this clean the map so it doesnt run out of memory
                map.clear();
            //resets the counter to 0
                wordCounter = 0;
                //simple display so it seems nice
            System.out.println(i);

        }
    } catch (IOException f){
            f.printStackTrace();
        }



        }

    }

Somewhere I read that because of Java using a virtual machine it makes the processing of data much slower. Would that be something to consider?

Answer 1

You should try to switch inner and outer loop, because it will be much faster to read each wikipedia articles in and count the word frequencies for all 500k words (you anyway have the 500k word list in memory all the time). What you do now is reading 500k times all articles into memory which is time consuming.

To sum up all usages of a word, you can use the map which already exists. Just read out the current sum for a given word, add the occurrences in the current article and write it back to the map. Right now you are just writing one entry to the map, convert it to a string and clear it right afterwards. I assume, you had the idea in mind to do it how I described.

Don't worry about Java execution speed in general, because the code will get compiled (just-in-time-compilation) to machine code eventually.

Answer 2

First of all you should really use more meaningful names for your variables.

wordCount says a lot more than map, uniqueWords a lot more than list and same for wordsInCurrentFile instead of list1.

Just renaming these makes it so much easier to follow what your program is doing.

---

You should follow Markus's advice to flip the loops. In the outer loop, you should iterate over each file. And then for each file, count the occurences for each of the words.

We can also optimise the use of our variables a bit so we don't even need the 2 lists at all.

Here's the main idea:

Map<String, int> wordCount = new HashMap<>();
File wordFile = new File( ...); //open the file with unique words
while(wordFile.hasNext()){
    wordCount.put(wordFile.next(), 0);
}

At this point we got an entry in our count map for each of the 500k words.

for( each file with wiki text) {

loop over each file in outer loop. Either the same way, opening the file like you did now based on a fixed name and number appended. Or by putting all the files in a specific directory and in java iterate over all files you find in that directory.

    while(file.hasNext()){
        String word = file.next();

At this point we're looping over each word in the file and want to update the total count.

        Map<String, Integer> wordCount = new HashMap<>();
        if (wordCount.containsKey(word)) {
            wordCount.put(word, wordCount.get(word)+1);
        }
    }
}

---

It may also be worth looking up how to list all files in a directory in java 8 (I'm not familiar with this myself yet).

And it's also a good idea to look up how to use a try-with-resources since java 7 which helps you handling closing the file after you're done with it. The way you wrote it now doesn't properly close the file if you encounter an error.