My program works with two types of files. File 1 contains 500,000 distinct words. File set 2 contains 173 text files, each containing 500 paragraphs, that I scraped from Wikipedia. The program counts how many times each word from the first file appears in the second set of files.
The main problem I have is that it's taking around 4 seconds per word to process so it will take around 24 days to complete all 500k words in a core5 7th gen 8gb ram laptop. Is it possible to make it more process efficient?
I am still learning Java so my knowledge is not that vast. I am using Java 8, with IntelliJ as my IDE.
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.*;
public class Main {
public static void main(String[] args) {
//This is the map that will contain each word
Map<String, Integer> map = new HashMap<>();
//int that will count how manny times the word is in the File set 2
int wordCounter = 0;
//List that contain arround 500k unrepeted words
List<String> list = new ArrayList<>();
//List that contains the current file words
List<String> list1 = new ArrayList<>();
try {
//scans the file that contains the 500k unrepeted words
Scanner s = new Scanner(new File("C:\\Users\\filepath"));
//while loop that add the words to a list so it can manipulate it latter on
while (s.hasNext()) {
list.add(s.next());
}
//random output to see the Set size
System.out.println(list.size());
//main loop that will cheek each word in the 500k file
for (int i = 0; i < list.size(); i++) {
//loop to se each file of words
for (int j = 0; j < 100; j++) {
try {
//read each file
Scanner d = new Scanner(new File("C:\\Users\\filepath" + j));
//add the information of each file
while (d.hasNext()) {
list1.add(d.next());
}
d.close();
//this code counts the number of words in all the files a
wordCounter = wordCounter + Collections.frequency(list1, list.get(i).toLowerCase());
//clears the list so it has more space and not run out of it
list1.clear();
} catch (IOException k) {
k.printStackTrace();
}
}
//adds the information to the map
map.put(list.get(i), wordCounter);
//this sorts the information and discard the words that only has 1 or less matches
if (wordCounter > 1) {
try {
FileWriter fw = new FileWriter("C:\\Users\\filePath", true);
PrintWriter pw = new PrintWriter(fw);
pw.append("\n");
pw.append(map.toString());
pw.close();
} catch (IOException f) {
f.printStackTrace();
}
}
//this clean the map so it doesnt run out of memory
map.clear();
//resets the counter to 0
wordCounter = 0;
//simple display so it seems nice
System.out.println(i);
}
} catch (IOException f){
f.printStackTrace();
}
}
}
Somewhere I read that because of Java using a virtual machine it makes the processing of data much slower. Would that be something to consider?
2 147 483 639
items? So unless your paragraphs have over 4.2M words each, you should be able to read an entire article into a single array, right? \$\endgroup\$ – Raystafarian Feb 26 '18 at 23:40