I am trying to read a large file ~200 MB (~300 million lines of text). I am using a relatively standard way of reading like:
for (int i = 1; i <= numProcs; i++) {
System.out.println("Parsing " + traceFilePrefix + i + ".prg");
System.out.println("##########################");
try {
String line;
int instrType;
String hexAddr;
ArrayList<Trace> traces = new ArrayList<Trace>();
int lineNum = 1;
BufferedReader br = new BufferedReader(new FileReader(traceFilePrefix + numProcs + "/" + traceFilePrefix + i + ".PRG"));
while ((line = br.readLine()) != null) {
line = line.toUpperCase();
// validate line format
if (line.matches(regexLineFormat)) {
StringTokenizer tokenizer = new StringTokenizer(line);
instrType = Integer.parseInt(tokenizer.nextToken());
hexAddr = tokenizer.nextToken();
traces.add(new Trace(instrType, hexAddr));
} else {
// line is in invalid format
System.err.println("ERR: line " + lineNum + " is invalid \"" + line + "\"");
}
lineNum++;
if (lineNum % 1000 == 0) System.out.println("line " + lineNum);
}
br.close();
processorsTraces.add(i-1, traces);
} catch (Exception e) {
e.printStackTrace();
}
}
I find that reading slows down at about line 20 million... in face it doesnt seem to progress at all ... how can I improve this? Whats the problem in this simple piece of code?
A latest run seem to run out of heap space ... I guess I cant store data in an arraylist like that?
This is supposed to be a simulator. It first parses trace files containing hexadecimal addresses of memory accesses. Then is supposed to run simulation of these data. This code snipplet only parses and stores the traces in an arraylist.
traces.add
and see if you title is correct. I feel that reading is not a problem at all. It should be a memory, an issue of storing the millions of small objects. Actually, the average size of your lines is 300/200 = 1.5 bytes per item. The overhead will be huge. Since java performance is based on JIT, it does not care about memory efficiency and you will have about 100 bytes overhead per evert 1.5 bytes of real data. You'll need about 30 GB to store all your 300 mln items. Java is not suitable for this task and you have a wrong title. \$\endgroup\$