Reading words and counting their occurence in Java

Question

I have very typical problem:

Given big text file, containing text. The task is to calculate occurence of each word and print this info from most occured to less.

Example input:

cat cat cat dog dog mouse

Correct answer:

cat -> 3
dog -> 2
mouse -> 1

First I solved this using Scanner. But this solutions seems to be slow, cause IO was blocking.

After some time and cookie spent I wrote this solution:

final Path path = FileSystems.getDefault().getPath("/tmp/", "file.txt");

Map<String, Long> occ =
  Files
    .lines(path)
    .flatMap(line -> Arrays.stream(line.split(delimRegex)))
    .filter(line -> line.length() > 0)
    .map(String::toLowerCase)
    .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));

occ
  .entrySet()
  .stream()
  .sorted((e1, e2) -> e2.getValue().compareTo(e1.getValue()))
  .forEach(e -> System.out.println(e.getKey() + " -> " + e.getValue()));

I think solution is good (but not sure). Is there any other optimization techniques I've ignored?

Answer 1

I have one comment: Files.lines() method has the following comment in the JDK documentation:

The returned stream encapsulates a Reader. If timely disposal of file system resources is required, the try-with-resources construct should be used to ensure that the stream's close method is invoked after the stream operations are completed.

in other words, in order to prevent resource leak, you need to construct the stream in a try-with-resources block

try (Stream<String> linesStream = Files.lines(path)) {
    Map<String, Long> occ = linesStream
        .flatMap(line -> Arrays.stream(line.split(delimRegex)))
        ....
}

Answer 2

One other comment about your code: variable names are not perfect:

  ...
  Files
    .lines(path)
    .flatMap(line -> Arrays.stream(line.split(delimRegex))) // OK, this is a line
    .filter(line -> line.length() > 0) // But this is a word
    ....

Also: consider using String::isEmpty instead of the lambda filter expression.

Answer 3

Some comments about things that you said in the question:

"First I solved this using Scanner. But this solutions seems to be slow, cause IO was blocking."

If you are reading from a file, I/O will not be blocking. It is possible that Scanner is using a buffer that is too small, but I think that is unlikely. You were probably doing something else wrong.

"I think solution is good (but not sure). Is there any other optimization techniques I've ignored?"

Using streams is not an optimization technique. Accepted wisdom is that using streams is currently a bit slower than an equivalent algorithm implemented using loops. The real advantage of streams is that they are simpler to write and understand than old-fashioned loops, especially when the transformation is complex. For example, you have implemented your problem with just two statements.

Not that I would rewrite your code. It is rarely worthwhile rewriting code to get a small performance boost. Developer time is more precious than computer time.

"After some time and cookie spent I wrote this solution:"

Here's the real problem. Too many cookies are bad for your health. Black coffee with no sugar is a much better aid for programming ... and it is better for your waistline and your teeth :-)