Merging word streams from files

Question

A follow-on, rags-to-riches implementation of The most efficient way to merge two lists in Java

The original requirements are to:

Identify the distinct values from two input files, and output the distinct values to an output file. There is no specification for the order of the output, only that each line should be unique in the results.

Special consideration should be made for efficiency.

I have implemented a more general specification:

1. merge multiple input files (at least one) to an output file
2. each line is treated as a line, not necessarily a "word". If the input files have just one word per line, then the output would be the same as the original specification.
3. take the input files from the commandline (the first file is the output file).

In my answer to the linked post I suggested that a Java 8 Streams implementation would be "nice". I have implemented that solution here. I am looking for suggestions on how to better utilize the new Java functionality, and any other suggestions you may have.

import java.io.BufferedWriter;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashSet;
import java.util.Set;
import java.util.stream.Stream;


@SuppressWarnings("javadoc")
public class Linemerge {

    /* Wrap the IOException in order to make convenient Stream usage. */
    private static final void writeWord(BufferedWriter writer, String word) {
        try {
            writer.write(word);
            writer.newLine();
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }

    private static void merge(Path source, Set<String> seen, BufferedWriter writer) throws IOException {
        try (Stream<String> words = Files.lines(source)) {
            words.filter(seen::add).forEach(word -> writeWord(writer, word));
        }
    }

    public static void main(String[] args) {
        if (args.length < 2) {
            throw new IllegalArgumentException("Need at least two file arguments: Destination Source {Source {Source {...}}}");
        }
        Path dest = Paths.get(args[0]);
        try (BufferedWriter writer  = Files.newBufferedWriter(dest)) {
            Set<String> seen = new HashSet<>();
            for (int i = 1; i < args.length; i++) {
               Path source = Paths.get(args[i]);
               if (Files.isRegularFile(source) && Files.isReadable(source)) {
                   System.out.println("Merging " + source);
                   merge(source, seen, writer);
               } else {
                   System.out.println("Unable to read (and ignoring) " + source);
               }
            }
        } catch (IOException e) {
            e.printStackTrace();
            System.exit(1);
        }
    }

}

Answer 1

"Need at least two file arguments: Destination Source {Source {Source {...}}}"

I think an easier way of documenting that, at least for *nix, is:

"Need at least two file arguments: DESTINATION [SOURCE]..."

You can also turn the Paths in your main() method into a Stream too:

public class Linemerge {
    // ...

    // suggestion note: had to wrap IOException -> UncheckedIOException too
    private static void merge(Path source, Set<String> seen, BufferedWriter writer) {
        try (Stream<String> words = Files.lines(source)) {
            words.filter(seen::add).forEach(word -> writeWord(writer, word));
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }

    private static final Predicate<Path> FILTER =
            f -> Files.isRegularFile(f) && Files.isReadable(f);

    private static void checkPath(Path path) {
        System.out.println((FILTER.test(path) ? "Merging"
                : "Unable to read (and ignoring)") + " " + path);
    }

    public static void main(String[] args) {
        if (args.length < 2) {
            throw new IllegalArgumentException(
                    "Need at least two file arguments: DESTINATION [SOURCE]...");
        }
        try (BufferedWriter writer = Files.newBufferedWriter(Paths.get(args[0]))) {
            Set<String> seen = new HashSet<>();
            Stream.of(args).skip(1).map(Paths::get).peek(Linemerge::checkPath)
                    .filter(FILTER).forEach(f -> merge(f, seen, writer));
        } catch (IOException e) {
            e.printStackTrace();
            System.exit(1);
        }
    }
}

Answer 2

If you're new to using Files.lines(), note that java.nio.* character I/O routines default to UTF-8, whereas java.io.* character I/O routines default to the local system's default encoding, so plan accordingly.

I don't know where your source files are coming from, but if they are not created by you, it is not uncommon to find UTF-8 encoded files that begin with a Byte-Order Mark, which for UTF-8 appears in the first three bytes in the file with values EF BB BF. If that is a possibility, you may want to copy the source code for 'Files.lines()' and use your own BufferedReader based on org.apache.commons.io.input.BOMInputStream

Since you are using a Set as a filter, the set will grow to contain all of the unique words. If you create one stream out of the multiple files, you could then use stream.collect(toSet()) and accomplish the same thing. You could get everything into one stream in two ways:

• Create an array containing each file's Stream, then create a single Stream from it by calling Stream.concat(myStreamsArray)
• Create a Stream<Path> containing the file paths, then create a single Stream of all of the lines by calling pathStream.flatMap(Files::lines)