Processing compressed .csv files

Question

I have some .csv files compressed in .bz2 format. I need to take a subset of the records (and data) and switch to .gz.

I am not happy with the performance. Is there a more efficient way to do it?

//For each file in a folder:
try (BufferedReader br = new BufferedReader(new InputStreamReader(
        new BZip2CompressorInputStream(new FileInputStream(fileIn))));
     BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
        new GZIPOutputStream(new FileOutputStream(fileOut))))) {

    String line;
    while ((line = br.readLine()) != null) {
        String[] parts = line.split(',');
        if(isLineToSkip(parts[0])) {
            continue;
        }
        String outLine = parts[0] + "," + parts[3]; 

        writer.append(outLine);
        writer.newLine();
    }
} . . . 

Answer 1

Well, your code is neat, and you use the Try-with-resources structures well. There is one potential bug, which is that there may be lines whihc have a 'successful' part[0], but no part[3] which would cause an IndexOutOfBoundsException.

As for the performance, the key here is Amdahl's Law.... essentially parallelization.

You have five CPU intensive parts to your problem:

1. decompression
2. Stream->Reader
3. Split & Filter
4. Writer->Stream
5. Compression

Assuming the transformations that you do in each of these are similarly expensive to do computationally, then you can probably make your system go 5 times faster by doing them each in parallel. Five threads:

• one of them reads the data from the file, and decompresses that in to chunks of bytes which it feeds in to a queue.
• the second takes chunks from the queue, and decodes it in to characters (UTF-8?), which it puts in to a char chunk queue
• the third takes char chunks, identifies and splits the lines, and filters what's junk.
• the fourth encodes the chars back to a byte stream, in chunks which it places on a queue
• the fifth compresses the byte chunks back to disk

This is going to net you a potential 5X improvement.

That's pretty complicated though.

What would be much simpler is to use a different axis to process the parallelism. Your comments indicate that you have multiple files to process.... you should do them each in multiple threads...

Consider the following structure:

private static final boolean processFile(final File fileIn) throws IOException {
    //For each file in a folder:
    try (BufferedReader br = new BufferedReader(new InputStreamReader(
            new BZip2CompressorInputStream(new FileInputStream(fileIn))));
         BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(
            new GZIPOutputStream(new FileOutputStream(fileOut))))) {

        String line;
        while ((line = br.readLine()) != null) {
            String[] parts = line.split(',');
            if(isLineToSkip(parts[0])) {
                continue;
            }
            String outLine = parts[0] + "," + parts[3]; 

            writer.append(outLine);
            writer.newLine();
        }
    }
    return true;
}

.......


ExecutorService service = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
List<Future<Boolean>> queued = new LinkedList<>();
for (final File toprocess : ....... ) {
    queued.add(service.submit(new Callable () {
        public Boolean call() throws IOException {
            return processFile(toprocess);
        }
    }));
}

for (Future<Boolean> future : queueud) {
    future.get();
}

With the above code, you will do one file in each thread, and the system CPU will run at 100% ... no matter how many CPU's you have. Essentially you are using your system fully.

Answer 2

My Java is a little rusty, and they're constantly changing the buffering and classes, but I'm tempted to say that you want a BufferedInputStream in between your compression input stream and your file stream, because the whole purpose is to not hit the disk so often. You should also look at setting specific large buffer sizes, probably 8 * 1024. Then I think you can remove the BufferedReader. Same thing goes for your output stack.

There are also CSV libraries I can recommend to better handle the CSV parsing and data, but that's outside of the scope of what you asked.