8
\$\begingroup\$

While looking at a recent question, I realized that there is no CSV parser in the standard Java library. I decided to write one that complies with RFC 4180. Highlights of the standard include:

  • Whitespace is significant.
  • Strings may be double-quoted. Inside a double-quoted string, newlines are treated literally, and a pair of consecutive double-quotes is interpreted as a literal double-quote.

This parser supports three data types: Integer, BigDecimal, and String. Unquoted strings that look like numbers are treated as numbers.

There is an option to treat the first row as headers, so that you can retrieve data fields by column name.

import java.io.IOException;
import java.io.Reader;
import java.math.BigDecimal;
import java.util.*;
import java.util.regex.Pattern;

/**
 * RFC 4180-compliant CSV parser.
 */
public class CSVParser implements AutoCloseable, Iterable<CSVParser.Row> {
    public class Row {
        private final List<Object> data;

        private Row(List<Object> data) {
            this.data = data;
        }

        /**
         * Gets the nth field (first field is numbered 1).
         */
        public Object get(int nth) {
            return this.data.get(nth - 1);
        }

        /**
         * Gets the value in this row for the field corresponding to
         * the named column.  Returns null if the CSVParser had no
         * headers or if no column with this name exists.
         */
        public Object get(String fieldName) {
            return CSVParser.this.headers == null ?
                null : this.data.get(CSVParser.this.headers.indexOf(fieldName));
        }

        /**
         * Each field may be an Integer, BigDecimal, or a String.
         */
        public List<Object> getData() {
            return this.data;
        }
    }

    //////////////////////////////////////////////////////////////////////

    private static final Pattern
        SEP = Pattern.compile(",|(?=\n)"),
        QUOTED_STRING_PATTERN = Pattern.compile("\"(?:(?:\"\")?+|[^\"])*\"(?=,|\n|$)", Pattern.DOTALL);

    private final Scanner scan;
    private final List<Object> headers;

    public CSVParser(Reader r, boolean withHeaderRow) {
        this.scan = new Scanner(r).useDelimiter(SEP);
        this.headers = withHeaderRow ? this.parseRow(true) : null;
    }

    private List<Object> parseRow(boolean noNumbers) {
        List<Object> row = new ArrayList<>();
        while (this.scan.findInLine("(?=.)") != null) {
            if (this.scan.hasNext("\".*")) {
                String q = this.scan.findWithinHorizon(QUOTED_STRING_PATTERN, 0);
                if (q == null) {
                    // Unterminated quoted string; treat it as unquoted.
                    row.add(this.scan.next());
                } else {
                    this.scan.skip(SEP);
                    row.add(dequote(q));
                }
            } else if (!noNumbers && this.scan.hasNextInt()) {
                row.add(this.scan.nextInt());
            } else if (!noNumbers && this.scan.hasNextBigDecimal()) {
                row.add(this.scan.nextBigDecimal());
            } else {
                row.add(this.scan.next());
            }
        }
        if (this.scan.hasNextLine()) {
            String eol = this.scan.nextLine();
            assert eol.isEmpty();
        }
        return Collections.unmodifiableList(row);
    }

    private static String dequote(String quoted) {
        assert quoted.startsWith("\"");
        assert quoted.endsWith("\"");
        return quoted.substring(1, quoted.length() - 1).replace("\"\"", "\"");
    }

    @SuppressWarnings("unchecked")
    public List<String> getHeaders() {
        return (List<String>)(List)this.headers;
    }

    public Iterator<Row> iterator() {
        return new Iterator<Row>() {
            public boolean hasNext() {
                return CSVParser.this.scan.hasNextLine();
            }

            public Row next() {
                return new Row(CSVParser.this.parseRow(false));
            }
        };
    }

    public void close() throws IOException {
        IOException error = this.scan.ioException();
        if (error != null) throw error;
        this.scan.close();
    }
}

Using a Scanner was trickier than I had expected, due to the context-dependent interpretation of the double-quotes and newlines. Also, a Scanner normally carelessly ignores whitespace; I had to discover a new technique to detect linebreaks without discarding input.

Regex-haters gonna hate my QUOTED_STRING_PATTERN. Any suggestions for reducing the hatred would be appreciated.

How is my RFC 4180 compliance? Also, what is your opinion of how I have chosen to handle weird input? (For example, if an unpaired " is encountered, then I treat the field as if it were unquoted.)

What's your opinion of my one-based field-numbering convention?

How about error handling? An IOException may be thrown on close().

Sample usage:

try (Reader fileReader = new FileReader(…);
     CSVParser csv = new CSVParser(fileReader, true)) {
    for (CSVParser.Row row : csv) {
        System.out.println(row.get("Price"));
    }
}

Or, using streams:

try (Reader fileReader = new FileReader(…);
     CSVParser csv = new CSVParser(fileReader, true)) {
    StreamSupport.stream(csv.spliterator(), false)
                 .map(row -> row.get("Price"))
                 .forEach(System.out::println);
}
\$\endgroup\$
  • \$\begingroup\$ Not relevant to the actual review, but surely there's an existing CSV parsing library that you can use? I know you only get five tags, but maybe one should be reinventing-the-wheel? :-) \$\endgroup\$ – Eric Stein Feb 16 '16 at 14:02
  • \$\begingroup\$ CSV ends up being much simpler to parse in a state machine... char by char \$\endgroup\$ – rolfl Feb 16 '16 at 22:51
1
\$\begingroup\$

In general I feel like you are trying to do too much in one class. There are two different parsing systems you have - parsing CSV to strings rows/fields, and then parsing some of them in to numbers.

You should separate those two systems out in to discrete places - in the more complicated use-cases, you will guess wrong, and in the simpler cases, the programmer can easily implement the number parsing anyway. By conflating the two parse operations you have lost the value of the generic List<String> on the rows, and your API is a lot more complicated than it needs to be.

This is a case-in-point, and illustrates all the bad things:

@SuppressWarnings("unchecked")
public List<String> getHeaders() {
    return (List<String>)(List)this.headers;
}

CSV is a text format - leave it that way. RFC4180 makes no mention of numeric data either.

You specifically ask:

How is my RFC 4180 compliance? Also, what is your opinion of how I have chosen to handle weird input? (For example, if an unpaired " is encountered, then I treat the field as if it were unquoted.)

That is specifically addressed in 4180, and is incorrect. RFC4180 has (2.6):

Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.

A quoted field with a line-break at the end, should extend to the next quote on the following lines. The quoted line-breaks should be treated literally.

\$\endgroup\$
  • \$\begingroup\$ "A quoted field with a line-break at the end, should extend to the next quote on the following lines. The quoted line-breaks should be treated literally." That is what the code does. The question is, how should I handle out-of-spec input? Just give up, or try to make alternate sense of it? \$\endgroup\$ – 200_success Feb 17 '16 at 0:13
  • 1
    \$\begingroup\$ Ahh, I got list in the regex the ... ;-) OK, about the trailing unclosed quote - the only reasonable thing to do is fail with an InvalidFormatException or something similar. If the data is supposed to be compliant, and it's not, then it's broken. \$\endgroup\$ – rolfl Feb 17 '16 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.