Parsing a single CSV line into a list of strings

Question

I've written this method to replace an older method that was much simpler, but used the regex split method and couldn't tell if a comma was in quotes/brackets/etc. and didn't read double quotes as single. I was hoping to cover most of the most used possible CSV formats except where a tab is used as a separator rather than a comma.

I haven't tested this yet, and I'm going to write a unit test later on, but I was hoping to get some feedback on it anyway. What I might be missing, what I might be able to do better or more concisely, etc.

It does seem a little long for my liking.

I used this as a guide while writing it.

private enum PositionState
{ inQuotes, inInvertedCommas, inBrackets, inSquareBrackets, inCurlyBrackets, inChevronBrackets }

private List<String> splitCSVLine(String toSplit)
{
    List<String> entries = new ArrayList<String>();
    boolean nextCharIsEscaped = false;
    boolean lastCharIsEscapingQuote = false;
    StringBuilder entryBuilder = new StringBuilder();
    List<PositionState> positionStates = new ArrayList<PositionState>();

    for(int i = 0; i < toSplit.length(); i++)
    {
        char iChar = toSplit.charAt(i);
        boolean printCharacter = true; // Whether or not the character should be printed into the current entry.

        switch(iChar)
        {
            case '\\':
            {
                if(!nextCharIsEscaped)
                    printCharacter = false;

                nextCharIsEscaped = !nextCharIsEscaped;
            } break;

            case ',':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else if(positionStates.isEmpty())
                {
                    entries.add(entryBuilder.toString());
                    entryBuilder = new StringBuilder();
                    printCharacter = false;
                }
            } break;

            case '\'':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                {
                    boolean isOpening = true;

                    for(int j = positionStates.size() - 1; j >= 0; j--)
                        if(positionStates.get(j) == PositionState.inInvertedCommas)
                        {
                            isOpening = false;

                            for(int k = positionStates.size() - 1; k >= j; k--)
                                positionStates.remove(k);

                            break;
                        }

                    if(isOpening)
                        positionStates.add(PositionState.inInvertedCommas);
                }
            } break;

            case '"':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else if(lastCharIsEscapingQuote)
                    lastCharIsEscapingQuote = false;
                else if(i < toSplit.length() - 1 && toSplit.charAt(i + 1) == '"')
                    lastCharIsEscapingQuote = true;
                else
                {
                    boolean isOpening = true;

                    for(int j = positionStates.size() - 1; j >= 0; j--)
                        if(positionStates.get(j) == PositionState.inQuotes)
                        {
                            isOpening = false;

                            for(int k = positionStates.size() - 1; k >= j; k--)
                                positionStates.remove(k);

                            break;
                        }

                    if(isOpening)
                        positionStates.add(PositionState.inQuotes);
                }
            } break;

            case '(':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                    positionStates.add(PositionState.inBrackets);
            } break;

            case ')':
            {
                for(int j = positionStates.size() - 1; j >= 0; j--)
                    if(positionStates.get(j) == PositionState.inBrackets)
                    {
                        for(int k = positionStates.size() - 1; k >= j; k--)
                            positionStates.remove(k);

                        break;
                    }
            } break;

            case '[':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                    positionStates.add(PositionState.inSquareBrackets);
            } break;

            case ']':
            {
                for(int j = positionStates.size() - 1; j >= 0; j--)
                    if(positionStates.get(j) == PositionState.inSquareBrackets)
                    {
                        for(int k = positionStates.size() - 1; k >= j; k--)
                            positionStates.remove(k);

                        break;
                    }
            } break;

            case '{':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                    positionStates.add(PositionState.inCurlyBrackets);
            } break;

            case '}':
            {
                for(int j = positionStates.size() - 1; j >= 0; j--)
                    if(positionStates.get(j) == PositionState.inCurlyBrackets)
                    {
                        for(int k = positionStates.size() - 1; k >= j; k--)
                            positionStates.remove(k);

                        break;
                    }
            } break;
        }

        if(printCharacter)
            entryBuilder.append(iChar);
    }

    entries.add(entryBuilder.toString());
    handleQuotes(entries);
    return entries;
}

/**
 * Removes the surrounding quotes from entries that are enclosed in them, and converts adjacent pairs of quotation
 * marks into single ones, where they don't represent an empty field and the first quotation mark isn't escaped.
 * @param entries The list of entries to handle quotes in.
 */
private void handleQuotes(List<String> entries)
{
    for(int i = 0; i < entries.size(); i++)
    {
        String iEntry = entries.get(i);
        String iEntryOriginal = iEntry;

        iEntry = iEntry.trim();

        if(iEntry.startsWith("\"") && iEntry.endsWith("\""))
            iEntry = iEntry.substring(1, iEntry.length() - 1);

        /*
            Go through iEntry looking for double quotes (""). Double-quotes representing empty fields have already
            been taken out with the previous check. Replace them with a single quote (") where the first quote isn't
            escaped. That is, where they aren't preceded by an escape character (\) that isn't itself escaped.
            This can be approximated by checking whether they're preceded by an odd or even number of the escape
            character.
        */

        for(int j = 0; j < iEntry.length() - 1; j++)
        {
            if(iEntry.charAt(j) == '"' && iEntry.charAt(j + 1) == '"')
            {
                int escapeCharacterCount = 0;

                for(int k = j - 1; k >= 0; k--)
                {
                    if(iEntry.charAt(k) == '\\')
                        escapeCharacterCount++;
                    else
                        break;
                }

                if(escapeCharacterCount % 2 == 0) // if escapeCharacterCount is even or 0
                    iEntry = new StringBuilder(iEntry).deleteCharAt(j + 1).toString();
            }
        }

        if(!iEntry.equals(iEntryOriginal))
            entries.set(i, iEntry);
    }
}

Answer 1

        case '}':
        {
            for(int j = positionStates.size() - 1; j >= 0; j--)
                if(positionStates.get(j) == PositionState.inCurlyBrackets)
                {
                    for(int k = positionStates.size() - 1; k >= j; k--)
                        positionStates.remove(k);

                    break;
                }
        } break;

Stop omitting braces. Understanding what that break does is harder when there's no braces, and it's easy to eventually screw up like this. This because most of the time, a statement being executed outside of some conditional block is not a syntax error - it's just a logic error, also known as a bug.

---

Bug

splitCSVLine("N/A,N\\A,ThirdField").size();

Length returned is 2. You have a bug regarding the handling of escape characters; that is, you do not stop escaping if the next character is an invalid character.

I don't feel comfortable trying to refactor buggy code, so I'll have to leave my answer at this.

Answer 2

It may be simpler to code in terms of single fields, instead of lines or "records" of CSV fields. Code sample below uses Java8 streams to build up a result; however, it's trivial to convert it to Java7 using an ArrayList<String> to build up the results.

package test;
import java.util.stream.Stream;

public class CSV {
  public static String[] split(final String text) {
    return (text.indexOf('"') < 0)? text.split(",")
                                  : splitCSV(Stream.<String>builder(), text, 0).build().toArray(String[]::new);
  }


  private static Stream.Builder<String> splitCSV(final Stream.Builder<String> accum, final String text, int start) {
    final int length = text.length();
    if (start >= length) return accum;

    final StringBuilder buf = new StringBuilder(length);
    boolean inquote = false;

    if (text.charAt(start) == '"') {
      inquote = true;
      start++;
    }

    int i = start;

  loop:
    for (; i < length; i++) {
      final char c = text.charAt(i);
      switch (c) {
        case '"':
          if (inquote) {
            if (i + 1 < length && text.charAt(i + 1) == '"') {
              i++;
              break;
            }
            else {
              inquote = false;
              continue loop;
            }
          }
          break;

        case '\\':
          if (i + 1 < length) {
            final char next = text.charAt(i + 1);
            switch (next) {
              case 'b': buf.append('\b'); i++; continue loop;
              case 'f': buf.append('\f'); i++; continue loop;
              case 'r': buf.append('\r'); i++; continue loop;
              case 'n': buf.append('\n'); i++; continue loop;
              case 't': buf.append('\t'); i++; continue loop;
              case '"': buf.append('"');  i++; continue loop;
            }
          }
          break;

        case ',':
          if (!inquote) {
              // value termination
            i++;
            break loop;
          }
          break;
      }

      buf.append(c);
    }

    return splitCSV(accum.add(buf.toString()), text, i);
  }
}

I tested it briefly using the Scala interpreter:

scala> import test.CSV
import test.CSV

scala> CSV.split("N/A,N\\A,ThirdField")
res0: Array[String] = Array(N/A, N\A, ThirdField)

scala> CSV.split(""""A quoted value","One with ""embedded"" quotes",One without,"",a,""")
res1: Array[String] = Array(A quoted value, One with "embedded" quotes, One without, "", a)

scala> res1(3)
res2: String = ""

scala> res2.length
res3: Int = 0

scala> CSV.split("""First,second,"third, with a comma",fourth has a \nnewline,"fifth:\tbeing tricky, has ""some real"" testing craziness!"
     | """
     | )
res4: Array[String] = Array(First, second, third, with a comma, fourth has a
newline, "fifth:  being tricky, has "some real" testing craziness!
")

scala> res4(2)
res5: String = third, with a comma

scala> CSV.split("""One,second has \"escaped\" quotes,third""")
res6: Array[String] = Array(One, second has "escaped" quotes, third)

Test Notes:

1. Scala has triple-quoted strings that allow embedded single quotes, newlines, and backslashes, etc., without escaping.

2. The Scala interpreter prints empty strings as a set of empty quotes (""), but non-empty strings without surrounding quotes.

Answer 3

With the help of csv parser library uniVocity-parsers, you can easily handle the quotes in csv data.

According to the tutorial here, you can set quotes to esapce and quotes as the delimiters.