3
\$\begingroup\$

I've written this method to replace an older method that was much simpler, but used the regex split method and couldn't tell if a comma was in quotes/brackets/etc. and didn't read double quotes as single. I was hoping to cover most of the most used possible CSV formats except where a tab is used as a separator rather than a comma.

I haven't tested this yet, and I'm going to write a unit test later on, but I was hoping to get some feedback on it anyway. What I might be missing, what I might be able to do better or more concisely, etc.

It does seem a little long for my liking.

I used this as a guide while writing it.

private enum PositionState
{ inQuotes, inInvertedCommas, inBrackets, inSquareBrackets, inCurlyBrackets, inChevronBrackets }

private List<String> splitCSVLine(String toSplit)
{
    List<String> entries = new ArrayList<String>();
    boolean nextCharIsEscaped = false;
    boolean lastCharIsEscapingQuote = false;
    StringBuilder entryBuilder = new StringBuilder();
    List<PositionState> positionStates = new ArrayList<PositionState>();

    for(int i = 0; i < toSplit.length(); i++)
    {
        char iChar = toSplit.charAt(i);
        boolean printCharacter = true; // Whether or not the character should be printed into the current entry.

        switch(iChar)
        {
            case '\\':
            {
                if(!nextCharIsEscaped)
                    printCharacter = false;

                nextCharIsEscaped = !nextCharIsEscaped;
            } break;

            case ',':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else if(positionStates.isEmpty())
                {
                    entries.add(entryBuilder.toString());
                    entryBuilder = new StringBuilder();
                    printCharacter = false;
                }
            } break;

            case '\'':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                {
                    boolean isOpening = true;

                    for(int j = positionStates.size() - 1; j >= 0; j--)
                        if(positionStates.get(j) == PositionState.inInvertedCommas)
                        {
                            isOpening = false;

                            for(int k = positionStates.size() - 1; k >= j; k--)
                                positionStates.remove(k);

                            break;
                        }

                    if(isOpening)
                        positionStates.add(PositionState.inInvertedCommas);
                }
            } break;

            case '"':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else if(lastCharIsEscapingQuote)
                    lastCharIsEscapingQuote = false;
                else if(i < toSplit.length() - 1 && toSplit.charAt(i + 1) == '"')
                    lastCharIsEscapingQuote = true;
                else
                {
                    boolean isOpening = true;

                    for(int j = positionStates.size() - 1; j >= 0; j--)
                        if(positionStates.get(j) == PositionState.inQuotes)
                        {
                            isOpening = false;

                            for(int k = positionStates.size() - 1; k >= j; k--)
                                positionStates.remove(k);

                            break;
                        }

                    if(isOpening)
                        positionStates.add(PositionState.inQuotes);
                }
            } break;

            case '(':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                    positionStates.add(PositionState.inBrackets);
            } break;

            case ')':
            {
                for(int j = positionStates.size() - 1; j >= 0; j--)
                    if(positionStates.get(j) == PositionState.inBrackets)
                    {
                        for(int k = positionStates.size() - 1; k >= j; k--)
                            positionStates.remove(k);

                        break;
                    }
            } break;

            case '[':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                    positionStates.add(PositionState.inSquareBrackets);
            } break;

            case ']':
            {
                for(int j = positionStates.size() - 1; j >= 0; j--)
                    if(positionStates.get(j) == PositionState.inSquareBrackets)
                    {
                        for(int k = positionStates.size() - 1; k >= j; k--)
                            positionStates.remove(k);

                        break;
                    }
            } break;

            case '{':
            {
                if(nextCharIsEscaped)
                    nextCharIsEscaped = false;
                else
                    positionStates.add(PositionState.inCurlyBrackets);
            } break;

            case '}':
            {
                for(int j = positionStates.size() - 1; j >= 0; j--)
                    if(positionStates.get(j) == PositionState.inCurlyBrackets)
                    {
                        for(int k = positionStates.size() - 1; k >= j; k--)
                            positionStates.remove(k);

                        break;
                    }
            } break;
        }

        if(printCharacter)
            entryBuilder.append(iChar);
    }

    entries.add(entryBuilder.toString());
    handleQuotes(entries);
    return entries;
}

/**
 * Removes the surrounding quotes from entries that are enclosed in them, and converts adjacent pairs of quotation
 * marks into single ones, where they don't represent an empty field and the first quotation mark isn't escaped.
 * @param entries The list of entries to handle quotes in.
 */
private void handleQuotes(List<String> entries)
{
    for(int i = 0; i < entries.size(); i++)
    {
        String iEntry = entries.get(i);
        String iEntryOriginal = iEntry;

        iEntry = iEntry.trim();

        if(iEntry.startsWith("\"") && iEntry.endsWith("\""))
            iEntry = iEntry.substring(1, iEntry.length() - 1);

        /*
            Go through iEntry looking for double quotes (""). Double-quotes representing empty fields have already
            been taken out with the previous check. Replace them with a single quote (") where the first quote isn't
            escaped. That is, where they aren't preceded by an escape character (\) that isn't itself escaped.
            This can be approximated by checking whether they're preceded by an odd or even number of the escape
            character.
        */

        for(int j = 0; j < iEntry.length() - 1; j++)
        {
            if(iEntry.charAt(j) == '"' && iEntry.charAt(j + 1) == '"')
            {
                int escapeCharacterCount = 0;

                for(int k = j - 1; k >= 0; k--)
                {
                    if(iEntry.charAt(k) == '\\')
                        escapeCharacterCount++;
                    else
                        break;
                }

                if(escapeCharacterCount % 2 == 0) // if escapeCharacterCount is even or 0
                    iEntry = new StringBuilder(iEntry).deleteCharAt(j + 1).toString();
            }
        }

        if(!iEntry.equals(iEntryOriginal))
            entries.set(i, iEntry);
    }
}
\$\endgroup\$
  • 1
    \$\begingroup\$ "I've written this method to replace an older method..." any reason not to just substitute with a third-party library? :) \$\endgroup\$ – h.j.k. Apr 20 '15 at 11:27
  • \$\begingroup\$ It's for a plugin (or, rather, a library that multiple of my own plugins reference) where using libraries not supported by the original api may be awkward, and I already have a framework for loading various filetypes in it, which this is part of. \$\endgroup\$ – Hanii Puppy Apr 20 '15 at 11:31
  • 1
    \$\begingroup\$ Cool, thanks for the explanation, hope you'll get answers soon! :) \$\endgroup\$ – h.j.k. Apr 20 '15 at 11:32
1
\$\begingroup\$
        case '}':
        {
            for(int j = positionStates.size() - 1; j >= 0; j--)
                if(positionStates.get(j) == PositionState.inCurlyBrackets)
                {
                    for(int k = positionStates.size() - 1; k >= j; k--)
                        positionStates.remove(k);

                    break;
                }
        } break;

Stop omitting braces. Understanding what that break does is harder when there's no braces, and it's easy to eventually screw up like this. This because most of the time, a statement being executed outside of some conditional block is not a syntax error - it's just a logic error, also known as a bug.


Bug

splitCSVLine("N/A,N\\A,ThirdField").size();

Length returned is 2. You have a bug regarding the handling of escape characters; that is, you do not stop escaping if the next character is an invalid character.

I don't feel comfortable trying to refactor buggy code, so I'll have to leave my answer at this.

\$\endgroup\$
1
\$\begingroup\$

It may be simpler to code in terms of single fields, instead of lines or "records" of CSV fields. Code sample below uses Java8 streams to build up a result; however, it's trivial to convert it to Java7 using an ArrayList<String> to build up the results.

package test;
import java.util.stream.Stream;

public class CSV {
  public static String[] split(final String text) {
    return (text.indexOf('"') < 0)? text.split(",")
                                  : splitCSV(Stream.<String>builder(), text, 0).build().toArray(String[]::new);
  }


  private static Stream.Builder<String> splitCSV(final Stream.Builder<String> accum, final String text, int start) {
    final int length = text.length();
    if (start >= length) return accum;

    final StringBuilder buf = new StringBuilder(length);
    boolean inquote = false;

    if (text.charAt(start) == '"') {
      inquote = true;
      start++;
    }

    int i = start;

  loop:
    for (; i < length; i++) {
      final char c = text.charAt(i);
      switch (c) {
        case '"':
          if (inquote) {
            if (i + 1 < length && text.charAt(i + 1) == '"') {
              i++;
              break;
            }
            else {
              inquote = false;
              continue loop;
            }
          }
          break;

        case '\\':
          if (i + 1 < length) {
            final char next = text.charAt(i + 1);
            switch (next) {
              case 'b': buf.append('\b'); i++; continue loop;
              case 'f': buf.append('\f'); i++; continue loop;
              case 'r': buf.append('\r'); i++; continue loop;
              case 'n': buf.append('\n'); i++; continue loop;
              case 't': buf.append('\t'); i++; continue loop;
              case '"': buf.append('"');  i++; continue loop;
            }
          }
          break;

        case ',':
          if (!inquote) {
              // value termination
            i++;
            break loop;
          }
          break;
      }

      buf.append(c);
    }

    return splitCSV(accum.add(buf.toString()), text, i);
  }
}

I tested it briefly using the Scala interpreter:

scala> import test.CSV
import test.CSV

scala> CSV.split("N/A,N\\A,ThirdField")
res0: Array[String] = Array(N/A, N\A, ThirdField)

scala> CSV.split(""""A quoted value","One with ""embedded"" quotes",One without,"",a,""")
res1: Array[String] = Array(A quoted value, One with "embedded" quotes, One without, "", a)

scala> res1(3)
res2: String = ""

scala> res2.length
res3: Int = 0

scala> CSV.split("""First,second,"third, with a comma",fourth has a \nnewline,"fifth:\tbeing tricky, has ""some real"" testing craziness!"
     | """
     | )
res4: Array[String] = Array(First, second, third, with a comma, fourth has a
newline, "fifth:  being tricky, has "some real" testing craziness!
")

scala> res4(2)
res5: String = third, with a comma

scala> CSV.split("""One,second has \"escaped\" quotes,third""")
res6: Array[String] = Array(One, second has "escaped" quotes, third)

Test Notes:

1. Scala has triple-quoted strings that allow embedded single quotes, newlines, and backslashes, etc., without escaping.

2. The Scala interpreter prints empty strings as a set of empty quotes (""), but non-empty strings without surrounding quotes.

\$\endgroup\$
  • \$\begingroup\$ I realise that in split(), it wasn't necessary to test to see if the input had any quotes; but that's the simple case and the naïve solution, so why not? \$\endgroup\$ – WeaponsGrade Apr 20 '15 at 19:22
  • \$\begingroup\$ Your implementation doesn't support the following: "Fields containing a comma must be escaped. (...) Unix style programs escape these values by inserting a single backslash character before each comma, i.e. a single cell with the text apples, carrots, and oranges becomes apples\, carrots\, and oranges." \$\endgroup\$ – Pimgd Apr 20 '15 at 20:31
  • \$\begingroup\$ It's trivial to add case ',': buf.append(','); i++; continue loop; to the escape-cases above. \$\endgroup\$ – WeaponsGrade Apr 20 '15 at 20:53
  • \$\begingroup\$ Except then you're still screwed due to the way your split function works. \$\endgroup\$ – Pimgd Apr 20 '15 at 21:26
  • \$\begingroup\$ Like I said above, you can remove it. Supposedly, this is a forum for developers, who understand how to trivially change code! \$\endgroup\$ – WeaponsGrade Apr 20 '15 at 22:55
1
\$\begingroup\$

With the help of csv parser library uniVocity-parsers, you can easily handle the quotes in csv data.

According to the tutorial here, you can set quotes to esapce and quotes as the delimiters.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.