Splitting a string into words or double-quoted substrings

Question

For various reasons, I'm parsing a string, this code will explain what I'm after:

string baseString = "This is a \"Very Long Test\"";

string[] strings = baseString.Split(' ');

List<String> stringList = new List<string>();
string temp = String.Empty;
foreach (var s in strings)
{
    if (!String.IsNullOrWhiteSpace(temp))
    {
        if (s.EndsWith("\""))
        {
            string item = temp + " " + s;
            stringList.Add(item.Substring(1,item.Length - 2));
            temp = string.Empty;
        }
        temp = temp + " " + s;
    }
    else if (s.StartsWith("\""))
    {
        temp = s;
    }
    else
    {
        stringList.Add(s);
    }

}

stringList.ForEach(Console.WriteLine);

The output should be:

This
is
a
Very Long Test

Basically, given a string, it will split it on spaces, unless its grouped into speech marks, the same way the command line does it.

Any better way to do this code?

Answer 1

Seems like a job for a regular expression:

string baseString = "This is a \"Very Long Test\"";
var re = new Regex("(?<=\")[^\"]*(?=\")|[^\" ]+");
var strings = re.Matches(baseString).Cast<Match>().Select(m => m.Value).ToArray();

What the regular expression (?<=")[^"]*(?=")|[^" ]+ does is that it either finds a sequence of zero or more characters that are not " ([^"]*) preceded by a " ((?<=")) and followed by a " ((?=")) or a sequence of one or more character that are not " or a space ([^" ]+).

For the sample input, it gives the same output as your version. The code itself is much simpler, but the regular expression might be hard to understand, especially if you're not used to them.

Answer 2

    string baseString = "This is a \"Very Long Test\"... not so long actually, eh?";
    string[] aux = baseString.Split('"');
    List<string> tokens = new List<string>();
    for (int i = 0; i < aux.Length; ++i)
        if (i % 2 == 0)
            tokens.AddRange(aux[i].Split(' '));
        else
            tokens.Add(aux[i]);

Notice that if there is a double quote in the middle of a word it will be split (a"ctuall"y would be a, ctuall, y in the final result). If the last double quote is unmatch it won't split from its position to the end of the string.

Answer 3

The usual approach to this kind of tokenising is to count the number of markers (it shares the same idea as hit scanning a polygon).

Scan the string one character at a time, checking if the character is a space or a quote mark, or any other character. If it's quote mark, toggle your quote flag indicating you are within a delimited string and output the current scanned token if transitioning from 'in delimited string' to 'not in delimited string'. If it's a space and you're 'not in a delimited string', output the currently scanned token. All other characters get added to the currently scanned token.

After the final character has been processed, if output the currently scanned token if there are any remaining characters that have been scanned but not output.

Untested, but the general approach:

StringBuilder currentToken = new StringBuilder();
bool inDelimitedString = false;
List<string> scannedTokens = new List<string>();
foreach (char c in source)
{
    switch(c)
    {
        case '"':
            if (inDelimitedString)
            {
                if (currentToken.Length > 0)
                {
                    scannedTokens.Add(currentToken.ToString());
                    currentToken.Clear();
                }
            }
            inDelimitedString = !inDelimitedString;
            break;
        case ' ':
            if (!inDelimitedString)
            {
                if (currentToken.Length > 0)
                {
                    scannedTokens.Add(currentToken.ToString());
                    currentToken.Clear();
                }
            }
            else
            {
                currentToken.Append(c);
            }
            break;
        default:
            currentToken.Append(c);
            break;
    }
}
if (currentToken.Length > 0)
{
    scannedTokens.Add(currentToken.ToString());
    currentToken.Clear();
}

You can extend the same idea to counting braces (e.g. '(' matching to ')', '[' to ']') which allow nesting, and your performance costs remain O(n).