3
\$\begingroup\$

UPDATE: I have refactored the code into a Gist using @Dmitry's answer as a guide. The update is much simpler to grok, implements IDisposable, and is roughly thirty lines shorter.


I wrote this over the weekend for fun and am looking for critique. Style and readability comments are welcome but what I truly need to know is:

  1. Does it function as advertised?
  2. Are there any lingering bugs that I've missed?
  3. Can you come up with a way to make it faster?

When I ask these of myself I get 1 = yes, 2 = no, and 3 = maaaaaybe. I'd like to add other features like skipping the header row, inferring data types, validating field counts, etc. but I'll be tackling that kind of thing via derivation or extension since such logic will be simpler to implement if based on an existing IEnumerable<IEnumerable<>> like this one.

FLAME ON;

Usage:

foreach (var row in DelimitedReader.Create(fileName)) {
    foreach (var field in row) {
        // do stuff
    }
}

Features:

  • Accurate: RFC4180 Compliant
  • Efficient: memory usage is (roughly) equal to the size of the largest row
  • Fast: average throughput of ~25 megabytes per second
  • Flexible: the default encoding and separator/escape characters can be user-defined
  • Lightweight: single 160 line class with no external dependencies

Code:

using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Text;

namespace ByteTerrace
{
    public class DelimitedReader : IEnumerable<IEnumerable<string>>
    {
        private const int DEFAULT_CHUNK_SIZE = 128;
        private const char DEFAULT_ESCAPE_CHAR = '"';
        private const char DEFAULT_SEPARATOR_CHAR = ',';

        private readonly char[] m_buffer;
        private readonly Encoding m_encoding;
        private readonly char m_escapeChar;
        private readonly string m_fileName;
        private readonly char m_separatorChar;

        public char[] Buffer {
            get {
                return m_buffer;
            }
        }
        public Encoding Encoding {
            get {
                return m_encoding;
            }
        }
        public char EscapeChar {
            get {
                return m_escapeChar;
            }
        }
        public string FileName {
            get {
                return m_fileName;
            }
        }
        public char SeparatorChar {
            get {
                return m_separatorChar;
            }
        }

        public DelimitedReader(string fileName, char separatorChar = DEFAULT_SEPARATOR_CHAR, char escapeChar = DEFAULT_ESCAPE_CHAR, Encoding encoding = null, int bufferSize = DEFAULT_CHUNK_SIZE) {
            m_buffer = new char[bufferSize];
            m_encoding = (encoding ?? Encoding.UTF8);
            m_escapeChar = escapeChar;
            m_fileName = fileName;
            m_separatorChar = separatorChar;
        }

        public IEnumerator<IEnumerable<string>> GetEnumerator() {
            return ReadFields().GetEnumerator();
        }
        IEnumerator IEnumerable.GetEnumerator() {
            return GetEnumerator();
        }
        IEnumerable<IEnumerable<string>> ReadFields() {
            return ReadFields(ReadAllChunks(FileName, Encoding, Buffer), SeparatorChar, EscapeChar);
        }

        public static DelimitedReader Create(string fileName, char separatorChar = DEFAULT_SEPARATOR_CHAR, char escapeChar = DEFAULT_ESCAPE_CHAR, Encoding encoding = null, int bufferSize = DEFAULT_CHUNK_SIZE) {
            return new DelimitedReader(fileName, separatorChar, escapeChar, encoding, bufferSize);
        }
        public static IEnumerable<char[]> ReadAllChunks(TextReader reader, char[] buffer) {
            var count = buffer.Length;
            var numBytesRead = 0;

            while ((numBytesRead = reader.ReadBlock(buffer, 0, count)) == count) {
                yield return buffer;
            }

            if (numBytesRead > 0) {
                Array.Resize(ref buffer, numBytesRead);

                yield return buffer;
            }
        }
        public static IEnumerable<char[]> ReadAllChunks(string fileName, Encoding encoding, char[] buffer) {
            return ReadAllChunks(new StreamReader(new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, FileOptions.SequentialScan), encoding), buffer);
        }
        public static string ReadField(StringBuilder buffer, int offset, int position, char escapeChar) {
            if (buffer[offset] == escapeChar) {
                if (position - offset != 2) {
                    return buffer.ToString(offset + 1, position - offset - 3);
                }
                else {
                    return string.Empty;
                }
            }
            else {
                return buffer.ToString(offset, position - offset - 1);
            }
        }
        public static IEnumerable<IEnumerable<string>> ReadFields(IEnumerable<char[]> chunks, char separatorChar = DEFAULT_SEPARATOR_CHAR, char escapeChar = DEFAULT_ESCAPE_CHAR) {
            var buffer = new StringBuilder();
            var fields = new List<string>();
            var endOfBuffer = 0;
            var escaping = false;
            var offset = 0;
            var position = 0;
            var head0 = '\0';
            var head1 = head0;

            foreach (var chunk in chunks) {
                buffer.Append(chunk, 0, chunk.Length);
                endOfBuffer = buffer.Length;

                while (position < endOfBuffer) {
                    head1 = head0;

                    if ((head0 = buffer[position++]) == escapeChar) {
                        escaping = !escaping;

                        if ((head0 == escapeChar) && (head1 == escapeChar)) {
                            endOfBuffer--;
                            position--;
                            buffer.Remove(position, 1);
                        }
                    }

                    if (!escaping) {
                        if ((head0 == '\n') || (head0 == '\r')) {
                            if ((head1 != '\r') || (head0 == '\r')) {
                                fields.Add(ReadField(buffer, offset, position, escapeChar));

                                yield return fields;

                                buffer.Remove(0, position);
                                endOfBuffer = buffer.Length;
                                fields.Clear();
                                offset = 0;
                                position = 0;
                            }
                            else {
                                offset++;
                            }
                        }
                        else if (head0 == separatorChar) {
                            fields.Add(ReadField(buffer, offset, position, escapeChar));
                            offset = position;
                        }
                    }
                }
            }


            if (buffer.Length > 0) {
                fields.Add(buffer.ToString());
            }

            if (fields.Count > 0) {
                yield return fields;
            }
        }
    }
}
\$\endgroup\$
2
\$\begingroup\$

I'd prefer to rely on the builtin functionality as much as possible. I want to believe that use of the builtin stuff makes my code more readable and probably faster.

So my proposal is:

public class DelimitedReader : IEnumerable<string[]>, IDisposable
{
    private readonly StreamReader reader;

    public DelimitedReader(string fileName, Encoding encoding = null)
        : this(new StreamReader(new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite),
            encoding ?? Encoding.UTF8, encoding == null))
    {
    }

    public DelimitedReader(StreamReader reader)
    {
        this.reader = reader;
    }

    public void Dispose()
    {
        reader.Dispose();
    }


    public char EscapeChar { get; set; } = '"';

    public char SeparatorChar { get; set; } = ',';


    private string[] ParseLine(string line)
    {
        List<string> fields = new List<string>();

        char[] charsToSeek = { EscapeChar, SeparatorChar };
        bool isEscaped = false;
        int prevPos = 0;

        while (prevPos < line.Length)
        {
            // If in the escaped mode, seek for the escape char only.
            // Otherwise, seek for the both chars.
            int nextPos = isEscaped
                ? line.IndexOf(EscapeChar, prevPos)
                : line.IndexOfAny(charsToSeek, prevPos);

            if (nextPos == -1)
            {
                // We reached the end of the line
                if (!isEscaped)
                {
                    // Add the rest of the line
                    fields.Add(line.Substring(prevPos, line.Length - prevPos).Trim());
                    break;
                }
                // If there is no closing escape char
                throw new InvalidDataException("The following line has invalid format: " + line);
            }

            char nextChar = line[nextPos];
            if (nextChar == EscapeChar)
            {
                // The next char is the escape char
                if (isEscaped)
                {
                    // If already in the escaped mode
                    fields.Add(line.Substring(prevPos, nextPos - prevPos)); // No Trim
                }
                isEscaped = !isEscaped; // Toggle mode
            }
            else
            {
                // The next char is the delimiter
                fields.Add(line.Substring(prevPos, nextPos - prevPos).Trim());  // Trim
            }

            prevPos = nextPos + 1;
        }

        return fields.ToArray();
    }

    public IEnumerator<string[]> GetEnumerator()
    {
        while (!reader.EndOfStream)
        {
            yield return ParseLine(reader.ReadLine());
        }
    }

    IEnumerator IEnumerable.GetEnumerator()
    {
        return GetEnumerator();
    }
}

In the class above I use the StreamReader.ReadLine method to read a file line by line, and the String.IndexOf/String.IndexOfAny methods to move within the line.

According to my test runs, this approach is a bit faster.

\$\endgroup\$
3
  • \$\begingroup\$ Excellent! I couldn't quite figure out how to use IndexOf in a forward only fashion because of the single character and somehow totally missed IndexOfAny. Points for sure, gonna play with this a bit before accepting. \$\endgroup\$ Nov 1 '16 at 21:11
  • \$\begingroup\$ After having a chance to mess with this quite a bit faster/slower completely depends on the file contents but is far simpler to understand. The problem I have marking it as the answer is that it is unable to successfully parse fields escaped empty fields such as "". Files such as download.cms.gov/nppes/NPI_Files.html end up with double the amount of fields. \$\endgroup\$ Nov 2 '16 at 15:29
  • \$\begingroup\$ This is what I came up with based on your answer: gist.github.com/ByteTerrace/89575400b7ea7e5e3ee774a70f529526. \$\endgroup\$ Nov 2 '16 at 16:15
1
\$\begingroup\$

Constructor

public DelimitedReader(string fileName, char separatorChar = DEFAULT_SEPARATOR_CHAR, char escapeChar = DEFAULT_ESCAPE_CHAR, Encoding encoding = null, int bufferSize = DEFAULT_CHUNK_SIZE)

This constructor is really huge. If you have so many optional parameters then turn them into properties. It'll be clearer to the user that to create a valid object he only needs to specify a single parameter.

Sill better would be an entirely new type holding all the optional parameters as DelimitedReadProperties.

You already plan to extend this class so moving the configuration into a specialized class might a good decision.

Nested function calls & missing Dispose()

return ReadAllChunks(new StreamReader(new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, FileOptions.SequentialScan), encoding), buffer);

The maintability of this line is horrible. Everything's nested in one long chain of calls.

The streams should be disposed.

Static APIs

Every API in this class is static. Why do you need to create an instace of the reader at all? It can be used with the same result and complexity by calling the methods directly.

\$\endgroup\$
3
  • \$\begingroup\$ Five parameters is a large constructor? The only reason that line is even remotely lengthy is due to my choice of naming conventions for the const fields. I could have telescoped constructors or properties but that doesn't really change anything. 100% agreed on Dispose (and will repair) but all that section does is marshal from one API to another. You're dead on about not needing an instance of the reader in order to parse but the class is designed to hold references to the buffer and other properties which allows me to derive a more advanced reader later on. 1-3 remain unanswered. \$\endgroup\$ Nov 1 '16 at 19:57
  • \$\begingroup\$ @Kittoes0124 I wonder how you are going to derive a more advanced reader from this one if you cannot override anything. The new type cannot offer anything new. There's not basic functionality. Nothing is private or protected. Mhmm... \$\endgroup\$
    – t3chb0t
    Nov 1 '16 at 20:10
  • \$\begingroup\$ Trivial, but you should get the point: static IEnumerable<IEnumerable<string>> ReadFields(DelimitedReader reader, int numberOfRowsToSkip) { return reader.Skip(numberOfRowsToSkip); } \$\endgroup\$ Nov 1 '16 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.