Creating a CSV stream from an object

Question

I have this code that gets a CSV string from a list of objects where the first row in the string is the object's properties and the rows underneath are the properties values. I then create a MemoryStream of the string so it's about the same as reading from an actual .csv file into a Stream. Is there any way to cutdown on this code and perhaps do more with LINQ?

public class Lead
{
    public string FirstName { get; set; }   
    public string MiddleName { get; set; }
    public string LastName { get; set; }
}
...

List<Lead> leads = new List<Lead>();
Lead lead1 = new Lead();
lead1.FirstName = "Joe";
lead1.MiddleName = "M";
lead1.LastName = "DiFabio";
leads.Add(lead1);

Lead lead2 = new Lead();
lead2.FirstName = "Dave";
lead2.MiddleName = "";
lead2.LastName = "Palacios";
leads.Add(lead2);

Lead lead3 = new Lead();
lead3.FirstName = "Andy";
lead3.MiddleName = "A";
lead3.LastName = "Nevers";
leads.Add(lead3);

List<string> headers = typeof(Lead).GetProperties().Select(x => x.Name).ToList();
string s = string.Join(",", headers);

foreach (Lead lead in leads)
{
    List<object> values = typeof(Lead).GetProperties().Select(prop => prop.GetValue(lead)).ToList();
    string ss = string.Join(",", values);
    s = $"{s}\n{ss}\n";
}

MemoryStream stream = new MemoryStream(Encoding.UTF8.GetBytes(s));

Answer 1

Let's do some performance tweaks.

I will measure the time simply with Stopwatch and first of all I'll cleanup your code to generate proper CSV (without reduntant empty lines as initial).

List<string> headers = typeof(Lead).GetProperties().Select(x => x.Name).ToList();
string s = string.Join(",", headers) + Environment.NewLine;

foreach (Lead lead in leads)
{
    List<object> values = typeof(Lead).GetProperties().Select(prop => prop.GetValue(lead)).ToList();
    string ss = string.Join(",", values);
    s += $"{ss}{Environment.NewLine}";
}

Environment.NewLine returns \r\n on Windows and \n on Linux.

Output will be

FirstName,MiddleName,LastName
Joe,M,DiFabio
Dave,,Palacios
Andy,A,Nevers

Looks like CSV :)

Then to measure performance I'll imagine that you have very large amount of data e.g. 30000 lines. I'll run the test in a loop of 10000 iterations.

TestOriginal

private static string TestOriginal(List<Lead> leads)
{
    List<string> headers = typeof(Lead).GetProperties().Select(x => x.Name).ToList();
    string s = string.Join(",", headers) + Environment.NewLine;

    for (int i = 0; i < 10000; i++)
        foreach (Lead lead in leads)
        {
            List<object> values = typeof(Lead).GetProperties().Select(prop => prop.GetValue(lead)).ToList();
            string ss = string.Join(",", values);
            s += $"{ss}{Environment.NewLine}";
        }
    return s;
}

Let's run.

Stopwatch sw = new Stopwatch();
sw.Start();
string s = TestOriginal(leads);
sw.Stop();
Console.WriteLine("Original: {0}ms", sw.ElapsedMilliseconds);

Original: 3162ms

Ugh. :)

Almost everyone knows that Reflection is slow. Let's somewhat optimize that.

ReflectionTweak

private static string ReflectionTweak(List<Lead> leads)
{
    PropertyInfo[] properties = typeof(Lead).GetProperties();
    List<string> headers = properties.Select(x => x.Name).ToList();
    string s = string.Join(",", headers) + Environment.NewLine;

    for (int i = 0; i < 10000; i++)
        foreach (Lead lead in leads)
        {
            List<object> values = properties.Select(prop => prop.GetValue(lead)).ToList();
            string ss = string.Join(",", values);
            s += $"{ss}{Environment.NewLine}";
        }
    return s;
}

Go go go

sw.Restart();
string sr = ReflectionTweak(leads);
sw.Stop();
Console.WriteLine("Reflection tweak: {0}ms, match: {1}", sw.ElapsedMilliseconds, s == sr);

Output with matching the string to original one

Reflection tweak: 3040ms, match: True

Better but still slow

One more idea - StringBuilder. Afaik, it's the fastest way to build a string (maybe except some unsafe/unmanaged code).

StringBuilderTweak

private static string StringBuilderTweak(List<Lead> leads)
{
    StringBuilder sb = new StringBuilder();
    PropertyInfo[] properties = typeof(Lead).GetProperties();
    sb.AppendLine(string.Join(",", properties.Select(x => x.Name)));

    for (int i = 0; i < 10000; i++)
        foreach (Lead lead in leads)
        {
            sb.AppendLine(string.Join(",", properties.Select(prop => prop.GetValue(lead))));
        }
    return sb.ToString();
}

Interesting...

sw.Restart();
string sb = StringBuilderTweak(leads);
sw.Stop();
Console.WriteLine("StringBuilder tweak: {0}ms, match: {1}", sw.ElapsedMilliseconds, s == sb);

Output

StringBuilder tweak: 24ms, match: True

WOW!!!

By the way, you may write the output directly to MemoryStream with StreamWriter.

StreamWriterTweak

private static MemoryStream StreamWriterTweak(List<Lead> leads)
{
    MemoryStream ms = new MemoryStream();
    // Encoding.UTF8 produces stream with BOM, new UTF8Encoding(false) - without BOM
    using (StreamWriter sw = new StreamWriter(ms, new UTF8Encoding(false), 8192, true)) 
    {
        PropertyInfo[] properties = typeof(Lead).GetProperties();
        sw.WriteLine(string.Join(",", properties.Select(x => x.Name)));

        for (int i = 0; i < 10000; i++)
            foreach (Lead lead in leads)
            {
                sw.WriteLine(string.Join(",", properties.Select(prop => prop.GetValue(lead))));
            }
    }
    ms.Position = 0;
    return ms;
}

Must be fast too...

sw.Restart();
MemoryStream stream = StreamWriterTweak(leads);
sw.Stop();
Console.WriteLine("StreamWriter tweak: {0}ms, match: {1}", sw.ElapsedMilliseconds, s == Encoding.UTF8.GetString(stream.ToArray()));

Output

StreamWriter tweak: 28ms, match: True

Yes!

---

The fastest

StringBuilder sb = new StringBuilder();
PropertyInfo[] properties = typeof(Lead).GetProperties();
sb.AppendLine(string.Join(",", properties.Select(x => x.Name)));

foreach (Lead lead in leads)
{
    sb.AppendLine(string.Join(",", properties.Select(prop => prop.GetValue(lead))));
}
string s = sb.ToString();

The whole code to reproduce and play more

internal class Program
{
    public static void Main(string[] args)
    {
        List<Lead> leads = new List<Lead>();
        Lead lead1 = new Lead();
        lead1.FirstName = "Joe";
        lead1.MiddleName = "M";
        lead1.LastName = "DiFabio";
        leads.Add(lead1);

        Lead lead2 = new Lead();
        lead2.FirstName = "Dave";
        lead2.MiddleName = "";
        lead2.LastName = "Palacios";
        leads.Add(lead2);

        Lead lead3 = new Lead();
        lead3.FirstName = "Andy";
        lead3.MiddleName = "A";
        lead3.LastName = "Nevers";
        leads.Add(lead3);

        Stopwatch sw = new Stopwatch();
        sw.Start();
        string s = TestOriginal(leads);
        sw.Stop();
        Console.WriteLine("Original: {0}ms", sw.ElapsedMilliseconds);
        sw.Restart();
        string sr = ReflectionTweak(leads);
        sw.Stop();
        Console.WriteLine("Reflection tweak: {0}ms, match: {1}", sw.ElapsedMilliseconds, s == sr);
        sw.Restart();
        string sb = StringBuilderTweak(leads);
        sw.Stop();
        Console.WriteLine("StringBuilder tweak: {0}ms, match: {1}", sw.ElapsedMilliseconds, s == sb);

        //MemoryStream stream = new MemoryStream(Encoding.UTF8.GetBytes(s));

        sw.Restart();
        MemoryStream stream = StreamWriterTweak(leads);
        sw.Stop();
        Console.WriteLine("StreamWriter tweak: {0}ms, match: {1}", sw.ElapsedMilliseconds, s == Encoding.UTF8.GetString(stream.ToArray()));

        Console.ReadKey();
    }

    private static string TestOriginal(List<Lead> leads)
    {
        List<string> headers = typeof(Lead).GetProperties().Select(x => x.Name).ToList();
        string s = string.Join(",", headers) + Environment.NewLine;

        for (int i = 0; i < 10000; i++)
            foreach (Lead lead in leads)
            {
                List<object> values = typeof(Lead).GetProperties().Select(prop => prop.GetValue(lead)).ToList();
                string ss = string.Join(",", values);
                s += $"{ss}{Environment.NewLine}";
            }
        return s;
    }

    private static string ReflectionTweak(List<Lead> leads)
    {
        PropertyInfo[] properties = typeof(Lead).GetProperties();
        List<string> headers = properties.Select(x => x.Name).ToList();
        string s = string.Join(",", headers) + Environment.NewLine;

        for (int i = 0; i < 10000; i++)
            foreach (Lead lead in leads)
            {
                List<object> values = properties.Select(prop => prop.GetValue(lead)).ToList();
                string ss = string.Join(",", values);
                s += $"{ss}{Environment.NewLine}";
            }
        return s;
    }

    private static string StringBuilderTweak(List<Lead> leads)
    {
        StringBuilder sb = new StringBuilder();
        PropertyInfo[] properties = typeof(Lead).GetProperties();
        sb.AppendLine(string.Join(",", properties.Select(x => x.Name)));

        for (int i = 0; i < 10000; i++)
            foreach (Lead lead in leads)
            {
                sb.AppendLine(string.Join(",", properties.Select(prop => prop.GetValue(lead))));
            }
        return sb.ToString();
    }

    private static MemoryStream StreamWriterTweak(List<Lead> leads)
    {
        MemoryStream ms = new MemoryStream();
        // Encoding.UTF8 produces stream with BOM, new UTF8Encoding(false) - without BOM
        using (StreamWriter sw = new StreamWriter(ms, new UTF8Encoding(false), 8192, true))
        {
            PropertyInfo[] properties = typeof(Lead).GetProperties();
            sw.WriteLine(string.Join(",", properties.Select(x => x.Name)));

            for (int i = 0; i < 10000; i++)
                foreach (Lead lead in leads)
                {
                    sw.WriteLine(string.Join(",", properties.Select(prop => prop.GetValue(lead))));
                }
        }
        ms.Position = 0;
        return ms;
    }

    public class Lead
    {
        public string FirstName { get; set; }
        public string MiddleName { get; set; }
        public string LastName { get; set; }
    }
}

Answer 2

@aepot has done a good job of giving you multiple approaches to get the results you need. I can't think of possible faster way than StringBuilder approach when using Reflection. Though these tests have been considered to be more generalized so it can be reused on multiple objects with no extra code required. However, if you really concerned about performance and you have a very maintainable objects that needs to be converted to CSV. You can adjust their models to fit your needs. For instance :

public class Lead
{
    public string FirstName { get; set; }
    public string MiddleName { get; set; }
    public string LastName { get; set; }
}

You can do this :

public class Lead
{
    public string FirstName { get; set; }
    
    public string MiddleName { get; set; }
    
    public string LastName { get; set; }
    
    public static string GetCSVHeader()
    {
        return $"FirstName,MiddleName,LastName";
    }
    
    public override string ToString()
    {
        return $"{FirstName},{MiddleName},{LastName}";
    }   
}

Now, you can do this using StringBuilder :

public string ToCSV(IEnumerable<Lead> leads)
{
    var sb = new StringBuilder();
    
    sb.AppendLine(Lead.GetCSVHeader());

    foreach(var lead in leads)
    {
        sb.AppendLine(lead.ToString());
    }

    return sb.ToString();
}

This would give you the fastest possible performance you can get without Reflection.

Answer 3

You could go straight to a Stream. To make this a bit more generic we can create an IFormatProvider and put the CSV code in there

public class CSVFormatter<T> : IFormatProvider, ICustomFormatter
{
    // list of converts from types to string
    private static IDictionary<Type, MethodInfo> Converters;
    private static Func<T, string[]> ConvertFunc;
    private static string Header;

    static CSVFormatter()
    {
        Converters = typeof(Convert).GetMethods()
             .Where(x => x.Name == nameof(System.Convert.ToString) && x.GetParameters().Length == 1)
             .ToDictionary(x => x.GetParameters()[0].ParameterType, x => x);

        var properties = typeof(T).GetProperties();
        ConvertFunc = CreateConvertFunc(properties);
        Header = string.Join(",", properties.Select(x => x.Name)) + Environment.NewLine;
    }

    // Creates the Func using Expression Trees to build it on the fly
    private static Func<T, string[]> CreateConvertFunc(PropertyInfo[] properties)
    {
        var parameter = Expression.Parameter(typeof(T), "item");
        var propToStrings = properties.Select<PropertyInfo, Expression>(p =>
        {
            Expression prop = Expression.Property(parameter, p);
            if (Converters.TryGetValue(p.PropertyType, out MethodInfo convertMethod))
            {
                prop = Expression.Convert(prop, typeof(string), convertMethod);
            }
            else if (p.PropertyType != typeof(string))
            {
                prop = Expression.Convert(prop, typeof(string), Converters[typeof(object)]);
            }

            var value = Expression.Variable(typeof(string), "value");
            return Expression.Block(new[] { value },
                Expression.Assign(value, prop),
                Expression.Condition(Expression.Equal(value, Expression.Constant(null, typeof(string))),
                Expression.Constant(string.Empty, typeof(string)), value));
        });
        var propValues = Expression.NewArrayInit(typeof(string), propToStrings);
        var func = Expression.Lambda<Func<T, string[]>>(propValues, parameter);
        return func.Compile();
    }

    public string Format(string format, object arg, IFormatProvider formatProvider)
    {
        if (format == "Header")
        {
            return Header;
        }
        if (arg is T)
        {
            var item = (T)arg;
            return string.Join(",", ConvertFunc(item)) + Environment.NewLine;
        }
        var formattable = arg as IFormattable;
        if (formattable != null)
        {
            return formattable.ToString(format, formatProvider);
        }
        return arg?.ToString() ?? string.Empty;
    }


    public object GetFormat(Type formatType) => formatType == typeof(ICustomFormatter) ? this : null;
}

While we do use reflection a bit we use Expression Trees to build up a func. Expression Tree are slow to compile but once compiled they are as fast as native code. I also added in handling if values null and if property isn't a string. If you only have a hand full of options then the expression trees are going to be slower but remember this is cached in a static field so would have the hit once and rest of time would be faster than reflection.

Now we can implement the Stream class

public class EnumerableStream<T> : Stream
{
    private readonly IEnumerator<T> source;
    private readonly IFormatProvider formatProvider;
    private byte[] buffer = new byte[0];
    public EnumerableStream(IEnumerable<T> source, IFormatProvider formatProvider)
    {
        this.source = source.GetEnumerator();
        this.formatProvider = formatProvider;
        // start with header in the buffer
        var customFormatter = formatProvider.GetFormat(typeof(ICustomFormatter)) as ICustomFormatter;
        try
        {
            var header = customFormatter?.Format("Header", null, formatProvider);
            if (!string.IsNullOrWhiteSpace(header))
            {
                this.buffer = Encoding.UTF8.GetBytes(header);
            }
        }
        catch (FormatException)
        {
           // no header so just don't set buffer
        }
    }

    public override bool CanRead => true;

    public override bool CanSeek => false;

    public override bool CanWrite => false;

    public override long Length => throw new NotSupportedException();

    public override long Position { get => throw new NotSupportedException(); set => throw new NotSupportedException(); }

    public override void Flush()
    {
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (buffer == null)
        {
            throw new ArgumentNullException(nameof(buffer));
        }
        if (offset < 0)
        {
            throw new ArgumentOutOfRangeException(nameof(offset));
        }
        if (count < 0)
        {
            throw new ArgumentOutOfRangeException(nameof(count));
        }
        if (buffer.Length - offset < count)
        {
            throw new ArgumentException(nameof(buffer));
        }
        if (count == 0)
        {
            return 0;
        }

        var read = 0;
        // read from leftover buffer first
        if (this.buffer.Length > 0)
        {
            if (this.buffer.Length <= count)
            {
                Array.Copy(this.buffer, 0, buffer, offset, this.buffer.Length);
                read = this.buffer.Length;
                this.buffer = new byte[0];
            }
            else
            {
                Array.Copy(this.buffer, 0, buffer, offset, count);
                read = count;
                var newBuffer = new byte[this.buffer.Length - count];
                Array.Copy(this.buffer, count, newBuffer, 0, newBuffer.Length);
                this.buffer = newBuffer;
            }
        }

        while (read < count)
        {
            if (this.source.MoveNext())
            {
                var data = Encoding.UTF8.GetBytes(string.Format(this.formatProvider, "{0}", this.source.Current));
                if (data.Length <= count - read)
                {
                    Array.Copy(data, 0, buffer, offset + read, data.Length);
                    read += data.Length;
                }
                else
                {
                    Array.Copy(data, 0, buffer, offset + read, count - read);
                    var newBuffer = new byte[data.Length - (count - read)];
                    Array.Copy(data, count - read, newBuffer, 0, newBuffer.Length);
                    this.buffer = newBuffer;
                    read = count;
                }
            }
            else
            {
                return read;
            }
        }
        return read;
    }


    public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
    public override void SetLength(long value) => throw new NotSupportedException();
    public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();

    protected override void Dispose(bool disposing)
    {
        if (disposing)
        {
            this.source.Dispose();
        }
        base.Dispose(disposing);
    }
}

Nothing too fancy going on in the Stream class just looping through the Enumerable and calling the formatter. Only trick is the Array copy and keeping the buffer in sync but that's not too difficult.

Then to make it easy we can create an extension method on IEnumerable

public static class EnumerableExtensions
{
    public static Stream ToCSVStream<TSource>(this IEnumerable<TSource> source)
    {
        if (source == null)
        {
            throw new ArgumentNullException(nameof(source));
        }
        return new EnumerableStream<TSource>(source, new CSVFormatter<TSource>());
    }
}

Once you have the List of Leads or IEnumerable and can be any just have to call

leads.ToCSVStream();

Since it's a stream can pass it to any method that is expecting a stream.