Buffered read from a large list

Question

There is a large list of email addresses and there is a limitation of max 100 email addresses to process once. This is the code I have written.

        //Create a list of email addresses to test
        List<string> emails = new List<string>();
        for (int i = 0; i < 10000; i++)
        {
            emails.Add(string.Format("email{0}@domain.com", i));
        }

        //buffered read
        int remainingCount = emails.Count;
        int maxToReadOnce = 100;
        int startIndex = 0;
        while(remainingCount > 0)
        {
            int countToRead = (remainingCount > maxToReadOnce) ? maxToReadOnce : remainingCount;
            List<string> myemails = emails.GetRange(startIndex, countToRead);
            startIndex = startIndex + countToRead;
            remainingCount = remainingCount - countToRead;
            Process(myemails); // Method that process the email addresses 
        }

Answer 1

Your code is pretty well written and I can't see anything too much I'd change...

(remainingCount > maxToReadOnce) ? maxToReadOnce : remainingCount

might be clearer as

Math.Min(maxToReadOnce, remainingCount) 

but other than that it all looks really sensible.

---

That said, by mixing the batching into the processing code both are harder to read. Batching (splitting/sharding/whatever you want to call it) comes up a lot - I have an extension method lying around for it:

public static IEnumerable<IEnumerable<T>> Batch<T>(this IEnumerable<T> enumerable, int batchSize)
{
    var c = 0;
    var batch = new List<T>();
    foreach (var item in enumerable)
    {
        batch.Add(item);
        if (++c % batchSize == 0)
        {
            yield return batch;
            batch = new List<T>();
        }
    }
    if (batch.Count != 0)
    {
        yield return batch;
    }
}

There are a whole bunch of variations on the above - feel free to make any changes... The ++c % batchSize == 0 has always bothered me a bit but I like the terseness of it.

With that extension method in place you can simplify your code to:

foreach (var batchOfEmails in emails.Batch(maxToReadOnce))
{
    Process(batchOfEmails);
}

(if you don't like var the type will be IEnumerable<IEnumerable<string>>)