4
\$\begingroup\$

I used the following to split sequence into batches:

source
    .Select((item, index) => new {Item = item, Index = index})
    .ToLookup(x => x.Index / batchSize, x => x.Item);

but it requires full materialization of the sequence - memory complexity could be high.

Are there anything ready to be used? I think there is nothing in MoreLinq for that.

Here is a custom solution I am trying now:

public static class Batch
{
    public static IEnumerable<T[]> Split<T>(this IEnumerable<T> source, int batchSize)
    {
        var it = source.GetEnumerator();
        for (var b = split(); b.Length > 0; b = split())
            yield return b;

        T[] split() => Enumerable
            .Range(0, batchSize)
            .TakeWhile(i => it.MoveNext())
            .Select(i => it.Current)
            .ToArray();
    }
}
\$\endgroup\$
  • \$\begingroup\$ This sounds like what you want, and there are some pretty simple answers: stackoverflow.com/questions/1258162/… \$\endgroup\$ – 410_Gone Oct 11 at 18:21
  • \$\begingroup\$ @410_Gone GroupBy will make full sequence being materialized before going to the next group - it will try to find everything potentially matching the very first one. I would like to read no more than necessary from the source sequence to reduce memory complexity. \$\endgroup\$ – Dmitry Nogin Oct 11 at 19:05
  • \$\begingroup\$ Doh, silly me. Nevermind. \$\endgroup\$ – 410_Gone Oct 11 at 19:09
  • 1
    \$\begingroup\$ If you keeping yours you should put GetEnumerable should be put in a using statement. \$\endgroup\$ – CharlesNRice Oct 11 at 19:17
  • 1
    \$\begingroup\$ @CharlesNRice could you please post an answer? I will upvote :) \$\endgroup\$ – Dmitry Nogin Oct 11 at 19:25
3
\$\begingroup\$

Since you mentioned morelinq in your question. It does have the Batch method which is similar. It's an IEnumerable<IEnumerable<TSource>> instead of IEnumerable<TSource[]> but if you look at the source code it's actually an IEnumerable of an array but you can't count on that as it's an implementation detail.

The only issue I see with your implementation is that source.GetEnumerator() returns back IEnumerator<T> which implements IDisposable and should be wrapped in a using statement.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.