4
\$\begingroup\$

I used the following to split sequence into batches:

source
    .Select((item, index) => new {Item = item, Index = index})
    .ToLookup(x => x.Index / batchSize, x => x.Item);

but it requires full materialization of the sequence - memory complexity could be high.

Are there anything ready to be used? I think there is nothing in MoreLinq for that.

Here is a custom solution I am trying now:

public static class Batch
{
    public static IEnumerable<T[]> Split<T>(this IEnumerable<T> source, int batchSize)
    {
        var it = source.GetEnumerator();
        for (var b = split(); b.Length > 0; b = split())
            yield return b;

        T[] split() => Enumerable
            .Range(0, batchSize)
            .TakeWhile(i => it.MoveNext())
            .Select(i => it.Current)
            .ToArray();
    }
}
\$\endgroup\$
  • \$\begingroup\$ This sounds like what you want, and there are some pretty simple answers: stackoverflow.com/questions/1258162/… \$\endgroup\$ – Der Kommissar Oct 11 '19 at 18:21
  • \$\begingroup\$ @410_Gone GroupBy will make full sequence being materialized before going to the next group - it will try to find everything potentially matching the very first one. I would like to read no more than necessary from the source sequence to reduce memory complexity. \$\endgroup\$ – Dmitry Nogin Oct 11 '19 at 19:05
  • \$\begingroup\$ Doh, silly me. Nevermind. \$\endgroup\$ – Der Kommissar Oct 11 '19 at 19:09
  • 1
    \$\begingroup\$ If you keeping yours you should put GetEnumerable should be put in a using statement. \$\endgroup\$ – CharlesNRice Oct 11 '19 at 19:17
  • 1
    \$\begingroup\$ @CharlesNRice could you please post an answer? I will upvote :) \$\endgroup\$ – Dmitry Nogin Oct 11 '19 at 19:25
3
\$\begingroup\$

Since you mentioned morelinq in your question. It does have the Batch method which is similar. It's an IEnumerable<IEnumerable<TSource>> instead of IEnumerable<TSource[]> but if you look at the source code it's actually an IEnumerable of an array but you can't count on that as it's an implementation detail.

The only issue I see with your implementation is that source.GetEnumerator() returns back IEnumerator<T> which implements IDisposable and should be wrapped in a using statement.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.