4
\$\begingroup\$

I used the following to split sequence into batches:

source
    .Select((item, index) => new {Item = item, Index = index})
    .ToLookup(x => x.Index / batchSize, x => x.Item);

but it requires full materialization of the sequence - memory complexity could be high.

Are there anything ready to be used? I think there is nothing in MoreLinq for that.

Here is a custom solution I am trying now:

public static class Batch
{
    public static IEnumerable<T[]> Split<T>(this IEnumerable<T> source, int batchSize)
    {
        var it = source.GetEnumerator();
        for (var b = split(); b.Length > 0; b = split())
            yield return b;

        T[] split() => Enumerable
            .Range(0, batchSize)
            .TakeWhile(i => it.MoveNext())
            .Select(i => it.Current)
            .ToArray();
    }
}
\$\endgroup\$
6
  • \$\begingroup\$ This sounds like what you want, and there are some pretty simple answers: stackoverflow.com/questions/1258162/… \$\endgroup\$ Oct 11, 2019 at 18:21
  • \$\begingroup\$ @410_Gone GroupBy will make full sequence being materialized before going to the next group - it will try to find everything potentially matching the very first one. I would like to read no more than necessary from the source sequence to reduce memory complexity. \$\endgroup\$ Oct 11, 2019 at 19:05
  • \$\begingroup\$ Doh, silly me. Nevermind. \$\endgroup\$ Oct 11, 2019 at 19:09
  • 1
    \$\begingroup\$ If you keeping yours you should put GetEnumerable should be put in a using statement. \$\endgroup\$ Oct 11, 2019 at 19:17
  • 1
    \$\begingroup\$ @CharlesNRice could you please post an answer? I will upvote :) \$\endgroup\$ Oct 11, 2019 at 19:25

1 Answer 1

3
\$\begingroup\$

Since you mentioned morelinq in your question. It does have the Batch method which is similar. It's an IEnumerable<IEnumerable<TSource>> instead of IEnumerable<TSource[]> but if you look at the source code it's actually an IEnumerable of an array but you can't count on that as it's an implementation detail.

The only issue I see with your implementation is that source.GetEnumerator() returns back IEnumerator<T> which implements IDisposable and should be wrapped in a using statement.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.