I used the following to split sequence into batches:
source
.Select((item, index) => new {Item = item, Index = index})
.ToLookup(x => x.Index / batchSize, x => x.Item);
but it requires full materialization of the sequence - memory complexity could be high.
Are there anything ready to be used? I think there is nothing in MoreLinq
for that.
Here is a custom solution I am trying now:
public static class Batch
{
public static IEnumerable<T[]> Split<T>(this IEnumerable<T> source, int batchSize)
{
var it = source.GetEnumerator();
for (var b = split(); b.Length > 0; b = split())
yield return b;
T[] split() => Enumerable
.Range(0, batchSize)
.TakeWhile(i => it.MoveNext())
.Select(i => it.Current)
.ToArray();
}
}
GroupBy
will make full sequence being materialized before going to the next group - it will try to find everything potentially matching the very first one. I would like to read no more than necessary from the source sequence to reduce memory complexity. \$\endgroup\$