I would suggest you use something like System.Threading.Tasks.Dataflow or System.Reactive from NuGet.
Dataflow in particular is made to feed data through a pipeline.
Consider passing the CancellationToken to the delegate.
Always validate arguments
Smaller more specializes methods can make the code easier to understand.
public static Task<...
thanks to @aepot for helping me clean this up.
Changes from original:
Tasks now start immediately when being added to tasks list to bypass the need to start them.
SemaphoreSlim is used over Parallel.ForEach to control MaxDegreeOfParallelism as to not not deadlock.
tasks.Where(x=> !x.IsFaulted) is removed because its sort of a race-condition where a task ...
A TPL inside async implementation: mixing of implementaion concepts makes the code complicated or even buggy. Finally Parallel.ForEach is blocking the current thread which isn't allowed in async programming.
Also avoid blocking operations on Task e.g. .Wait() or .Result for not completed Task.
But I suggest totally different approach. Consider the simplified ...