I'm implementing a queue to update leaderboards when score data changes in the background. Updating the leaderboard is a relatively expensive operation, so if we say "Update leaderboard 5" and then immediately say "Update leaderboard 5" again before the first one is finished, we should just update leaderboard 5 one time.
To do this I'm using TPL dataflow to create an ActionBlock
which takes an id to update, then calls the subscriptions
delegate which does the actual work. I'm using a BufferBlock
to post to, which checks a HashSet
to see if the leaderboard is already pending update, and if so posts the data to the ActionBlock
, otherwise posts it to a NullTarget
. When a leaderboard is updated, I remove the id from dupeMap
so it can be processed again if data changes.
The goal is to minimize the duplicates, if there's a few edge cases where the same leaderboard might be updated twice, it's not a huge deal. Here's my code, I'd appreciate any comments or feedback since I'm not a complete TPL-Dataflow master (this is the first time I've used anything besides ActionBlock<T>
.
public class LeaderboardQueue : ILeaderboardQueue
{
private Func<int, Task> subscriptions;
private readonly ActionBlock<int> block;
private readonly BufferBlock<int> filter;
private readonly HashSet<int> dupeMap = new(); // Ensure the same leaderboard isn't being recalc'ed twice in the queue, which would be a waste of time
public LeaderboardQueue()
{
block = new ActionBlock<int>(async activityId =>
{
dupeMap.Remove(activityId);
await subscriptions(activityId);
},
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3 });
filter = new BufferBlock<int>();
filter.LinkTo(block, dupeMap.Add);
filter.LinkTo(DataflowBlock.NullTarget<int>());
}
public async Task MarkLeaderboardDirty(int activityId)
{
await filter.SendAsync(activityId);
}
public void Subscribe(Func<int, Task> fnProcess)
{
subscriptions += fnProcess;
}
public async Task Shutdown()
{
filter.Complete();
await filter.Completion;
block.Complete();
await block.Completion;
}
}
I also have a question on the Shutdown
method. Am I correct to Complete()
both blocks, or do I just need to complete one of them? Thanks!
BufferBlock
asfilter
? What does it filter? \$\endgroup\$Complete
on the first block andawait
theCompletion
of the last block. \$\endgroup\$LinkTo
call:new DataflowLinkOptions { PropagateCompletion = true }
\$\endgroup\$dupeMap.Remove(activityId);
call your queue implementation has a race condition. If the processing ofactivityId#1
completed sooner than the second call ofMarkLeaderboardDirty
withactivityId#1
then you process it twice. If the second call happened sooner than the processing then it won't be re-processed. \$\endgroup\$