Efficiently remove duplicate elements from the list

Question

I have a legacy code in which ids List has some duplicate value coming and this method is being called from lot of difference places as of now -

public async Task<List<T>> Execute<T>(IList<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class
{
    var holder = new List<Task<T>>(ids.Count);
    var removeNull = new List<T>(ids.Count);
    for (int i = 0; i < ids.Count; i++)
    {
        var id = ids[i];
        holder.Add(ProcessData(policy, ct => mapperFunc(ct, id)));
    }

    var responses = await Task.WhenAll(holder);
    for (int i = 0; i < responses.Length; i++)
    {
        var response = responses[i];
        if (response != null)
        {
            removeNull.Add(response);
        }
    }
    return removeNull;
}

I am trying to change above method such that I can remove duplicate stuff from ids list so I came up with below code which does that but I wanted to see if there is any better way we can write below code?

public async Task<List<T>> Execute<T>(IList<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class
{
    var noDupsList = new HashSet<int>(ids).ToList();
    var holder = new List<Task<T>>(noDupsList.Count);
    var removeNull = new List<T>(noDupsList.Count);
    for (int i = 0; i < noDupsList.Count; i++)
    {
        var id = noDupsList[i];
        holder.Add(ProcessData(policy, ct => mapperFunc(ct, id)));
    }

    var responses = await Task.WhenAll(holder);
    for (int i = 0; i < responses.Length; i++)
    {
        var response = responses[i];
        if (response != null)
        {
            removeNull.Add(response);
        }
    }
    return removeNull;
}

Update

Seeing the answer that came I thought it's good idea to mention how my actual code is - This is how my original code is :

public async Task<List<T>> Execute<T>(IList<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc, string logMessage) where T : class
{
    var noDupsList = new HashSet<int>(ids).ToList();
    var holder = new List<Task<T>>(noDupsList.Count);
    var removeNull = new List<T>(noDupsList.Count);
    using (var logMetric = new LogMetric(_logger, TITLE, "DatabaseCall"))
    {
        logMetric.Message = logMessage;
        for (int i = 0; i < noDupsList.Count; i++)
        {
            var id = noDupsList[i];
            holder.Add(ProcessData(policy, ct => mapperFunc(ct, id)));
        }

        var responses = await Task.WhenAll(holder);
        for (int i = 0; i < responses.Length; i++)
        {
            var response = responses[i];
            if (response != null)
            {
                removeNull.Add(response);
            }
        }
        logMetric.StatusCode = (removeNull.Count == 0) ? 204 : 200;
    }
    return removeNull;
}

Answer 1

I think you used a good way to filter out the duplicates, but I don't see why you feel the need to call .ToList() after that. A HashSet is perfectly iterable.

As such, I believe I would write your lengthy code somewhat differently as:

public async Task<IReadonlyList<T>> Execute<T>(IEnumerable<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class 
{
    return (await Task.WhenAll(
            new HashSet<int>(ids).Select( id => ProcessData( policy, ct => mapperFunc( ct, id ) ) ) ) )
        .Where( response => response != null )
        .ToList();
}

It will still remove all the duplicates, await all entries using the Task.WhenAll and remove the null values.

As for knowing if it is more efficient I can't tell without testing. I think it is more readable, though to be fair, I would probably rather assign the results to a variable before filtering them out, saving you some brackets.

public async Task<IReadonlyList<T>> Execute<T>(IEnumerable<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class 
{
    var responses = await Task.WhenAll(
            new HashSet<int>(ids).Select( id => ProcessData( policy, ct => mapperFunc( ct, id ) ) ) );

    return responses
        .Where( response => response != null )
        .ToList();
}