# Efficiently remove duplicate elements from the list

I have a legacy code in which ids List has some duplicate value coming and this method is being called from lot of difference places as of now -

public async Task<List<T>> Execute<T>(IList<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class
{
var holder = new List<Task<T>>(ids.Count);
var removeNull = new List<T>(ids.Count);
for (int i = 0; i < ids.Count; i++)
{
var id = ids[i];
holder.Add(ProcessData(policy, ct => mapperFunc(ct, id)));
}

var responses = await Task.WhenAll(holder);
for (int i = 0; i < responses.Length; i++)
{
var response = responses[i];
if (response != null)
{
}
}
return removeNull;
}


I am trying to change above method such that I can remove duplicate stuff from ids list so I came up with below code which does that but I wanted to see if there is any better way we can write below code?

public async Task<List<T>> Execute<T>(IList<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class
{
var noDupsList = new HashSet<int>(ids).ToList();
var holder = new List<Task<T>>(noDupsList.Count);
var removeNull = new List<T>(noDupsList.Count);
for (int i = 0; i < noDupsList.Count; i++)
{
var id = noDupsList[i];
holder.Add(ProcessData(policy, ct => mapperFunc(ct, id)));
}

var responses = await Task.WhenAll(holder);
for (int i = 0; i < responses.Length; i++)
{
var response = responses[i];
if (response != null)
{
}
}
return removeNull;
}


Update

Seeing the answer that came I thought it's good idea to mention how my actual code is - This is how my original code is :

public async Task<List<T>> Execute<T>(IList<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc, string logMessage) where T : class
{
var noDupsList = new HashSet<int>(ids).ToList();
var holder = new List<Task<T>>(noDupsList.Count);
var removeNull = new List<T>(noDupsList.Count);
using (var logMetric = new LogMetric(_logger, TITLE, "DatabaseCall"))
{
logMetric.Message = logMessage;
for (int i = 0; i < noDupsList.Count; i++)
{
var id = noDupsList[i];
holder.Add(ProcessData(policy, ct => mapperFunc(ct, id)));
}

var responses = await Task.WhenAll(holder);
for (int i = 0; i < responses.Length; i++)
{
var response = responses[i];
if (response != null)
{
}
}
logMetric.StatusCode = (removeNull.Count == 0) ? 204 : 200;
}
return removeNull;
}

• Use Linq it will shorten your code and give you more readability to your code. For instance, var uniqueIdsList = ids.Distinct().ToList(); would give you the unique ids directly. – iSR5 Oct 31 '20 at 18:38
• Is Linq efficient here? I have been told its quite expensive and this method will be called at hight throughput so just wanted to make sure. But apart from that do you think we can rewrite this in better way to solve this problem or just change that one line to use linq instead of set? – AndyP Oct 31 '20 at 18:46
• Linq is efficient in mostly, it would do the same thing, except it would be shorter, readable, and extendable. You can try to test it, and see how it would preform with the reset of the code. Compare, then decide. – iSR5 Oct 31 '20 at 19:22
• The problem of given you a decent answer here, is that I have no idea what ProcessData actually should do, neither do I know whether you can change the signature of your method. I would quite possibly keep the return type to Task<IEnumerable<T>> or Task<IReadonlyList<T>> if I have to, and the input type restricted to IEnumerable<int> ids. To be fair you can wrap the entire code in 1 somewhat long one liner with linq with using a Select (to get the tasks) and a Where (to filter out the nulls). Is performance a must here? – Icepickle Oct 31 '20 at 23:33
• Where is logMessage coming? Do you really want to log a number here? If you really need to know if nulls got removed, I guess you could check the hashset length vs the filtered responses length, but I am not getting the logic here completely – Icepickle Nov 1 '20 at 10:22

I think you used a good way to filter out the duplicates, but I don't see why you feel the need to call .ToList() after that. A HashSet is perfectly iterable.

As such, I believe I would write your lengthy code somewhat differently as:

public async Task<IReadonlyList<T>> Execute<T>(IEnumerable<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class
{
new HashSet<int>(ids).Select( id => ProcessData( policy, ct => mapperFunc( ct, id ) ) ) ) )
.Where( response => response != null )
.ToList();
}


It will still remove all the duplicates, await all entries using the Task.WhenAll and remove the null values.

As for knowing if it is more efficient I can't tell without testing. I think it is more readable, though to be fair, I would probably rather assign the results to a variable before filtering them out, saving you some brackets.

public async Task<IReadonlyList<T>> Execute<T>(IEnumerable<int> ids, Policy policy, Func<CancellationToken, int, Task<T>> mapperFunc) where T : class
{
var responses = await Task.WhenAll(
new HashSet<int>(ids).Select( id => ProcessData( policy, ct => mapperFunc( ct, id ) ) ) );

return responses
.Where( response => response != null )
.ToList();
}

• Thanks for your suggestion. I can update my question which has ProcessData method as well so that it can clear your confusion if needed but I have a question on your suggestion. Will this have any problem related to closure issue? Earlier I had closure issue with my original code so that's why I needed to initialize variable like this - var id = ids[i]; to fix that so just making sure your suggestion will not have same problem? – AndyP Nov 1 '20 at 0:31
• @AndyP, No you shouldn't have a problem there, id is already local because of the lambda in select – Icepickle Nov 1 '20 at 0:52
• After checking your answer which I think is really good I was gonna try integrating in my solution but I realize one thing. I removed one thing from my original code which I thought is not relevant here but looking at your answer now I am confuse on how to have that in your suggestion. So I updated my question to provide my full code which has this line using (var logMetric = new LogMetric(_logger, TITLE, "DatabaseCall")) wrapped around my first for loop. – AndyP Nov 1 '20 at 1:53
• I apologize about this as I should have added this in earlier. Now with that detail can we still use your suggestion or we need to change it? If yes, then how I can wrap around using block in your suggestion? Will it be wrapped around var responses block? – AndyP Nov 1 '20 at 1:54
• @AndyP yes, wrap it inside the usings or use a local using statement – Icepickle Nov 1 '20 at 1:57