Filter out nulls from sequence of nullables to produce sequence of non-nullables

Question

While introducing nullable reference types to our enterprise application's codebase, I found that we are often using LINQ on sequences of nullable reference types where we filter out the nulls using .Where(item => item is not null). The compiler does not see that the resulting sequence should not have nullable items anymore, though, because the Where method has the same type of input and output.

I came up with an extension method that solves the issue:

using System.Collections.Generic;
using System.Linq;

namespace Company.Product.Application.Component.Extensions;

public static class EnumerableOfNullableExtensions
{
    public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source)
    {
        return source.Where(item => item is not null)!;
    }
}

^{See in SharpLab.}

It can be applied to sequences of nullable value types, too, but the items in the resulting sequence are sadly still nullable. An extra .Select(item => item.Value) is needed because after filtering out the nulls, the representation of the remaining values still needs to change, unlike for the reference types¹.

The extension can also be applied to non-nullable types.

So I came up with an improved version that handles nullable value types well and even included an extra extension for casting sequences of nullable reference types to sequences of the corresponding non-nullable type:

using System.Collections.Generic;
using System.Linq;

namespace Company.Product.Application.Component.Extensions;

public static class EnumerableOfNullableExtensions
{
    public static IEnumerable<T> CastNotNull<T>(this IEnumerable<T?> source)
        where T : class
    {
        return source!;
    }

    public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source)
        where T : class
    {
        return source.Where(item => item is not null).CastNotNull();
    }
    
    public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source)
        where T : struct
    {
        return source.Where(item => item.HasValue).Select(item => item!.Value);
    }
}

^{See in SharpLab.}

None of the overloads of the WhereNotNull method accepts non-nullable value types anymore. Non-nullable reference types are still accepted, but that is consistent with the design of nullable reference types. Non-nullable reference type T is assignable to T? and variance rules apply (covariance in the case of IEnumerable).

I noticed that if the null check needs to be performed e.g. on a property of the objects in the sequence, the next use of the property still needs a damnit operator. I believe that this cannot be solved generally.

Is there a way to further improve my improved version of the extensions? Are there any issues with them I do not see?

---

¹: The statement that no Select call is needed for nullable reference types holds only since C# 10 (or since C# 9?). Till then, T? in the signature of a generic method was treated just like T even for reference types. There is a related Q&A on Stack Overflow, which is constrained to C# 8.0. We have no such constraint -- currently, we are using .NET 6 with C# 10, but we will upgrade to .NET 7 and C# 11 once they become generally available. Calling .Select(e => e!) wastes performance -- it wraps the whole enumerable into another enumerable, generates an instance of the lambda, and actually executes it for each element of the collection. Better evade it if possible.

Answer 1

Learn more about nullability

Of what I can see so far from your further elaboration and comments/replies everything comes with pros and cons, but there are a few misconceptions especially with Linq and .Net/C# versions.

Going from here I'll solely address this as a .Net 6 scenario. So to clear the weeds a bit and lay out some background:

source.Where(item => item is not null)!

The Where method does nothing to the elements it is processing and returns the same type coming out as it did going in.

public IEnumerable<T> Where<T>(Predicate<T> condition){}

Even if you filter to only items that are not null, the type system has nothing to forward, since the Where method doesn't do type changes.

Back to

source.Where(item => item is not null)!

The ! at the end still probably isn't satisfying the compiler; there most definitely should be a warning saying so that "it" is not convinced that you've actually made your code null safe. General advice about the ! operator: just try not to use it and learn about what nullability is, and learn coding patterns to clean it up. You'll far more likely end up using it far less than you think.

Linq is optimized but not performant

There is plenty of documentation and benchmarks out there about if you want to write the most performant code: you don't use Linq. Linq is a handy syntax to query objects in an expressive manner that usually should be fairly efficient. So I can't say it is worth the argument to get picky about chaining on another Linq method after another; if anything it's common enough to see Linq chains of 4-8 methods chained together.

.Select(e => e!)

The Select, Cast, and OfType methods are meant for changing type of each element from an enumerable. The same is true of creating another enumerable and a lambda: both are created once. The Select wrapper is just a super simple cheap class, and the lambda is stored and reused for all elements. FAR better than a method that takes an expression per element, yuck.

Compiler can't forward nullability through a Linq chain

Going with this as the example:

public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source)
{
    return source.Where(item => item is not null).Select(item => item!);
}

The compiler can't help you here, nor can the type system make this simpler. In fact what the compiler discovers about an element in the Where method is essentially thrown away when it processes the Select method. What you can do is make all the type information about nullability accessible to both parts of the transformation, i.e. check and switch to non-null. To do this we employ, as in other comments, a delayed enumerable foreach. Rather than make our own enumerable class we use yield to instruct the compiler to treat the method as a delayed enumerable that doesn't cause extra loops of processing of each element.

public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source)
{
    foreach(var item in source)//item might be null
    {
        if(item is T notNullItem)//non exception throwing conversion with success check
        {
            yield return notNullItem;
        }
    }
}

Answer 2

I have a further improvement for all your improvements: You can also check inside the extension methods whether the list itself is null, so you don't have to do it outside.

So instead of:

if (list != null)
{
    foreach (var item in list.WhereNotNull())
    {
        ...
    }
}

You can just write this:

foreach (var item in list.WhereNotNull())
{
    ...
}

The null check inside WhereNotNull() can look like this:

namespace System.Collections.Generic
{
    public static class ExtensionMethods
    {
        public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T> source)
            => source?.Where(item => item != null) ?? Enumerable.Empty<T>();
    }
}

Also note that I use System.Collections.Generic for the namespace. So if you're using an IEnumerable<T> somewhere, you have the extension methods available right away without having to add yet another using. That would be another proposal from me.