3
\$\begingroup\$

I want to get a recursive list of files in a directory but I want relative paths and not the full paths that Directory.GetDirectories() would return using the AllDirectories search option. This is because I want to move the list of files to another location and perform a custom check on each file. I could try converting each absolute path to a relative path but that's a bit messy, so I decided to create a recursive function that generates relative paths as it goes.

public static string[] GetRelativeFilesRecursive(string path)
{
    IEnumerable<string> GetRelativeFilesRecursive(string basePath, string subFolder)
    {
        IEnumerable<string> getRelativeArray(Func<string, IEnumerable<string>> func)
        {
            return func(Path.Combine(basePath, subFolder)).Select(file => Path.Combine(subFolder, Path.GetFileName(file)));
        }

        IEnumerable<string> files = getRelativeArray(Directory.GetFiles);
        IEnumerable<string> directories = getRelativeArray(Directory.GetDirectories);

        foreach (string directory in directories)
        {
            files = files.Concat(GetRelativeFilesRecursive(basePath, directory));
        }

        return files;
    }

    return GetRelativeFilesRecursive(path, "").ToArray();
}

I'm using nested functions to make it more modular, i.e. I'd rather have one function that I can copy and paste as a whole than have multiple functions that could become separated. I'm using IEnumerable<string> instead of string[] because otherwise I'd have to call ToArray() on every Concat() call and I'm worried that would be bad for performance.

What do you think? Is this a good way to do this?

\$\endgroup\$
4
\$\begingroup\$

I would return IEnumerable<string> in the outmost function too to avoid the call to ToArray() in order to let the client decide how to handle the output:

public static IEnumerable<string> GetRelativeFilesRecursive(string path)
{
  ...

  ...

  return GetRelativeFilesRecursive(path, "");
}

As I see it, you're producing relative paths to the initial root path.

I think, it can be done a little more easier in a way like this:

IEnumerable<string> GetRelativePaths(string root)
{
  int rootLength = root.Length + (root[root.Length - 1] == '\\' ? 0 : 1);

  foreach (string path in Directory.GetFiles(root, "*", SearchOption.AllDirectories))
  {
    yield return path.Remove(0, rootLength);
  }
}
\$\endgroup\$
  • \$\begingroup\$ When I said that was messy, I was kind of worried the paths that GetFiles() returns could be formatted in some way so that the root path becomes a different length. But I just ran some tests by inserting white space, .., etc. and it seems that GetFiles() will use the unaltered path input no matter what. So this may be safe after all. Do you have any insight into GetFiles()'s internal implementation? \$\endgroup\$ – Kyle Delaney Mar 29 '18 at 13:24
  • \$\begingroup\$ @KyleDelaney Not really, but you can search the source code at referencesource.microsoft.com. Search for System.IO.Directory? \$\endgroup\$ – Henrik Hansen Mar 29 '18 at 13:53
  • \$\begingroup\$ I found the method, but the source code appears to be rather difficult to navigate. \$\endgroup\$ – Kyle Delaney Apr 16 '18 at 17:56
5
\$\begingroup\$

Building on Henrick's answer, using Ling to return the IEnumerable:

return Directory.GetFiles(root, "*", SearchOption.AllDirectories)
              .Select(p => p.Remove(0, rootLength));
\$\endgroup\$
1
\$\begingroup\$

Review from the future

This review if from the future so you don't see it right now, it isn't there yet ;-)

I will write it in a couple of weeks or months when the new System.Memory package is official. But if you think you will like it then, you can leave your future vote anyway ;-P


The components necessary for the changes I describe here are not released yet and the new features are still in beta but they are very interesing and worth trying.


Span<T>

So, we finally can use the new [ReadOnly]Span<T> and [ReadOnly]Memory<T> types. But what are they?

Span<T> is a new type we are adding to the platform to represent contiguous regions of arbitrary memory, with performance characteristics on par with T[]. Its APIs are similar to the array, but unlike arrays, it can point to either managed or native memory, or to memory allocated on the stack. [2]

One of the APIs that makes it so useful is the Slice method that is like a Substring for string but without copying anything.

but

The full API surface of Span is not yet finalized [2]

The Slice however should stay.

Unfortunatelly we cannot use this type here becuase

Span is a ref-like type as it contains a ref field, and ref fields can refer not only to the beginning of objects like arrays, but also to the middle of them. [1]

These references are called interior pointers, and tracking them is a relatively expensive operation for the .NET runtime’s garbage collector. As such, the runtime constrains these refs to only live on the stack, as it provides an implicit low limit on the number of interior pointers that might be in existence. [1]

and because

As a result, Span instances can only live on the stack, not on the heap. This means you can’t box spans. [1]

but the most import point against it in our use-case is that

you can’t use Span as a generic argument, as instances of that type argument could end up getting boxed or otherwise stored to the heap (and there’s currently no “where T : ref struct” constraint available). [1]

and we need something that we can use for IEnumerable<T> to enumerate paths.


Memory<T> to the rescue

if Span<T> can’t be stored to the heap and thus can’t be persisted across asynchronous operations, what’s the answer? Memory<T>. [1]

This can be use just like a Span<T> so we can put it inside IEnumerable<T> and rewrite the method for enumerating paths like this:

public static IEnumerable<ReadOnlyMemory<char>> EnumeratePaths(string path, SearchOption searchOption)
{
    return
        Directory
            .EnumerateFiles(path, "*", searchOption)
            .Concat(Directory.EnumerateDirectories(path, "*", searchOption))
            .Select(name => new ReadOnlyMemory<char>(name.ToArray()));
}

It concatenates the results of EnumerateFiles and EnumerateDirectories and returns them as ReadOnlyMemory<char>.

In order to get the relative path, we neither use the Replace nor the Substring methods but instead the Slice:

public static ReadOnlyMemory<char> RelativePath(this ReadOnlyMemory<char> path, int relativePathStart)
{
    return path.Slice(relativePathStart, path.Length - relativePathStart);
}

With it, we can create any relative path we want without copying strings:

static void Main(string[] args)
{
    var rootPath = @"C:\foo\bar";
    foreach (var path in EnumeratePaths(rootPath, SearchOption.AllDirectories))
    {
        Console.WriteLine(path.RelativePath(rootPath.Length).Span.ToString());
    }

    Console.ReadKey();
}

String slicing might be in some scenarios a very expensive operation so it's good to know that there is a new feature that can significant improve its performance, when necessary.

Currently, System.String.Substring is the main .NET API for creating slices of a string, but the API is inefficient as it creates a new string to represent the slice and copies the characters from the original string to the new string slice. Because of this inefficiency, high performance servers shy away from using this API, where they can (in their internals), and pay the cost in the publicly facing APIs. [2]

There are also other use-cases that can be optimized with the new types like parsing or formatting.


What’s Next?

The types, methods, runtime optimizations, and other elements discussed here are on track to being included in .NET Core 2.1. [1]

After that, they are expected to make their way into the .NET Framework. Let's hope they will...


Sources

  1. C# - All About Span: Exploring a New .NET Mainstay
  2. Span<T> - dotnet/corefxlab
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.