I want test review on my unit test code. As I got feeling like this unit test code given in Listing 2 is quite not trustworthy.
My production code(Listing 1) is accessing or precisely downloading the files from Azure's DataLake Storage Gen2. Those files are placed by an upstream system of our application. The files are organized hierarchy of directories.
A typical hierarchy of the file will be:
sample-container/process_Id_12/task_id_34/year2024/month7/day10/file.csv
The functionality of Subject Under test (GetSalesFiles method of DataLakeService class) is navigating to correct directory, identify the correct file by a pattern and finally get it as stream. The DataLakeService object will hold information/credentials to interact with ADLS and also hold other state like paths need to be navigated.
More detail on My Azure Data Lake storage:
sample-container is filesystem in My Data Lake Storage. This will be called as container in case of Azure Blob Storage. I consider this as a root folder. Here I have used the library Azure.Storage.Files.DataLake to access ADLS and it's source code is here
Listing 1:-
using Azure.Storage.Files.DataLake;
public class DataLakeService
{
//this object is pointing to the directory "sample-container/download/"
private DataLakeDirectoryClient downloadClient;
private string salesPath;
private string auditPath;
public DataLakeService(DataLakeDirectoryClient dataLakeDirectoryClient, PathConfiguration pathConfig)
{
this.downloadClient = dataLakeDirectoryClient;
this.salesPath = pathConfig.SalesPath; // "process_Id_12/task_id_34/"
this.auditPath = pathConfig.AuditPath; // "process_Id_12/task_id_50/"
}
private static Func<DataLakeDirectoryClient, DataLakeFileSystemClient> _getParentFileSystemClient ;
/// <summary>
/// Set this purpose for mocking purpose only. By Default this delegate property
/// will provide Azure Data Lake Storage library's static method to retrieve
/// the parent file system client. For production code no need to set this
/// property.
/// </summary>
public static Func<DataLakeDirectoryClient, DataLakeFileSystemClient> GetParentFileSystemClient
{
private get
{
if (_getParentFileSystemClient is null)
{
return SpecializedDataLakeExtentions.GetParentFileSystemClient;
}
return _getParentFileSystemClient;
}
set { _getParentFileSystemClient = value; }
} //For making the code testable later point of time from project beginning,
//I have created this property. This will be used only internal
public IAsyncEnumerable<Stream> GetSalesFiles(DateTime processingDate)
{
DataLakeDirectoryClient dir = downloadClient.GetSubDirectoryClient(this.salesPath)
.GetSubDirectoryClient($"year{processingDate.Year}")
.GetSubDirectoryClient($"month{processingDate.Month}");
if(!dir.Exists()){
return AsyncEnumerable.Empty<Stream>();
}
var pathItems = dir.GetPathsAsync(true);
await foreach(var pathItem in pathItems)
{
string absoluteFullPathToFile = pathItem.Name; //e.g., sample-container/download/process_Id_12/task_id_34/year2024/month7/day10/file.csv
string fileName = System.IO.Path.GetFileName(absoluteFullPathToFile);
if(!IsRegexPatternMatch(fileName))
{
continue;
}
var continer = GetParentFileSystemClient(downloadClient);
string pathToDirectory = System.IO.Path.GetDirectoryName(absoluteFullPathToFile);
var directoryClient = container.GetDirectoryClient(pathToDirectory);
var fileClient = directoryClient.GetFileClient(fileName);
yield return await fileClient.OpenReadAsync();
}
}
}
To unit test GetSalesFiles method, I have mockednote1 created a test-double for DataLakeDirectoryClient dependency using Moq. But it feels like I am deceiving myself with the mock setups for DataLakeDirectoryClient's method calls.
note1:- I am not sure what I should call that test-double I have set up with Moq framework. Initially I thought it is a mock since it is not handwritten. But now I am thinking this is a Stub as it returns objects (especially file stream in this case).
My unit testing code is below
Listing 2:-
using Nunit.Framework;
using Moq;
[TestFixture]
public class DataLakeServiceTests
{
[Test]
public async Task GetSalesFiles_WhenSingleFileInStorage_ThenReturnAStream()
{
//Arrange
var mockDirectoryClient = new Mock<DataLakeDirectoryClient>();
mockDirectoryClient
.Setup(dc => dc.GetSubDirectoryClient(It.IsAny<string>()))
.Returns(mockDirectoryClient.Object);
mockDirectoryClient
.Setup(dc => dc.Exists(default(CancelationToken)))
.Returns(Response.FromValue(true, Mock.Of<Response>()));
mockDirectoryClient
.Setup(dc => dc.GetPathsAsync(true, false, default))
.Returns(() => {
var pathItem1 = DataLakeModelFactory
.PathItem("sample-container/download/process_Id_12/task_id_34/year2024/month7/day10/file.csv",
false, new DateTime(2024, 7, 10), ETag.All, 256, "", "", "");
var pathItem2 = DataLakeModelFactory
.PathItem("sample-container/download/process_Id_12/task_id_34/year2024/month7/day10/file.txt",
false, new DateTime(2024, 7, 10), ETag.All, 256, "", "", "");
var page1 = Page<PathItem>.FromValues(values: new[] { pathItem1, pathItem2 },
continuationToken: null, response: Mock.Of<Response>());
return AsyncPageable<PathItem>.FromPages(new[] { page1 });
});
var mockFileSystemClient = new Mock<DataLakeFileSystemClient>();
mockFileSystemClient.Setup(fsc => fsc.GetDirectoryClient(It.IsAny<string>()))
.Returns(mockDirectoryClient.Object);
var mockFileClient = new Mock<DataLakeFileClient>();
Stream stream = BinaryData.FromString("This is a line.\n This is the second line\n").ToStream();
mockFileClient.Setup(fc => fc.OpenReadAsync(0, default, default, default))
.Returns(Task.FromResult(stream));
mockDirectoryClient
.Setup(dc => dc.GetFileClient(It.IsAny<string>()))
.Returns(mockFileClient.Object);
PathConfiguration pathConfigs = new("process_Id_12/task_id_34/", "process_Id_12/task_id_50/");
DataLakeService.GetParentFileSystemClient = dc => mockFileSystemClient.Object;
DataLakeService dataLakeService = new(mockDirectotyClient.Object, pathConfigs);
//Act
var files = dataLakeService.GetSalesFiles(new DateTime(2024, 7, 1));
//Assert
int fileCount = 0;
await foreach (var file in files)
{
fileCount++;
}
Assert.AreEqual(1, fileCount);
}
}
I am thinking of writing a Fake Data Lake Directory Client to mimic the hierarchical file system navigation so that my test will fail when business logic change come i.e. when paths are changing. Here I thought the navigating into the correct directory structure is one of the business functionality of the SUT. Do I need Something like this for emulating ADLS ?
