I have a queue of jobs in an ASP.NET CORE Application. Each of the files are in a restricted area and I have to Login first, thats why I use a Cookie Container for my download request.

The Job is logging into a platform and then starts to download the file, given an URL and a Cookie Container from the Login.

The download method is looking like this and has a Func which reports the progress in 5 % steps:

public async Task DownloadFile(string url, string destination, Func<int, Task> reportStatus)
{
await reportStatus(0);

var request = WebRequest.CreateHttp(url);
var response = (HttpWebResponse)await request.GetResponseAsync();
var requestStream = response.GetResponseStream();

{
using (var fileStream = File.OpenWrite(destination))
{
const int bufferSize = 2048;
byte[] bytes;
short oldPercentage = 4;
do
{
await fileStream.WriteAsync(bytes, 0, bytes.Length);

var percentage = (short)(bytesRead * 100 / response.ContentLength);
if (oldPercentage < percentage)
{
await reportStatus(percentage);
oldPercentage = (short)(oldPercentage + 5);
}
} while (bytes.Length != 0);
}
}

await reportStatus(100);
}


Small problems I am currently facing:

• Download speed feels quite slow in comparison to my internet connection.
• I was wondering if there are better ways downloading a file from a restricted area using .NET Core functionality.
• i think parameter jobId is useless here May 19 '17 at 8:16
• maybe speed will increase if you write all bytes to file once in the end May 19 '17 at 8:21
• For simplicity I have made the source code a bit smaller, thats why Job id isn't used. I have removed it from the code now. May 19 '17 at 8:42
• @Disappointed memory usage will also then increase
– cat
May 19 '17 at 10:15
• I have rolled back your edit as it would invalidate existing answers, in accordance with our policy.See What you may and may not do after receiving answers May 19 '17 at 13:03

You mentioned that the download is slower than you'd expect, I'll focus on just that for a quick review.

var percentage = (short)(bytesRead * 100 / response.ContentLength);
if (oldPercentage < percentage)
{
await reportStatus(percentage);
oldPercentage = (short)(oldPercentage + 5);
}


await reportStatus(percentage);


You're waiting for the task reporting the status to finish before reading the rest of the stream! What if reportStatus were logging the progress to a database or doing something else really time consuming?

I think you should just execute the callback and not care when/how it finishes:

reportStatus(percentage); // Don't wait


Oh and oldPercentage is really confusing when you scan the code... I'd rename it to something like reportingThreshold.

if (percentage > reportingThreshold)
{
reportStatus(percentage);
reportingThreshold = (short)(reportingThreshold+ 5);
}

• Yeah, I saw that right after I posted this question. It actually is a bottleneck, because I do write to the Job database everytime I report the progress. However it is only updated every 5 %, so it can't be the main problem here when it comes to big files (e.g. 100 MB downloading with a 50mbit connection). Anyway, thanks I will try to implement your change :) I will get into some threading issues though. Because who will guarantee that 5 % DB update is committed before a 10 % DB update :) May 19 '17 at 8:51
• @Jannik - Add your updates to a queue in reportProgress and process the queue separately to maintain the order. I can assure you that the latency on a DB call 20 times during a download will have a noticeable effect on speed.
– RobH
May 19 '17 at 9:13

As a matter of fact you should use the IProgress<T> interface for reporting progress that

Defines a provider for progress updates.

With it you wouldn't be tempted to await it because the Report method is void.

Example:

public async Task DownloadFile(string url, string destination, IProgress<int> progress)
{
..
progress.Report(percentage);
..
}