1
\$\begingroup\$

I'm trying to move a large number of files. I get the path of source and destination from database.

It takes hours to move images, and I was wondering if there is a faster way for moving large numbers of files.

Table example:

source destination
C:\x D:\y
C:\z D:\y

This the code I'm using.

var datasource = @"00.000.0.00"; 
var database = "maydatabase"; 
var username = "power"; 
var password = "123"; 
string connString = @"Data Source=" + datasource + ";Initial Catalog="
           + database + ";Persist Security Info=True;User ID=" + username + ";Password=" + password;
SqlConnection conn = new SqlConnection(connString);
     static void moveImg(string getimg, SqlConnection conn) 
        {
                using (SqlCommand command = new SqlCommand(getimg, conn)) 
                {
                    command.CommandTimeout = 0;
                    using (var reader = command.ExecuteReader())
                    {
                        while (reader.Read())
                        {
                            if ((File.Exists(reader["source"].ToString())) && !(File.Exists(reader["destination"].ToString())))

                            {

                                    File.Move(reader["source"].ToString(), reader["destination"].ToString(), true);
                            }

                        }

                    }

                }
        }

Note

The number of rows in database can exceed 1 million, meaning 1 million images to move. Each image has size between 16KB and 60KB.

The query takes 1.30 mins to run. I don't know how can I get the timing for how long a file move takes, as not every file is same size. I can tell from the log file that it moves around 4 to 6 files in 1 second.

\$\endgroup\$
3
  • \$\begingroup\$ Welcome to CodeReview! Which .NET and C# version are you using? \$\endgroup\$ Nov 3, 2021 at 7:48
  • \$\begingroup\$ @PeterCsala Thanks, I'm using .net core 3.1 and C# 8.0 \$\endgroup\$
    – MickeyMan
    Nov 3, 2021 at 8:05
  • \$\begingroup\$ Is moving that many files something you do on a regular basis or a one time action? How comes that you have so many files to move? Could you shed some light into it? \$\endgroup\$
    – t3chb0t
    Dec 31, 2021 at 23:43

2 Answers 2

0
\$\begingroup\$

Here are my observations and suggestions:

connString

string connString = $@"Data Source={datasource};Initial Catalog={database};Persist Security Info=True;User ID={username};Password={password}";
  • Since C# 8 the ordering of @ and $ does not matter

using

using var command = new SqlCommand(getimg, conn);
command.CommandTimeout = 0;
using var reader = command.ExecuteReader();

Fetch

  • I would suggest to separate database read from actual move operations
  • Your database connection should not last long
static IEnumerable<(string Source, string Destination)> FetchMoveParameters(string getimg, SqlConnection conn)
{
    var result = new List<(string Source, string Destination)>();
    using var command = new SqlCommand(getimg, conn);
    command.CommandTimeout = 0;
    using var reader = command.ExecuteReader();
    while (reader.Read())
    {
        result.Add(reader["source"].ToString(), reader["destination"].ToString());
    }

    return result;
}
  • I've used ValueTuples here but you can create your own structure for that

Move

  • If you want to increase the performance of move operations then you should consider to use parallelism
  • You have to make sure that there are no two (or more) conflicting instructions in the fetched move parameters to avoid race condition
var moveParams = FetchMoveParameters(...);
moveParams.AsParallel().ForAll(moveParam =>
{
    //move source to destination
});

Alternative

Parallel.ForEach(moveParams, moveParam =>
{
    //move source to destination
});
  • And as always measure, measure and measure
    • Make sure that the newly applied changes are giving some performance gain
\$\endgroup\$
11
  • 1
    \$\begingroup\$ Thanks, will try it and feed back. \$\endgroup\$
    – MickeyMan
    Nov 3, 2021 at 9:10
  • 1
    \$\begingroup\$ @MickeyMan, I will like to see your feedback. I suspect that vectoring the move operations will most likely be slower, especially on a clunky old and fragmented HDD, but I have often been wrong :-) \$\endgroup\$
    – jdt
    Nov 3, 2021 at 9:38
  • \$\begingroup\$ @upkajdt Thank you a lot! It works Parallel.ForEach big increase in performance it takes 250mb from memory but file moving way faster, I have another issue now with serilog as it create a log file for each thread if you a way to solve let me know thanks again. \$\endgroup\$
    – MickeyMan
    Nov 4, 2021 at 9:51
  • 1
    \$\begingroup\$ Peter thanks....my bad XD \$\endgroup\$
    – MickeyMan
    Nov 4, 2021 at 13:56
  • 1
    \$\begingroup\$ @MickeyMan, Sorry but I have no idea what is happening in your serilog. Can you perhaps disable it while moving the files? \$\endgroup\$
    – jdt
    Nov 4, 2021 at 14:05
0
\$\begingroup\$

Quick suggestion:

  1. Iterate through all the records in the database
  2. Group them by folder
  3. See if there are other items in the folders
  4. Moving folders is exponentially faster than individual files
  5. If the files are on different drives, or you are not allowed to change the directory structure, you are pretty much SOL and will just have to be patient =)
\$\endgroup\$
2
  • \$\begingroup\$ I can't move it as a folder because there are other files within the folder that I don't want to move and also I'm not allowed to change the structure. I'm doing this on azure server I'm supposed to backup the files by the date of creation meaning I move all the files that been created in JUN for example 0.5 million file to another partition with much slower drive and much cheaper than the SSD we are using this partition is shared on this server so it takes traffic too. I think there is no way to improve it as traffic will make it slow and also the writing to the slow partition right ? \$\endgroup\$
    – MickeyMan
    Nov 2, 2021 at 22:49
  • \$\begingroup\$ You might get a marginal improvement in speed by batching operations etc. Perhaps somebody else has some sort of magic way to increase performance, let's see :-) \$\endgroup\$
    – jdt
    Nov 2, 2021 at 23:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.