1
\$\begingroup\$

I'm trying to compare several files using md5 hash with powershell 2.0. The code works, the problem is that it's to slow. It takes longer in step 3. It's about 500 files to compare. Can you see a way to get it faster, like, don't do the step 3 every time?

write-host "1"
    $COMP_ORI=$LOCAL_HOME+"\"+$PROG+"\"+$COMPARE
    $file_ori = Get-ChildItem -Path $COMP_ORI -name
write-host "2"
    $COMP_DEST="\\"+$HOSTIP[$i]+"\"+$PROG_PATH
    $file_dest = Get-ChildItem -PATH $COMP_DEST -name
write-host "3"
    for ($i=0; $i -lt $file_ori.Count; $i++) {
write-host "compare md5" $i
    if ( Get-ChildItem -PATH $COMP_DEST -name -Include $file_ori[$i] ) {
        $md5 = New-Object -TypeName system.Security.Cryptography.MD5CryptoServiceProvider
write-host "4"
        $hash_ori = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($COMP_ORI+"\"+$file_ori[$i])))
write-host "5"
        $hash_dest = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($COMP_DEST+"\"+$file_ori[$i])))
write-host "6"
    if ($hash_ori -ne $hash_dest) {
        $out=$file_ori[$i]
        $out=$out+" IS DIFFERENT"
        output message_solo $out
    }
}

}

\$\endgroup\$
4
  • \$\begingroup\$ What do you mean by "It takes longer in step 3"? Is step 3 the code between write-host "2" and write-host "3", or is it everything after write-host "3"? \$\endgroup\$ Jul 13, 2016 at 13:37
  • \$\begingroup\$ After write-host "3", I think more specifically in the if after write-host "compare md5" $i \$\endgroup\$ Jul 13, 2016 at 13:40
  • 1
    \$\begingroup\$ Is there a reason you're not using Get-Filehash? \$\endgroup\$
    – alx9r
    Jul 14, 2016 at 3:34
  • \$\begingroup\$ I have placed an example using Get-Filehash, a hashtable and skipping all files that do not have a common size (as suggested by @Dangph) here: Powershell Speed: How to speed up ForEach-Object MD5/hash check \$\endgroup\$
    – iRon
    Mar 1, 2020 at 9:28

4 Answers 4

3
\$\begingroup\$

There are two obvious optimisations.

The first is to remove the line

    $file_dest = Get-ChildItem -PATH $COMP_DEST -name

since $file_dest is unused. I assume that it was previously used instead of

Get-ChildItem -PATH $COMP_DEST -name -Include $file_ori[$i]

The second is to do the remote hashing remotely. Rather than fetch 500 files (of unknown size) across the network in order to calculate their MD5 sums, use remote PowerShell to calculate their MD5 sums and just return the filenames and the hashes. You might even be able to manage to get the two computers to calculate the hashes in parallel, giving a two-fold speedup.

\$\endgroup\$
1
  • \$\begingroup\$ Thank you for your answer. The first line you suggest to remove is used a few lines below, just not seen in this example, my bad. I'll see how the remote PowerShell is used and how I can to this, that you for your time! \$\endgroup\$ Jul 13, 2016 at 14:08
2
\$\begingroup\$

Some possible optimizations:

  • Compare the file sizes of the two files before the comparison. This step would take near zero time. If the sizes are different, then you know that the files must be different, so you won't need to compare the contents. If they are the same, then you will of course still need to compare the contents.
  • Store the MD5 codes in files. That is, if you have MyFile.dat, then store the MD5 in a file called MyFile.dat.md5. You would just compare these MD5 files instead of the actual files. The MD5 file could be created when the files are created, or they could be created some other time. When you create the MD5 file, you could set its LastWriteTime to be the same as that of the main file. If you find that these two time stamps don't match later, then you know that the MD5 is stale and needs to be regenerated. (Whether this scheme will be an optimization or not will depend on your particular circumstances.)
  • If you do need to compare two files, it would probably be faster to read through the two simultaneously, comparing as you go. If you find a difference, then you can stop. (The way you are doing it, you need to read through both files in their entirety as well as doing a fairly costly computation.) I would open both files as FileStreams, and read a chunk at a time from each comparing the two chunks. I would make the chunk size a parameter, and I would find the best value for that parameter experimentally.
\$\endgroup\$
1
  • 1
    \$\begingroup\$ Some good inputs about comparing files that I will use in another code I have, thank you \$\endgroup\$ Jul 15, 2016 at 9:15
1
\$\begingroup\$
  1. Do you have to create 'md5' each time in the loop (~500 times on your data)? Can you create it once outside of the loop and then reuse?

  2. Do you have to read the file into memory and then spoon-feed it into md5? Is there any way to invoke md5 as if you'd do it from CLI so it reads file on its own? Something like $hash_ori = $($(CertUtil -hashfile C:\TEMP\MyDataFile.img MD5)[1] -replace " ","")

(no idea about Powershell per se, but that would make sense, say, in bash)

\$\endgroup\$
3
  • \$\begingroup\$ That's exactly my idea ...the problem is that I'm not seeing how to do it ... \$\endgroup\$ Jul 13, 2016 at 12:21
  • \$\begingroup\$ @PedroSales, added example of calling md5 as if using CLI \$\endgroup\$ Jul 13, 2016 at 13:21
  • \$\begingroup\$ Thank you for your input, but the problem still maintains because I still have to go over all the files and verify if they exist in the directory one by one, and that is what is taking to long \$\endgroup\$ Jul 13, 2016 at 13:28
0
\$\begingroup\$

I went for another approach since the problem wasn't the MD5 time. I sent the 2 directories content into an array and then all the operation are made using those arrays. The time went from about an hour to 10 min for 450 files.

$COMP_CS=$LOCAL_HOME+"\"+$PROG+"\"+$COMPARE
$FILE_CS=@(Get-ChildItem -Path $COMP_CS -name -exclude sqlnet.log)
$COMP_BASE="\\"+$HOSTIP[$i]+"\"+$PROG_PATH
$FILE_BASE=@(Get-ChildItem -PATH $COMP_BASE -name -exclude sqlnet.log)
    for ($i=0; $i -lt $FILE_CS.length; $i++) {
        if ($FILE_CS -contains $FILE_BASE[$i]) {
            $md5 = New-Object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
            $HASH_CS = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($COMP_CS+"\"+$FILE_CS[$i])))
            $HASH_BASE = [System.BitConverter]::ToString($md5.ComputeHash([System.IO.File]::ReadAllBytes($COMP_BASE+"\"+$FILE_CS[$i])))
            if ($HASH_CS -ne $HASH_BASE) {
                $out=$FILE_CS[$i]
                $out=$out+" IS DIFFERENT"
                output message_solo $out
            }
        } 
    }
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.