Replicate / Sync files from source to destination with powershell only

Question

I have lots of live and disaster recovery servers, which should have the same files and folders. I searched on the web but could not find a Powershell script which has all of the functionality on the same script. So I have created one in the below function.

Please do not use it in Production without fully testing it!!!!!

This script replicates / synchronizes files and folders recursively from source location to destination location with the sync switch.

If you use -Sync $True then the script will remove all of the files and folders on destination location which do not exist in the source location.

Usage:

1. Sync-Files -SourceRoot "C:\Test" -DestinationRoot "\\Server\Test"
2. Sync-Files "C:\Test" "\\Server\Test"
3. Sync-Files -SourceRoot "C:\Test" -DestinationRoot "\\Server\Test" -Sync $true

Source:

    <#
    .Synopsis
    synchronies files between two location
    .DESCRIPTION
    This function is synchronising all files from source location to destination location. 
    Script has ability to fully sync files by deleting the destination files if not exists in the source directory.

    if -Sync is set $true script will delete files which exists in the destination directory only.
    Please do not run this script before test in test/dev environment.

    .EXAMPLE
    Sync-Files C:\Test \\SERVER\Test
    .EXAMPLE
    Sync-Files -SourceRoot "C:\Source" -Destination "\\SERVER\DestinationDirectory\

    .EXAMPLE
    Sync-Files -SourceRoot "C:\Source" -Destination "\\SERVER\DestinationDirectory\ -Sync $True
    #>
    function Sync-Files
    {
    [CmdletBinding()]
    Param
    (
    
    [Parameter(Mandatory=$true,
               ValueFromPipelineByPropertyName=$true,
               Position=0)]
    $SourceRoot,

    
    [Parameter(Mandatory=$true,
               ValueFromPipelineByPropertyName=$true,
               Position=1)]
    $DestinationRoot,

    [Parameter(
               ValueFromPipelineByPropertyName=$true,
               Position=3)]
    [bool]$Sync

    

    
    )
    Begin
    {
    $SourceRootFiles = Get-ChildItem $SourceRoot -Recurse
    $DestinationRootFiles = Get-ChildItem $DestinationRoot -Recurse
    $CompareObjects = Compare-Object -ReferenceObject $SourceRootFiles -DifferenceObject $DestinationRootFiles -ErrorAction SilentlyContinue -IncludeEqual 
      
    }
    Process
    {
    ForEach($CompareObject in $CompareObjects){

        $FinalPath = Join-Path $DestinationRoot $CompareObject.InputObject.FullName.Substring($SourceRoot.length)
              
        if($CompareObject.SideIndicator -eq "<="){

            if($CompareObject.InputObject.PSIsContainer){
                Copy-Item -Path $CompareObject.InputObject.FullName -Destination $FinalPath -Force
            }

            Elseif(!(Test-Path $FinalPath)){
        
                Copy-Item -Path $CompareObject.InputObject.FullName -Destination $FinalPath -Force
        
            }
   
        }

        if($CompareObject.SideIndicator -eq "=="){
           
            if((Get-FileHash $CompareObject.InputObject.FullName).Hash -ne (Get-FileHash $FinalPath).Hash){
                
                Copy-Item $CompareObject.InputObject.FullName $FinalPath -Force
            }

        }

        if($CompareObject.SideIndicator -eq "=>"){
            if($Sync -eq $true){
                if(Test-Path $CompareObject.InputObject.FullName){
                    Remove-Item $CompareObject.InputObject.FullName -Force -Recurse -Verbose
                }
            }
            
        } 

    }
    }
    End
    {

    }
    }

    Sync-Files -SourceRoot "C:\Test" -DestinationRoot "\\Server\Test" -Sync $true

Answer 1

Frankly, robocopy.exe is such a good tool for this job and has been a part of the base Windows install since Vista/2008 that I would just use that instead of reinventing the wheel (especially when you're replacing a top fuel dragster wrinkle slick with an iron-shod wooden wagon wheel). Yes, robocopy only uses datetime (LastWriteTime) and file size (Length) to detect changes, but if that doesn't work you're doing something rather strange. That and the fact that Compare-Object is an awful command to work with because extremely unclear exactly what it's doing.

However, from a quick look:

$SourceRootFiles = Get-ChildItem $SourceRoot -Recurse
$DestinationRootFiles = Get-ChildItem $DestinationRoot -Recurse

These will both ignore Hidden and System files without the -Force parameter.

$CompareObjects = Compare-Object -ReferenceObject $SourceRootFiles -DifferenceObject $DestinationRootFiles -ErrorAction SilentlyContinue -IncludeEqual 

As far as I'm aware the above only compares file name. At the very least, you'll want to specify the parameters you're comparing with Compare-Object. It does not compare all properties like you might expect. You've got to specify them:

$CompareObjects = Compare-Object -ReferenceObject $SourceRootFiles -DifferenceObject $DestinationRootFiles -ErrorAction SilentlyContinue -IncludeEqual -Property Name, LastWriteTime, Length

However, even then, Compare-Object isn't going to work here when files move within the directory structure.

Example:

# Create two directory trees
$SourceDir = mkdir DirectoryA
$DestDir = mkdir DirectoryB
$DestDirSub = mkdir "$DestDir\DirectoryC"

# Create some files in the SourceDir
Push-Location $SourceDir
0..9 | ForEach-Object {
    Set-Content -Path $_ -Value $null
}
Pop-Location

# Copy the files
Get-ChildItem -Path $SourceDir -File | Copy-Item -Destination $DestDir

# Move some files to a subdirectory
Get-ChildItem -Path $DestDir -File | Select-Object -Last 5 |  Move-Item -Destination $DestDirSub

OK, now lets get the list of files. 0-9 should be in DirectoryA, 0-4 in DirectoryB, and 5-9 in DirectoryB\DirectoryC. Now do the comparison:

# Fetch file lists
$Source = Get-ChildItem -Path $SourceDir -Recurse
$Dest = Get-ChildItem -Path $DestDir -Recurse

# Do comparison
Compare-Object -ReferenceObject $Source -DifferenceObject $Dest -IncludeEqual -Property Name, LastWriteTime, Length

I get:

InputObject SideIndicator
----------- -------------
0.txt       ==
1.txt       ==
2.txt       ==
3.txt       ==
4.txt       ==
5.txt       ==
6.txt       ==
7.txt       ==
8.txt       ==
9.txt       ==
DirectoryC  =>

And that's why I hate Compare-Object.

You can't just specify -Property FullName, LastWriteTime, Length or Compare-Object [...] -Property * either, since it's got the root part of the path in the properties. You'll need to do something like this:

# Normalize the source and destination paths
$SourcePath = $SourceDir.FullName.TrimEnd('\');
$DestPath = $DestDir.FullName.TrimEnd('\');

# Fetch file lists
$Source = Get-ChildItem -Path $SourcePath -Recurse | Select-Object -Property *, @{n = 'RelativeName';e ={$_.FullName.Substring($SourcePath.Length)}}
$Dest = Get-ChildItem -Path $DestPath -Recurse | Select-Object -Property *, @{n = 'RelativeName';e ={$_.FullName.Substring($DestPath.Length)}}

# Do comparison
Compare-Object -ReferenceObject $Source -DifferenceObject $Dest -IncludeEqual -Property Name, LastWriteTime, Length, RelativeName

Alternately, you could do something like this using Tuples as the key in a Dictionary:

$SourceTable = @{}
$Source = Get-ChildItem -Path $SourceDir -Recurse | ForEach-Object { 
    $t = [System.Tuple]::Create($_.FullName.Substring($SourceDir.FullName.TrimEnd('\').Length), $_.LastWriteTime, $_.Length)
    $SourceTable.Add($t, $_.FullName)
}

$DestTable = @{}
$Dest = Get-ChildItem -Path $DestDir -Recurse | ForEach-Object { 
    $t = [System.Tuple]::Create($_.FullName.Substring($DestDir.FullName.TrimEnd('\').Length), $_.LastWriteTime, $_.Length)
    $DestTable.Add($t, $_.FullName)
}

# Files in source and dest and could be filehashed
$SourceTable.GetEnumerator() | Where-Object { $DestTable.ContainsKey($_.Key) }

# Files in source and not in dest
$SourceTable.GetEnumerator() | Where-Object { !$DestTable.ContainsKey($_.Key) }

# Files in dest and not in source
$DestTable.GetEnumerator() | Where-Object { !$SourceTable.ContainsKey($_.Key) }

That will be really fast once it's loaded all the tables, but it's really pretty gross to set up.