2
\$\begingroup\$

The code below copies large amount of data. It does so in 2 steps. I was wondering if both step1 and step2 can be done in parallel.

<#
    The code below copies from a network fileshare to another share in the same location but to a different folder.
        Robocopy is used to do the file move of all the old files that are no longer required. This is done by a script block.
        After the files are moved then another script block moves the rest of the files that were left behind.
        Logging of files moved by robocopy and folder/filenames of files moved/copied.
        My reason for code review to finetune for better performance to reduce the file move/copy times, since the fileshare size that i am trying to process is 25Tb.
#>
        

#Initialize variables

        $RoboLogging = [System.Collections.Hashtable]::Synchronized(@{})
        $AllFolders = [System.Collections.Hashtable]::Synchronized(@{})
        $ItemsProcessed = [System.Collections.Hashtable]::Synchronized(@{})
        $dest = "\\servername\fileshare_old\oldfiles"
        
#Multithreading the file transfer process 

        $numThreads       = 50
        $maxSessions      = 50
        $RoboCopyFilePath = "D:test\logs"
        $SrcFilePath      = "D:\test\SrcFiles.csv"
        
        $folders = @()
        
        #Start the time 
        $stopwatch.Start()
        
        #TEST ONLY
        $src = "D:\Test\Copy\Processed"
        $dest = "D:\Test\Copy\archived"
    
#Script block 1 that does robocopy asynchrously using runspacepool and Process only files before 2020
        
        $Copy_Folders = {
        Param  (
                [string]$folderName,
                [string]$dest,
                [hashtable]$AllFolders = (@{})
                )
                        
             $allFilteredFiles = ls $folderName 
                          
             $roboLogs = robocopy $folderName $dest /E /S /nfl /np /mt:16 /ndl 
            /MINAGE:20200101 /MOVE /LOG /J /NOOFFLOAD /R:5 /W:5 /REG /COPY:DAT
        
            $allFilteredFiles | % {        
                $($AllFolders).Add($_ , $roboLogs)            
            }                                
        }
        
#Script block 2 to filter files that were left behind by robocopy due to file modified 
date being later that 2020.
    
        $Process_Folders = {
        
            Param (
            [string]$folderName,
            [hashtable]$ItemsProcessed = (@{})
            )
        

    $allFilteredFiles = Get-ChildItem  -Path $folderName -Recurse -Include("*.misc") -Force  
        
                $allFilteredFiles | % {
        
                #$AllFiltered += $_
                $fileName = $_.FullName
                $temp = $fileName      
        
                if([System.IO.Path]::GetExtension($fileName) -eq ".misc"){
                    #Logic to filter orphaned files left behind by Robocopy
                        Copy-Item $fileName  $dest 
                        Remove-Item $fileName
                        $ItemsProcessed.Add($_, $fileName)              
                    }
                    
            }  
                        
        }

#Initialize Runspaces

        $Runspace = [RunspaceFactory]::CreateRunspacePool(1, $numThreads)
        $Runspace.ApartmentState = 'STA'
        $Runspace.Open()
        
        $Jobs = @()
        
        $folders = ls $src
        $folders = $folders | Sort-Object -Property{$_.Name -as [int]}
        
#invoke firt script_block for robocopy to move the files
        
        $folders | % {
        
            $folderName = $_.FullName    
            Write-Output $folderName
            $Job = [powershell]::Create().AddScript($Copy_Folders).AddParameter("folder", $folderName).AddParameter("dest", $dest).AddParameter("AllFolders", $AllFolders)
            $Job.RunspacePool = $Runspace
            
        
            $Jobs += New-Object PSObject -Property @{
                RunNum = $_
                Job = $Job
                Result = $Job.BeginInvoke()      
            }
        
        

    Write-Host ("Thread Count: {0} Processing asynchrounusly: {1}  Processed: {2} Elapsed `Time: {3}"-f $Jobs.Count, $folderName, $AllFolders.Count, $stopwatch.Elapsed) -ForegroundColor Yellow
            
            }
        
        $Counter = 1
        
        Do{
            Write-Host " " $Counter " " -NoNewline
            Start-Sleep -Seconds 1
            $Counter++
        }While ( $Jobs.Result.IsCompleted -contains $false)  #Jobs.Result is a collection
        
        Write-Output " "
        
#EndInvoke returns the objects from the background threads

        $return = $jobs | % {    
            $_.Job.RunspacePool.Dispose()       
            $_.job.Dispose()    
        }
        
        Write-Host "Number of Folders added from Job: " $jobs.Count
        $jobs.clear()
        $Runspace.Close()
        [GC]::Collect() 
        
#Initialize collections

        $copiedArch = [System.Collections.Generic.HashSet[string[]]]@()
        $processedArch = [System.Collections.Generic.HashSet[string[]]]@()
        $RLogs = [System.Collections.Generic.HashSet[string[]]]@()
        $Counter = 0
        
#Now create the second runspacepool or processing the files that have a newer 
modified date, but were still created before 2020
    
        $RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $numThreads)
        $RunspacePool.ApartmentState = 'STA'
        $RunspacePool.Open()
        $Jobs = @()
        
        $flag = $false
        $threads = @()
        $mrnNames = $mrnNames | Sort
        $folderNames = @()
        
#Initalize the Runspace and RunspacePool and add the Scriptblock
                
        [string[]]$Excludes = @('-1')
        
#Process folder names and feed into the thread to process and degug to see the items that are processed asynchrously  

        $folders | % {
        
            Get-ChildItem $_.FullName -Directory -Exclude $Excludes | %{ 
            $allowed = $true
            foreach ($exclude in $Excludes) { 
                if ((Split-Path $_.FullName -Parent) -ilike $exclude) { 
                    $allowed = $false
                    break
                }
            }
        
            if ($allowed){
                $folderName = $_.FullName
                $job = 
           [powershell]::Create().AddScript($Process_Folders).AddParameter("folderName", 
           $folderName).AddParameter("ItemsProcessed", $ItemsProcessed)
                $job.RunspacePool = $RunspacePool
                        
                $threads += New-Object PSObject -Property @{
                    RunNum = $_
                    Job = $job
                    Result = $job.BeginInvoke()
                }
                       
                Write-Host ("Thread Count: {0} Processing asynchronously: {1}  Processed: {2} Elapsed Time: {3}"-f $threads.Count, $folderName, $ItemsProcessed.Count, $stopwatch.Elapsed) -ForegroundColor Yellow                                   
            }
            }
        }
                        
        Write-Host "Waiting.." -NoNewline
        $count = 0
        
#Wait for the thread to finish robocopy and threads is a collection
    
    Do{
            Write-Host "." -NoNewline
            Start-Sleep -Seconds 1
        }While ( $threads.Result.IsCompleted -contains $false) 
                 
        Write-Output " "
        
        $AllMovedFiles = @()
        $AllfilesToProcess = @()
        
#Empty the jobs

        $return = $threads | % {         
            $_.Job.RunspacePool.Dispose()  
            $_.Job.Dispose()    
        }

#Clear memory

        if($threads.Count -ne 0){
            $threads.clear()
            [GC]::Collect() 
        }

#Clear runspace and close the Runspace and return resources

        Write-Host "Closing the Runspacepool"
        $RunspacePool.Close()
        
#Add logs of folders/files moved

        $null = $AllFolders.GetEnumerator() | ForEach { $copiedArch.Add($($_.Key).FullName), 
        $RLogs.Add($_.Value) }
        
        $RLogs.getenumerator() |
            Sort-Object value -descending |
            Format-Table |
            Out-String |
            Add-Content -Path $RoboCopyFilePath 
        
        Sleep 2
        $AllFolders.Clear()
        
#Add logs of folders/files copied. 

        $null = $ItemsProcessed.GetEnumerator() | ForEach { $processedArch.Add($($_.Value)) }
        
        $processedArch.getenumerator() |
            Sort-Object value -descending |
            Format-Table |
            Out-String |
            Add-Content -Path $RoboCopyFilePath 
        
        Sleep 2
        $ItemsProcessed.Clear()

\$\endgroup\$
1
  • \$\begingroup\$ Welcome to Code Review! I changed the title so that it describes what the code does per site goals: "State what your code does in your title, not your main concerns about it.". Feel free to edit and give it a different title if there is something more appropriate. \$\endgroup\$ Commented Dec 6, 2023 at 23:56

1 Answer 1

1
\$\begingroup\$

This is complex code. Complex enough that I can't tell if it does what your spec suggests.

Choose a simpler approach to scheduling copies.


We are copying from Processed to archive. Narrative commentary explains time is of the essence, and copying proceeds in two steps:

  1. robocopy ancient files from before 2020, deleting the originals
  2. copy remaining (modern, filtered, undeleted) files

We're not using named functions, nor even separately named scripts, just "# script block {1, 2}". I will assign {1, 2} the names {ancient, modern}. The question is: Can we safely overlap the ancient + modern copying?

This would require predicting in advance which files belong to which task, before launching any copying jobs, so we can verify they are disjoint sets. Regrettably the "is_ancient?" predicate is delegated to the robocopy tool, using a /MINAGE switch that compares against 2020-01-01. Later, the modern *.misc files seem to be selected without examining age -- they are selected by virtue of surviving robocopy's /MOVE operation.

After that, many tedious things happen. It's not clear what occurs, exactly, despite a profusion of comments about "processing", "waiting", "emptying jobs", "clearing", plus logging. There is no overall plan outlined. I have no idea why we can't just fork a few rsync's into the background and be done with it, as they would take care to not interfere with one another.


bottlenecks

Disk I/O, network bandwidth, and CPU cycles are all potential bottleneck resources. You may have several hosts that are offering such resources or that could be pressed into service.

I haven't seen any throughput or elapsed time measurements, nor any evidence of where we believe the bottleneck is currently located. If latency is holding us back despite available idle bandwidth, then increasing the multi-programming level can speed things up. But we should understand what's happening, and we should verify that running a 2nd or 3rd concurrent copying process consumed the idle resource we believed it would. Even after reading $numThreads = 50, I still doubt that you have a bottleneck that your 50 idle CPU cores can remedy, and I really have to wonder about a TCP implementation that obtains more goodput from 45 connections than it does from 35. The code should document which bottleneck resource it is addressing, and what the current measured performance is. In future, elements of the system will be added or replaced, so the details will change and the code must, as well.

Making changes to the offered load will eventually move the bottleneck from here to there, and then we're back at the "diagnose and make a new plan" stage.


Background: stragglers

When handing out a bunch of jobs to N workers, we inevitably find that the jobs do not each take identical amount of time. At some point we are mostly done -- one job completes and that worker goes idle. Now we're working on N-1 jobs concurrently. We call these jobs "stragglers".

Then another job completes, and we're only keeping N-2 cores busy.

At last the final job completes, and we click the stopwatch to declare "done!"

A scheduler will strive to minimize the adverse impact of long-running stragglers.


Moving 25 TiB is clearly going to take a while.

scheduling

This whole task seems to revolve around scheduling N workers. (Current proposal is N=2 workers.)

First thing to do is identify the files that need copying. Make a list. We will want details: pathname, st_mtime, file size.

Second thing to do is label each file with a worker type. Apparently ancient ones from a previous decade need "robocopy", while modern ones may be copied in a different way. The label will affect which worker we assign a file to.

Third, sort by file size. (That is, sort by anticipated elapsed time to copy, predicted by file size.) We will do the big ones first. This minimizes the straggler effect.

Finally, schedule the jobs. Start N workers, have them dynamically pull work items from the queue, and keep doing that until queue is empty.

If you must, produce a static schedule instead. Scan the jobs list in descending size order, and assign each job round-robin to the next of the N workers. This approach is less resilient to situations where a worker is unexpectedly slow compared to its peers, for example due to other workloads ("cross traffic") running on a server. It's also not a great approach when some workers have less network bandwidth than other workers.

Ideally each worker would be able to accommodate either label, a robocopy /move or the other kind of copy. If you must, start one or more workers dedicated to just this label or to that label.

Recommend you make a record of some directory listings, so you can go back and see state of the world before copying began, and after the queue drained.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.