Get owner of ~5 million files and folders using Powershell and Robocopy

Question

I've written a Powershell script that uses Robocopy to get the full path of all files on a drive and then run it through a foreach loop to get the owner for each file and generate it in a CSV. The code works, but is very slow when I run it against so many items. Can someone help me make it as efficient as possible? I also use "File System Security PowerShell Module 4.2.3" to use Get-NTFSOwner which gets me around the 256 character limit, which is also why I use Robocopy to generate the paths.

Import-Module NTFSSecurity


#Gets initial file and folder list
 $Folderpath = "G:\DEPT"
 $Folders = Get-Childitem $Folderpath

 ForEach ($Folder in $Folders) {Start-Job -ScriptBlock {
          Param($Folder,$Folderpath)
          robocopy "$FolderPath\$Folder" NULL /L /S /NJH /FP /NC /XJ /NS /NJH /NC /NJS /LOG:D:\AutoAssign\Data\RoboLogs\$Folder.txt

           } -ArgumentList $Folder,$Folderpath}


#Wait unitl all jobs are finished
Get-Job | Wait-Job


#Trims out white space and empty lines
  $Folderpath = "D:\AutoAssign\Data\RoboLogs"
  $Files = Get-Childitem $Folderpath

ForEach ($File in $Files) {Start-Job -ScriptBlock {
          Param($File,$Folderpath,$Files)
          $Trim = Get-Content "$Folderpath\$File"
          $Trim.trim() | Where { $_ } | Out-File "$Folderpath\$File"
           } -ArgumentList $File,$Folderpath,$Files }



#Wait unitl all jobs are finished
Get-Job | Wait-Job


#Gets Owner for each file And Create CSV


$Dir = "D:\AutoAssign\Data\RoboLogs"
$Files = Get-ChildItem $Dir
ForEach ($File in $Files){ Start-Job -ScriptBlock {
    Param($Dir,$Files,$File)


$OutputFile = "D:\AutoAssign\Data\Final\$File.csv"
$Results = @()

$Paths = Get-Content $Dir\$File
ForEach ($Path in $Paths){
 $Owner = get-ntfsowner $Path | Select Owner | ft -hidetableheaders | Out-String
 $Owner = $Owner.Trim()

      $Properties = @{

      Path = $Path
      Owner = $Owner

      }

If ($Owner -ne "BUILTIN\Administrators" -and $Owner -ne $null-and $Owner -ne "Domain\Domain Admins" -and $Owner -notlike "S-1*"){
    $Results += New-Object psobject -Property $properties}

    }


$Results | Export-Csv -notypeinformation -Path $OutputFile

} -ArgumentList $Dir,$Files,$File #Ends Initial ForEach

}  #Ends Job Script Block


#Wait unitl all jobs are finished
Get-Job | Wait-Job


#Merge all files into one CSV
Get-ChildItem  D:\AutoAssign\Data\Final | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv D:\AutoAssign\Data\G.csv -NoTypeInformation -Append

#Delete all original files
Get-ChildItem D:\AutoAssign\Data\RoboLogs | Remove-Item
Get-ChildItem D:\AutoAssign\Data\Final | Remove-Item

Answer 1

Your robocopy command lists /NC and /NJH twice.

$Folders and $Folder contain objects with properties, use these instead of treating them as strings (which implicit casts them as such) or pipe to Select-Object -ExpandProperty FullName Same with $Files and $File

I'd not reuse variable names but define neccessary ones from begin. Proper indenting eases reading/understanding the code IMO better than excess white space.

The following script isn't tested at all, see it as rough template:

## Q:\Test\2018\08\12\cr_201527.ps1
Import-Module NTFSSecurity

$Logs  = 'D:\AutoAssign\Data\RoboLogs\'
$Final = 'D:\AutoAssign\Data\Final\'
$Folderpath = "G:\DEPT"

#Gets initial file and folder list
$Folders = Get-Childitem $Folderpath

ForEach ($Folder in $Folders) {
    Start-Job -ScriptBlock { Param($Folder,$Logs)
        robocopy "$($Folder.FullName)" NULL /L /S /NJH /FP /NC /XJ /NS /NJH /NC /NJS /LOG:"$Logs$($Folder.Name).txt"
    } -ArgumentList $Folder,$Logs
}

#Wait until all jobs are finished
Get-Job | Wait-Job

<# functionality integrated into next sction
    #Trims out white space and empty lines
#>

#Gets Owner for each file And Create CSV

ForEach ($File in (Get-ChildItem $Logs)){
    Start-Job -ScriptBlock { Param($Logs,$File,$Final)
        $OutputFile = Join-Path $Final "$($File.Name).csv"
        $Results = ForEach ($Path in ((Get-Content $File.FullName).Trim() -ne '')){
            $Owner = (Get-NtfsOwner $Path).Owner.Trim()
            If ($Owner -ne "BUILTIN\Administrators" -and
                $Owner -ne $null                    -and
                $Owner -ne "Domain\Domain Admins"   -and
                $Owner -notlike "S-1*"){
                [PSCustomObject]@{
                    Path = $Path
                    Owner = $Owner
                }
            }
        }
        $Results | Export-Csv -NoTypeInformation -Path $OutputFile
    } -ArgumentList $Logs,$File,$Final #Ends Initial ForEach

}  #Ends Job Script Block

#Wait unitl all jobs are finished
Get-Job | Wait-Job

#Merge all files into one CSV
Get-ChildItem  D:\AutoAssign\Data\Final | Select-Object -ExpandProperty FullName |
    Import-Csv | Export-Csv D:\AutoAssign\Data\G.csv -NoTypeInformation -Append

#Delete all original files
Get-ChildItem D:\AutoAssign\Data\RoboLogs | Remove-Item
Get-ChildItem D:\AutoAssign\Data\Final | Remove-Item