I am using Powershell 7.

We have the following PowerShell script that will parse some very large file.

I no longer want to use 'Get-Content' as this is to slow.

The script below works, but it takes a very long time to process even a 10 MB file.

I have about 200 files 10MB file with over 10000 lines.

Sample Log:

#Fields:1
#Fields:2
#Fields:3
#Fields:4
#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.22:15650,<,DATA,
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.25:15650,<,DATA,

Script:

$Output = @()
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-Item  $LogFilePath
$Count = @($logfiles).count

ForEach ($Log in $LogFiles)
{
    $Int = $Int + 1
    
    $Percent = $Int/$Count * 100

    Write-Progress -Activity "Collecting Log details" -Status "Processing log File $Int of $Count - $LogFile" -PercentComplete $Percent 

    Write-Host "Processing Log File  $Log" -ForegroundColor Magenta
    Write-Host
    $FileContent = Get-Content $Log | Select-Object -Skip 5
    ForEach ($Line IN $FileContent)
    {

        $Socket = $Line  | Foreach {$_.split(",")[5] }

        $IP = $Socket.Split(":")[0]

        $Output += $IP

    } 
} 
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object

Write-Host "List of noted remove IPs:" 
$Output
Write-Host 
$Output | Out-File $PWD\Output.txt