Powershell function to search CSV logs for certain regexes

Question

I have a parser that goes through two different logs, both .csv files, and checks for certain lines based off the regex code that I have chosen.

This parser works fine, it just takes about a minute to parse through about 100 files. This parser is based off another parser I have that only parses through one log type and it is incredibly fast, like 200 files in maybe 15 secs.

This one grabs the IDNumber from the beginning of the filename (1234-randomfile.csv), then adds the files location to a variable ($Validate), then based on the regex, adds files to certain variables ($Scriptdone, $Updatedone, $Failed) and starts the checks to see if they have them.

Like I said, it works, but it is slow. If you have any input on a way to quicken this up, or maybe clean up my code (I am still learning), it will all be greatly appreciated!

function Get-MR4RES {
[CmdletBinding()]
param (
    [Parameter(Position = 0,
        Mandatory = $True)]
    [ValidateNotNullorEmpty()]
    [ValidateScript( {Test-Path -Path $_ -PathType 'Any'})]
    [String]
    $Files,

    [Parameter(Position = 1,
        Mandatory = $false)]
    [String]
    $CSVPath) # End Param

begin {

    # Setting Global Variables
    $Scriptcompletedsuccess = '.+Script\scompleted\ssuccessfully.+' # 3:44:15 End function called, Script completed successfully at  3:44:15 on Tue 07/03/2018
    $Updatecomplete = '\w+\s+\:\s\[\d+\:\d+\:\d+\]\s+\w+\scomplete' # STATUS  : [03:43:07]   Update complete
    $FailedValidaton = '.+check\sfail.+'
    $Fail1 = 'Validation Failed'
    $Fail2 = 'Failed'
    $Good1 = 'Script completed'
    $Good2 = 'Update completed'
    $array = @('IDNumber, Results')
    $counter = 0
    $FileList = (Get-ChildItem -Path $Files -File -Filter "*.log").FullName
    $Done = ''

} # End begin

process {

    # Do the following code in all the files in the filelist
    foreach ($File in $fileList) {

        # Test files variables to ensure is directory to ensure progress bar will be operational and needed
        if ((Get-Item $Files) -is [System.IO.DirectoryInfo]) {

            # Counts once per each file variable in filelist variable
            $counter++

            # Progress bar indicates the name of the current file and calculates percent based on current count verses total files in $filelist
            Write-Progress -Activity 'Analyzing Files' -CurrentOperation $File -PercentComplete (($counter / $FileList.count) * 100)

        }

        # Calculates ID number based on filename, file name is -filtered in beginning to only contain properly named files
        $IDNumber = [System.IO.Path]::GetFileName("$File").split('-')[0]

        # Puts file into Variable to be IF Else
        $Validate = Get-Content -Path $File

        $Scriptdone = $Validate | Where-Object {$_ -match $Scriptcompletedsuccess}
        $Updatedone = $Validate | where-object {$_ -match $Updatecomplete}
        $Failed = $Validate | Where-Object {$_ -match $FailedValidaton}

        # Check if the file HAS a FAILED validation
        if($Failed){

            # Creates an array of the data from each file that failed
            $array += -join ("$IDNumber",', ',"$Fail1")
        }
        Elseif($Scriptdone){
            $Done = '' # Clear the $Done variable
            $Done = $Good1

            # Creates an array of the data from each file that script completed
            $array += -join ("$IDNumber",', ',"$Done")

        } # if the parser found "Update complete"
        Elseif($Updatedone){
            $Done = '' # Clear the $Done variable
            $Done = $Good2

            # Creates an array of the data from each file that update is done
            $array += -join ("$IDNumber",', ',"$Done")

        } # End of Successful
        Else{
            # Creates an array of the data from each file that failed
            $array += -join ("$IDNumber",', ',"$Fail2")
        }

    } # End of foreach

} # End process section

End {

    # If CSVPath is used in get-command 
    if ($PSBoundParameters.ContainsKey('CSVPath')) {

        # Pipe the array data to a CSV
        Add-Content -Path $CSVPath -Value $array -Encoding ascii

    }

    # If no CSVPath is used in get-command
    else {

        # Out-put to console
        Write-Output $array

    } # End of else

} # End of the End

} # End of function

Answer 1

Just a quick off the cuff and building off of what Dangph started but the triple scan is probably killing performance.

$Scriptdone = $Validate | Where-Object {$_ -match $Scriptcompletedsuccess}
$Updatedone = $Validate | where-object {$_ -match $Updatecomplete}
$Failed = $Validate | Where-Object {$_ -match $FailedValidaton}

Each one is reading through the entire file to find one thing. One route you can try out is using a foreach(You can shorthand it with % as I'll do below) and a switch.

$size = $array.Length
#Assumes there's only one line that will match a given regex per file
#If not, it'll add duplicates which can be stripped at the end with an $array | sort -unique
$validate |% {
    switch -regex($_){
        $Scriptcompletedsuccess {$array += "$IDNumber, $Good1"; break}
        $updatecomplete {$array += "$IDNumber, $Good2"; break}
        $Failedvalidation {$array += "$IDNumber, $Fail1"; break}
        default {}
    }

#Checks to see if array has grown, if it hasn't, no matches were found
#Bit hacky and there's probably a better way to do it.
if($size -eq $array.length){
    $array += -join ("$IDNumber",', ',"$Fail2")
}

Oh yeah, another performance boost (not sure how much of one though) would be changing the array into an ArrayList. They can append whereas the array resizes on each add.

Answer 2

You have to find out where the slow part is before you can speed it up. Just by looking at it, I would say it is either this part where you load the whole file into memory:

$Validate = Get-Content -Path $File

Or it's this part where you scan through the in-memory array:

$Scriptdone = $Validate | Where-Object {$_ -match $Scriptcompletedsuccess}
$Updatedone = $Validate | where-object {$_ -match $Updatecomplete}
$Failed = $Validate | Where-Object {$_ -match $FailedValidaton}

Or maybe both contribute to the slowness.

You should do some experiments to determine which part is slow. You could for instance comment out the Get-Content line, and just load in one of the files once at the beginning. Does that speed it up?

You can also try commenting out the scanning lines. Does that speed it up?

An observation:

$Scriptdone, $Updatedone, $Failed appear to be mutually exclusive. You don't need to find $Scriptdone if $Failed is true, for instance. You could restructure your code like this to remove the redundant processing:

$Failed = $Validate | Where-Object {$_ -match $FailedValidaton}

if($Failed){
    # ...
}
else {
    $Scriptdone = $Validate | Where-Object {$_ -match $Scriptcompletedsuccess}

    if($Scriptdone){
        #...
    }
    else {
        $Updatedone = $Validate | where-object {$_ -match $Updatecomplete}

        if($Updatedone){
            # ...
        }
        else {
            # ...
        }
    }
}

Some questions:

• Where do the lines you are looking for appear in the files? Can they be anywhere, or are they at some particular place (the beginning or the end for instance)?

• How big are the files?

Edit:

Based on the answers to those questions (see the comments), I have to say I don't understand why it is slow.

I don't think a Get-Content on 100 4MB files should take any time at all. I am sceptical that it is the cause. Since I don't know what what the source of the slowness is, I can't really suggest much except to do more experimentation to work out what it is.

I can just throw some random ideas out there.

• Try using the -Raw switch on the Get-Content to load the whole in one chunk:

  $Validate = Get-Content -Path $File -Raw

  $Scriptdone = $Validate -match $Scriptcompletedsuccess

• Try using Select-String to search through the files:

  Note: Delete the Get-Content line for this idea.

  $Scriptdone = Select-String $Scriptcompletedsuccess $File | | Select-Object -First 1

  The Select-Object -First 1 is optional, but it should speed things up because the search will stop as soon as the first match is found.

• My last idea is to try simplifying the regular expressions, just as an experiment. Sometimes some regular expressions can be slow. I don't think that should be the case with yours, but you never know.

Ultimately you have to track down the source of the slowness before you can fix the problem.