Powershell to break apart large flat files (e.g. csv)

Question

I created the below script to help break apart large flat files; to save myself effort opening files which are several gigs in a text editor and manually split them.

So far I've only tested on sample files, so I can't yet comment on the performance, but wanted to share my work so far / get feedback on whether I've missed any good opportunities for optimisation.

In the process section of the Split-FlatFileSub routine I've used writeline in several places; I could have put this statement later on, but would have had to add additional statements to handle potential header rows; which I did not want to do in the interests of performance (i.e. I've tried to ensure that the standard loop has as little as possible to do; with more involved operations only happening when the next file's required, as this will be less often).

clear-host
[Reflection.Assembly]::LoadWithPartialName("System.IO") | out-null

function Split-FlatFile {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true)]
        [string]$Filename
        ,
        [Parameter(Mandatory = $false)]
        [switch]$NoHeader        #assume header by default
        ,
        [Parameter(Mandatory = $false)]
        [int]$PageSize = 1048576 #includes the header row / is the excel 2007+ row count
    )
    begin {
        [long]$LinesPerPage = $PageSize 
        if (-not $NoHeader.IsPresent) {
            $LinesPerPage--
        }
        if ($LinesPerPage -le 0) {
            throw "You need to be able to have at least 1 line per page!  Current lines per page is: $LinesPerPage"
        }
        if (-not (test-path -Path $Filename)) {
            throw "File '$Filename' does not exist"
        }
        $file = Get-Item $Filename
        [string]$OutputFilenameFormat = join-path -path ($file.PSParentPath -replace 'Microsoft\.PowerShell\.Core\\FileSystem\:\:(.*)','$1') -ChildPath ("{0}.{1}{2}" -f $file.BaseName,'{0:000000}',$file.Extension)
    }
    process {
        [System.IO.File]::ReadLines($sourceFN) `
        | Split-FlatFileSub -OutFilenameFormat $OutputFilenameFormat -NoHeader:$NoHeader -LinesPerPage $LinesPerPage
    }
}

#not called directly by the user; but invoke via the parent routine above
function Split-FlatFileSub {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [string]$Line
        ,
        [Parameter(Mandatory = $true)]
        [string]$OutFilenameFormat
        ,
        [Parameter(Mandatory = $false)]
        [switch]$NoHeader        #assume header by default
        ,
        [Parameter(Mandatory = $true)]
        [int]$LinesPerPage = 1048575 #excludes the header row / is the excel 2007+ row count - 1
    )
    begin {
        [string]$header = $null
        [long]$currentLineNo = $LinesPerPage + 1
        [long]$currentFileNo = 1
        [string]$fn = ($OutputFilenameFormat -f $currentFileNo)
        [System.IO.TextWriter]$writer = [System.IO.File]::CreateText($fn) #assumption: filename does not exist
    }
    process {
        if($currentLineNo -lt $LinesPerPage) {
            $currentLineNo++
            $writer.WriteLine($Line) #would writelineasync be better?  Experiment later
        } else {
            if($currentLineNo -eq $LinesPerPage) {
                $currentFileNo++ 
                $fn = ($OutputFilenameFormat -f $currentFileNo)
                $writer.Dispose()
                [System.IO.TextWriter]$writer = [System.IO.File]::CreateText($fn) #assumption: filename does not exist
                $writer.WriteLine($Header)
                $writer.WriteLine($Line)
            } else {
                if (-not $NoHeader.IsPresent) {
                    $header = $Line
                    $writer.WriteLine($Header)
                } 
            }
            $currentLineNo = 0
            $fn
        }
        #I could do the $writer.writeline here, but that would mean more logic to check if already written (i.e. for first line)
        #probably a cleaner way to skip the header line; but for now not worth worrying about
    }
    end {
        $writer.Dispose()
    }
}


[string]$SourceFn = 'c:\temp\raw\FlatFileSplitterTestDoc.dat'
Split-FlatFile -Filename $SourceFn -PageSize 10 #simple demo to test with before throwing huge docs at this

Answer 1

[Reflection.Assembly]::LoadWithPartialName("System.IO") | out-null

I'm almost certain that this statement is not needed at all. You should be able to reference [System.IO.File] without loading an assembly.

[System.IO.File]::ReadLines($sourceFN) | 
Split-FlatFileSub -OutFilenameFormat $OutputFilenameFormat -NoHeader:$NoHeader -LinesPerPage $LinesPerPage

Try to avoid using backtick ` as a line continuation character. It's hard to see, and usually unnecessary. In the case of a pipeline, you can end the line with the pipe | and then continue on the next line; it tends to be a lot more clear.

#not called directly by the user; but invoke via the parent routine above
function Split-FlatFileSub {

You could define this function inside the other function, to truly use it as an internal function. If you're writing a module you could just not export it.