3
\$\begingroup\$

In this SO question I provided two answers for the batch conversion of more than 100K txt to xlsx files

I suspect any automation of Excel is too slow, but would appreciate if there is a superior (i.e. faster) approach using powershell.

$files = Get-ChildItem C:\Temp\*.txt
Write "Loading Files..."

$Excel = New-Object -ComObject Excel.Application
$Excel.visible = $false
$Excel.DisplayAlerts = $false

ForEach ($file in $files)
{     

 $WorkBook = $Excel.Workbooks.Open($file.Fullname)
 $NewFilepath = $file.Fullname -replace ".{4}$"
 $NewFilepath =  $NewFilepath + ".xls"
 $Workbook.SaveAs($NewFilepath,56)   

}
  Stop-Process -processname EXCEL
  $Excel.Quit()
\$\endgroup\$
  • 1
    \$\begingroup\$ I always ask if these things need to be in excel. CSVs open in excel just fine and if you remove the Com dependency then options open up. I shall assume these text files are delimited in some way? There are a couple of non performance things I could point out. \$\endgroup\$ – Matt Jan 14 '17 at 4:40
  • \$\begingroup\$ @Matt this query is about whether the powershell code (im new to this), could be better, as opposed to whether excel is the right tool in the original question that another party asked \$\endgroup\$ – brettdj Jan 14 '17 at 8:41
  • \$\begingroup\$ I know. I still wanted to ask. The code can be reviewed outside of that request though. \$\endgroup\$ – Matt Jan 14 '17 at 14:41
3
\$\begingroup\$

As far as performance goes there are two general approaches to address this

Parallel processing

Running the com object is dragging in itself. Processing 10's of 1000's of files with one Excel instance will be draining. PowerShell does support multiple avenues of mitigating this

I am going to show an approach using Jobs. What this will do is group up identified files. Each group will be past to its own Excel job for processing. Some comments in the code below explain it more.

# Root directory containing your files.
$path = "E:\temp\csv"

# Get Current EXCEL Process ID's so they are not affected but the scripts cleanup
# SilentlyContinue in case there are no active Excels
$currentExcelProcessIDs = (Get-Process excel -ErrorAction SilentlyContinue).Id

# Collect the files
$files = Get-ChildItem -Path $path -Filter "*.txt" -File

# Split the files up into processing groups. For Each group and excel process will be started. 
$numberOfGroups = 5 
$maxGroupMemberSize = [math]::Ceiling($files.Count / $numberOfGroups)

# Create as many file groups
$fileGroups = 0..($numberOfGroups - 1) | Foreach-object{
    $groupIndexStart = $maxGroupMemberSize * $_
    # Use the unary comma operator to be sure an array is returned and not unrolled
    ,$files[$groupIndexStart..($groupIndexStart + $maxGroupMemberSize - 1)]
}

# Create a job for each file group.
for($jobCount = 0; $jobCount -lt $fileGroups.Count; $jobCount++){
    # Start a unique Excel instance for this group of files.
    Start-Job -Name "Excel$jobCount" -ScriptBlock {
        param($files)
        $excelFileFormat = 56 #xlExcel8 format

        # Create a new Excel Instance
        $excel = New-Object -ComObject Excel.Application
        $excel.Visible = $false
        $excel.DisplayAlerts = $false

        ForEach ($file in $files){     
            $workbook = $Excel.Workbooks.Open($file.Fullname)
            $newFilepath = $file.Fullname -replace "\..*$",".xls"
            $workbook.SaveAs($newFilepath, $excelFileFormat)
        }

        # Quit this instance and return its memory
        $excel.Quit()
        while([System.Runtime.Interopservices.Marshal]::ReleaseComObject($workbook)){}
        while([System.Runtime.Interopservices.Marshal]::ReleaseComObject($excel)){}
        Remove-Variable "workbook","excel"

    } -ArgumentList (,($fileGroups[$jobCount])) | Out-Null
}

# Wait for the jobs to be completed and remove them from inventory since they won't have output we need
Get-Job -Name "Excel*" | Wait-Job | Receive-Job

# Remove any stale Excel processes created by this scripts execution
Get-Process excel -ErrorAction SilentlyContinue | Where-Object{$currentExcelProcessIDs -notcontains $_.id} | Stop-Process

Excel automation is full of pitfalls but the general approach seems to be working. For one com objects can prevent from being closed. Best thing to do is just kill any remaining processes. There is logic that can help to be sure any Excel opened before the script was run are not affected. Jobs are done in the way as well to not affect processing of other scripts.

This all would be better as a script using proper parameters but showing the logic of jobs is the point here.

Automate without Excel

There are a number of libraries out there that can create Excel documents without the need for COM objects or Excel itself. These would perform faster as well. EPPlus comes to mind and even [ImportExcel module] (which uses EPPlus) would be solid choices as well. I don't know what your text files look like so you will need to experiment.

Get-Content E:\temp\csv\data.txt | Export-Excel -Path "e:\temp\csv\file.xlsx"

Export-Excel works better with objects so, depending on your data structure, you would do this instead.

Import-Csv E:\temp\csv\data.txt | Export-Excel -Path "e:\temp\csv\file.xlsx"

Import-CSV is generally a slow process so you would want to try using streamreader and Convert-FromCSV if performance suffers.

Magic Numbers

Your 56 would be an example of that. You have to look it up to know what it represents. So 56 is for Excel Version 8. At a minimum that should be its own variable. My code above uses

$excelFileFormat = 56 #xlExcel8 format 

If you wanted to go a little crazier you could actually import the assembly to access the enum

[reflection.assembly]::LoadWithPartialName("Microsoft.Office.InterOp.Excel") | Out-Null
[Microsoft.Office.Interop.Excel.XlFileFormat]::xlExcel8

BaseName

 $NewFilepath = $file.Fullname -replace ".{4}$"
 $NewFilepath =  $NewFilepath + ".xls"

This is what you are doing to replace the extension of the file. File objects have a basename property that is the file name without the extension. I know that you are dealing with txt files, so you already know how many characters on the end to remove, but using .basename would remove that complexity in other areas as well.

$NewFilepath = "$($file.BaseName).xls"
\$\endgroup\$
  • \$\begingroup\$ Kudos for an exemplary answer. \$\endgroup\$ – brettdj Jan 22 '17 at 1:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.