1
\$\begingroup\$

This code is to gather a sorted list of IDs (first by ID, then by date) recursively from a directory that submitted a particular type of file (that, as you can tell, includes the word "Magic") and then outputs the IDs paired with the modified date (the closest proxy I have for the submission date) to a CSV. (Where I then dedupe by ID to get the earliest submission for each ID/find average number of unique IDs submitted a month over the year/ the total submitted for the year):

Get-ChildItem -Recurse -Include *Magic* |  
Where-Object {$_.lastWriteTime -gt '01/01/2018' -AND $_.lastWriteTime -lt '12/31/2018'} | 
Select-Object -Property Directory, lastWriteTime |
ForEach-Object -Begin {
     $holder=@{} 
     $results=@()
} -Process {
    $holder.id= [regex]::match($_.Directory,'(\d{8})')
    $holder.modifiedDate = Get-Date  $_.lastWriteTime -Format 'yyyy/MM/dd' 
    $results+=[pscustomobject]$holder
} -End {
    $results
    }  | 
Sort-Object -Property @{Expression = "id"; Descending = $True;}, @{Expression = "modifiedDate"; Descending = $False} | 
Export-csv C:\Users\REDACTED\Desktop\REDACTED.csv

This is my first attempt at writing something in powershell purely using the command prompt (which is to say, not scripting in the ISE as if it were Ruby using ifs and so forth) How could I have done this better and should I have organized the pipeline differently? I tried to avoid using variables like would have in the past and focused purely on piping the data I was interested in from beginning to end.

EDIT

Based on the comment I got I made these changes. I wasn't able to get rid of all the pipelines and I haven't been able to test if it's faster or not, but I have a feeling even this version has some garbage that could go.

$sourceFiles = Get-ChildItem -Recurse -Include *Magic*
$filteredFiles = @()
ForEach($file in $sourceFiles){
    If($file.LastWriteTime.Year -eq 2018){
        $filteredFiles+=[PSCustomObject]@{
            Agency = [regex]::match($file.Directory,'(\d{8})')
            ModifiedDate = Get-Date  $file.lastWriteTime -Format 'yyyy/MM/dd' 
        }
    }
}
$filteredfiles | Sort-Object -Property @{Expression = "Agency"; Descending = 
$True;}, @{Expression = "ModifiedDate"; Descending = $False} |
Export-csv C:\Users\REDACTED\Desktop\REDACTED.csv -NoTypeInformation
\$\endgroup\$
  • \$\begingroup\$ i'm curious why you want to do this in such an awkward manner that sacrifices speed for "one liner! ooo!". building and tearing down the pipeline adds to the total processing time when there are many objects - it's worth while to sacrifice speed when you need to reduce RAM use, but otherwise, it is generally not sensible. [grin] ///// on the improvement side of things, your date test seems to be checking for "in the year 2018". you can simplify that by using $_.LastWriteTime.Year -eq 2018. otherwise, things seem OK. \$\endgroup\$ – Lee_Dailey Jan 26 at 2:36
  • \$\begingroup\$ another idea ... your $Holder hashtable seems unneeded. plus, it seems likely to give you nearly random property order. just create the items in the [PSCustomObject] call. ///// yet another - your $resutls collection seems unneeded. simply put the [PSCustomObject] call on it's own line to send that to the pipeline. \$\endgroup\$ – Lee_Dailey Jan 26 at 2:40
  • \$\begingroup\$ @Lee_Dailey So, should I have done something more like $sourceFiles = Get-ChildItem -Recurse -Include *Magic* $filteredFiles = Where-Object {$sourceFiles.lastWriteTime.Year -eq 2018} and then found a way to get the ordered pairs of directories and dates from the files? I was using the pipeline because I was patterning things after a tutorial I watched, but didn't realize it was the reason everything was so incredibly slow \$\endgroup\$ – jasonmadesomething Jan 28 at 16:46
  • 1
    \$\begingroup\$ i would replace the -Include with -Filter, then add -File to ensure i only got back files, and last would explicitly indicate the path with either -LiteralPath or -Path. the 1st will handle embedded [] chars, the 2nd allows wildcards. ///// next i would remove the += since adding to arrays is SLOW. simply use $Result = foreach ($Thing in $Collection) to assign the whole set of objects to the collection all at once. ///// last i would use a [PSCustomObject] for object-building instead of building a new object with Select-Object. a PSCO structure is usually easier to read. \$\endgroup\$ – Lee_Dailey Jan 28 at 18:25
  • \$\begingroup\$ @Lee_Dailey make an answer using the latter comment. It should comply with OP's request. \$\endgroup\$ – JosefZ Feb 10 at 11:41
1
\$\begingroup\$

here is a solution that uses ...

-Filter instead of -Include
this is usually faster since the filtering is done by the provider [the filesystem in this case] instead of by the cmdlet.

[PSCustomObject] object construction
faster than Select-Object for building custom objects, since it does not require building/tearing-down a pipeline. plus the Select-Object cmdlet has a great deal of complex logic that is not needed here.

$AgencyList = foreach ($FL_Item in $FileList) loop assignment
this is VASTLY faster than $AgencyList += for large collections, but only slightly faster for small ones.

$SourceDir = $env:TEMP
$Filter = '*itunes*'
$TargetYear = 2019
$ExportFile = "$env:TEMP\jasonmadesomething_AgencyList.csv"

$GCI_Params = @{
    LiteralPath = $SourceDir
    Filter = $Filter
    File = $True
    Recurse = $True
    }
$FileList = Get-ChildItem @GCI_Params |
    Where-Object {$_.LastWriteTime.Year -eq $TargetYear}

$AgencyList = foreach ($FL_Item in $FileList)
    {
    [PSCustomObject]@{
        # i don't have any suitable directory info
        #    so i used the BaseName of the file
        Agency = $FL_Item.BaseName.Split('.')[0] -replace '^Itunes_', ''
        LastWriteTime = $FL_Item.LastWriteTime
        }
    }

# this gives a list sorted by ...
#    Agency [alfa sorted 'A' to 'Z' and/or '0' to '9']
#    LastWriteTime [oldest 1st]
$AgencyList = $AgencyList |
    Sort-Object -Property Agency, LastWriteTime

# on screen
$AgencyList

# to CSV
$AgencyList |
    Export-Csv -LiteralPath $ExportFile -NoTypeInformation

truncated on screen output ...

Agency                  LastWriteTime         
------                  -------------         
AlbumAutoRating_Disable 2019-01-07 12:22:00 PM
AlbumAutoRating_Disable 2019-01-14 12:20:36 PM
AlbumAutoRating_Disable 2019-01-21 12:20:41 PM
AlbumAutoRating_Disable 2019-01-28 12:22:14 PM
AlbumAutoRating_Disable 2019-02-04 12:21:55 PM
[*...snip...*] 
R-PC-SC_Save            2019-01-24 12:38:09 PM
R-PC-SC_Save            2019-02-07 12:37:53 PM

truncated CSV file content ...

"Agency","LastWriteTime"
"AlbumAutoRating_Disable","2019-01-07 12:22:00 PM"
"AlbumAutoRating_Disable","2019-01-14 12:20:36 PM"
"AlbumAutoRating_Disable","2019-01-21 12:20:41 PM"
"AlbumAutoRating_Disable","2019-01-28 12:22:14 PM"
"AlbumAutoRating_Disable","2019-02-04 12:21:55 PM"
[*...snip...*] 
"R-PC-SC_Save","2019-01-24 12:38:09 PM"
"R-PC-SC_Save","2019-02-07 12:37:53 PM"
\$\endgroup\$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.