Combine and condense PowerShell arrays

Question

I need to combine 3 sets of arrays that contain PsCustomObject elements, such that the unique elements of each array is only listed once, but has properties that indicate the source.

# create test cases
$a = [PsCustomObject]@{Id='AAA';Name='Dev'}
$b = [PsCustomObject]@{Id='BBB';Name='Dev/Test'}
$c = [PsCustomObject]@{Id='CCC';Name='Dev/Prod'}
$d = [PsCustomObject]@{Id='DDD';Name='Dev/Test/Prod'}
$e = [PsCustomObject]@{Id='EEE';Name='Test'}
$f = [PsCustomObject]@{Id='FFF';Name='Test/Prod'}
$g = [PsCustomObject]@{Id='GGG';Name='Prod'}

# add to arrays
$dev=@()
$dev += $a
$dev += $b
$dev += $c
$dev += $d

$test=@()
$test += $b
$test += $d
$test += $e
$test += $f

$prod=@()
$prod+=$c
$prod+=$d
$prod+=$f
$prod+=$g

# array to contain the results
$result=@()

# process dev list; decorate w/ additional properties
$dev | % { 
    Add-Member -InputObject $_ -NotePropertyName "DEV" -NotePropertyValue "Y" 
    Add-Member -InputObject $_ -NotePropertyName "TEST" -NotePropertyValue $null
    Add-Member -InputObject $_ -NotePropertyName "PROD" -NotePropertyValue $null
    # add to results
    $result += $_
}

# process test list
$test | % {

    $match = $result -match $_.Id

    # if a test element matches an element in the results
    if ($match) {
        # set the column
        $match[0].TEST='Y'
    }
    # if this element is not in the results
    else {
        # additional properties
        Add-Member -InputObject $_ -NotePropertyName "DEV" -NotePropertyValue $null
        Add-Member -InputObject $_ -NotePropertyName "TEST" -NotePropertyValue "Y"
        Add-Member -InputObject $_ -NotePropertyName "PROD" -NotePropertyValue $null
        # add to results
        $result += $_    
    }
}

# process test list
$prod | % {

    $match = $result -match $_.Id

    # if a test element matches an element in the results
    if ($match) {
        # set the column
        $match[0].PROD='Y'
    }
    else {
        # additional properties
        Add-Member -InputObject $_ -NotePropertyName "DEV" -NotePropertyValue $null
        Add-Member -InputObject $_ -NotePropertyName "TEST" -NotePropertyValue $null
        Add-Member -InputObject $_ -NotePropertyName "PROD" -NotePropertyValue 'Y'
        # add to results
        $result += $_    
    }
}

# display the results
$result | format-table -AutoSize

the results of the script:

Id  Name          DEV TEST PROD
--  ----          --- ---- ----
AAA Dev           Y            
BBB Dev/Test      Y   Y        
CCC Dev/Prod      Y        Y   
DDD Dev/Test/Prod Y   Y    Y   
EEE Test              Y        
FFF Test/Prod         Y    Y   
GGG Prod                   Y

I realize that I could refactor the search logic into a single function, but my concern is the search logic itself.

Is there a more-efficient way to perform this logic?

TheMadTechnician · Accepted Answer · 2017-10-23 19:18:36Z

Assuming your objects have a unique property, such as the Id property in your example, you can expand each array in a single pipeline and use Select -Unique to create a combined array, and add your tracking in the same command as such:

$result = $dev,$test,$prod|%{$_}|Select *,@{l='Dev';e={$_.Id -in $dev.id}},@{l='Prod';e={$_.id -in $prod.id}},@{l='Test';e={$_.id -in $test.id}} -Unique

That cuts the time down to a bit under half what you are seeing. I ran Measure-Command 100 times against your code vs my code (using the same test build in each, the only difference being how $result was generated), and your code on my machine took about 16.9ms on average, and my code took 7.4ms on average.

1..100|%{Measure-Command {# create test cases
$a = [PsCustomObject]@{Id='AAA';Name='Dev'}
$b = [PsCustomObject]@{Id='BBB';Name='Dev/Test'}
$c = [PsCustomObject]@{Id='CCC';Name='Dev/Prod'}
$d = [PsCustomObject]@{Id='DDD';Name='Dev/Test/Prod'}
$e = [PsCustomObject]@{Id='EEE';Name='Test'}
$f = [PsCustomObject]@{Id='FFF';Name='Test/Prod'}
$g = [PsCustomObject]@{Id='GGG';Name='Prod'}

# add to arrays
$dev = $a,$b,$c,$d

$test = $b,$d,$e,$f

$prod = $c,$d,$f,$g

$results = $dev,$test,$prod|%{$_}|Select *,@{l='Dev';e={$_.Id -in $dev.id}},@{l='Prod';e={$_.id -in $prod.id}},@{l='Test';e={$_.id -in $test.id}} -Unique

$Results|FT -Auto
}}|Measure-Object -Average -Property TotalMilliseconds|% Average

7.408363

There could be a difference in processor speed here. What do you get when you average on Ticks rather than TotalMilliseconds ? I get 42995.2 Ticks for mine and 97630.5 for yours — Jonathon Anderson, Oct 23 '17 at 20:19
Oh, no, I can confirm that your code is faster, and I'm sure that it consumes less memory, but mine does not modify the original objects, and would work with objects that are not in shared memory space, so if they import three CSV files or something mine would still work, where I'm pretty sure yours would fail. — TheMadTechnician, Oct 23 '17 at 20:56
Yea, you're right. I wasn't sure how they were creating objects. Mine is definitely not terribly flexible. — Jonathon Anderson, Oct 24 '17 at 14:47

Sᴀᴍ Onᴇᴌᴀ · Accepted Answer · 2017-10-20 17:14:32Z

How about the code below?

Note: this assumes that you are actually passing the same object into the arrays, as you are doing here. If the arrays are being populated on the fly, and each will have it's own instances of objects with the same ID, then this method won't work.

# create test cases
    $a = [PsCustomObject]@{Id='AAA';Name='Dev'          ;DEV='';TEST='';PROD=''}
    $b = [PsCustomObject]@{Id='BBB';Name='Dev/Test'     ;DEV='';TEST='';PROD=''}
    $c = [PsCustomObject]@{Id='CCC';Name='Dev/Prod'     ;DEV='';TEST='';PROD=''}
    $d = [PsCustomObject]@{Id='DDD';Name='Dev/Test/Prod';DEV='';TEST='';PROD=''}
    $e = [PsCustomObject]@{Id='EEE';Name='Test'         ;DEV='';TEST='';PROD=''}
    $f = [PsCustomObject]@{Id='FFF';Name='Test/Prod'    ;DEV='';TEST='';PROD=''}
    $g = [PsCustomObject]@{Id='GGG';Name='Prod'         ;DEV='';TEST='';PROD=''}

    # add to arrays
    $dev=@()
    $dev += $a
    $dev += $b
    $dev += $c
    $dev += $d

    $test=@()
    $test += $b
    $test += $d
    $test += $e
    $test += $f

    $prod=@()
    $prod+=$c
    $prod+=$d
    $prod+=$f
    $prod+=$g

    # array to contain the results
    $result=@()

    # process dev list; decorate w/ additional properties
    $dev | % { 
        $_.DEV="Y"
        # add to results
        $result += $_
    }

    # process test list
    $test | % {
        $_.TEST="Y"
        # add to results
        if (!$result.Contains($_)){
        $result += $_ 
        }
    }

    # process test list
    $prod | % {
        $_.PROD="Y"
        # add to results
        if (!$result.Contains($_)){
        $result += $_ 
        }
    }

    # display the results
    $result | format-table -AutoSize

Measure-Command output from yours:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 38
Ticks             : 388914
TotalDays         : 4.50131944444444E-07
TotalHours        : 1.08031666666667E-05
TotalMinutes      : 0.00064819
TotalSeconds      : 0.0388914
TotalMilliseconds : 38.8914

Measure-Command output from mine:

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 16
Ticks             : 167563
TotalDays         : 1.93938657407407E-07
TotalHours        : 4.65452777777778E-06
TotalMinutes      : 0.000279271666666667
TotalSeconds      : 0.0167563
TotalMilliseconds : 16.7563

wOxxOm · Accepted Answer · 2017-10-24 00:53:40Z

Generate arrays via foreach instead of += which recreates the entire array each time
Declare arrays using simplified syntax: just a list of comma-separated elements

$dev = $a, $b, $c, $d
$test = $b, $d, $e, $f
$prod = $c, $d, $f, $g

$result = @(
    foreach ($_ in $dev) { Add-Member 'DEV' 'Y' -InputObject $_ -PassThru }
    foreach ($_ in $test) { Add-Member 'TEST' 'Y' -InputObject $_ -PassThru }
    foreach ($_ in $prod) { Add-Member 'PROD' 'Y' -InputObject $_ -PassThru }
) | Group Id | ForEach { $_.Group[0] }

# help Format-Table recognize all added properties
Add-Member ([ordered]@{
    DEV = $result[0].DEV
    TEST = $result[0].TEST
    PROD = $result[0].PROD
}) -Force -InputObject $result[0]

$result | ft -auto

While the above code is approximately 2 times faster than the original one, it's possible to make it 5 times faster in PowerShell 5 and newer by adding properties directly via .NET method so the second block would become as follows:

$addedId = @{}
$result = Sort Id -input @(
    foreach ($sourceName in 'DEV', 'TEST', 'PROD') {
        $source = Get-Variable $sourceName -value
        $prop = [PSNoteProperty]::new($sourceName, 'Y')
        foreach ($item in $source) {
            $item.PSObject.Properties.Add($prop)
            if (!$addedId[$item.Id]) {
                $addedId[$item.Id] = $true
                $item
            }
        }
    }
)

Stack Exchange Network

current community

your communities

more stack exchange communities

Combine and condense PowerShell arrays

3 Answers 3

Your Answer

Not the answer you're looking for? Browse other questions tagged powershell or ask your own question.

Hot Network Questions

Combine and condense PowerShell arrays

3 Answers 3

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged powershell or ask your own question.

Related

Hot Network Questions