I tried this back and forth but unfortunately I can't see any performance difference between using the Group-Object approach and this :( I assume that this is because the vast majority of the time is spent reading the owner information with .GetAccessControl() and that the time saving from Group-Object is comparatively negligeble. Although I do admit I haven't tried this in production yet so it could still be the case that this approach is somewhat faster should the Group-Object start taking hours. I will update when I have verified this!

Wow, thank you! I'm still fairly new to programming in general and powershell in particular so the [System.Linq.Enumerable] was complete news to me.