Algorithm

Generating combinations of a set is a classic problem. You chose the recursive route, which incurs auxiliary space: All those sets that were not returned will consume temporary memory. You may also be limited in the number of recursive calls by the size of the stack in your system. To overcome this, and with the help of some bit manipulation, you could create a generator that gives you only the next combination. After consuming that combination, you could ask for the next one.

Dangerous extension

This extension calls for trouble, since there is no check for non emptiness:

extension Set {
    
    func removingFirst() -> (element: Element, set: Set) {
        
        var set = self
        let element = set.removeFirst()
    
        return (element: element, set: set)
    }
}

With association to your combinations(_:) it's fine since you're checking; BUT! It wouldn't be safe to used it outside this context, since removing from an empty set would result in a fatal error.

The data structures that are more suitable to be consumed from the left and append elements to the right, are: a Linked list or a Deque.

Linting (Code presentation)

It would be nice if the use of new lines before and after the curly brackets was consistent (especially after the return keyword). At the end of the day, it's a matter of personal taste.

Naming

• firstElement: It could be misleading since a set is not an ordered collection, maybe use head instead.
• diminishedSet: To me, it feels unnatural/unfamiliar. tail would be a more suitable alternative.

Alternative implementation

Here is a pretty common way to generate a power set:

func combinations<T>(of elements: Set<T>) -> [[T]] {
    var allCombinations: [[T]] = []
    for element in elements {
        let oneElementCombo = [element]
        for i in 0..<allCombinations.count {
            allCombinations.append(allCombinations[i] + oneElementCombo)
        }
        allCombinations.append(oneElementCombo)
    }
    return allCombinations
}

Note that the return value is [[T]] to avoid unnecessary hashing, since the uniqueness is guaranteed by the fact that elements is a Set<T>.

Benchmarks

Here are some benchmarks on TIO with -O optimization on a 20-element set: