Removing duplicate Unicode characters

Question

Motivation

I came across an interesting question on SO: determine-if-string-has-all-unique-characters and thought about providing an extension method to enable duplicate removal given some kind of normalization.

Description

The goals:

• Remove any duplicate characters (keep the first occurence) and return the updated string.
• The method should be able to handle diacritics and extended unicode characters.
• The method should allow the consumer to control normalization to find duplicates (for instance, case-sensitivity).

Questions

• Looking for general feedback on C# conventions
• Performance feedback
• Am I reinventing the wheel?

Code

public static class StringExtension
{
    public static string RemoveDuplicateChars(
        this string text, Func<string, string> normalizer = null)
    {
        var output = new StringBuilder();
        var entropy = new HashSet<string>();
        var iterator = StringInfo.GetTextElementEnumerator(text);

        if (normalizer == null)
        {
            normalizer = x => x.Normalize();
        }

        while (iterator.MoveNext())
        {
            var character = iterator.GetTextElement();
            if (entropy.Add(normalizer(character)))
            {
                output.Append(character);
            }
        }

        return output.ToString();
    }
}

Unit Tests

Let's test a string that contains variations on the letter A, including the Angstrom sign Å. The Angstrom sign has unicode codepoint u212B, but can also be constructed as the letter A with the diacritic u030A. Both represent the same character.

[TestClass]
public class Fixtures
{
    [TestMethod]
    public void Fixture()
    {
        // ÅÅAaA -> ÅAa
        Assert.AreEqual("ÅAa", "\u212BA\u030AAaA"
            .RemoveDuplicateChars());

        // ÅÅAaA -> ÅA
        // Note that the ToLowerInvariant is used to normalize characters
        // when searching for duplicates, it does not mean the output gets
        // transformed to lower case.
        Assert.AreEqual("ÅA", "\u212BA\u030AAaA"
            .RemoveDuplicateChars(x => x.Normalize().ToLowerInvariant()));
    }
}

Answer 1

There are not much to say other than the usual missing argument null check:

It is valid to write the following:

  string test = null;
  test.RemoveDuplicateChars();

and RemoveDuplicateChars will be called with a null for the this argument text. Therefore you'll have to test for null:

public static string RemoveDuplicateChars(
    this string text, Func<string, string> normalizer = null)
{
  if (text == null)
    return text;

  ...

or throw an exception...

---

The default initialization of normalizer could be a little less verbose:

  normalizer = normalizer ?? ((x) => x.Normalize());

---

A minor detail: Angstrom and Å: is also represented by \u00C5, which your code interprets as equal to Angstrom, but MS Word interprets them as different when using its Find-function?