Motivation
I came across an interesting question on SO: determine-if-string-has-all-unique-characters and thought about providing an extension method to enable duplicate removal given some kind of normalization.
Description
The goals:
- Remove any duplicate characters (keep the first occurence) and return the updated string.
- The method should be able to handle diacritics and extended unicode characters.
- The method should allow the consumer to control normalization to find duplicates (for instance, case-sensitivity).
Questions
- Looking for general feedback on C# conventions
- Performance feedback
- Am I reinventing the wheel?
Code
public static class StringExtension
{
public static string RemoveDuplicateChars(
this string text, Func<string, string> normalizer = null)
{
var output = new StringBuilder();
var entropy = new HashSet<string>();
var iterator = StringInfo.GetTextElementEnumerator(text);
if (normalizer == null)
{
normalizer = x => x.Normalize();
}
while (iterator.MoveNext())
{
var character = iterator.GetTextElement();
if (entropy.Add(normalizer(character)))
{
output.Append(character);
}
}
return output.ToString();
}
}
Unit Tests
Let's test a string that contains variations on the letter A
, including the Angstrom sign Å
. The Angstrom sign has unicode codepoint u212B
, but can also be constructed as the letter A
with the diacritic u030A
. Both represent the same character.
[TestClass]
public class Fixtures
{
[TestMethod]
public void Fixture()
{
// ÅÅAaA -> ÅAa
Assert.AreEqual("ÅAa", "\u212BA\u030AAaA"
.RemoveDuplicateChars());
// ÅÅAaA -> ÅA
// Note that the ToLowerInvariant is used to normalize characters
// when searching for duplicates, it does not mean the output gets
// transformed to lower case.
Assert.AreEqual("ÅA", "\u212BA\u030AAaA"
.RemoveDuplicateChars(x => x.Normalize().ToLowerInvariant()));
}
}