Nice and clean implementation. I tested this with the tests (the new ones) of my implementation and they all passed (skipping the tests with chunkSize == 0).  

The code could be sligthly more readable by having some vertical space to group related code, for instance after validating the input.  

You could improve this a little bit by just having this  

    if (chunkSize == 1)
    {
        yield return value;
        return;
    }  

after the validation, in this way you wouldn't need to create a `StringBuilder` nor having the `enumerator`.