Nice and clean implementation. I tested this with the tests (the new ones) of my implementation and they all passed (skipping the tests with chunkSize == 0).
The code could be sligthly more readable by having some vertical space to group related code, for instance after validating the input.
You could improve this a little bit by just having this
if (chunkSize == 1)
{
yield return value;
return;
}
after the validation, in this way you wouldn't need to create a StringBuilder
nor having the enumerator
.