To be frank: Your implementation does not target the topic token scanning.
ItIt is targeted at serialization / deserialization.
But
But the example that it is targeted at is no example of token scanning neither.
Tokens Tokens in the example would be the '.', [0-9]+, token scanning would first
buildbuild a token tree made of this tokens before interpreting the tokens in a
grammaticalgrammatical context. The implementation you choose is very clear. For the example used it could be irritating to the reader as it introduces other non - trivialtrivial concepts but I personally tend to teach clear and structured programming over examples that lead to bad code design (and mark it as "is extremely crude" ;)
.
I personally would not recommend recursion in operator implementations though, speciallyespecially not in stream operators. You would not want to call a stream operator and get an error such as "too many recursions".