Return to Revisions

This is a worthy goal - strtok() has a terrible interface, and sorely needs a stateless, thread-safe, non-destructive replacement.

It's inconvenient for the caller to be given ownership of allocated objects. I'll propose an alternative interface you might like to consider, that's based on existing practice in the standard library:

struct view { const char *s, size_t len };
size_t sepstr(struct view *result, size_t n, const char *str, const char *delim)

• result points to an array of size n supplied by the caller. A null pointer may be passed if n is zero.
• str and delim are the same as your buf and sep but renamed to be consistent with strtok() documentation
• The return value is the number of (possibly empty) tokens found (i.e. one more than the number of delimiters). This may be greater than n, in which case, tokens after the nth are not stored in result.

This interface means that a caller can use their own memory management (which might not be malloc() and free(), but can still determine the necessary buffer size, in a similar manner to snprintf().

One disadvantage here is that because we're returning a view into an existing string, we need a length instead of using ordinary null-terminated strings. The caller may need to copy them into working memory to get null termination. On the upside, it doesn't need to allocate memory for all the results at once if it only operates on one at a time.

If you combine this with the suggestion to use standard strstr() and similar functions, you should end up with a really good replacement for strtok()