# String tokenization and replacing a numbered field

I created various functions for string manipulation in Lua, where the strings are composed of tokens separated by an ASCII character (which may be the point, comma, semicolon etc.). One of these is the function puttok()

local function tokenize(C, strng)
local sInput = strng or ""
local sChar = string.format('%c', C)
local tReturn = {}
for sWord in string.gmatch(sInput, "[^"..sChar.."]+") do
table.insert(tReturn, tonumber(sWord) or sWord)
end
return tReturn
end

local function puttok(text,token,N,C)
local char = string.format("%c", C)
local n
local result
local tokens = tokenize(C, text)
if (N == 0) or (N > #tokens) then
result = text
end
if (N) then
n = (N > 0) and N or #tokens + N + 1
table.remove(tokens,n)
table.insert(tokens,n,token)
result = table.concat(tokens,char)
end
return result
end


Where

• text = string to manipulate
• token = token (string) we want insert
• N = position in which will be added to the token
• C = ASCII code of the token separator

This function is to replace a token with another. Here's an example

local text = "Violets.are.white"
text = puttok(text,"blue",3,46)
print(text)
—› Violets.are.blue


The token "blue" was inserted in position 3 in place of "white," and "Violets" and "are" are respectively in position 1 and 2. The tokens are separated by a dot (.) with ASCII code 46.

Is there a way to improve this code?
Would metatables be usefull in this case?

1. Don't use the expensive string.format where string.char can do it all.

2. You should accept a string (of length 1) as separator in tokenize and puttok, and have a default.

3. Swapping the middle two arguments of puttok allows you to default the replacement to "".

4. Your second conditional is useless, as it will always be taken: The first one results in a runtime-error if N is not a number.
I wonder why you don't use an early return there anyway?

5. First removing an element and then inserting a different one at the same index is really inefficient, resulting in a need to move all higher-numbered elements first down and then back.
tokens[n] = token

6. tokenize(C, strng) => explode(delimiter, text)