mIRC scripting like string token manipulation in Lua

Question

I'm trying to reproduce in Lua the mIRC scripting manipulating tokens function:

local function tokenize(C, text)
    local char = string.format("%c", C)
    local t = {}
    for w in string.gmatch(tostring(text), "[^"..char.."]+") do
    w = tonumber(w) or tostring(w)
    table.insert(t,w)
    end
    return t
end

local function gettok(strng, position, separator, range)
  local char = string.format("%c", separator)
  local tokens = tokenize(separator, strng)
  local result, n, r, start, stop
  if (position ~= 0) then
    if (position > 0) then n = position else n = #tokens + position + 1 end
    if (range) and (position ~= range) then
      if (range > 0) then r = range

      elseif (range == 0) or ((n + range) > #tokens) then r = #tokens
      else r = n + (range + 1)
      end
      if (n == r) then
        result = tokens[i]
      else
        start = (r >= n) and n or r
        stop = (r <= n) and n or r
        for i = start, stop do
          result = (not result) and tokens[i] or tostring(result..char..tokens[i])
        end
      end
    else
      for i = 1, #tokens do
        if (i == n) then result = tokens[i] end
      end
    end
  else result = strng
  end
  return result
end

And this is the way it should work:

gettok(strng, position, separator, range)

Where

• strng = string to manipulate
• position = position of the token inside the string. If lesser than 0, it will be considered the position from the last token to the first. If equal to 0, returns the whole string.
• separator = ASCII code of the token separator
• range = optional: if specified, returns the token from position to range. If equal to 0, return all tokens from position to the end of the string.

---

local text = "apple.banana.cherry.grape.orange"

• apple

  gettok(text,1,46)
  

• grape

  gettok(text,-2,46)
  

• banana.cherry.grape

  gettok(text,2,46,4)
  

• cherry.grape.orange

  gettok(text,-1,46,-3)
  

Could you give me some advice on improving the code?

Answer 1

1. You don't need to use tostring inside string.match or other methods provided by string meta-table.
2. I'd suggest allowing users to pass the character as well as ASCII-code for the separator. This is of course, you own choice and nothing needs to be done in the code to enforce this.

3. The following loop:

   for i = 1, #tokens do
       if (i == n) then result = tokens[i] end
   end
   

   is entirely useless and can be replaced with result = tokens[n].

4. I do not understand your reasons behind the following conversion:

   w = tonumber(w) or tostring(w)
   

   w will be type string by default.

5. You can use table.concat effectively; instead of creating string buffers.
6. Instead of writing long winded if-else blocks, put the smaller blocks forward and return results as soon as you arrive at one.
7. In lua, you do not necessarily need to put the if-conditions inside parentheses.
8. What does a negative range imply? I tested your function with gettok(text, -3, 46, -2) expecting cherry.grape as output, instead; I received banana.cherry.
9. For your start and stop values, use math.max and math.min.

The rewritten code follows. Please note that I've used my own convention of variable naming.

local function Tokenize( sChar, sInput )
    local tReturn = {}
    for sWord in string.gmatch( sInput, "[^"..sChar.."]+" ) do
        table.insert( tReturn, tonumber(sWord) or sWord )
    end
    return tReturn
end

local function GetTok( sInput, iPosition, Separator, iRange )
    local Separator = string.format( '%c', Separator )
    local tTokens = Tokenize( Separator, sInput )
    if iPosition == 0 then
        return sInput
    end
    local iStart, iStop = ( iPosition > 0 ) and iPosition or ( #tTokens + iPosition + 1 )
    if not iRange or iPosition == iRange then
        return tTokens[ iStart ]
    end
    if iRange > 0 then
        iStop = iRange
    elseif iRange == 0 or ( iStart + iRange ) > #tTokens then
        iStop = #tTokens
    else
        iStop = iStart + iRange + 1
    end
    if iStart == iStop then
        return tTokens[ iStart ]
    end
    if iStart > iStop then
        iStart, iStop = iStop, iStart
    end
    return table.concat( tTokens, Separator, iStart, iStop )
end