I'm trying to reproduce in Lua the mIRC scripting manipulating tokens function:

local function tokenize(C, text)
    local char = string.format("%c", C)
    local t = {}
    for w in string.gmatch(tostring(text), "[^"..char.."]+") do
    w = tonumber(w) or tostring(w)
    return t

local function gettok(strng, position, separator, range)
  local char = string.format("%c", separator)
  local tokens = tokenize(separator, strng)
  local result, n, r, start, stop
  if (position ~= 0) then
    if (position > 0) then n = position else n = #tokens + position + 1 end
    if (range) and (position ~= range) then
      if (range > 0) then r = range

      elseif (range == 0) or ((n + range) > #tokens) then r = #tokens
      else r = n + (range + 1)
      if (n == r) then
        result = tokens[i]
        start = (r >= n) and n or r
        stop = (r <= n) and n or r
        for i = start, stop do
          result = (not result) and tokens[i] or tostring(result..char..tokens[i])
      for i = 1, #tokens do
        if (i == n) then result = tokens[i] end
  else result = strng
  return result

And this is the way it should work:

gettok(strng, position, separator, range)


  • strng = string to manipulate
  • position = position of the token inside the string. If lesser than 0, it will be considered the position from the last token to the first. If equal to 0, returns the whole string.
  • separator = ASCII code of the token separator
  • range = optional: if specified, returns the token from position to range. If equal to 0, return all tokens from position to the end of the string.

local text = "apple.banana.cherry.grape.orange"
  • apple

  • grape

  • banana.cherry.grape

  • cherry.grape.orange


Could you give me some advice on improving the code?


1 Answer 1

  1. You don't need to use tostring inside string.match or other methods provided by string meta-table.
  2. I'd suggest allowing users to pass the character as well as ASCII-code for the separator. This is of course, you own choice and nothing needs to be done in the code to enforce this.
  3. The following loop:

    for i = 1, #tokens do
        if (i == n) then result = tokens[i] end

    is entirely useless and can be replaced with result = tokens[n].

  4. I do not understand your reasons behind the following conversion:

    w = tonumber(w) or tostring(w)

    w will be type string by default.

  5. You can use table.concat effectively; instead of creating string buffers.
  6. Instead of writing long winded if-else blocks, put the smaller blocks forward and return results as soon as you arrive at one.
  7. In lua, you do not necessarily need to put the if-conditions inside parentheses.
  8. What does a negative range imply? I tested your function with gettok(text, -3, 46, -2) expecting cherry.grape as output, instead; I received banana.cherry.
  9. For your start and stop values, use math.max and math.min.

The rewritten code follows. Please note that I've used my own convention of variable naming.

local function Tokenize( sChar, sInput )
    local tReturn = {}
    for sWord in string.gmatch( sInput, "[^"..sChar.."]+" ) do
        table.insert( tReturn, tonumber(sWord) or sWord )
    return tReturn

local function GetTok( sInput, iPosition, Separator, iRange )
    local Separator = string.format( '%c', Separator )
    local tTokens = Tokenize( Separator, sInput )
    if iPosition == 0 then
        return sInput
    local iStart, iStop = ( iPosition > 0 ) and iPosition or ( #tTokens + iPosition + 1 )
    if not iRange or iPosition == iRange then
        return tTokens[ iStart ]
    if iRange > 0 then
        iStop = iRange
    elseif iRange == 0 or ( iStart + iRange ) > #tTokens then
        iStop = #tTokens
        iStop = iStart + iRange + 1
    if iStart == iStop then
        return tTokens[ iStart ]
    if iStart > iStop then
        iStart, iStop = iStop, iStart
    return table.concat( tTokens, Separator, iStart, iStop )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.