Optimization of a custom interpreter

Question

Ok, so I want to find out if my approach towards a custom interpreter is pragmatic and if not how to implement it in a better way. this is purely for fun and practice and to find out more about how to implement a good interpreter. when I get a good grip on what to do I want to implement this into a Unity game.

Any tips, concepts, or ideas would be greatly appreciated.

Here I have my Tokenizer:

public static List<Tuple<string, string>> GetTokens(string input)
        {
            List<Tuple<string, string>> ret = new List<Tuple<string, string>>();
            Regex r = new Regex("(?<Comma>\\,)" +
                "|(?<Dot>\\.)" +
                "|(?<SemiColon>\\;)" +
                "|(?<DoubleDot>\\:)" +
                "|(?<Increment>\\+\\+)" +
                "|(?<Decrement>\\-\\-)" +
                "|(?<SystemCommand> *deviceListCount *| *deviceList *| *devices *| *device *| *str *| *int *| *dev *| *bool *| *print *| *wait *| *device *| *if *| *while *| *loop *)" +
                "|(?<OpenBracket>\\()" +
                "|(?<CloseBracket>\\))" +
                "|(?<OpenBlockBracket>\\[)" +
                "|(?<CloseBlockBracket>\\])" +
                "|(?<DeviceCommand> *On *| *Off *| *Open *| *Close *| *Move *| *Detect *)" +
                "|(?<Integer>\\d+)" +
                "|(?<Equals> *[=] *)" +
                "|(?<Greater>\\>)" +
                "|(?<Smaller>\\<)" +
                "|(?<Exclamation>\\!)" +
                "|(?<String>[aA-zZ0-9 ]*)");
            foreach (Match item in r.Matches(input))
            {
                for (int i = 1; i < item.Groups.Count; i++)
                {
                    string v = item.Groups[i].Value;
                    if (v != "")
                    {
                        ret.Add(new Tuple<string, string>(r.GroupNameFromNumber(i), v));
                    }
                }
            }
            List<Tuple<string, string>> ret1 = new List<Tuple<string, string>>();
            foreach (var item in ret)
            {
                ret1.Add(new Tuple<string, string>(item.Item1, item.Item2.Trim()));                
            }
            return ret1;
        }

Here I have the Method that Executes my interpreter's commands (i am not finished adding all the commands I want to find out if this approach is viable before I finish it) I am mainly concerned about the code below:

public static void RunCode(string input)
        {   
            var g = GetTokens(input);
            for (int i = 0; i < g.Count; i++)
            {
                if (g[i].Item2 == "print")
                {
                    Console.WriteLine(GetVariable(UntilNestedEnd(g, ref i)));
                }
                else if (g[i].Item2 == "loop")
                {
                    int cnt = int.Parse(g[i + 2].Item2);
                    i += 2;
                    string nested = UntilNestedEnd(g, ref i);
                    for (int x = 0; x < cnt; x++)
                    {
                        RunCode(nested);
                    }
                }
                else if (g[i].Item2 == "str")
                {
                    SetVar(ref g,"str",ref i);
                }
                else if (g[i].Item2 == "int")
                {
                    SetVar(ref g, "int", ref i);
                }
                else if (g[i].Item2 == "bool")
                {
                    SetVar(ref g, "bool", ref i);
                }
                else if (g[i].Item2 == "dev")
                {
                    SetVar(ref g, "dev", ref i);
                }
                else if (g[i].Item2 == "if")
                {
                    var left = GetVariable(g[i + 2].Item2);
                    string op = g[i + 3].Item2;
                    dynamic Right = null;
                    if (g[i + 4].Item2 == "True"| g[i + 4].Item2 == "true")
                    {
                        Right = true;
                    }
                    else if (g[i + 4].Item2 == "False" | g[i + 4].Item2 == "false")
                    {
                        Right = false;
                    }
                    else
                    {
                        Right = GetVariable(g[i + 4].Item2);
                    }
                    i += 4;
                    string nested = UntilNestedEnd(g, ref i);
                    switch (op)
                    {
                        case "=":
                            if (left == Right)
                            {
                                RunCode(nested);
                            }
                            break;
                        case "!":
                            if (left != Right)
                            {
                                RunCode(nested);
                            }
                            break;
                        case ">":
                            if (float.Parse(left) > float.Parse(Right))
                            {
                                RunCode(nested);
                            }
                            break;
                        case "<":
                            if (float.Parse(left) < float.Parse(Right))
                            {
                                RunCode(nested);
                            }
                            break;
                        default:
                            break;

                    }
                }
            }
        }

Here i have a Method that retrieves nested commands:

public static string UntilNestedEnd(List<Tuple<string, string>> t, ref int i)
        {            
            string inner = "";
            int nested = 0;
            while (true)
            {
                if (i < t.Count-1)
                {
                    i++;
                    if (t[i].Item2 == ")")
                    {
                        nested--;
                        if (nested > 0)
                        {
                            inner += t[i].Item2;
                        }
                    }
                    else if (t[i].Item2 == "(")
                    {
                        if (nested > 0)
                        {
                            inner += t[i].Item2;
                        }
                        nested++;
                    }
                    else
                    {
                        inner += t[i].Item2;
                    }
                    if (nested == 0)
                    {
                        break;
                    }                    
                }
            }
            return inner;
        }       

Here are the rest of the Methods i use to store or retrieve variables:

        public static Dictionary<string, string> devices = new Dictionary<string, string>();
        public static Dictionary<string, string> memory = new Dictionary<string, string>();
        public static Dictionary<string, bool> booleans = new Dictionary<string, bool>();
        public static dynamic GetVariable(string key)
        {
            if (memory.ContainsKey(key))
            {
                return memory[key];
            }
            else if (booleans.ContainsKey(key))
            {
                return booleans[key];
            }
            else if (devices.ContainsKey(key))
            {
                return devices[key];
            }
            else
            {
                return key;
            }
        }
        public static void SetVar(ref List<Tuple<string, string>> g, string type,ref int i)
        {
            i++;
            string v = g[i].Item2;
            i += 2;
            string m = g[i].Item2;            
            switch (type)
            {
                case "str":
                    if (memory.ContainsKey(v))
                    {
                        memory[v] = m;
                    }
                    else
                    {
                        memory.Add(v, m);
                    }                    
                    break;
                case "int":
                    if (memory.ContainsKey(v))
                    {
                        memory[v] = m;
                    }
                    else
                    {
                        memory.Add(v, m);
                    }
                    break;
                case "bool":
                    if (booleans.ContainsKey(v))
                    {
                        booleans[v] = bool.Parse(m);
                    }
                    else
                    {
                        booleans.Add(v, bool.Parse(m));
                    }                    
                    break;
                case "dev":
                    if (devices.ContainsKey(v))
                    {
                        devices[v] = m;
                    }
                    else
                    {
                        devices.Add(v, m);
                    }
                    break;
                default:
                    break;
            }
        }

Lastly here is the Main program with some example commands:

static void Main(string[] args)
        {            
            RunCode("if:1=1(loop:2(loop:2(loop:2(print(Hello There)))))");
            Console.ReadLine();
        }
```

Answer 1

First of all: I know, it is fun to implement your own parser from scratch and it is also reasonable for getting a better understanding of how parsers work... however if your goal is to build a custom language for your unity game, I strongly recommend to use one of the available parser generators (e.g. ANTLR).

Concept

Your approach is:

• Create Tokens

• Evaluate Tokens

That is the normal "naive approach" when trying to parse something for the first time... However, usually parsers have one step between them where tokens are build up to an abstract syntax tree. The abstract syntax tree is simply spoken an hierarchically object mode that represents the parsed code.

Evaluating the abstract syntax tree is relatively simple, because the complexity of nested expressions prioritized by brackets and so on are implicitly represented by the hierarchy.

If you want to improve your implementation, I recommend creating such an abstract syntax tree before evaluating it.

Code

• Error handling code is totally missing - there is no guarantee, that the tokens are in the right order
• GetTokens returns a tuple. I would use a custom type "Token { Name:string , Vale:string }" instead, because it is much more readable.

Examples for AST realizations

• The .Net Framework has a build-in object model: Expression Trees
• The Roslyn Compiler provides function for creating AST from C#/F#/VB code: https://github.com/dotnet/roslyn/wiki/Getting-Started-C%23-Syntax-Analysis
• Googel provides lots of information about the topic + lots of example implementations. A simple calculator may be a good point to start with

Answer 2

in order to make a smart interpreter make parser recursivecan handle expressions and statements , generate abstract syntax tree and evaluate semantic errors(data type and casting errors) here you are only dealing with syntax's set your symbols ie

const int L_PAREN = 19;
const int R_PAREN = 20;
const int WHILE = 21;
const int DO = 22;
const int IF = 23;
const int ELSE = 24;
const int THEN = 25;
const int END = 26;
const int PRINT = 27;
const int ERROR = 37; 

first your token generator must be modified in general

 function geToken ()
          {
      if operant == sometype then
      return token(token_type,current_line,current_column)
         else if another_operant ==some_type
          do the same;
         } 

after that make a parser

       currentToken = lex.getToken();
       if(currentToken.type() == Id //or int or char or something
            then
            either add to an abstract syntax tree
         evaluvate(token,"expected '='"); 

you can use a peephole optimizer if you are using complex expressions anyway i will provide a link pls check it out it is c++ don't worry it is simple you can convert them into c# or java other wise you will get an idea of making a better interpreter.thanks! this is mini lang very mini