# A lexer in C++ for analysing regex-like text

I'm creating a parser, and I have just finished my lexer. I wanted to ask if there is anything I should change, add, or reconsider in my code! (I don't think the grammars matter much, since it is only a lexer)

lexical_analyzer.h

#pragma once

#include <iostream>
#include <string>
#include <vector>

// kinds of lexemes (i.e. tokens)
namespace la_enum
{
enum token
{
STRING              // anything that isn't something below
, AND               // +
, CHAR_REPEATED     // *
, LEFT_PARENTHESIS  // (
, RIGHT_PARENTHESIS // )
, ANY_CHAR          // .
, COUNTER           // {N}
, IGNORE_CASE       // \I
, SINGLE_CAPTURE    // \O{N}
};
}

class lexical_analyzer
{
public:
lexical_analyzer(std::string patternInput) :
pattern(patternInput)
{
}

private:
void addLexemes(); // adds lexemes and tokens from the pattern to the vectors

bool isSingleSymbol(char); // checks if it is an operator with ONE symbol

void addString(); // is used to store strings (operands)

void addOperator(la_enum::token, std::string&, int, int); // is used to store operators

std::string pattern; // input pattern
std::string characterBuffer; // buffer for operators with more than one symbol

// 'lexemes' and 'tokens' have synched indexes
std::vector<std::string> lexemes; // stores lexemes
std::vector<la_enum::token> tokens; // stores tokens
};


lexical_analyzer.cpp

#include "lexical_analyzer.h"

// adds lexemes and tokens from the pattern to the vectors
{
for (int i = 0; i != pattern.size(); i++)
{
// adds lexems and tokens from the char buffer (strings)
if (isSingleSymbol(pattern[i]))

switch (pattern[i])
{
case ('+') :
break;
case ('*') :
break;
case ('.') :
break;
case ('(') :
break;
case (')') :
break;
default :
if (pattern[i] == '{')
{
// checks if it's the right syntax '{N}'...
if (isdigit(pattern[i + 1]) && pattern[i + 2] == '}')
{
i += 2;
}
else
{   // ...otherwise it counts as a string and is added to the buffer
characterBuffer.push_back(pattern[i]);
}
}
else if (pattern[i] == '\\')
{
// checks if it's the right syntax '\I'...
if (pattern[i + 1] == 'I')
{
i++;
}
// checks if it's the right syntax '\O{N}'...
else if (pattern[i + 1] == 'O' && pattern[i + 2] == '{' && isdigit(pattern[i + 3]) && pattern[i + 4] == '}')
{
i += 4;
}
else
{   // ...otherwise it counts as a string and is added to the buffer
characterBuffer.push_back(pattern[i]);
}
}
else // If the symbol isn't one of those above
// it counts as a string (operand) and is added to the buffer
{
characterBuffer.push_back(pattern[i]);
}
}
}
// check one last time if the buffer has content which is then added to the vectors
// prints tokens and lexemes
for (int i = 0; i != lexemes.size(); i++)
{
std::cout << "Token: \"" << tokens[i] << "\" Lexeme: \"" << lexemes[i] << "\"" << std::endl;
}
}

// checks if it is an operator with ONE symbol
bool lexical_analyzer::isSingleSymbol(char c)
{
if (c == '+' || c == '*' || c == '(' || c == ')' || c == '.')
return true;
else
return false;
}

// checks if the buffer has content
// which is then added as lexeme and token
{
if (!characterBuffer.empty())
{
lexemes.push_back(characterBuffer);
tokens.push_back(la_enum::STRING);
}
}

// adds an operator as a lexeme in the 'lexeme' vector, and token in 'tokens' vector
void lexical_analyzer::addOperator(la_enum::token tok, std::string& str, int pos, int sz)
{
lexemes.push_back(std::string(str, pos, sz));
tokens.push_back(tok);
characterBuffer.clear();
}


main.cpp

#include <iostream>

#include "lexical_analyzer.h"

int main()
{
//std::string in;
//std::getline(std::cin, in);
//lexical_analyzer(std::move(in));

lexical_analyzer("Hell. (MY)\I n..e (is+was) Melwin.\O{0}");

return 0;
}


output:

Token: "0" Lexeme: "Hell"
Token: "5" Lexeme: "."
Token: "0" Lexeme: " "
Token: "3" Lexeme: "("
Token: "0" Lexeme: "MY"
Token: "4" Lexeme: ")"
Token: "0" Lexeme: "I n"
Token: "5" Lexeme: "."
Token: "5" Lexeme: "."
Token: "0" Lexeme: "e "
Token: "3" Lexeme: "("
Token: "0" Lexeme: "is"
Token: "1" Lexeme: "+"
Token: "0" Lexeme: "was"
Token: "4" Lexeme: ")"
Token: "0" Lexeme: " Melwin"
Token: "5" Lexeme: "."
Token: "0" Lexeme: "O"
Token: "6" Lexeme: "{0}"

• you could use a scoped enum: en.cppreference.com/w/cpp/language/enum rather than namespaced. And maybe some form of map to get from char to enum rather than enum defined in one place and then a big switch statement. The map might be more sophisticated with lambda/callbacks or similar for a design which could scale to something more complex? Jan 23 '20 at 14:14
• @OliverSchonrock I had been thinking about using a map in this project, and you might be on to something there. I will see what I can find. But you're saying I could remove the switch statement altogether and use a map instead? Jan 23 '20 at 14:34
• I have not spent much time on the code, but yes, it look like all the simple - non-default - cases could be handled in one or 2 lines with a map. worth exploring. I agree with @ratchet freak that if the number of tokens will stay at this level you don't need a map. I depends on the future of this code. for 200 tokens a map would probably be more maintainable Jan 23 '20 at 14:52
• @OliverSchonrock I will explore it! thanks! Jan 23 '20 at 15:23
• Lexer tools are already prevalent. Have you though about using "Lex"? Jan 27 '20 at 23:52

You should clear characterBuffer at the end of addString instead of addOperator.

There is no need for characterBuffer to be a member field. Instead make it a local in addLexemes and pass it (by ref) when needed.

There is a very consistent patter that every time you call addOperator you call addString right before. Therefor you can put addString in addOperator.

// first adds the string in 'precedingStringBuffer' as string if not empty
// then adds an operator as a lexeme in the 'lexeme' vector, and token in 'tokens' vector
void lexical_analyzer::addOperator(std::string& precedingStringBuffer, la_enum::token tok, const std::string& str, int pos, int sz)
{

The COUNTER and SINGLE_CAPTURE only allow for a single digit between the braces, this can be troublesome if you ever need something more than 9 in there.