4
\$\begingroup\$

So I've been trying to implement a Token class in C++. At first I wanted to use a simple enum class to store the Kind of the Token.

But then I came across a problem. Some Tokens (such as keywords, identifiers, numeric literals, etc.) would need to store an external value along with their Kind, while others wouldn't. I tried to get around the problem by creating a struct for each Token Kind and then using an std::variant for the Kind type. Here's the code I ended up with:

#include <cstdint>
#include <string>
#include <variant>

class Token
{
  public:
    struct Num
    {
        std::int64_t val;
    };

    struct Float
    {
        double val;
    };

    struct String
    {
        std::string val;
    };

    struct Keyword
    {
        std::string val;
    };

    struct Identity
    {
        std::string val;
    };

    struct LParen
    {
    };
    struct RParen
    {
    };
    struct LBrace
    {
    };
    struct RBrace
    {
    };
    struct LBracket
    {
    };
    struct RBracket
    {
    };
    struct Semicolon
    {
    };
    struct Colon
    {
    };
    struct Dot
    {
    };

    struct Operator
    {
        enum class Name : std::uint_fast8_t
        {
            Add = 0,
            Sub,
            Mult,
            Div,
            Pow
        };

        Name val;
    };

    struct EndOfFile
    {
    };

    struct Invalid
    {
    };

    using Kind =
        std::variant<Num, Float, String, Keyword, Identity, LParen, RParen, LBrace, RBrace,
                     LBracket, RBracket, Semicolon, Colon, Dot, Operator, EndOfFile, Invalid>;

    struct Span
    {
        std::uint_fast64_t row;
        std::uint_fast64_t col;
    };

    Token() = delete;
    Token(Kind const& kind, Span span) : m_kind(kind), m_span(span) {}

    auto kind() const -> Kind { return m_kind; }
    auto span() const -> Span { return m_span; }

    auto isEOF() const -> bool { return std::holds_alternative<EndOfFile>(m_kind); }
    auto isValid() const -> bool { return !std::holds_alternative<Invalid>(m_kind); }

  private:
    Kind m_kind;
    Span m_span;
};

Is this a valid way to do what I originally intended to do? Or is it way too complicated and there's a simpler way?

\$\endgroup\$
0

1 Answer 1

4
\$\begingroup\$

Seems great to me — a nice way to simulate algebraic data types in C++.

The only improvement I can think of:

struct Operator
{
    enum class Name : std::uint_fast8_t
    {
        Add = 0,
        Sub,
        Mult,
        Div,
        Pow
    };

    Name val;
};

Why not make Operator itself an enum class?

I might consider changing

Token(Kind const& kind, Span span) : m_kind(kind), m_span(span) {}

to

template <typename T>
Token(T&& kind, Span span)
    : m_kind(std::forward<T>(kind)) // #include <utility>
    , m_span(span)
{
}

to avoid the overhead of constructing an extra variant.


The OP mentioned grouping the symbols together into an enum class in the comments. Whether this is beneficial depends on how the symbols will be used. It is easier to handle all the symbols together:

enum class Symbol {
    LParen, RParen, LBrace, RBrace, LBracket, RBracket, Semicolon, Colon, Dot,
};

// ...

std::ostream& operator<<(std::ostream& os, Symbol symbol) {
    static const std::unordered_map<Symbol, char> table {
        { Symbol::LParen, '(' },
        { Symbol::RParen, ')' },
        // etc.
    };

    return os << table.at(symbol);
}

std::ostream& operator<<(std::ostream& os, const Token& token) {
    std::visit(Overloaded {
        // ...
        [&](Symbol symbol) { os << symbol; }, // uniform handling
        // ...
    }, token.kind);
    // ...
}

versus

struct LParen {};
struct RParen {};
struct LBrace {};
struct RBrace {};
// etc.

// ...

std::ostream& operator<<(std::ostream& os, const Token& token) {
    std::visit(Overloaded {
        // ...
        [&](LParen) { os << '('; },
        [&](RParen) { os << ')'; },
        [&](LBrace) { os << '{'; },
        [&](RBRace) { os << '}'; }, // manual handling
        // ...
    }, token.kind);
    // ...
}

But it is harder to check if a token is a given symbol:

enum class Symbol {
    LParen, RParen, LBrace, RBrace, LBracket, RBracket, Semicolon, Colon, Dot,
};

// ...

bool is_symbol(const Token& token, Symbol symbol) {
    if (auto p = std::get_if<Symbol>(token)) {
        return *p == symbol;
    } else {
        return false;
    }
}

versus simply a call to std::holds_alternative.

The same applies to the operators.

\$\endgroup\$
8
  • \$\begingroup\$ Ah yeah, changing operator itself to an enum class is something I should do. Also, I don't quite get what you did with the constructor. I don't even know if it's possible to make a template constructor for a non-template class \$\endgroup\$
    – Famiu
    May 30, 2021 at 4:22
  • \$\begingroup\$ @Famiu It is indeed possible - I edited the answer to clarify. The difference here is that the argument is directly forwarded to the constructor of std::variant, whereas in your version, a temporary std::variant is constructed and then copied to m_kind. \$\endgroup\$
    – L. F.
    May 30, 2021 at 4:25
  • \$\begingroup\$ I see, thanks a lot for that. \$\endgroup\$
    – Famiu
    May 30, 2021 at 4:28
  • \$\begingroup\$ Do you think I should group all symbols together inside a symbols enum, or is it fine how it is? \$\endgroup\$
    – Famiu
    May 30, 2021 at 4:29
  • \$\begingroup\$ @Famiu Ultimately, it depends on the usage pattern. Grouping them together is beneficial when the code handles all symbols in a uniform manner, but it would be slightly more work (a helper function, that is) to check if a token is a specific symbol (get_if + ==). \$\endgroup\$
    – L. F.
    May 30, 2021 at 4:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.