Questions tagged [lexical-analysis]

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

Filter by
Sorted by
Tagged with
7
votes
2answers
325 views

Tokenize and detect syntactic errors in simple math expressions

I thought that I didn't learn anything new for over a month or two, so I decided to start off with compilers. The code is really crappy, but at least it works. Problem definition: Given a bunch of ...
20
votes
3answers
967 views

Complex tokenizer in C#

As we have our Community Oriented Development Extravaganza, Requiring Extreme Vigor Inciting Extraordinary Winners 2017 Challenge. Finally I had some time to write a tokenizer for my own language. It ...
14
votes
4answers
3k views

Compiler Tokenizer implementation in C#

I'm writing a compiler for a couple of months now, this is the tokenization part of the lexer. I would like a code review to improve my coding style and learn new techniques to pretty up my code and ...
13
votes
1answer
2k views

Tokenizer for my programming language

Here's my attempt at porting the Lua codebase for my programming language to C++(11). This is just the first step, the tokenizer, and I wanted to remove all the bad performance / practices / code ...
4
votes
2answers
94 views

Turn characters into sensible things (tokenizing/lexing)

I'm writing a programming language. It's something I've been working on and off on for the last year or so, but recently, I've decided to buckle down and do it. As part of the language, I wrote a ...