Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [lexical-analysis]

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

Filter by
Sorted by
Tagged with
10 votes
6 answers
2k views

Efficient C Tokenizer/Lexer in C++

I wanted to create my own compiler for C and it's the first prototype of the Lexer. Any thoughts on improvements? Note: I forgot to add comment lines in header files while I was coding, but I assure ...
Egemen Yalın's user avatar
8 votes
1 answer
278 views

C++ - Lexer for the Monkey Programming Language from the book "Writing An Interpreter In Go"

Below is a C++ version of a lexer for the Monkey programming language, originally written in Go, as presented in the book Writing An Interpreter In Go. LOC: 262. (Excluding the tests.) Language: Copy ...
Harith's user avatar
  • 7,054
4 votes
1 answer
108 views

C99 - Lexer for the Monkey Programming Language from the book "Writing An Interpreter In Go"

Below is a C version of a lexer for the Monkey programming language, originally written in Go, as presented in the book Writing An Interpreter In Go. LOC: 371. (Excluding the tests.) Language: Copy ...
Harith's user avatar
  • 7,054
-4 votes
2 answers
61 views

How to refactor "long" if statment in lexer? [closed]

I have a Tokenizer doing lexer thing. It is used like String line = "class Main {" ...
codeDog's user avatar
3 votes
1 answer
250 views

ANTLR4 grammar for Conventional Commits spec

I would like to create a grammar for the Conventional Commits spec and I would love to hear any feedback for what I wrote. The spec has some ambiguities, I think, hence my usage of "island ...
Michael's user avatar
  • 133
6 votes
2 answers
190 views

Programming Language Source Lexer

I have a lexer (program that turns source code into tokens) written in Go that I am re-writing in Rust. The lexer should take in a sequence of tokens, generally different special characters, integers, ...
John Allison's user avatar
1 vote
1 answer
156 views

Rust state-machine lexer

I tried implementing a lexer in rust that peeks ahead at the next character and makes a decision based on that. However, i am told that this is bad practice, and instead i should be using finite-state-...
xxxxxxxxxx's user avatar
0 votes
1 answer
169 views

supporting unicode in rust lexer

I am working on a lexer in rust. It has been through many different iterations, going from iterating over a Vec<char> to ...
xxxxxxxxxx's user avatar
4 votes
1 answer
420 views

lexer for programming languages in rust

I'm relatively new to rust, so you don't need to go easy on the criticism. right now, this lexes 1MB of text (well, trims a 1MB-long word) in 30ms. is there any way I can make this faster? cursor.rs: <...
xxxxxxxxxx's user avatar
-1 votes
1 answer
125 views

top-down lexer in rust

the other day, i decided to build a top-down lexer in rust, just for fun. this is what i have so far: ...
John doe's user avatar
3 votes
1 answer
276 views

Generating indent and dedent tokens for python-style indenting code

I found lexy, a C++ template-based parser combinator, that helps a lot to build a language parser. The problem is that lexy can only deal with context free grammar, and python indentation is not ...
jokoon's user avatar
  • 293
2 votes
1 answer
149 views

Simple Manual Lexer in Haskell

As a beginner exercise, I made small manual lexer that recognizes three types of inputs: integers: /[-]?[0-9]+/ strings, inside double quotes, with backslash ...
Pierre Arlaud's user avatar
1 vote
1 answer
98 views

Lexer for shell-like language [rust]

I am working on writing a rust implementation of a sh-like language. Rather than posting 600 lines of code here, you can just go to the GitHub repository. Is this a ...
CATboardBETA's user avatar
1 vote
1 answer
206 views

C# Language Lexer

Here is a Lexer for a programming language I am working on. Any feedback would be appreciated. I only started learning C# a couple of days ago, so please excuse my newbie code :) ...
conaticus's user avatar
  • 125
3 votes
1 answer
271 views

Simple tokenizer in C

I implemented a simple tokenizer. Would love to hear your feedback on code style, best practices: ...
kovac's user avatar
  • 269
6 votes
1 answer
226 views

Lexer for a Scheme-like language in rust

I was/am working on an interpreter for a scheme-like language. Just some time back I shifted my implementation from C++ to Rust, which I just started learning. I know there are parser libraries like ...
Bhargav Kulkarni's user avatar
2 votes
3 answers
429 views

Optimize lexer/parser bottleneck in C++

I am writing a parser for a custom mesh format for fluid dynamics simulation library, the mesh file contains 3D points (vertices) for the simulation mesh, for example: ...
Algo's user avatar
  • 339
5 votes
1 answer
208 views

How could this indentation-based lexer be simplified?

I finally got this lexer working, full implementation here. How can the lex function be simplified? The goal is to take a tree of indentation-based text, and ...
Lance's user avatar
  • 351
1 vote
1 answer
273 views

A Token class for a lexical parser

In a larger project, I need a (rather simple) expression parser able to accept numerical values, operators and string identifiers. Fine, a lexical parser fed with some input which gives tokens one at ...
Serge Ballesta's user avatar
4 votes
1 answer
572 views

Token class implementation with std::variant

So I've been trying to implement a Token class in C++. At first I wanted to use a simple enum class to store the Kind of the ...
Famiu's user avatar
  • 196
5 votes
2 answers
1k views

Tokenize arithmetic expression in C

This program tokenizes an arithmetic expression specified in infix notation. It is one of my personal projects and I would love to receive expert advice. After compiling, you can run the program with ...
Andy Sukowski-Bang's user avatar
4 votes
1 answer
249 views

Idiomatic Go Lexer

I have written some Go code that produces the output I would like, but I am unsure if the code itself "smells" good. I would like to know if it contains any anti-patterns, exemplifies bad ...
Brent Pappas's user avatar
12 votes
2 answers
318 views

A simple parser generator

I need to parse simple DSLs in a few projects. Since I don't know BNF or other grammars, I figured an alternative would be to use a simple parser generator. I'm looking for improvements to the lexer/...
C5H8NNaO4's user avatar
  • 201
4 votes
1 answer
63 views

Hand Coded State Driven Lexical Analyzer in C With Unit Test Part B

This review is presented in 3 questions due to the amount of code: Part A contains the Lexical Analyzer and the main portion of the unit test code. Part B (this question) contains the lower level ...
pacmaninbw's user avatar
  • 23.8k
9 votes
2 answers
293 views

Hand Coded State Driven Lexical Analyzer in C With Unit Test Part A

This code review is presented in 3 questions due to the amount of code: Part A (this question) contains the Lexical Analyzer and the main portion of the unit test code. Part B contains the lower ...
pacmaninbw's user avatar
  • 23.8k
2 votes
1 answer
226 views

Tokenizer FSM in Rust - better way to do match statements?

I'm learning Rust and have decided to code up a toy YANG parser as an exercise. In case it's relevant, I mainly work in C and Python. While working on the tokenizer/lexer, I tried to make it Rust-y, ...
Kyle_S-C's user avatar
  • 233
3 votes
0 answers
380 views

Lexical Analyzer In C++

So this is my first time writing a Lexer, and I want to make sure I'm doing it right. The lexer is not complete for a programming language right now, because I think I can easily add more stuff later. ...
nishantc1527's user avatar
3 votes
1 answer
381 views

Tokenizer / lexer

I wanted to make a calculator in python so I first wrote a tokenizer. I have written one before but this time I tried to refine it a bit. Any thoughts on improvements, things I could of done better. <...
Sheep_Wizard's user avatar
1 vote
0 answers
47 views

implementing parser - improvment

I have been trying to develop a program which implements a parser, i.e., syntactic analysis. Here is what I have done so far: parser.c ...
Amiram Goldblum's user avatar
2 votes
2 answers
88 views

Optimization of a custom interpreter

Ok, so I want to find out if my approach towards a custom interpreter is pragmatic and if not how to implement it in a better way. this is purely for fun and practice and to find out more about how to ...
TyperMan1133's user avatar
5 votes
1 answer
853 views

Lexer written in Rust

So I ported a lexer I wrote in C++ over to rust, as I'm starting to learn rust. Since I'm very new though, I don't know any Idioms and good practices in rust. So if anyone could point out some (...
Ian Rehwinkel's user avatar
3 votes
1 answer
302 views

A lexer in C++ for analysing regex-like text

I'm creating a parser, and I have just finished my lexer. I wanted to ask if there is anything I should change, add, or reconsider in my code! (I don't think the grammars matter much, since it is only ...
Hampus Lundberg's user avatar
4 votes
1 answer
456 views

Is this a good way to write a lexer?

I don't know whether if this is the right place to ask this. I am semi-beginner in C. I always wanted to build my own Programming Language. Here I have built a lexical analyser completely myself. ...
rsonx's user avatar
  • 143
5 votes
0 answers
139 views

Scanning a string into a list of tokens

Here is my code: ...
dopatraman's user avatar
3 votes
1 answer
164 views

Lex Scheme in Rust

I'm writing a Scheme interpreter in Rust, and the first step is the parser. I've finished the lexer, and would like to see what people think of it before I go any further. ...
Simon Brahan's user avatar
2 votes
1 answer
1k views

Quadratic "Complete the Square" Solver in Python 3

I quickly wrote this program to transform quadratic expressions in general form into vertex form. \$ax^2 + bx + c \;=\; a(x-h)^2 + k,\quad\text{where}\quad h = -\frac{b}{2a} \quad\text{and}\quad k = ...
nokko's user avatar
  • 43
5 votes
1 answer
426 views

Simplest lexer for c in go

I started learning compilers development and as pet project decided to implement c compiler. The first step is implementing simplest lexer. Also, I decided to try new programming language and chose go....
Greg Elledge's user avatar
5 votes
1 answer
181 views

Work-In-Progress Lexer

So, purely out of desire to learn, I decided to take on a project of making my own very basic programming language from scratch. After which, I made a lexer. I've tested the lexer (but only a bit) ...
Famiu's user avatar
  • 196
5 votes
2 answers
821 views

Tokenizer Class for handling .csv files in c++17

I am a hobbyist computer programmer trying to learn modern C++ (in this case C++17). I thought it might be an interesting challenge to write a tokenizer class for ...
davidbear's user avatar
  • 577
3 votes
1 answer
188 views

Function to parse VB.NET Like pattern

VB.NET has a Like operator with a paradigm similar to the standard SQL LIKE expression (SQL Server, SQLite, MySQL), but with a ...
Zev Spitz's user avatar
  • 438
7 votes
5 answers
362 views

Simple tokenizer v2 - reading all matching chars at once

I have rewritten my tokenizer according to most of the suggestions from the previous question here. API It now reads all chars as long as they match the pattern. I use three types of attributes to ...
t3chb0t's user avatar
  • 44.3k
3 votes
2 answers
497 views

Simple tokenizer v1 - reading char by char

I was feeling like writing a new Uri parser. The previous one was too limited and wasn't able to parse the authority part. This one is also based on the pretty ...
t3chb0t's user avatar
  • 44.3k
6 votes
2 answers
249 views

Compact command line argument parser : Revisited

This question is a follow up to my previous one which can be found here. The user dfhwze suggested me to look into compiler construction and recommended me to write a lexer and a parser that would ...
766F6964's user avatar
  • 901
6 votes
1 answer
4k views

Lexer using C++ and Regex

I'm trying to create a very basic compiler, and I'm working on the lexer. I decided to store keywords and operators in an unordered_set, and use regex to match identifiers and literals (no comments ...
Rish's user avatar
  • 2,341
1 vote
2 answers
544 views

Java lexical analyser

I created a lexical analyser in Java recently, but I don't think the performance is very good. The code works, but when I debugged the program, it take around ~100 milliseconds for only two tokens... ...
Minkizz's user avatar
  • 31
7 votes
4 answers
2k views

Lexer for C++ in python

I wrote a simple C++ lexer in python. Although it already works but I kinda feel I didn't follow the pythonic way. Any suggestion for improvements? Sample C++ Code Sample output Here's my code: ...
Nitwit's user avatar
  • 501
4 votes
0 answers
267 views

Writing a tokenizer in SWI-Prolog

To teach myself how to do meta-programming using Prolog, I've been playing around with translating ABNF syntax into DCG syntax. I'm still at the lexing and tokenising stage, but hope to advance to the ...
joeblog's user avatar
  • 243
3 votes
1 answer
278 views

Parsing a C++ function declaration with Perl 6 grammar

I wrote a Perl 6 grammar to parse a C++ function. My final goal would be to parse an entire header. The aim is not to correct C++ syntax errors, but to parse valid C++. Do you have some advice or ...
tejas's user avatar
  • 321
2 votes
0 answers
159 views

Lexer function to handle the '/' character

I'm writing a toy compiler, which is in a very early stage of development. The lexer contains the following block of code: ...
Myk's user avatar
  • 141
4 votes
2 answers
271 views

Lexer for Apache logs in Python

I recently learned Python using the book by Mark Summerfeld and I am now doing a few katas of mine when I learn a new language. This helps to get the “how the language works” and I wrote a simple ...
Michaël Le Barbier's user avatar