Questions tagged [lexical-analysis]
Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.
70
questions
-4
votes
2
answers
60
views
How to refactor "long" if statment in lexer? [closed]
I have a Tokenizer doing lexer thing.
It is used like
String line = "class Main {"
...
- 1
3
votes
1
answer
180
views
ANTLR4 grammar for Conventional Commits spec
I would like to create a grammar for the Conventional Commits spec and I would love to hear any feedback for what I wrote.
The spec has some ambiguities, I think, hence my usage of "island ...
- 133
6
votes
2
answers
163
views
Programming Language Source Lexer
I have a lexer (program that turns source code into tokens) written in Go that I am re-writing in Rust.
The lexer should take in a sequence of tokens, generally different special characters, integers, ...
- 163
1
vote
1
answer
121
views
Rust state-machine lexer
I tried implementing a lexer in rust that peeks ahead at the next character and makes a decision based on that.
However, i am told that this is bad practice, and instead i should be using finite-state-...
- 57
0
votes
1
answer
126
views
supporting unicode in rust lexer
I am working on a lexer in rust.
It has been through many different iterations, going from iterating over a Vec<char> to ...
- 57
4
votes
1
answer
305
views
lexer for programming languages in rust
I'm relatively new to rust, so you don't need to go easy on the criticism.
right now, this lexes 1MB of text (well, trims a 1MB-long word) in 30ms.
is there any way I can make this faster?
cursor.rs:
<...
- 57
-1
votes
1
answer
98
views
top-down lexer in rust
the other day, i decided to build a top-down lexer in rust, just for fun.
this is what i have so far:
...
- 27
3
votes
1
answer
207
views
Generating indent and dedent tokens for python-style indenting code
I found lexy, a C++ template-based parser combinator, that helps a lot to build a language parser.
The problem is that lexy can only deal with context free grammar, and python indentation is not ...
- 293
2
votes
1
answer
131
views
Simple Manual Lexer in Haskell
As a beginner exercise, I made small manual lexer that recognizes three types of inputs:
integers: /[-]?[0-9]+/
strings, inside double quotes, with backslash ...
- 121
1
vote
1
answer
77
views
Lexer for shell-like language [rust]
I am working on writing a rust implementation of a sh-like language.
Rather than posting 600 lines of code here, you can just go to the GitHub repository.
Is this a ...
- 192
1
vote
1
answer
154
views
C# Language Lexer
Here is a Lexer for a programming language I am working on. Any feedback would be appreciated. I only started learning C# a couple of days ago, so please excuse my newbie code :)
...
- 125
3
votes
1
answer
179
views
Simple tokenizer in C
I implemented a simple tokenizer. Would love to hear your feedback on code style, best practices:
...
- 269
6
votes
1
answer
204
views
Lexer for a Scheme-like language in rust
I was/am working on an interpreter for a scheme-like language. Just some time back I shifted my implementation from C++ to Rust, which I just started learning. I know there are parser libraries like ...
- 137
2
votes
3
answers
352
views
Optimize lexer/parser bottleneck in C++
I am writing a parser for a custom mesh format for fluid dynamics simulation library, the mesh file contains 3D points (vertices) for the simulation mesh, for example:
...
- 339
5
votes
1
answer
181
views
How could this indentation-based lexer be simplified?
I finally got this lexer working, full implementation here. How can the lex function be simplified? The goal is to take a tree of indentation-based text, and ...
- 351
1
vote
1
answer
180
views
A Token class for a lexical parser
In a larger project, I need a (rather simple) expression parser able to accept numerical values, operators and string identifiers. Fine, a lexical parser fed with some input which gives tokens one at ...
- 947
4
votes
1
answer
497
views
Token class implementation with std::variant
So I've been trying to implement a Token class in C++. At first I wanted to use a simple enum class to store the Kind of the ...
- 196
5
votes
2
answers
903
views
Tokenize arithmetic expression in C
This program tokenizes an arithmetic expression specified in infix notation. It is one of my personal projects and I would love to receive expert advice.
After compiling, you can run the program with ...
4
votes
1
answer
186
views
Idiomatic Go Lexer
I have written some Go code that produces the output I would like, but I am unsure if the code itself "smells" good. I would like to know if it contains any anti-patterns, exemplifies bad ...
- 186
12
votes
2
answers
293
views
A simple parser generator
I need to parse simple DSLs in a few projects. Since I don't know BNF or other grammars, I figured an alternative would be to use a simple parser generator.
I'm looking for improvements to the lexer/...
- 201
4
votes
1
answer
62
views
Hand Coded State Driven Lexical Analyzer in C With Unit Test Part B
This review is presented in 3 questions due to the amount of code:
Part A contains the Lexical Analyzer and the main portion of the unit test code.
Part B (this question) contains the lower level ...
- 23.5k
9
votes
2
answers
270
views
Hand Coded State Driven Lexical Analyzer in C With Unit Test Part A
This code review is presented in 3 questions due to the amount of code:
Part A (this question) contains the Lexical Analyzer and the main portion of the unit test code.
Part B contains the lower ...
- 23.5k
2
votes
1
answer
207
views
Tokenizer FSM in Rust - better way to do match statements?
I'm learning Rust and have decided to code up a toy YANG parser as an exercise. In case it's relevant, I mainly work in C and Python.
While working on the tokenizer/lexer, I tried to make it Rust-y, ...
- 233
3
votes
0
answers
296
views
Lexical Analyzer In C++
So this is my first time writing a Lexer, and I want to make sure I'm doing it right. The lexer is not complete for a programming language right now, because I think I can easily add more stuff later. ...
- 171
3
votes
1
answer
324
views
Tokenizer / lexer
I wanted to make a calculator in python so I first wrote a tokenizer. I have written one before but this time I tried to refine it a bit. Any thoughts on improvements, things I could of done better.
<...
- 33
1
vote
0
answers
47
views
implementing parser - improvment
I have been trying to develop a program which implements a parser, i.e., syntactic analysis.
Here is what I have done so far:
parser.c
...
2
votes
2
answers
83
views
Optimization of a custom interpreter
Ok, so I want to find out if my approach towards a custom interpreter is pragmatic and if not how to implement it in a better way.
this is purely for fun and practice and to find out more about how to ...
- 21
5
votes
1
answer
810
views
Lexer written in Rust
So I ported a lexer I wrote in C++ over to rust, as I'm starting to learn rust. Since I'm very new though, I don't know any Idioms and good practices in rust. So if anyone could point out some (...
- 205
3
votes
1
answer
262
views
A lexer in C++ for analysing regex-like text
I'm creating a parser, and I have just finished my lexer. I wanted to ask if there is anything I should change, add, or reconsider in my code! (I don't think the grammars matter much, since it is only ...
4
votes
1
answer
323
views
Is this a good way to write a lexer?
I don't know whether if this is the right place to ask this.
I am semi-beginner in C. I always wanted to build my own Programming Language. Here I have built a lexical analyser completely myself. ...
- 143
5
votes
0
answers
134
views
3
votes
1
answer
159
views
Lex Scheme in Rust
I'm writing a Scheme interpreter in Rust, and the first step is the parser. I've finished the lexer, and would like to see what people think of it before I go any further.
...
- 281
2
votes
1
answer
1k
views
Quadratic "Complete the Square" Solver in Python 3
I quickly wrote this program to transform quadratic expressions in general form into vertex form.
\$ax^2 + bx + c \;=\; a(x-h)^2 + k,\quad\text{where}\quad h = -\frac{b}{2a} \quad\text{and}\quad k = ...
- 43
5
votes
1
answer
371
views
Simplest lexer for c in go
I started learning compilers development and as pet project decided to implement c compiler.
The first step is implementing simplest lexer.
Also, I decided to try new programming language and chose go....
- 53
5
votes
1
answer
147
views
Work-In-Progress Lexer
So, purely out of desire to learn, I decided to take on a project of making my own very basic programming language from scratch. After which, I made a lexer.
I've tested the lexer (but only a bit) ...
- 196
5
votes
2
answers
743
views
Tokenizer Class for handling .csv files in c++17
I am a hobbyist computer programmer trying to learn modern C++ (in this case C++17). I thought it might be an interesting challenge to write a tokenizer class for ...
- 577
3
votes
1
answer
166
views
Function to parse VB.NET Like pattern
VB.NET has a Like operator with a paradigm similar to the standard SQL LIKE expression (SQL Server, SQLite, MySQL), but with a ...
- 418
7
votes
5
answers
348
views
Simple tokenizer v2 - reading all matching chars at once
I have rewritten my tokenizer according to most of the suggestions from the previous question here.
API
It now reads all chars as long as they match the pattern. I use three types of attributes to ...
- 44.1k
3
votes
2
answers
472
views
Simple tokenizer v1 - reading char by char
I was feeling like writing a new Uri parser. The previous one was too limited and wasn't able to parse the authority part. This one is also based on the pretty ...
- 44.1k
6
votes
2
answers
217
views
Compact command line argument parser : Revisited
This question is a follow up to my previous one which can be found here.
The user dfhwze suggested me to look into compiler construction and recommended me to write a lexer and a parser that would ...
- 901
6
votes
1
answer
3k
views
Lexer using C++ and Regex
I'm trying to create a very basic compiler, and I'm working on the lexer.
I decided to store keywords and operators in an unordered_set, and use regex to match identifiers and literals (no comments ...
- 1,751
1
vote
2
answers
508
views
Java lexical analyser
I created a lexical analyser in Java recently, but I don't think the performance is very good.
The code works, but when I debugged the program, it take around ~100 milliseconds for only two tokens...
...
- 31
7
votes
4
answers
2k
views
Lexer for C++ in python
I wrote a simple C++ lexer in python. Although it already works but I kinda feel I didn't follow the pythonic way. Any suggestion for improvements?
Sample C++ Code
Sample output
Here's my code:
...
- 501
3
votes
0
answers
240
views
Writing a tokenizer in SWI-Prolog
To teach myself how to do meta-progaming using Prolog, I've been playing around with translating ABNF syntax into DCG syntax. I'm still at the lexing and tokenising stage, but hope to advance to the "...
- 203
3
votes
1
answer
259
views
Parsing a C++ function declaration with Perl 6 grammar
I wrote a Perl 6 grammar to parse a C++ function. My final goal would be to parse an entire header. The aim is not to correct C++ syntax errors, but to parse valid C++.
Do you have some advice or ...
- 311
2
votes
0
answers
157
views
Lexer function to handle the '/' character
I'm writing a toy compiler, which is in a very early stage of development. The lexer contains the following block of code:
...
- 141
4
votes
2
answers
266
views
Lexer for Apache logs in Python
I recently learned Python using the book by Mark Summerfeld and I am now doing a few katas of mine when I learn a new language. This helps to get the “how the language works” and I wrote a simple ...
4
votes
1
answer
107
views
Lexer for a small language in D
I am writing an interpreter for a simple mathematical programming language.
Below is a simple program written in it that prints 41.22:
...
- 143
12
votes
4
answers
2k
views
Simple C99 Lexer
I'm building a compiler for fun, and this is my first pass at the lexer for it. It should handle all tokens for C99, with the exception of the preprocessor tokens. It's very minimal, only grabbing the ...
- 1,132
5
votes
1
answer
925
views
Basic Equation Tokenizer
Recently I've been doing some experimenting with RPN and the shunting-yard algorithm, in order to test these systems more appropriately I planned on writing a tokenizer and then using these tokens to ...
user122352