Questions tagged [lexical-analysis]

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

Filter by
Sorted by
Tagged with
3 votes
1 answer
50 views

ANTLR4 grammar for Conventional Commits spec

I would like to create a grammar for the Conventional Commits spec and I would love to hear any feedback for what I wrote. The spec has some ambiguities, I think, hence my usage of "island ...
  • 133
6 votes
2 answers
123 views

Programming Language Source Lexer

I have a lexer (program that turns source code into tokens) written in Go that I am re-writing in Rust. The lexer should take in a sequence of tokens, generally different special characters, integers, ...
1 vote
1 answer
77 views

Rust state-machine lexer

I tried implementing a lexer in rust that peeks ahead at the next character and makes a decision based on that. However, i am told that this is bad practice, and instead i should be using finite-state-...
0 votes
1 answer
69 views

supporting unicode in rust lexer

I am working on a lexer in rust. It has been through many different iterations, going from iterating over a Vec<char> to ...
4 votes
1 answer
116 views

lexer for programming languages in rust

I'm relatively new to rust, so you don't need to go easy on the criticism. right now, this lexes 1MB of text (well, trims a 1MB-long word) in 30ms. is there any way I can make this faster? cursor.rs: <...
-1 votes
1 answer
69 views

top-down lexer in rust

the other day, i decided to build a top-down lexer in rust, just for fun. this is what i have so far: ...
3 votes
1 answer
108 views

Generating indent and dedent tokens for python-style indenting code

I found lexy, a C++ template-based parser combinator, that helps a lot to build a language parser. The problem is that lexy can only deal with context free grammar, and python indentation is not ...
  • 293
2 votes
1 answer
78 views

Simple Manual Lexer in Haskell

As a beginner exercise, I made small manual lexer that recognizes three types of inputs: integers: /[-]?[0-9]+/ strings, inside double quotes, with backslash ...
1 vote
1 answer
64 views

Lexer for shell-like language [rust]

I am working on writing a rust implementation of a sh-like language. Rather than posting 600 lines of code here, you can just go to the GitHub repository. Is this a ...
1 vote
1 answer
112 views

C# Language Lexer

Here is a Lexer for a programming language I am working on. Any feedback would be appreciated. I only started learning C# a couple of days ago, so please excuse my newbie code :) ...
3 votes
1 answer
115 views

Simple tokenizer in C

I implemented a simple tokenizer. Would love to hear your feedback on code style, best practices: ...
  • 269
6 votes
1 answer
164 views

Lexer for a Scheme-like language in rust

I was/am working on an interpreter for a scheme-like language. Just some time back I shifted my implementation from C++ to Rust, which I just started learning. I know there are parser libraries like ...
2 votes
3 answers
238 views

Optimize lexer/parser bottleneck in C++

I am writing a parser for a custom mesh format for fluid dynamics simulation library, the mesh file contains 3D points (vertices) for the simulation mesh, for example: ...
  • 339
5 votes
1 answer
142 views

How could this indentation-based lexer be simplified?

I finally got this lexer working, full implementation here. How can the lex function be simplified? The goal is to take a tree of indentation-based text, and ...
  • 351
1 vote
1 answer
74 views

A Token class for a lexical parser

In a larger project, I need a (rather simple) expression parser able to accept numerical values, operators and string identifiers. Fine, a lexical parser fed with some input which gives tokens one at ...
4 votes
1 answer
369 views

Token class implementation with std::variant

So I've been trying to implement a Token class in C++. At first I wanted to use a simple enum class to store the Kind of the ...
  • 196
5 votes
2 answers
624 views

Tokenize arithmetic expression in C

This program tokenizes an arithmetic expression specified in infix notation. It is one of my personal projects and I would love to receive expert advice. After compiling, you can run the program with ...
4 votes
1 answer
134 views

Idiomatic Go Lexer

I have written some Go code that produces the output I would like, but I am unsure if the code itself "smells" good. I would like to know if it contains any anti-patterns, exemplifies bad ...
12 votes
2 answers
262 views

A simple parser generator

I need to parse simple DSLs in a few projects. Since I don't know BNF or other grammars, I figured an alternative would be to use a simple parser generator. I'm looking for improvements to the lexer/...
  • 201
4 votes
1 answer
60 views

Hand Coded State Driven Lexical Analyzer in C With Unit Test Part B

This review is presented in 3 questions due to the amount of code: Part A contains the Lexical Analyzer and the main portion of the unit test code. Part B (this question) contains the lower level ...
  • 21.9k
9 votes
2 answers
234 views

Hand Coded State Driven Lexical Analyzer in C With Unit Test Part A

This code review is presented in 3 questions due to the amount of code: Part A (this question) contains the Lexical Analyzer and the main portion of the unit test code. Part B contains the lower ...
  • 21.9k
2 votes
1 answer
180 views

Tokenizer FSM in Rust - better way to do match statements?

I'm learning Rust and have decided to code up a toy YANG parser as an exercise. In case it's relevant, I mainly work in C and Python. While working on the tokenizer/lexer, I tried to make it Rust-y, ...
  • 223
3 votes
0 answers
255 views

Lexical Analyzer In C++

So this is my first time writing a Lexer, and I want to make sure I'm doing it right. The lexer is not complete for a programming language right now, because I think I can easily add more stuff later. ...
3 votes
1 answer
233 views

Tokenizer / lexer

I wanted to make a calculator in python so I first wrote a tokenizer. I have written one before but this time I tried to refine it a bit. Any thoughts on improvements, things I could of done better. <...
1 vote
0 answers
45 views

implementing parser - improvment

I have been trying to develop a program which implements a parser, i.e., syntactic analysis. Here is what I have done so far: parser.c ...
2 votes
2 answers
80 views

Optimization of a custom interpreter

Ok, so I want to find out if my approach towards a custom interpreter is pragmatic and if not how to implement it in a better way. this is purely for fun and practice and to find out more about how to ...
5 votes
1 answer
729 views

Lexer written in Rust

So I ported a lexer I wrote in C++ over to rust, as I'm starting to learn rust. Since I'm very new though, I don't know any Idioms and good practices in rust. So if anyone could point out some (...
3 votes
1 answer
219 views

A lexer in C++ for analysing regex-like text

I'm creating a parser, and I have just finished my lexer. I wanted to ask if there is anything I should change, add, or reconsider in my code! (I don't think the grammars matter much, since it is only ...
4 votes
1 answer
210 views

Is this a good way to write a lexer?

I don't know whether if this is the right place to ask this. I am semi-beginner in C. I always wanted to build my own Programming Language. Here I have built a lexical analyser completely myself. ...
  • 143
5 votes
0 answers
124 views

Scanning a string into a list of tokens

Here is my code: ...
3 votes
1 answer
148 views

Lex Scheme in Rust

I'm writing a Scheme interpreter in Rust, and the first step is the parser. I've finished the lexer, and would like to see what people think of it before I go any further. ...
2 votes
1 answer
824 views

Quadratic "Complete the Square" Solver in Python 3

I quickly wrote this program to transform quadratic expressions in general form into vertex form. \$ax^2 + bx + c \;=\; a(x-h)^2 + k,\quad\text{where}\quad h = -\frac{b}{2a} \quad\text{and}\quad k = ...
  • 43
5 votes
1 answer
275 views

Simplest lexer for c in go

I started learning compilers development and as pet project decided to implement c compiler. The first step is implementing simplest lexer. Also, I decided to try new programming language and chose go....
5 votes
1 answer
109 views

Work-In-Progress Lexer

So, purely out of desire to learn, I decided to take on a project of making my own very basic programming language from scratch. After which, I made a lexer. I've tested the lexer (but only a bit) ...
  • 196
5 votes
2 answers
604 views

Tokenizer Class for handling .csv files in c++17

I am a hobbyist computer programmer trying to learn modern C++ (in this case C++17). I thought it might be an interesting challenge to write a tokenizer class for ...
  • 547
3 votes
1 answer
144 views

Function to parse VB.NET Like pattern

VB.NET has a Like operator with a paradigm similar to the standard SQL LIKE expression (SQL Server, SQLite, MySQL), but with a ...
  • 418
7 votes
5 answers
313 views

Simple tokenizer v2 - reading all matching chars at once

I have rewritten my tokenizer according to most of the suggestions from the previous question here. API It now reads all chars as long as they match the pattern. I use three types of attributes to ...
  • 44.1k
3 votes
2 answers
443 views

Simple tokenizer v1 - reading char by char

I was feeling like writing a new Uri parser. The previous one was too limited and wasn't able to parse the authority part. This one is also based on the pretty ...
  • 44.1k
6 votes
2 answers
183 views

Compact command line argument parser : Revisited

This question is a follow up to my previous one which can be found here. The user dfhwze suggested me to look into compiler construction and recommended me to write a lexer and a parser that would ...
  • 795
6 votes
1 answer
3k views

Lexer using C++ and Regex

I'm trying to create a very basic compiler, and I'm working on the lexer. I decided to store keywords and operators in an unordered_set, and use regex to match identifiers and literals (no comments ...
  • 1,689
1 vote
2 answers
467 views

Java lexical analyser

I created a lexical analyser in Java recently, but I don't think the performance is very good. The code works, but when I debugged the program, it take around ~100 milliseconds for only two tokens... ...
  • 31
7 votes
4 answers
2k views

Lexer for C++ in python

I wrote a simple C++ lexer in python. Although it already works but I kinda feel I didn't follow the pythonic way. Any suggestion for improvements? Sample C++ Code Sample output Here's my code: ...
  • 461
3 votes
0 answers
217 views

Writing a tokenizer in SWI-Prolog

To teach myself how to do meta-progaming using Prolog, I've been playing around with translating ABNF syntax into DCG syntax. I'm still at the lexing and tokenising stage, but hope to advance to the "...
  • 203
3 votes
1 answer
229 views

Parsing a C++ function declaration with Perl 6 grammar

I wrote a Perl 6 grammar to parse a C++ function. My final goal would be to parse an entire header. The aim is not to correct C++ syntax errors, but to parse valid C++. Do you have some advice or ...
  • 311
1 vote
0 answers
97 views

Lexer for my "Obganism" language

After having rewritten this lexer for the Nth time, I finally feel confident enough to show it publicly. I'm looking for advices to improve my code. The lexer you are going to see is the first step ...
user avatar
2 votes
0 answers
148 views

Lexer function to handle the '/' character

I'm writing a toy compiler, which is in a very early stage of development. The lexer contains the following block of code: ...
  • 141
4 votes
2 answers
246 views

Lexer for Apache logs in Python

I recently learned Python using the book by Mark Summerfeld and I am now doing a few katas of mine when I learn a new language. This helps to get the “how the language works” and I wrote a simple ...
4 votes
1 answer
101 views

Lexer for a small language in D

I am writing an interpreter for a simple mathematical programming language. Below is a simple program written in it that prints 41.22: ...
  • 143
12 votes
4 answers
2k views

Simple C99 Lexer

I'm building a compiler for fun, and this is my first pass at the lexer for it. It should handle all tokens for C99, with the exception of the preprocessor tokens. It's very minimal, only grabbing the ...
  • 1,132
5 votes
1 answer
763 views

Basic Equation Tokenizer

Recently I've been doing some experimenting with RPN and the shunting-yard algorithm, in order to test these systems more appropriately I planned on writing a tokenizer and then using these tokens to ...
user avatar