Questions tagged [lexical-analysis]

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

Filter by
Sorted by
Tagged with
2
votes
3answers
107 views

Optimize lexer/parser bottleneck in C++

I am writing a parser for a custom mesh format for fluid dynamics simulation library, the mesh file contains 3D points (vertices) for the simulation mesh, for example: ...
1
vote
1answer
49 views

A Token class for a lexical parser

In a larger project, I need a (rather simple) expression parser able to accept numerical values, operators and string identifiers. Fine, a lexical parser fed with some input which gives tokens one at ...
4
votes
1answer
69 views

Idiomatic Go Lexer

I have written some Go code that produces the output I would like, but I am unsure if the code itself "smells" good. I would like to know if it contains any anti-patterns, exemplifies bad ...
11
votes
2answers
168 views

A simple parser generator

I need to parse simple DSLs in a few projects. Since I don't know BNF or other grammars, I figured an alternative would be to use a simple parser generator. I'm looking for improvements to the lexer/...
4
votes
1answer
47 views

Hand Coded State Driven Lexical Analyzer in C With Unit Test Part B

This review is presented in 3 questions due to the amount of code: Part A contains the Lexical Analyzer and the main portion of the unit test code. Part B (this question) contains the lower level ...
8
votes
2answers
172 views

Hand Coded State Driven Lexical Analyzer in C With Unit Test Part A

This code review is presented in 3 questions due to the amount of code: Part A (this question) contains the Lexical Analyzer and the main portion of the unit test code. Part B contains the lower ...
3
votes
1answer
149 views

A lexer in C++ for analysing regex-like text

I'm creating a parser, and I have just finished my lexer. I wanted to ask if there is anything I should change, add, or reconsider in my code! (I don't think the grammars matter much, since it is only ...
5
votes
1answer
80 views

Work-In-Progress Lexer

So, purely out of desire to learn, I decided to take on a project of making my own very basic programming language from scratch. After which, I made a lexer. I've tested the lexer (but only a bit) ...
3
votes
1answer
128 views

Function to parse VB.NET Like pattern

VB.NET has a Like operator with a paradigm similar to the standard SQL LIKE expression (SQL Server, SQLite, MySQL), but with a ...
1
vote
2answers
335 views

Java lexical analyser

I created a lexical analyser in Java recently, but I don't think the performance is very good. The code works, but when I debugged the program, it take around ~100 milliseconds for only two tokens... ...
3
votes
0answers
151 views

Writing a tokenizer in SWI-Prolog

To teach myself how to do meta-progaming using Prolog, I've been playing around with translating ABNF syntax into DCG syntax. I'm still at the lexing and tokenising stage, but hope to advance to the "...
1
vote
0answers
90 views

Lexer for my "Obganism" language

After having rewritten this lexer for the Nth time, I finally feel confident enough to show it publicly. I'm looking for advices to improve my code. The lexer you are going to see is the first step ...
4
votes
1answer
90 views

Lexer for a small language in D

I am writing an interpreter for a simple mathematical programming language. Below is a simple program written in it that prints 41.22: ...
12
votes
4answers
1k views

Simple C99 Lexer

I'm building a compiler for fun, and this is my first pass at the lexer for it. It should handle all tokens for C99, with the exception of the preprocessor tokens. It's very minimal, only grabbing the ...
5
votes
1answer
581 views

Basic Equation Tokenizer

Recently I've been doing some experimenting with RPN and the shunting-yard algorithm, in order to test these systems more appropriately I planned on writing a tokenizer and then using these tokens to ...
2
votes
1answer
1k views

how to classify tweets in python based on handful of domains [closed]

I am currently working on a project that classifies tweets based on certain categories of information they belong to (7 given categories for his project). For example, a tweet with the keywords "I ...
7
votes
1answer
1k views

SQL lexer in JavaScript

Context Some time ago I wrote a very simple SQL interpreter for Javascript for fun one weekend and have since started using it in production in a few projects for work, and as requirements change for ...
3
votes
1answer
106 views

Mustache lexer class

I am trying to create a mustache interpreter. I have started implementing the lexer parts. Here are the classes: Lexer.h ...
22
votes
4answers
1k views

Complex tokenizer in C#

As we have our Community Oriented Development Extravaganza, Requiring Extreme Vigor Inciting Extraordinary Winners 2017 Challenge. Finally I had some time to write a tokenizer for my own language. It ...
6
votes
3answers
2k views

Simple language tokenizer in C#

I had some spare time so I have decided to write a simple tokenizer in C# for my own programming language. The tokenizer is not finished, but it is working and I would like to hear your opinion on ...
7
votes
2answers
680 views

Tokenize and detect syntactic errors in simple math expressions

I thought that I didn't learn anything new for over a month or two, so I decided to start off with compilers. The code is really crappy, but at least it works. Problem definition: Given a bunch of ...
15
votes
2answers
782 views

Counting density of Python code

I created a tool to examine Python 3 source code. It measures the density of program. More info in docstring (to not reinvent the wheel). What could be done to increase its usability? Any edge cases ...
5
votes
1answer
209 views

Compiler Tokenizer implementation in C# - follow-up

I have posted this question 2 days ago and got amazing feedback on it. Hands back to the keyboard, I believe I took the suggestions into account and improved the classes drastically. In this question,...
14
votes
4answers
3k views

Compiler Tokenizer implementation in C#

I'm writing a compiler for a couple of months now, this is the tokenization part of the lexer. I would like a code review to improve my coding style and learn new techniques to pretty up my code and ...
8
votes
1answer
1k views

A simple regex based lexer

Recently I've been interested in language design and text processing. And a common program needed in both of those fields is a lexer. And since I found myself writing a new, slightly different lexer ...
2
votes
1answer
47 views

Transform poor CSS into a more beatiful form in Dart

I have this script that opens a file containing (possibly) bad CSS, lexes the content of it and creates a new file with the same code in a more pleasant way to look at. It formats it and sorts the ...
1
vote
0answers
87 views

Extending a simple Tree parser and lexer

Yesterday, I decided to learn about scala's Parser-Combinator library. After a while, I was able to write a simple Tree parser and lexer which understands trees written like this: ...
4
votes
2answers
110 views

Turn characters into sensible things (tokenizing/lexing)

I'm writing a programming language. It's something I've been working on and off on for the last year or so, but recently, I've decided to buckle down and do it. As part of the language, I wrote a ...
5
votes
1answer
172 views

Schemey - lexer

1Note: This is the first post in - hopefully - a series of posts I plan to make. Project background Schemey is a project I've been working on for the past 3-4 weeks. It is an implementation of a ...
2
votes
0answers
67 views

Parsing key:value metadata in a file

I have written some Flex and Bison code for parsing a file. The following is a simplified example of the input file as well as the code used for parsing. There are a few things I am not satisfied ...
13
votes
1answer
2k views

Tokenizer for my programming language

Here's my attempt at porting the Lua codebase for my programming language to C++(11). This is just the first step, the tokenizer, and I wanted to remove all the bad performance / practices / code ...
8
votes
2answers
6k views

Yet another lexical analyser

The following program is a lexical analyser for a simple and small grammar. Please read my code and answer me these questions: Is my code easy to understand? Is my code well organized? Are there ...
10
votes
1answer
158 views

Lexer for a language I'm working on

I've recently started working on making my own programming language and I just wrapped up its lexer. I'm too young to take any official training in C, compiler construction, or computer science so I'm ...
7
votes
2answers
155 views

Lexer for an experimental tiny language in Go

I am implementing a lexer for an experimental language by the name "Phoenix", the language supports four primary arithmetic expression for integers only (add, subtract, multiply, divide), variable ...
4
votes
1answer
2k views

Simple lexical analysis - basic calculator

I recently found an interesting series that describes the specifics of how interpreters and compilers work, explaining each step (with code) and encouraging the reader to do exercises. My code is ...