The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [lexical-analysis]

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

Filter by
Sorted by
Tagged with
20
votes
3answers
970 views

Complex tokenizer in C#

As we have our Community Oriented Development Extravaganza, Requiring Extreme Vigor Inciting Extraordinary Winners 2017 Challenge. Finally I had some time to write a tokenizer for my own language. It ...
15
votes
2answers
567 views

Counting density of Python code

I created a tool to examine Python 3 source code. It measures the density of program. More info in docstring (to not reinvent the wheel). What could be done to increase its usability? Any edge cases ...
14
votes
4answers
3k views

Compiler Tokenizer implementation in C#

I'm writing a compiler for a couple of months now, this is the tokenization part of the lexer. I would like a code review to improve my coding style and learn new techniques to pretty up my code and ...
13
votes
1answer
2k views

Tokenizer for my programming language

Here's my attempt at porting the Lua codebase for my programming language to C++(11). This is just the first step, the tokenizer, and I wanted to remove all the bad performance / practices / code ...
12
votes
4answers
749 views

Simple C99 Lexer

I'm building a compiler for fun, and this is my first pass at the lexer for it. It should handle all tokens for C99, with the exception of the preprocessor tokens. It's very minimal, only grabbing the ...
10
votes
1answer
147 views

Lexer for a language I'm working on

I've recently started working on making my own programming language and I just wrapped up its lexer. I'm too young to take any official training in C, compiler construction, or computer science so I'm ...
8
votes
2answers
5k views

Yet another lexical analyser

The following program is a lexical analyser for a simple and small grammar. Please read my code and answer me these questions: Is my code easy to understand? Is my code well organized? Are there ...
8
votes
1answer
547 views

A simple regex based lexer

Recently I've been interested in language design and text processing. And a common program needed in both of those fields is a lexer. And since I found myself writing a new, slightly different lexer ...
7
votes
2answers
326 views

Tokenize and detect syntactic errors in simple math expressions

I thought that I didn't learn anything new for over a month or two, so I decided to start off with compilers. The code is really crappy, but at least it works. Problem definition: Given a bunch of ...
6
votes
3answers
815 views

Simple language tokenizer in C#

I had some spare time so I have decided to write a simple tokenizer in C# for my own programming language. The tokenizer is not finished, but it is working and I would like to hear your opinion on ...
6
votes
1answer
840 views

SQL lexer in JavaScript

Context Some time ago I wrote a very simple SQL interpreter for Javascript for fun one weekend and have since started using it in production in a few projects for work, and as requirements change for ...
6
votes
2answers
130 views

Lexer for an experimental tiny language in Go

I am implementing a lexer for an experimental language by the name "Phoenix", the language supports four primary arithmetic expression for integers only (add, subtract, multiply, divide), variable ...
5
votes
1answer
306 views

Basic Equation Tokenizer

Recently I've been doing some experimenting with RPN and the shunting-yard algorithm, in order to test these systems more appropriately I planned on writing a tokenizer and then using these tokens to ...
5
votes
1answer
140 views

Schemey - lexer

1Note: This is the first post in - hopefully - a series of posts I plan to make. Project background Schemey is a project I've been working on for the past 3-4 weeks. It is an implementation of a ...
5
votes
1answer
173 views

Compiler Tokenizer implementation in C# - follow-up

I have posted this question 2 days ago and got amazing feedback on it. Hands back to the keyboard, I believe I took the suggestions into account and improved the classes drastically. In this question,...
5
votes
1answer
61 views

Work-In-Progress Lexer

So, purely out of desire to learn, I decided to take on a project of making my own very basic programming language from scratch. After which, I made a lexer. I've tested the lexer (but only a bit) ...
4
votes
1answer
76 views

Lexer for a small language in D

I am writing an interpreter for a simple mathematical programming language. Below is a simple program written in it that prints 41.22: ...
4
votes
2answers
94 views

Turn characters into sensible things (tokenizing/lexing)

I'm writing a programming language. It's something I've been working on and off on for the last year or so, but recently, I've decided to buckle down and do it. As part of the language, I wrote a ...
4
votes
1answer
931 views

Simple lexical analysis - basic calculator

I recently found an interesting series that describes the specifics of how interpreters and compilers work, explaining each step (with code) and encouraging the reader to do exercises. My code is ...
3
votes
1answer
113 views

Function to parse VB.NET Like pattern

VB.NET has a Like operator with a paradigm similar to the standard SQL LIKE expression (SQL Server, SQLite, MySQL), but with a ...
3
votes
1answer
93 views

Mustache lexer class

I am trying to create a mustache interpreter. I have started implementing the lexer parts. Here are the classes: Lexer.h ...
3
votes
0answers
76 views

Writing a tokenizer in SWI-Prolog

To teach myself how to do meta-progaming using Prolog, I've been playing around with translating ABNF syntax into DCG syntax. I'm still at the lexing and tokenising stage, but hope to advance to the "...
2
votes
1answer
45 views

Transform poor CSS into a more beatiful form in Dart

I have this script that opens a file containing (possibly) bad CSS, lexes the content of it and creates a new file with the same code in a more pleasant way to look at. It formats it and sorts the ...
2
votes
1answer
1k views

how to classify tweets in python based on handful of domains [closed]

I am currently working on a project that classifies tweets based on certain categories of information they belong to (7 given categories for his project). For example, a tweet with the keywords "I ...
2
votes
0answers
59 views

Parsing key:value metadata in a file

I have written some Flex and Bison code for parsing a file. The following is a simplified example of the input file as well as the code used for parsing. There are a few things I am not satisfied ...
1
vote
2answers
76 views

Java lexical analyser

I created a lexical analyser in Java recently, but I don't think the performance is very good. The code works, but when I debugged the program, it take around ~100 milliseconds for only two tokens... ...
1
vote
0answers
79 views

Lexer for my “Obganism” language

After having rewritten this lexer for the Nth time, I finally feel confident enough to show it publicly. I'm looking for advices to improve my code. The lexer you are going to see is the first step ...
1
vote
0answers
69 views

Extending a simple Tree parser and lexer

Yesterday, I decided to learn about scala's Parser-Combinator library. After a while, I was able to write a simple Tree parser and lexer which understands trees written like this: ...