Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [lexical-analysis]

Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.

3
votes
0answers
61 views

Writing a tokenizer in SWI-Prolog

To teach myself how to do meta-progaming using Prolog, I've been playing around with translating ABNF syntax into DCG syntax. I'm still at the lexing and tokenising stage, but hope to advance to the "...
1
vote
0answers
71 views

Lexer for my “Obganism” language

After having rewritten this lexer for the Nth time, I finally feel confident enough to show it publicly. I'm looking for advices to improve my code. The lexer you are going to see is the first step ...
4
votes
1answer
68 views

Lexer for a small language in D

I am writing an interpreter for a simple mathematical programming language. Below is a simple program written in it that prints 41.22: ...
12
votes
4answers
580 views

Simple C99 Lexer

I'm building a compiler for fun, and this is my first pass at the lexer for it. It should handle all tokens for C99, with the exception of the preprocessor tokens. It's very minimal, only grabbing the ...
5
votes
1answer
239 views

Basic Equation Tokenizer

Recently I've been doing some experimenting with RPN and the shunting-yard algorithm, in order to test these systems more appropriately I planned on writing a tokenizer and then using these tokens to ...
2
votes
1answer
1k views

how to classify tweets in python based on handful of domains [closed]

I am currently working on a project that classifies tweets based on certain categories of information they belong to (7 given categories for his project). For example, a tweet with the keywords "I ...
6
votes
1answer
747 views

SQL lexer in JavaScript

Context Some time ago I wrote a very simple SQL interpreter for Javascript for fun one weekend and have since started using it in production in a few projects for work, and as requirements change for ...
3
votes
1answer
88 views

Mustache lexer class

I am trying to create a mustache interpreter. I have started implementing the lexer parts. Here are the classes: Lexer.h ...
19
votes
3answers
881 views

Complex tokenizer in C#

As we have our Community Oriented Development Extravaganza, Requiring Extreme Vigor Inciting Extraordinary Winners 2017 Challenge. Finally I had some time to write a tokenizer for my own language. It ...
5
votes
3answers
723 views

Simple language tokenizer in C#

I had some spare time so I have decided to write a simple tokenizer in C# for my own programming language. The tokenizer is not finished, but it is working and I would like to hear your opinion on ...
7
votes
2answers
288 views

Tokenize and detect syntactic errors in simple math expressions

I thought that I didn't learn anything new for over a month or two, so I decided to start off with compilers. The code is really crappy, but at least it works. Problem definition: Given a bunch of ...
15
votes
2answers
530 views

Counting density of Python code

I created a tool to examine Python 3 source code. It measures the density of program. More info in docstring (to not reinvent the wheel). What could be done to increase its usability? Any edge cases ...
5
votes
1answer
153 views

Compiler Tokenizer implementation in C# - follow-up

I have posted this question 2 days ago and got amazing feedback on it. Hands back to the keyboard, I believe I took the suggestions into account and improved the classes drastically. In this question,...
12
votes
4answers
2k views

Compiler Tokenizer implementation in C#

I'm writing a compiler for a couple of months now, this is the tokenization part of the lexer. I would like a code review to improve my coding style and learn new techniques to pretty up my code and ...
7
votes
1answer
465 views

A simple regex based lexer

Recently I've been interested in language design and text processing. And a common program needed in both of those fields is a lexer. And since I found myself writing a new, slightly different lexer ...
3
votes
1answer
43 views

Transform poor CSS into a more beatiful form in Dart

I have this script that opens a file containing (possibly) bad CSS, lexes the content of it and creates a new file with the same code in a more pleasant way to look at. It formats it and sorts the ...
1
vote
0answers
64 views

Extending a simple Tree parser and lexer

Yesterday, I decided to learn about scala's Parser-Combinator library. After a while, I was able to write a simple Tree parser and lexer which understands trees written like this: ...
4
votes
2answers
94 views

Turn characters into sensible things (tokenizing/lexing)

I'm writing a programming language. It's something I've been working on and off on for the last year or so, but recently, I've decided to buckle down and do it. As part of the language, I wrote a ...
5
votes
1answer
137 views

Schemey - lexer

1Note: This is the first post in - hopefully - a series of posts I plan to make. Project background Schemey is a project I've been working on for the past 3-4 weeks. It is an implementation of a ...
2
votes
0answers
56 views

Parsing key:value metadata in a file

I have written some Flex and Bison code for parsing a file. The following is a simplified example of the input file as well as the code used for parsing. There are a few things I am not satisfied ...
13
votes
1answer
1k views

Tokenizer for my programming language

Here's my attempt at porting the Lua codebase for my programming language to C++(11). This is just the first step, the tokenizer, and I wanted to remove all the bad performance / practices / code ...
8
votes
2answers
5k views

Yet another lexical analyser

The following program is a lexical analyser for a simple and small grammar. Please read my code and answer me these questions: Is my code easy to understand? Is my code well organized? Are there ...
10
votes
1answer
146 views

Lexer for a language I'm working on

I've recently started working on making my own programming language and I just wrapped up its lexer. I'm too young to take any official training in C, compiler construction, or computer science so I'm ...
6
votes
2answers
126 views

Lexer for an experimental tiny language in Go

I am implementing a lexer for an experimental language by the name "Phoenix", the language supports four primary arithmetic expression for integers only (add, subtract, multiply, divide), variable ...
4
votes
1answer
892 views

Simple lexical analysis - basic calculator

I recently found an interesting series that describes the specifics of how interpreters and compilers work, explaining each step (with code) and encouraging the reader to do exercises. My code is ...