Lexer for expression evaluator

Question

I started learning my first functional programming language (Haskell) yesterday and I have been working on a lexer for an expression evaluator. The lexer is now basically feature complete, but I'm not sure what I can do to improve the code.

import Data.Char
import Data.List

data TokenType = Identifier |
                 RealNumberLiteral |
                 PlusSign |
                 MinusSign |
                 Asterisk |
                 ForwardSlash |
                 Caret |
                 LeftParenthesis |
                 RightParenthesis
    deriving (Show)

data Token = Token TokenType String
    deriving (Show)

read_token :: String -> Token
read_token [] = error "Unexpectedly reached the end of the source code while reading a token."
read_token source_code@(next_character:_)
    | isSpace next_character = read_token (dropWhile isSpace source_code)
    | isAlpha next_character = Token Identifier (takeWhile isAlpha source_code)
    | isDigit next_character = let token_lexeme = (takeWhile (\x -> isDigit x || x == '.') source_code)
                               in let period_count = length (filter (=='.') token_lexeme)
                                  in Token RealNumberLiteral (if period_count <= 1
                                                                  then token_lexeme
                                                                  else error "There can only be one period in a real number literal.")
    | next_character == '+' = Token PlusSign "+"
    | next_character == '-' = Token MinusSign "-"
    | next_character == '*' = Token Asterisk "*"
    | next_character == '/' = Token ForwardSlash "/"
    | next_character == '^' = Token Caret "^"
    | next_character == '(' = Token LeftParenthesis "("
    | next_character == ')' = Token RightParenthesis ")"
    | otherwise = error ("Encountered an unexpected character (" ++ [next_character] ++ ") while reading a token.")

append_read_tokens :: [Token] -> String -> [Token]
append_read_tokens tokens source_code
    | null source_code = tokens
    | isSpace (head source_code) = append_read_tokens tokens (dropWhile isSpace source_code)
    | otherwise = let next_token@(Token next_token_type next_token_lexeme) = read_token source_code
                  in append_read_tokens (tokens ++ [next_token]) (drop (length next_token_lexeme) source_code)

tokenize :: String -> [Token]
tokenize [] = []
tokenize source_code = append_read_tokens [] source_code

Answer 1

• No need of TokenType: data Token = Identifier String | RealNumberLiteral String | PlusSign ...

• No need of nested let statements, see http://learnyouahaskell.com/syntax-in-functions#let-it-be

• camelCase naming is preferred everywhere.

• generally it is more efficient to aggregate lists using cons (: operator), and reverse at the end, if needed. It is about tokens ++ [next_token] fragment.

• read_token could return a tuple of token and the rest string, no need to drop (length next_token_lexeme) source_code after that.

Note, I didn't inspect code logic at all.