4
\$\begingroup\$

This may seem a little small, but it's actually a self-contained crate in my project.

This crate contains definitions for the source tokens in the language I'm working on, Nafi. As of current, it's definitely quite minimal, because I learned from an earlier attempt that I want to move in small chunks up the full tree to working language, in order to keep myself motivated with tangible progress.

This tokens crate may grow slightly before my "proper" submission, but the structure of this library is here so I thought it fit to submit for a review. I actually have a working lexer for most (but not yet all) of these tokens, but that's not quite ready yet.

You can view the current auto-generated documentation or see the full project on GitHub.

Cargo.toml

[package]
name = "nafi-tokens"
version = "0.0.1"
publish = false

[dependencies.num]
version = "0.1"
default-features = false
features = [ "bigint" ]

lib.rs

//! Tokens of Nafi source
#![forbid(bad_style, missing_debug_implementations, unconditional_recursion, future_incompatible)]
#![deny(missing_docs, unsafe_code, unused)]
#![feature(conservative_impl_trait)]

extern crate num;

mod symbol;
mod literal;

pub use literal::{BigUint, Literal, StringFragments};
pub use symbol::Symbol;

/// A token in the source code. Simply chunking the source into units to then parse.
#[derive(Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Token {
    #[doc(hidden)] _Unknown(usize),
    Whitespace(usize),
    Symbol(usize, Symbol),
    Literal(usize, Literal),
    Keyword(usize, Keyword),
    Identifier(usize, String),
}

impl Token {
    /// The start location of this token.
    pub fn position(&self) -> usize {
        match *self {
            Token::_Unknown(pos) |
            Token::Whitespace(pos) |
            Token::Symbol(pos, _) |
            Token::Literal(pos, _) |
            Token::Keyword(pos, _) |
            Token::Identifier(pos, _) => pos,
        }
    }
}

/// A reserved identifier-like in the source code.
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Keyword {
    Let,
    Mutable,
    If,
    Else,
}

literal.rs

pub use num::bigint::BigUint;
use std::borrow::Cow;

/// A literal in the source code, e.g. a string or number.
#[derive(Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Literal {
    Integer(BigUint),
    String(StringFragments),
}

impl From<BigUint> for Literal {
    fn from(uint: BigUint) -> Self { Literal::Integer(uint) }
}

impl From<String> for Literal {
    fn from(s: String) -> Self { Literal::String(s.into()) }
}

impl<'a> From<&'a str> for Literal {
    fn from(s: &'a str) -> Self { Literal::String(s.into()) }
}

impl From<StringFragments> for Literal {
    fn from(fragments: StringFragments) -> Self { Literal::String(fragments) }
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum StringFragment {
    Str(String),
    InvalidEscape(String),
}

impl<S: Into<String>> From<S> for StringFragment {
    fn from(s: S) -> Self { StringFragment::Str(s.into()) }
}

/// A String that also remembers invalid escapes inside it.
#[derive(Clone, Debug, Default, Eq, PartialEq)]
pub struct StringFragments {
    fragments: Vec<StringFragment>,
}

impl<S: Into<String>> From<S> for StringFragments {
    fn from(s: S) -> Self { StringFragments { fragments: vec![s.into().into()] } }
}

impl StringFragments {
    /// Create a new, empty string.
    pub fn new() -> StringFragments { Default::default() }

    /// Push a character onto the end of this string.
    pub fn push(&mut self, char: char) {
        let len = self.fragments.len();
        if len == 0 {
            self.fragments.push(StringFragment::Str(char.to_string()));
        } else {
            if let StringFragment::Str(_) = self.fragments[len - 1] {
                if let StringFragment::Str(ref mut string) = self.fragments[len - 1] {
                    string.push(char);
                }
            } else {
                self.fragments.push(StringFragment::Str(char.to_string()));
            }
        }
    }

    /// Push a string onto the end of this string.
    pub fn push_str<'a, S: Into<Cow<'a, str>>>(&mut self, str: S) {
        let len = self.fragments.len();
        if len == 0 {
            self.fragments
                .push(StringFragment::Str(str.into().into_owned()));
        } else {
            if let StringFragment::Str(_) = self.fragments[len - 1] {
                if let StringFragment::Str(ref mut string) = self.fragments[len - 1] {
                    string.push_str(str.into().as_ref());
                }
            } else {
                self.fragments
                    .push(StringFragment::Str(str.into().into_owned()))
            }
        }
    }

    /// Push an invalid escape onto the end of this string.
    pub fn push_invalid_escape<S: Into<String>>(&mut self, escape: S) {
        self.fragments
            .push(StringFragment::InvalidEscape(escape.into()))
    }

    /// Try to turn this string into a normal string.
    ///
    /// Fails if any invalid escapes are present.
    pub fn try_into_string(self) -> Result<String, InvalidEscapes> {
        if self.fragments.len() == 1 {
            if let StringFragment::Str(_) = self.fragments[0] {
                if let Some(StringFragment::Str(string)) = self.fragments.into_iter().next() {
                    return Ok(string);
                } else {
                    unreachable!()
                }
            }
        }
        return Err(InvalidEscapes(
            self.fragments
                .into_iter()
                .filter_map(|fragment| match fragment {
                    StringFragment::InvalidEscape(escape) => Some(escape),
                    StringFragment::Str(_) => None,
                })
                .collect(),
        ));
    }
}

/// The invalid escapes in a string literal.
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct InvalidEscapes(Vec<String>);

impl InvalidEscapes {
    /// Create an iterator over the invalid escapes.
    ///
    /// You get what was attached after the `\`.
    /// E.g. `\w` gives `w` and `\u{INVALID}` gives `u{INVALID}`
    pub fn iter<'a>(&'a self) -> impl Iterator<Item = &'a str> { self.0.iter().map(String::as_str) }
}

symbol.rs

/// A symbol in the source code, e.g. `+-={}[]<>` (or others)
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Symbol {
    ExclamationMark,
    // QuotationMark, // will never happen -- superseded by string literal
    NumberSign,
    DollarSign,
    PercentSign,
    Ampersand,
    // Apostrophe, // will never happen -- superseded by quote literal
    LeftParenthesis,
    RightParenthesis,
    Asterisk,
    PlusSign,
    Comma,
    HyphenMinus,
    FullStop,
    Solidus,
    Colon,
    Semicolon,
    LessThanSign,
    EqualsSign,
    GreaterThanSign,
    QuestionMark,
    CommercialAt,
    LeftSquareBracket,
    ReverseSolidus,
    RightSquareBracket,
    CircumflexAccent,
    LowLine,
    GraveAccent,
    LeftCurlyBracket,
    VerticalLine,
    RightCurlyBracket,
    Tilde,
    Other(char),
}

impl Symbol {
    /// The character in the source
    pub fn as_char(&self) -> char {
        use Symbol::*;
        match *self {
            ExclamationMark => '!',
            NumberSign => '#',
            DollarSign => '$',
            PercentSign => '%',
            Ampersand => '&',
            LeftParenthesis => '(',
            RightParenthesis => ')',
            Asterisk => '*',
            PlusSign => '+',
            Comma => ',',
            HyphenMinus => '-',
            FullStop => '.',
            Solidus => '/',
            Colon => ':',
            Semicolon => ';',
            LessThanSign => '<',
            EqualsSign => '=',
            GreaterThanSign => '>',
            QuestionMark => '?',
            CommercialAt => '@',
            LeftSquareBracket => '[',
            ReverseSolidus => '\\',
            RightSquareBracket => ']',
            CircumflexAccent => '^',
            LowLine => '_',
            GraveAccent => '`',
            LeftCurlyBracket => '{',
            VerticalLine => '|',
            RightCurlyBracket => '}',
            Tilde => '~',
            Other(char) => char,
        }
    }
}

As the rest of the code is rather simple, I'm most interested in literal.rs and the code supporting the string literal -- StringFragment(s).

NOTE: "Code is fine, move on" is a viable answer. But there's always something else you can say as well.

\$\endgroup\$
  • \$\begingroup\$ meta note: I tagged this community-challenge because it is part of my project that will be submitted to the C.O.D.E.R.E.V.I.E.W. challenge. If this question is deemed to not fit the tag, because it doesn't actually complete the challenge in any capacity yet, I'm fine with its removal. \$\endgroup\$ – CAD97 Sep 5 '17 at 3:54
  • \$\begingroup\$ Looks good to me. Your comments are helpful. \$\endgroup\$ – J_H Sep 5 '17 at 5:14
  • \$\begingroup\$ @CAD97 I'm not sure if you intended on entering the CODE REVIEW challenge, but I've extended the entry deadline to the end of this week, since your question was posted before the original deadline. Please answer the linked Meta question if you would like to enter. (Possible +500 bounty.) \$\endgroup\$ – Der Kommissar Dec 4 '17 at 21:24
  • \$\begingroup\$ @202_accepted I could put this in but it's not a functional language yet. The plan was to get a working language to submit but I don't have one yet. \$\endgroup\$ – CAD97 Dec 4 '17 at 21:26
  • \$\begingroup\$ @CAD97 You generate a token stream, right? (That's at least one point.) Do you generate an AST (not familiar with RUST)? That's another. This is the basis of a direct interpreter, so that's a third. At this moment you would be in the lead. ;) \$\endgroup\$ – Der Kommissar Dec 4 '17 at 21:27
1
\$\begingroup\$

I only see minor things; most of the code appears to be straight-forward shuffling around of data.

  1. I would not use char (or any other type) as a variable name. The risk for confusion is too high in my opinion.

  2. Your enum variant is called Str, but it holds a String. Since enough people are confused with &str vs String, it's worth it to be consistent.

  3. I disagree with the current rustfmt formatting for one expression blocks, so I'd advocate for placing a separate block inside the match arms:

    .filter_map(|fragment| {
        match fragment {
            StringFragment::InvalidEscape(escape) => Some(escape),
            StringFragment::Str(_) => None,
        }
    })
    
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.