This may seem a little small, but it's actually a self-contained crate in my project.
This crate contains definitions for the source tokens in the language I'm working on, Nafi. As of current, it's definitely quite minimal, because I learned from an earlier attempt that I want to move in small chunks up the full tree to working language, in order to keep myself motivated with tangible progress.
This tokens crate may grow slightly before my "proper" community-challenge submission, but the structure of this library is here so I thought it fit to submit for a review. I actually have a working lexer for most (but not yet all) of these tokens, but that's not quite ready yet.
You can view the current auto-generated documentation or see the full project on GitHub.
Cargo.toml
[package]
name = "nafi-tokens"
version = "0.0.1"
publish = false
[dependencies.num]
version = "0.1"
default-features = false
features = [ "bigint" ]
lib.rs
//! Tokens of Nafi source
#![forbid(bad_style, missing_debug_implementations, unconditional_recursion, future_incompatible)]
#![deny(missing_docs, unsafe_code, unused)]
#![feature(conservative_impl_trait)]
extern crate num;
mod symbol;
mod literal;
pub use literal::{BigUint, Literal, StringFragments};
pub use symbol::Symbol;
/// A token in the source code. Simply chunking the source into units to then parse.
#[derive(Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Token {
#[doc(hidden)] _Unknown(usize),
Whitespace(usize),
Symbol(usize, Symbol),
Literal(usize, Literal),
Keyword(usize, Keyword),
Identifier(usize, String),
}
impl Token {
/// The start location of this token.
pub fn position(&self) -> usize {
match *self {
Token::_Unknown(pos) |
Token::Whitespace(pos) |
Token::Symbol(pos, _) |
Token::Literal(pos, _) |
Token::Keyword(pos, _) |
Token::Identifier(pos, _) => pos,
}
}
}
/// A reserved identifier-like in the source code.
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Keyword {
Let,
Mutable,
If,
Else,
}
literal.rs
pub use num::bigint::BigUint;
use std::borrow::Cow;
/// A literal in the source code, e.g. a string or number.
#[derive(Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Literal {
Integer(BigUint),
String(StringFragments),
}
impl From<BigUint> for Literal {
fn from(uint: BigUint) -> Self { Literal::Integer(uint) }
}
impl From<String> for Literal {
fn from(s: String) -> Self { Literal::String(s.into()) }
}
impl<'a> From<&'a str> for Literal {
fn from(s: &'a str) -> Self { Literal::String(s.into()) }
}
impl From<StringFragments> for Literal {
fn from(fragments: StringFragments) -> Self { Literal::String(fragments) }
}
#[derive(Clone, Debug, Eq, PartialEq)]
enum StringFragment {
Str(String),
InvalidEscape(String),
}
impl<S: Into<String>> From<S> for StringFragment {
fn from(s: S) -> Self { StringFragment::Str(s.into()) }
}
/// A String that also remembers invalid escapes inside it.
#[derive(Clone, Debug, Default, Eq, PartialEq)]
pub struct StringFragments {
fragments: Vec<StringFragment>,
}
impl<S: Into<String>> From<S> for StringFragments {
fn from(s: S) -> Self { StringFragments { fragments: vec![s.into().into()] } }
}
impl StringFragments {
/// Create a new, empty string.
pub fn new() -> StringFragments { Default::default() }
/// Push a character onto the end of this string.
pub fn push(&mut self, char: char) {
let len = self.fragments.len();
if len == 0 {
self.fragments.push(StringFragment::Str(char.to_string()));
} else {
if let StringFragment::Str(_) = self.fragments[len - 1] {
if let StringFragment::Str(ref mut string) = self.fragments[len - 1] {
string.push(char);
}
} else {
self.fragments.push(StringFragment::Str(char.to_string()));
}
}
}
/// Push a string onto the end of this string.
pub fn push_str<'a, S: Into<Cow<'a, str>>>(&mut self, str: S) {
let len = self.fragments.len();
if len == 0 {
self.fragments
.push(StringFragment::Str(str.into().into_owned()));
} else {
if let StringFragment::Str(_) = self.fragments[len - 1] {
if let StringFragment::Str(ref mut string) = self.fragments[len - 1] {
string.push_str(str.into().as_ref());
}
} else {
self.fragments
.push(StringFragment::Str(str.into().into_owned()))
}
}
}
/// Push an invalid escape onto the end of this string.
pub fn push_invalid_escape<S: Into<String>>(&mut self, escape: S) {
self.fragments
.push(StringFragment::InvalidEscape(escape.into()))
}
/// Try to turn this string into a normal string.
///
/// Fails if any invalid escapes are present.
pub fn try_into_string(self) -> Result<String, InvalidEscapes> {
if self.fragments.len() == 1 {
if let StringFragment::Str(_) = self.fragments[0] {
if let Some(StringFragment::Str(string)) = self.fragments.into_iter().next() {
return Ok(string);
} else {
unreachable!()
}
}
}
return Err(InvalidEscapes(
self.fragments
.into_iter()
.filter_map(|fragment| match fragment {
StringFragment::InvalidEscape(escape) => Some(escape),
StringFragment::Str(_) => None,
})
.collect(),
));
}
}
/// The invalid escapes in a string literal.
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct InvalidEscapes(Vec<String>);
impl InvalidEscapes {
/// Create an iterator over the invalid escapes.
///
/// You get what was attached after the `\`.
/// E.g. `\w` gives `w` and `\u{INVALID}` gives `u{INVALID}`
pub fn iter<'a>(&'a self) -> impl Iterator<Item = &'a str> { self.0.iter().map(String::as_str) }
}
symbol.rs
/// A symbol in the source code, e.g. `+-={}[]<>` (or others)
#[derive(Copy, Clone, Debug, Eq, PartialEq)]
#[allow(missing_docs)]
pub enum Symbol {
ExclamationMark,
// QuotationMark, // will never happen -- superseded by string literal
NumberSign,
DollarSign,
PercentSign,
Ampersand,
// Apostrophe, // will never happen -- superseded by quote literal
LeftParenthesis,
RightParenthesis,
Asterisk,
PlusSign,
Comma,
HyphenMinus,
FullStop,
Solidus,
Colon,
Semicolon,
LessThanSign,
EqualsSign,
GreaterThanSign,
QuestionMark,
CommercialAt,
LeftSquareBracket,
ReverseSolidus,
RightSquareBracket,
CircumflexAccent,
LowLine,
GraveAccent,
LeftCurlyBracket,
VerticalLine,
RightCurlyBracket,
Tilde,
Other(char),
}
impl Symbol {
/// The character in the source
pub fn as_char(&self) -> char {
use Symbol::*;
match *self {
ExclamationMark => '!',
NumberSign => '#',
DollarSign => '$',
PercentSign => '%',
Ampersand => '&',
LeftParenthesis => '(',
RightParenthesis => ')',
Asterisk => '*',
PlusSign => '+',
Comma => ',',
HyphenMinus => '-',
FullStop => '.',
Solidus => '/',
Colon => ':',
Semicolon => ';',
LessThanSign => '<',
EqualsSign => '=',
GreaterThanSign => '>',
QuestionMark => '?',
CommercialAt => '@',
LeftSquareBracket => '[',
ReverseSolidus => '\\',
RightSquareBracket => ']',
CircumflexAccent => '^',
LowLine => '_',
GraveAccent => '`',
LeftCurlyBracket => '{',
VerticalLine => '|',
RightCurlyBracket => '}',
Tilde => '~',
Other(char) => char,
}
}
}
As the rest of the code is rather simple, I'm most interested in literal.rs and the code supporting the string literal -- StringFragment(s).
NOTE: "Code is fine, move on" is a viable answer. But there's always something else you can say as well.
