Configuration file parser

Question

https://github.com/pixlark/vars-parser

#![allow(unused_variables)]
#![allow(dead_code)]
#![allow(non_camel_case_types)]

#[cfg(test)]
mod vars_parser {
    use std::str::Chars;
    use std::iter::Peekable;
    use std::collections::HashMap;

    /// Useful wrapper for Peekable<Chars> which returns an EOF char
    /// when the iterator is empty
    struct Stream<'a> {
        stream: &'a mut Peekable<Chars<'a>>,
    }

    impl<'a> Stream<'a> {
        fn peek(&mut self) -> char
        {
            let c: Option<&char> = self.stream.peek();
            match c {
                Some(c) => *c,
                None => '\0'
            }
        }
        fn next(&mut self) -> char
        {
            self.stream.next().unwrap_or('\0')
        }
    }

    #[derive(Debug)]
    enum Token {
        EOF,
        Assignment,
        Name(String),
        String_Literal(String),
        Int_Literal(i64),
        Float_Literal(f64),
    }

    /// Pull from stream into buffer until name is terminated or EOF
    /// reached
    fn scan_name(stream: &mut Stream) -> String
    {
        let mut string = String::new();
        while !stream.peek().is_whitespace() && stream.peek() != ':' && stream.peek() != '\0' {
            string.push(stream.next());
        }
        return string;
    }

    /// Pull from stream into buffer until string is terminated or EOF
    /// reached
    fn scan_string(stream: &mut Stream) -> String
    {
        let mut string = String::new();
        while stream.peek() != '\0' && stream.peek() != '"' {
            string.push(stream.next());
        }
        return string;
    }

    #[derive(Debug)]
    enum Number {
        Integer(i64),
        Float(f64),
        Not_A_Number,
    }

    /// Read int/float from stream. Returns Number::Not_A_Number if
    /// scanning fails.
    fn scan_number(stream: &mut Stream) -> Number
    {
        let mut buffer = String::new();
        let mut fractional: bool = false;
        if stream.peek() == '-' || stream.peek() == '+' {
            buffer.push(stream.next());
        }
        while stream.peek().is_numeric() || stream.peek() == '.' {
            if stream.peek() == '.' { fractional = true; }
            buffer.push(stream.next());
        }
        if fractional {
            let result = buffer.parse::<f64>();
            match result {
                Ok(ok) => Number::Float(ok),
                Err(_) => Number::Not_A_Number
            }
        } else {
            let result = buffer.parse::<i64>();
            match result {
                Ok(ok) => Number::Integer(ok),
                Err(_) => Number::Not_A_Number
            }
        }
    }

    /// Central part of lexer. Advances stream by arbitrary amount
    /// until the next token is lexed.
    fn next_token(stream: &mut Stream) -> Result<Token, String>
    {
        let c = stream.peek();
        if c.is_whitespace() {
            stream.next();
            return next_token(stream);
        }
        if c.is_alphabetic() || c == '_' {
            return Ok(Token::Name(scan_name(stream)));
        }
        if c.is_numeric() || c == '.' || c == '-' || c == '+' {
            let num = scan_number(stream);
            return match num {
                Number::Integer(n) => Ok(Token::Int_Literal(n)),
                Number::Float(f) => Ok(Token::Float_Literal(f)),
                Number::Not_A_Number => Err("Unable to parse literal".to_string())
            }
        }
        match c {
            '#' => {
                stream.next();
                let mut n = stream.next();
                while n != '\n' && n != '\0' {
                    n = stream.next();
                }
                next_token(stream)
            }
            ':' => {
                stream.next();
                if stream.next() == '=' {
                    Ok(Token::Assignment)
                } else {
                    Err("Expected = after :".to_string())
                }
            },
            '"' => {
                stream.next();
                let s = Token::String_Literal(scan_string(stream));
                if stream.next() == '"' {
                    Ok(s)
                } else {
                    Err("String literal unterminated".to_string())
                }
            },
            '\0' => {
                Ok(Token::EOF)
            },
            _ => {
                Err("Unrecognized char".to_string())
            }
        }
    }

    #[derive(Debug)]
    pub enum Value {
        String(String),
        Integer(i64),
        Float(f64),
    }

    #[derive(Debug)]
    struct Declaration {
        name: String,
        value: Value,
    }

    /// Variation on Result which can specify EOF reached to terminate
    /// main loop
    #[derive(Debug)]
    enum Parse_Result {
        Ok(Declaration),
        Err(String),
        EOF,
    }

    /// Reads arbitrary amount of tokens from next_token() until a new
    /// declaration is found
    fn parse_declaration(stream: &mut Stream) -> Parse_Result
    {
        let mut decl = Declaration { name: "".to_string(), value: Value::Integer(0) };
        {
            let token = next_token(stream);
            match token {
                Ok(ok) => {
                    match ok {
                        Token::Name(s) => decl.name = s,
                        Token::EOF => return Parse_Result::EOF,
                        _ => return Parse_Result::Err("Expected name at beginning of declaration".to_string())
                    }
                },
                Err(e) => return Parse_Result::Err(e)
            }
        }
        {
            let token = next_token(stream);
            match token {
                Ok(ok) => {
                    match ok {
                        Token::Assignment => (),
                        _ => return Parse_Result::Err("Expected := after name in declaration".to_string())
                    }
                },
                Err(e) => return Parse_Result::Err(e)
            }
        }
        {
            let token = next_token(stream);
            match token {
                Ok(ok) => {
                    match ok {
                        Token::String_Literal(s) => {
                            decl.value = Value::String(s);
                        },
                        Token::Int_Literal(n) => {
                            decl.value = Value::Integer(n);
                        },
                        Token::Float_Literal(f) => {
                            decl.value = Value::Float(f);
                        },
                        _ => return Parse_Result::Err("Expected literal at end of declaration".to_string())
                    }
                },
                Err(e) => return Parse_Result::Err(e)
            }   
        }
        Parse_Result::Ok(decl)
    }

    /// Reads as many declarations from a source string as it can and
    /// stores them in a HashMap
    pub fn parse_vars(source: String) -> Result<HashMap<String, Value>, String>
    {
        let mut stream = Stream { stream: &mut source.chars().peekable() };
        let mut decls: HashMap<String, Value> = HashMap::new();
        loop {
            let result = parse_declaration(&mut stream);
            match result {
                Parse_Result::Ok(ok) => decls.insert(ok.name, ok.value),
                Parse_Result::EOF => break,
                Parse_Result::Err(e) => return Err(e)
            };
        }
        Ok(decls)
    }

    #[test]
    fn test_parsing()
    {
        let source: String = "
        # Comment
        variable_str   := \"string literal\"
        variable_int   := -15
        variable_float := 105.3".to_string();
        let vars = match parse_vars(source) {
            Ok(ok) => ok,
            Err(e) => panic!(e)
        };
        {
            let key: String = "variable_str".to_string();
            match vars.get(&key) {
                Some(val) => {
                    match val {
                        Value::String(s) => {
                            assert_eq!(s, "string literal");
                        },
                        _ => panic!("String literal didn't parse correctly")
                    }
                },
                None => panic!("Name didn't get parsed correctly")
            }
        }
        {
            let key: String = "variable_int".to_string();
            match vars.get(&key) {
                Some(val) => {
                    match val {
                        Value::Integer(n) => {
                            assert_eq!(*n, -15);
                        },
                        _ => panic!("Integer literal didn't parse correctly")
                    }
                },
                None => panic!("Name didn't get parsed correctly")
            }
        }
        {
            let key: String = "variable_float".to_string();
            match vars.get(&key) {
                Some(val) => {
                    match val {
                        Value::Float(f) => {
                            assert_eq!(*f, 105.3);
                        },
                        _ => panic!("Float literal didn't parse correctly")
                    }
                },
                None => panic!("Name didn't get parsed correctly")
            }
        }
    }
}

I've used C for a long time but have been experimenting with Rust, this is the first somewhat significant thing I've written in Rust. Mostly wondering how well I managed to take advantage of Rust's capabilities, as this is the kind of thing I would usually write in C.

Answer 1

First off, congratulations on writing a complete project in Rust. To get something working in a language you're fairly new to is a great achievement—I remember the struggles I had with Rust initially.

There are a few points I noticed which caught my attention. I've listed them below with some explanation so you can understand how you could make your code more idiomatic and easily maintainable:

1. Compiler lints.

   #![allow(unused_variables)]
   #![allow(dead_code)]
   #![allow(non_camel_case_types)]
   

   Generally, unless you have an important reason to ignore Rust's style guidelines and lints, you should listen to them. That's twice as important if you're writing a library to be used by others if you want to provide a consistent experience—with Rust I know with 99% certainty that any code I include from other sources uses similar style choices. Being part of that 1% makes things frustrating for everyone.

   I noticed that your public API does follow the style guide, so technically you're not exposing any of that to your potential users, but it's probably more maintainable if you just follow the usual style everywhere.

   Even though you're writing a library, the dead code lint will still be helpful to you. Any code which is not public or used by something public will be considered 'dead'. You should delete anything that is indeed dead.

2. Testing modules.

   #[cfg(test)]
   mod vars_parser {
   

   Here, you've inadvertently made your library only accessible during cargo test runs. As noted in the book, "[t]he #[cfg(test)] annotation on the tests module tells Rust to compile and run the test code only when you run cargo test, not when you run cargo build." I assume you'd like your code to be usable in real projects, so you'd need to separate your code into a testing module and your actual project. Roughly, like so in src/lib.rs:

   // your code here, in the crate root
   pub fn frobnicate(foo: String) -> bool {
       true
   }
   
   #[cfg(test)]
   mod tests {
       #[test]
       fn test_something() {
           unimplemented!();
       }
   }
   

3. Miscellaneous style.

   fn scan_name(stream: &mut Stream) -> String
   {
   

   Standard Rust style is to have the opening brace on the same line as the fn declaration. Additionally, 4 spaces are used for indentation in Rust rather than a tab (your editor can probably auto-convert this if you configure it). Consider trying rustfmt to see how it formats your code. It's generally pretty good at its job.

4. pub fn parse_vars(source: String) -> Result<HashMap<String, Value>, String>
   

   You could change source to be a &str without breaking anything. Accepting a borrowed string here makes it easier for your API consumers because they don't have to allocate a new String. If they already have a String, passing it as a reference is easy enough.

5. The Stream struct.

   I don't feel your implementation of the Stream struct is very 'Rusty'. Mapping Nones to \0 feels very much like a habit brought over from C of null-terminated strings. Getting rid of that allows you to use the Iterator API to write far more succinct code. For example, here's your scan_name() function:

   fn scan_name(stream: &mut Stream) -> String
   {
       let mut string = String::new();
       while !stream.peek().is_whitespace() && stream.peek() != ':' && stream.peek() != '\0' {
           string.push(stream.next());
       }
       return string;
   }
   

   And here's how I might rewrite it, taking an iterator:

   fn scan_name(stream: &mut Peekable<Chars>) -> String {
       stream.take_while(|character| !character.is_whitespace() && *character != ':').collect()
   }
   

   Try it on the Playground if you like; notice that you only need a Chars iterator and not a Peekable<Chars<'a>> or your Stream type. I think most of your other functions could be written in a more functional style, and that is far more idiomatic than imperative C-style code. Far more beautiful, too, in my opinion, but of course I have a slightly biased opinion.

6. Existing parser libraries. If you wanted to experiment, you could try using an existing parser library such as nom. If you want to write a parser fairly quickly and be sure that there are no problems with it, you could give that a try.