# Parsing library interface

I have been working on a parser generator, in which I want to employ a sort of fluent configuration interface - avoiding the pre-compile step that usually comes with parser generators. It's supposed to be a tool not just for language construction, but also for generalized parsing tasks that you usually end up coding manually since it's not worth the effort to drag in a full-blown language construction kit.

What I'm concerned about is if I'm missing the mark, since the code quickly becomes a bit complex when configuring a parser. Here's my demo for a JSON parser:

using System.Collections.Generic;
using Piglet.Parser;

namespace Piglet.Demo.Parser
{
public class JsonParser
{
public class JsonElement
{
public string Name { get; set; }
public object Value { get; set; }
};

public class JsonObject
{
public List<JsonElement> Elements { get; set; }
};

public static void Run()
{
var parser = ParserFactory.Configure<object>(configurator =>
{
var quotedString = configurator.Terminal("\"(\\.|[^\"])*\"",    f => f.Substring(1, f.Length - 2));
var doubleValue = configurator.Terminal(@"\d+\.\d+",            f => double.Parse(f));
var integerValue = configurator.Terminal(@"\d+",                f => int.Parse(f));

var jsonObject = configurator.NonTerminal();
var optionalElementList = configurator.NonTerminal();
var elementList = configurator.NonTerminal();
var element = configurator.NonTerminal();
var value = configurator.NonTerminal();
var array = configurator.NonTerminal();
var optionalValueList = configurator.NonTerminal();
var valueList = configurator.NonTerminal();

jsonObject.Productions(p => p.Production("{", optionalElementList, "}")
.OnReduce( f => new JsonObject { Elements = (List<JsonElement>) f[1]} ));

optionalElementList.Productions(p =>
{
p.Production(elementList)
.OnReduce(f => f[0]);

p.Production()
.OnReduce(f => new List<JsonElement>());
});

elementList.Productions(p =>
{
p.Production(elementList, ",", element)
.OnReduce(f => {
var list = (List<JsonElement>)f[0];
return list;
});

p.Production(element)
.OnReduce(f => new List<JsonElement> { (JsonElement)f[0] });
});

element.Productions(p => p.Production(quotedString, ":", value)
.OnReduce(f => new JsonElement { Name = (string)f[0], Value = f[2]}));

value.Productions(p =>
{
p.Production(quotedString)  .OnReduce(f => f[0]);
p.Production(integerValue)  .OnReduce(f => f[0]);
p.Production(doubleValue)   .OnReduce(f => f[0]);
p.Production(jsonObject)    .OnReduce(f => f[0]);
p.Production(array)         .OnReduce(f => f[0]);
p.Production("true")        .OnReduce(f => true);
p.Production("false")       .OnReduce(f => false);
p.Production("null")        .OnReduce(f => null);
});

array.Productions(p => p.Production("[", optionalValueList, "]")
.OnReduce( f => ((List<object>)f[1]).ToArray() ));

optionalValueList.Productions(p =>
{
p.Production(valueList)
.OnReduce(f => f[0]);
p.Production()
.OnReduce(f => new List<object>());
});

valueList.Productions(p =>
{
p.Production(valueList, ",", value)
.OnReduce(f => {
var list = (List<object>)f[0];
return list;
});
p.Production(value)
.OnReduce(f => new List<object>{f[0]});
});

configurator.LexerSettings.EscapeLiterals = true;
configurator.LexerSettings.Ignore = new [] { @"\s+" };
});

var jObject = (JsonObject)parser.Parse("{ \"Property1\":\"value\", \"IntegerProperty\" : 1234 }");
}
}
}


Questions:

1. Is it clear what this configuration achieves to you?
2. Is there any way you can see to improve the interface of the library?

Of course, this should be considerably more understandable if you have previous experience of parser generators, but even if you don't it would be valuable to me to know how much of this someone unfamiliar with parser generators can figure out from just seeing the code.

The full library is on GitHub, if you feel the need to delve deeper into the internals. But since this is the interface that users of the library would see, this is probably the most important part I want to get right.

To clarify exactly what I would like people to take a shot at is not the actual JSON parsing grammar in this example, or the Run method in general. Rather it is the structure of the ParserFactory and the way the Production, OnParse and such methods are called. For example:

1. Is my way of designing the library configuration interface good?
2. Does it use the lambdas in a good way?
3. Are the method names understandable?
4. Do you understand what it does given whatever background you have on context free grammar parsing?

I like the idea of a Fluent Parser, but I find the syntax you're using very confusing, which probably isn't what you wanted to hear.

Some feedback:

• Productions, Production, Onreduce, these terms have no meaning for me. So they don't feel fluent at all. They might be fluent in the world of a parser, but as you want to abstract the parser away, I think you need to think more in terms of the resulting structure or the expressions. Kind of Parser.Add("ElementList").AddOption().Start().FollowedByLiteral("{").FollowedByList<int>(",").FollowedByLiteral("}"); or something, that expresses the structure being parsed, but not the process underneath. (not the best example, but still more understandable than Productions.AddProduction.OnReduce(f => f[0][1].ToString());
• What's meant by f => f[1]? It's unclear for the reader, can't you write this as OnReduce<returnType>().SkipLiteral("}").Take<returnType>(); or something else that stays fluent and removed the need for these lambda's if possible?
• I'd try to refactor OnReduce (whatever it does) to be a generic method, so that OnReduce(f => ((List&lt;object&gt;)f[1]).ToArray()) would become OnReduce<object[]>(f => f[1]); And then you need to figure out a way to remove the f => f[1] part.
• Very good input! That's exactly the sort of stuff I want to know. Thanks a lot – Dervall Feb 20 '12 at 20:17
• The f=>f[0] will select the first parameter used in the production of the parser as the resulting value of a rule. The syntax I have right now is pretty much a code version of whatever is found in flex and bison. I really like the FollowedBy method, with the type parameter! – Dervall Feb 20 '12 at 20:27
• I really like your idea :) But don't have much time on my hand... if you manage to "abstract away" the parser details and leave us to describe the structure of what we're parsing, then I'd probably use your library a lot, it is a lot easier to read than that of many others. Does the parser then generate something that evaluates the data being fed to it lazily, or do you need to use it as a serializer? – jessehouwing Feb 20 '12 at 20:34
• Well, it takes a stream of characters and parses according to whatever given rules. The output is always a single object of the given type that you declared the parser to produce. It will consume the input in one swoop, since that's needed to confirm the validity of the grammar, usually. It'll execute it's actions as soon as possible though, if that makes sense to you. – Dervall Feb 20 '12 at 20:39
• Makes sense, was just wondering if the root element were a list, if it would support IEnumerable for improved streaming. Sometimes you just don't care about the correctness of the grammar if performance is important. Just expect it to be correct and handle the issues when they occur ;) – jessehouwing Feb 20 '12 at 21:37

Refactor the Run()-method into seperate methods. It's barely readable now.

• I agree. That would be the first step I would take as well. – dreza Feb 18 '12 at 20:49
• Oh, I think you're missing the point. The Run method is not really what I want reviewed. That is just the demo code. What I want peoples opinion on is if the way of using the configurator object is looking intuitive, and the way of configuring productions and reduction rules for the grammar is a nice way of doing it. I'll clarify the question – Dervall Feb 18 '12 at 21:31

The Productions() method is clearly something with side effects, since you are calling it and not saving any return value; shouldn't its name include a verb, then?

Something like SetProductions() or AddProductions().

• It probably should, I have been too seduced by functional programming and lambdas to miss what actually was my goal. I'm working on redesigning the configuration interface. The new version will probably have this named IsMadeUpBy() which is followed by a bunch of FollowedBy. Something to that effect. Thanks for your input! – Dervall Feb 21 '12 at 10:54