Perl 6: Grammar issues (simple defines)

Question

My goal is to parse defines in Objective-C code:

#define wowFont [UIFont fontWithName:(DEV)?@"one":@"two" size:(DEV? 10: 12)]
#define wowColor [UIColor colorWithRed:(DEV?240/255.0f:255.0f) green:((DEV)? 230.0f / 255.0f) blue: 10.0f]

I try to write grammar for it:

grammar Defines::Grammar {
    token TOP {
        \s*<define_expression>\n
    }
    token define_expression {

        ^^ #define $<keyword>=\w+ \s+ $<value> = <general_expression>
    }

    token general_expression{
        # here, determine parenthes
        <simple_expression>|[(]<general_expression>[)]|<value_expression>
    }

    token simple_expression{
        # here, determine ?: behaviour
        \s*<general_expression>\s*[?]\s*<general_expression>\s*[:]\s*<general_expression>\s*
    }

    token value_expression {
        # stop here, need to determine ?: operator here
        <numeric_expression>|<string_expression>|<function_invocation_expression>|<bareword_expression>
    }

    token function_invocation_expression {
        [\[] (<class>|<bareword_expression>) \s+ <function_body>\s* [\]]
    }

    token function_body {
        <bareword_expression>|<bareword_expression> \s* [:] <general_expression>
    }

    token bareword_expression {
        [\w_]+
    }

    token string_expression {
        \@["][^"]["]
    }

    token numeric_expression {
        # we could +-*/ numeric expressions, also we can put lead sigil [-]. put parentheses and it could be a <number>
        <numeric_expression> [*/+-] <numeric_expression> | [-+]<numberic_expression> | [(] <numeric_expression> [)] | <number>
    }

    token number {  
        # number expression here, ok
        # not needed legal sign here, numeric_expression already have it
        \d+ [\.] \d+
    }
}

Could anybody help to improve it?

P.S: As a part of answer (I hope that somebody will find this documents helpful) I place here a link to Perl 6 book (I found out that Grammars section is good enough for beginning). Perl 6 Book

Answer 1

It doesn't look like this code is currently working.

AFAIK Objective-C uses the normal C preprocessor. The preprocessor works on a token level, and does not know about stuff like method calls. It is entirely irrelevant that [ starts a method call or an array subscript, for the preprocessor it's just some token. So as a rough sketch:

token TOP {
    ^^ \s* '#define' \s* $<name>=<bareword> \s* $<value>=(<token>+) $$
}

Notice that my # is inside a string and is thus not interpreted as a comment!

There are however some special cases to consider:

1. Arguments to the macros:

   #define MIN(a, b) (((a) < (b)) ? a : b)
   

   Here, the a and b tokens have special meaning to the preprocessor.

2. Line continuations:

   #define MIN(a, b)         \
       (((a) < (b)) ? a : b)
   

3. Token concatenation with ##:

   // "MANGLE(foo, bar)" → foobar
   #define MANGLE(prefix, name) prefix##name
   

The next problem is that \s is any Unicode whitespace, including newlines. It would be recommendable to restrict that to horizontal whitespace. It would make sense to override the ws rule:

token ws {
    <!ww> \h*
}

With support for line continuations:

token ws {
    <!ww> [ \h | \\\n ]*
}

We can now use sigspace (significant whitespace) to simplify our tokens as rules, e.g:

rule TOP {
    ^^ '#define' $<name>=<bareword> $<value>=(<token>+) $$
}

What is a token? Identifier, number, character, symbol, or string. You attempt to match a string with \@["][^"]["], but this fails for various reasons:

• In Perl 6, the […] denotes a non-capturing group, but not a character class. These now look like <[…]>. Especially, a negated character class looks like <-[…]>.

• If that were a Perl 5 regex, you'd only be allowing a single letter inside the string, because the [^"] charclass is not being repeated.

If we also want to handle escapes inside the string, a string token might be matched by

token string {
    \@\" [ <-[\"\\]> | \\. ]* \"
}

Similar errors also occur in other tokens.

Answer 2

I was about to submit this when I saw that you had visited #perl6. I'll leave this post as is because it is supposed to help others that might one day read this as well as you.

TL;DR I recommend you: make sure you're ready for an immature compiler, ecosystem, doc, etc; use Rakudo Star and the Rakudo debugger; start small and take small steps, making each step work before moving forward; rely very heavily on #perl6 -- they welcome any and all comers, including "newbies" and lots of basic questions.

---

First, let me make sure you are not wasting your time due to wrong expectations. I am pretty sure there are currently only a handful of folk using P6 in production and they are doing so for only a small range of use cases. Powerful parsing is one of them, so what you are attempting is reasonable, but it's slow and you need to be willing to patiently deal with the many other frustrations that can result from P6's immaturity. If this hasn't put you off, welcome aboard! :)

Second, let me make sure I am not wasting your time. Your code doesn't compile using the tools I'm using. I used a procedure you can repeat to test your code with the latest Rakudo Perl 6 compiler HEAD:

• I joined #perl6, the freenode IRC channel. (A very friendly channel. All are welcome.)
• I tested code by entering a line of the form "m: ...". (Replace the "..." with code, or a suitable "pastebin" URL as explained below, otherwise you'll get a "Stub code executed" exception. An evalbot spots such lines, runs the code using the HEAD of the Rakudo compiler using the MoarVM backend, and displays the result.)
• I ran your code privately (recommended if you don't want to "spam" the channel with attempts that aren't interesting to others) so you won't be able to see this in the public IRC log. To run code privately enter the command /msg camelia r: say $*PERL, switch to the "camelia" window, and enter lines there.
• So I entered "m: https://gist.github.com/raiph/11084741" which attempted to compile a copy of your code. It failed to compile.

Third, let's radically improve your debugging options. One option when debugging grammars is to use the Grammar::Tracer module. This is occasionally useful but for most purposes there's a much better option: Rakudo's good-trending-great debugger. Use the debugger all the time unless you have a compelling reason not to. The Rakudo debugger is not available using the on channel evalbots I just discussed above; it requires that you install something. I recommend the Rakudo Star package unless you have a compelling reason to use something else. Rakudo Star includes the debugger, modules, and doc including info on how to start the debugger.

Fourth, start simple and build up from there, only making another small step forward when what you currently have works. For example, a first step might be:

say so '#define foo [blah]' ~~ rule { '#define' <ident> .* }

prints:

True

meaning the string on the left of the ~~ was "accepted" by the thing on the right.

The thing on the right is a rule which is one of several convenient forms of regex introduced by P6. The P6 regex engine never backtracks once it's matched an atom in a rule and it replaces appropriate spaces in your rule with implicit calls to the whitespace matching rule called <ws>. The above line is equivalent to:

say so '#define foo [blah]' ~~ regex { :ratchet '#define' <.ws> <ident> <.ws> .* }

(The . in <.ws> tells P6 to call the <ws> rule without saving the text it matched in the match object being built up by the processing of the regex.)