7
\$\begingroup\$

I wrote (as part of a greater work) a Boost.Spirit grammar that would parse string literals, including support for the various escape sequences known from C/C++ (\n, \x7f, \341, \u017f, \U00010451).

At some point I encountered some problems, mostly due to my lack of understanding either Boost.Spirit or Boost.Phoenix to the detail required to have full control over what I'm doing, and despairing at the rather non-descriptive error messages Boost.Spirit generates. ;-) User sehe was very helpful over at StackOverflow, and my grammar is now functional.

However, some things are still bothering me:

  1. The functor cp2utf8_f does the conversion of a UChar32 to UTF-8 byte sequence. However, as a struct inside the grammar, it is not exactly re-usable. I would like to have it as a stand-alone function, but have failed to make it work.
  2. The escapes rule basically does the same thing in five different ways -- determine a UChar32 code point, and pass it to the functor (see above) using semantic actions, which appends it to the result string. This should really be a rule with an UChar32 result, which is then passed to the functor at the point the rule is called (to avoid the five-fold repitition of the functor call). Again, I had an idea of how it should work, but it didn't.
  3. The error handlers (straight from the tutorial) currently print to std::cout. That's not nice; I'd rather have the error message generated by the handler thrown as exception (let's say std::runtime_error for the sake of this review). Again, my lack of in-depth understanding of what is going on here exactly makes me scratch my head at why the compiler complains about "invalid use of void exception" when I replace the std::cout << ... with throw std::runtime_error( ... ). Or rather, I have a good guess at why (it's a function template call that is choking on a statement that evaluates to void), but no idea as to how to work around it.

Any other suggestions (like, how to better learn to fish in Boost.Spirit instead of asking you to hand me the fish...) are likewise welcome. I know the "test driver" main() is crude; I didn't want to make this longer than necessary by going through the full rigor of setting up a unit test with result verification etc.

#define BOOST_SPIRIT_UNICODE

#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>

#include <unicode/unistr.h>
#include <unicode/bytestream.h>

#include <string>
#include <stdexcept>

namespace qi = boost::spirit::qi;
namespace spirit = boost::spirit;
namespace phoenix = boost::phoenix;

using boost::spirit::unicode::char_;
using boost::spirit::eol;

template < typename Iterator >
struct QuotedString : qi::grammar< Iterator, std::string() >
{
    QuotedString() : QuotedString::base_type( quoted_string )
    {
        quoted_string = '"' > *( +( char_ - ( '"' | eol | '\\' ) ) | escapes ) > '"';

        // (2.) severely violating don't-repeat-yourself here...
        escapes = '\\' > ( escaped_character[ cp2utf8( spirit::_val, spirit::_1 ) ]
                         | ( "x" > qi::uint_parser< UChar32, 16, 2, 2 >()[ cp2utf8( spirit::_val, spirit::_1 ) ] )
                         | ( "u" > qi::uint_parser< UChar32, 16, 4, 4 >()[ cp2utf8( spirit::_val, spirit::_1 ) ] )
                         | ( "U" > qi::uint_parser< UChar32, 16, 8, 8 >()[ cp2utf8( spirit::_val, spirit::_1 ) ] )
                         | qi::uint_parser< UChar32,  8, 1, 3 >()[ cp2utf8( spirit::_val, spirit::_1 ) ]
                         );

        escaped_character.add
        (  "a", 0x07 ) // alert
        (  "b", 0x08 ) // backspace
        (  "f", 0x0c ) // form feed
        (  "n", 0x0a ) // new line
        (  "r", 0x0d ) // carriage return
        (  "t", 0x09 ) // horizontal tab
        (  "v", 0x0b ) // vertical tab
        ( "\"", 0x22 ) // literal quotation mark
        ( "\\", 0x5c ) // literal backslash
        ;

        qi::on_error< qi::fail >
        (
            escapes,
            // (3.) I don't want std::cout from deep within the bowels of my application
#if 0
            throw std::runtime_error( "...the error message..." ) );
#else
            std::cout << phoenix::val( "Illegal escape sequence. Expecting " )
                      << spirit::_4
                      << phoenix::val( " here: \"" )
                      << phoenix::construct< std::string >( spirit::_3, spirit::_2 )
                      << phoenix::val( "\"" )
                      << std::endl
        );
#endif

        qi::on_error< qi::fail >
        (
            quoted_string,
            std::cout << phoenix::val( "Illegal string literal. (Unterminated string?)" )
                      << std::endl
        );
    }

    // (1.) This could be useful elsewhere.
    // (And I don't like *result* and *cp2utf8* lying around here
    // when a stand-alone function should do just as well.)
    struct cp2utf8_f
    {
        template < typename ... > struct result { using type = void; };

        void operator()( std::string & a, UChar32 codepoint ) const
        {
            icu::StringByteSink< std::string > bs( &a );
            icu::UnicodeString::fromUTF32( &codepoint, 1 ).toUTF8( bs );
        }
    };

    phoenix::function< cp2utf8_f > cp2utf8;

    qi::rule< Iterator, std::string() > quoted_string;
    qi::rule< Iterator, std::string() > escapes;
    qi::symbols< char const, UChar32 > escaped_character;
};

#ifdef TEST

#include <vector>

int main()
{
    std::vector< std::string > tests = { "\"plain string\"",
                                         "\"hex\\x20escape\"",
                                         "\"oct\\40escape (two digits)\"",
                                         "\"oct\\040escape (two digits, leading zero)\"",
                                         "\"oct \\105scape (three digits)\"",
                                         "\"unterminated string",
                                         "\"embedded\\nnewline\"",
                                         "\"4-digit unicode \u0065scape\"",
                                         "\"8-digit unicode \U00000065scape\"",
                                         "\"illegal \\character escape\"",
                                         "\"illegal \\xhex escape\"",
                                         "\"illegal \\8ct escape\"",
                                         "\"illegal \\unicode escape\"",
                                         "\"illegal \\Unicode escape\"" };
    QuotedString< std::string::const_iterator > qs;
    for ( auto input : tests )
    {
        std::string result;
        parse( input.cbegin(), input.cend(), qs, result );
        std::cout << result << "\n";
    }
    return 0;
}

#endif
\$\endgroup\$
  • \$\begingroup\$ And yes, I know that my \x escapes (taking exactly two hex digits) don't follow the letter of the C/C++ standard (taking as many hex digits as there are). Mine are actually useful. ;-) \$\endgroup\$ – DevSolar Aug 31 '15 at 11:21
6
\$\begingroup\$

3. The error handlers

The problem with the throw expression, as the compiler kindly reminded you, is that they're void-expressions.

Even if it compiled, it would not do what you want: it'd throw during the grammar constructor...

The repeating story here is that semantic actions (and error handlers in this case) require Phoenix actors (a.k.a. lazy or deferred functions), so that spirit knows how to evalute them against the spirit context when needed. The simple case:

qi::on_error< qi::fail >
(
    quoted_string,
    phoenix::throw_(
        phoenix::construct<std::runtime_error>( "Illegal string literal. (Unterminated string?)" )
    )
);

The more complex version requires stream concatenation. You could do this with a local/let-expression, but I'd keep it simple and extract a Phoenix function make_error_message:

qi::on_error< qi::fail >
(
    escapes,
    phoenix::throw_(
        phoenix::construct<std::runtime_error>( make_error_message(qi::_4, qi::_3, qi::_2) )
    )
);

Now, you can just code that function in any which way you like:

struct make_error_message_f {
    template <typename ...> struct result { using type = std::string; };

    template <typename Info, typename F, typename L>
    std::string operator()(Info const& info, F f, L l) const {
        std::ostringstream oss;
        oss << "Illegal escape sequence. Expecting " << info << " here: \"" << std::string(f,l) << "\"";
        return oss.str();
    }
};

phoenix::function<make_error_message_f> make_error_message;

See below for ways to make make_error_message a function that's adapted for Phoenix use.


1. Using a global function

However, as a struct inside the grammar, it is not exactly re-usable. I would like to have it as a stand-alone function, but have failed to make it work.

You can of course just relay the implementation of cp2utf8_f::operator() to a re-usable function of your choice. Of course, that makes the cp2utf8_f function object merely red-tape code. If you don't mind putting traits in the Phoenix extensions namespaces, you can use the existing adaptation macros:

namespace my_helpers {
    void cp2utf8(std::string& a, UChar32 codepoint)
    {
        icu::StringByteSink<std::string> bs(&a);
        icu::UnicodeString::fromUTF32(&codepoint, 1).toUTF8( bs );
    }

    template<typename Iterator>
        std::string make_error_message(boost::spirit::info const& info, Iterator first, Iterator last) {
            std::ostringstream oss;
            oss << "Illegal escape sequence. Expecting " << info << " here: \"" << std::string(first,last) << "\"";
            return oss.str();
        }
}

BOOST_PHOENIX_ADAPT_FUNCTION(void,        cp2utf8_,            my_helpers::cp2utf8,            2)
BOOST_PHOENIX_ADAPT_FUNCTION(std::string, make_error_message_, my_helpers::make_error_message, 3)
// (And I don't like *result* and *cp2utf8* lying around here
// when a stand-alone function should do just as well.)

They're private inner types. They inlining food. What's the cost you measured?

Personally, I prefer the localized function objects because they give you more control and prevent namespace pollution. Note that on sufficiently advanced version you may be able to drop the inner result_type/result<>::type constructs (see RESULT_OF docs).


2. Reducing WET-ness (repetition)

Is this what you had in mind:

escapes = '\\' > ( 
          escaped_character
        | ("x" > qi::uint_parser<UChar32, 16, 2, 2>())
        | ("u" > qi::uint_parser<UChar32, 16, 4, 4>())
        | ("U" > qi::uint_parser<UChar32, 16, 8, 8>())
        | (      qi::uint_parser<UChar32,  8, 1, 3>()) 
      ) [ cp2utf8_( qi::_val, qi::_1 ) ]
;

DEMO

Includes the improvements described, and also some excess scope/namespace pollution issues.

Live on Coliru

#define BOOST_SPIRIT_UNICODE

#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>

#include <unicode/unistr.h>
#include <unicode/bytestream.h>

#include <string>
#include <stdexcept>

namespace qi = boost::spirit::qi;

using boost::spirit::unicode::char_;
using boost::spirit::eol;

namespace my_helpers {
    void cp2utf8(std::string& a, UChar32 codepoint)
    {
        icu::StringByteSink<std::string> bs(&a);
        icu::UnicodeString::fromUTF32(&codepoint, 1).toUTF8( bs );
    }

    template<typename Iterator>
        std::string make_error_message(boost::spirit::info const& info, Iterator first, Iterator last) {
            std::ostringstream oss;
            oss << "Illegal escape sequence. Expecting " << info << " here: \"" << std::string(first,last) << "\"";
            return oss.str();
        }
}

BOOST_PHOENIX_ADAPT_FUNCTION(void,        cp2utf8_,            my_helpers::cp2utf8,            2)
BOOST_PHOENIX_ADAPT_FUNCTION(std::string, make_error_message_, my_helpers::make_error_message, 3)

template < typename Iterator >
struct QuotedString : qi::grammar< Iterator, std::string() >
{
    QuotedString() : QuotedString::base_type( quoted_string )
    {
        quoted_string = '"' > *( +( char_ - ( '"' | eol | '\\' ) ) | escapes ) > '"';

        escapes = '\\' > ( 
                  escaped_character
                | ("x" > qi::uint_parser<UChar32, 16, 2, 2>())
                | ("u" > qi::uint_parser<UChar32, 16, 4, 4>())
                | ("U" > qi::uint_parser<UChar32, 16, 8, 8>())
                | (      qi::uint_parser<UChar32,  8, 1, 3>()) 
              ) [ cp2utf8_( qi::_val, qi::_1 ) ]
        ;

        escaped_character.add
            (  "a", 0x07 ) // alert
            (  "b", 0x08 ) // backspace
            (  "f", 0x0c ) // form feed
            (  "n", 0x0a ) // new line
            (  "r", 0x0d ) // carriage return
            (  "t", 0x09 ) // horizontal tab
            (  "v", 0x0b ) // vertical tab
            ( "\"", 0x22 ) // literal quotation mark
            ( "\\", 0x5c ) // literal backslash
        ;

        namespace phx = boost::phoenix;

        qi::on_error< qi::fail > (
            escapes,
            phx::throw_(
                phx::construct<std::runtime_error>( make_error_message_(qi::_4, qi::_3, qi::_2) )
            )
        );

        qi::on_error< qi::fail > (
            quoted_string,
            phx::throw_(
                phx::construct<std::runtime_error>( "Illegal string literal. (Unterminated string?)" )
            )
        );
    }

  private:
    qi::rule< Iterator, std::string() > quoted_string;
    qi::rule< Iterator, std::string() > escapes;
    qi::symbols< char const, UChar32 > escaped_character;
};

int main()
{
    QuotedString<std::string::const_iterator> qs;

    for (std::string const input : {
            "\"plain string\"",
            "\"hex\\x20escape\"",
            "\"oct\\40escape (two digits)\"",
            "\"oct\\040escape (two digits, leading zero)\"",
            "\"oct \\105scape (three digits)\"",
            "\"unterminated string",
            "\"embedded\\nnewline\"",
            "\"4-digit unicode \u0065scape\"",
            "\"8-digit unicode \U00000065scape\"",
            "\"illegal \\character escape\"",
            "\"illegal \\xhex escape\"",
            "\"illegal \\8ct escape\"",
            "\"illegal \\unicode escape\"",
            "\"illegal \\Unicode escape\"" } )
    {
        std::string result;
        std::cout << "-------\n" << "\n";
        std::cout << "input:  '" << input << "'\n";
        try {
            parse( input.cbegin(), input.cend(), qs, result );
            std::cout << "result: '" << result << "'\n";
        } catch(std::exception const& e) {
            std::cout << "error:  " << e.what() << "\n";
        }
    }
}

Output:

-------

input:  '"plain string"'
result: 'plain string'
-------

input:  '"hex\x20escape"'
result: 'hex escape'
-------

input:  '"oct\40escape (two digits)"'
result: 'oct escape (two digits)'
-------

input:  '"oct\040escape (two digits, leading zero)"'
result: 'oct escape (two digits, leading zero)'
-------

input:  '"oct \105scape (three digits)"'
result: 'oct Escape (three digits)'
-------

input:  '"unterminated string'
error:  Illegal string literal. (Unterminated string?)
-------

input:  '"embedded\nnewline"'
result: 'embedded
newline'
-------

input:  '"4-digit unicode escape"'
result: '4-digit unicode escape'
-------

input:  '"8-digit unicode escape"'
result: '8-digit unicode escape'
-------

input:  '"illegal \character escape"'
error:  Illegal escape sequence. Expecting <alternative><symbols><expect>"x"<unsigned-integer><expect>"u"<unsigned-integer><expect>"U"<unsigned-integer><unsigned-integer> here: "character escape""
-------

input:  '"illegal \xhex escape"'
error:  Illegal escape sequence. Expecting <unsigned-integer> here: "hex escape""
-------

input:  '"illegal \8ct escape"'
error:  Illegal escape sequence. Expecting <alternative><symbols><expect>"x"<unsigned-integer><expect>"u"<unsigned-integer><expect>"U"<unsigned-integer><unsigned-integer> here: "8ct escape""
-------

input:  '"illegal \unicode escape"'
error:  Illegal escape sequence. Expecting <unsigned-integer> here: "nicode escape""
-------

input:  '"illegal \Unicode escape"'
error:  Illegal escape sequence. Expecting <unsigned-integer> here: "nicode escape""
\$\endgroup\$
  • \$\begingroup\$ coliru doesn't readily provide libicu, it seems. Note that Boost Regex contains utf8 converting iterators as well. \$\endgroup\$ – sehe Aug 31 '15 at 12:52
  • 1
    \$\begingroup\$ Boost.Regex relies on ICU in turn for its Unicode capabilities, and asserting that Boost has been compiled with ICU support on various targets has been a royal PITA for me in the past. That's why I prefer going through ICU directly. That part about Phoenix being required because otherwise the error would be thrown during grammar construction has been a real eye-opener. Seems like that's been the part that kept me from really understanding Spirit so far. This will take some digesting though. Much obliged! \$\endgroup\$ – DevSolar Aug 31 '15 at 13:07
  • \$\begingroup\$ Getting back to cp2utf8... that BOOST_PHOENIX_ADAPT_FUNCTION() macro looks nice. However, it fails to compile: error: cp2utf8_ has not been declared / my_helpers::cp2utf8 is not a type / expected identifier before numeric constant...? (g++ 4.8.4, in case that matters.) \$\endgroup\$ – DevSolar Sep 1 '15 at 7:59
  • 1
    \$\begingroup\$ If at all, it's probably more dependent on Boost version. Try #define BOOST_SPIRIT_USE_PHOENIX_V3 or include the function.hpp header from Boost Phoenix. \$\endgroup\$ – sehe Sep 1 '15 at 8:20
  • \$\begingroup\$ The #define did the trick. (Boost version was 1.54.) \$\endgroup\$ – DevSolar Sep 1 '15 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.