1
\$\begingroup\$

The Unicode Consortium has actually defined a specification for

the provision of a parsing and/or lexing engine for identifiers, such as programming language variables or domain names.

(Unicode® Standard Annex #31, Unicode Identifier and Pattern Syntax)

Since I'm currently making a language of my own, I want to follow the Unicode standard. Then I don't have to decide which characters are valid in an identifier and which aren't. 🙃

Since I'm using to create my lexer/parser, though, I needed to teach ANTLR to create an identifier along with the standard. The basic rule is simple enough:

Identifier : XID_Start XID_Continue ;

However, XID_Start and XID_Continue are a little more complicated, as ANTLR doesn't know anything about Unicode character properties. So, I wrote a script using ICU4J to generate lexer fragments containing the correct characters. It's complicated a bit by the fact that ANTLR talks UTF-16 (because Java does), but this script does create valid antlr code as output.

import com.ibm.icu.lang.UCharacter
import com.ibm.icu.lang.UProperty
import com.ibm.icu.text.UTF16
import com.ibm.icu.text.UnicodeSet
import java.io.File

fun main(args: Array<String>) {
    val (xidStart, xidContinue) = generateXIDSets()

    assert(
            xidContinue.containsAll(xidStart)
    ) { "XID_Start contains characters not in XID_Continue. This shouldn't be possible." }

    File("src/xid.g4").writeText(
            """// This grammar provides fragments for the XID_Start and XID_Continue Unicode character properties.
              !lexer grammar xid;
              !
              !fragment XID_Start:
              !    ${antlrify(xidStart)}
              !
              !fragment XID_Continue:
              !    XID_Start
              !  | ${antlrify(xidContinue.cloneAsThawed().removeAll(xidStart))}
              !""".trimMargin("!")
    )
    println("OK!")
}

data class XIDSets(val xidStart: UnicodeSet = UnicodeSet(), val xidContinue: UnicodeSet = UnicodeSet())

fun generateXIDSets(): XIDSets {
    val xidSets = XIDSets()

    (0..0x10FFFF)
            .filter { UCharacter.hasBinaryProperty(it, UProperty.XID_START) }
            .forEach { xidSets.xidStart.add(it) }
    xidSets.xidStart.compact().freeze()

    (0..0x10FFFF)
            .filter { UCharacter.hasBinaryProperty(it, UProperty.XID_CONTINUE) }
            .forEach { xidSets.xidContinue.add(it) }
    xidSets.xidContinue.compact().freeze()

    return xidSets
}

fun antlrify(set: UnicodeSet): String {
    val ranges = set.ranges()
            .map { entryRange -> entryRange.codepoint to entryRange.codepointEnd }
            .toList()

    val bmp = ranges
            .filter { (first, second) -> first <= 0xffff && second <= 0xffff }
            .map { (first, second) ->
                if (first == second) {
                    "'\\u%04x'".format(first)
                } else {
                    "'\\u%04x' .. '\\u%04x'".format(first, second)
                }
            }
    val emp = ranges
            .filter { (first, second) -> first > 0xffff || second > 0xffff }
            .map { (first, second) ->
                if (first == second) {
                    "'\\u%04x\\u%04x'".format(
                            UTF16.getLeadSurrogate(first).toInt(),
                            UTF16.getTrailSurrogate(first).toInt()
                    )
                } else {
                    val firstLeadSurrogate = UTF16.getLeadSurrogate(first).toInt()
                    val secondLeadSurrogate = UTF16.getLeadSurrogate(second).toInt()
                    if (firstLeadSurrogate == secondLeadSurrogate) {
                        "'\\u%04x' '\\u%04x' .. '\\u%04x'".format(
                                firstLeadSurrogate,
                                UTF16.getTrailSurrogate(first).toInt(),
                                UTF16.getTrailSurrogate(second).toInt()
                        )
                    } else {
                        assert(secondLeadSurrogate == firstLeadSurrogate + 1)
                        "'\\u%04x' '\\u%04x' .. '\\u%04x' | '\\u%04x' '\\u%04x' .. '\\u%04x'".format(
                                firstLeadSurrogate,
                                UTF16.getTrailSurrogate(first).toInt(),
                                UCharacter.MAX_LOW_SURROGATE.toInt(),
                                secondLeadSurrogate,
                                UCharacter.MIN_LOW_SURROGATE.toInt(),
                                UTF16.getTrailSurrogate(second).toInt()
                        )
                    }
                }
            }

    return (bmp + emp).joinToString(
            separator = "\n  | ",
            postfix = "\n  ;"
    )
}

I'll take feedback on the Kotlin code or the generated ANTLR code. Notes on idiomatic style and correctness are ideal, though anything you have to say is appreciated! I'd like to be able to claim conformance to UAX-R1 (version 25), so another pair of eyes checking for mistakes would be great.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.