3
\$\begingroup\$

I've been working on an ANTLR grammar that defines some tokens in terms of Unicode character categories (fileformat.info page). The Unicode Consortium makes a full data tab-separated-values .txt available at http://www.unicode.org/notes/tn36/Categories.txt.

The below scripts work together to take Categories.txt and turn it into ANTLR Lexer fragments. extract_fragment.py Takes a character class code and turns it into a fragment, and gen_fragments.py uses the former script to generate a full ANLTR .g4 for all character categories.

extract_fragment.py

# Categories.txt from http://www.unicode.org/notes/tn36/Categories.txt

from sys import argv

ANTLR_UNICODE_TOKEN_FORMAT = "'\\u{:04X}'"
TEN_BITS = 2 ** 10 - 1


def is_bmp(point):
    """
    Is a character on the Basic Multilingual Plane?
    (Can it be represented as one UTF-16 code point.)
    :param point: The character to check
    :return: truthy if the character lies on the BMP, falsy otherwise
    """
    return ord(point) < 0x10000


def high_surrogate(point):
    if is_bmp(point):
        return None
    return 0xD800 + (TEN_BITS & (ord(point) - 0x10000 >> 10))


def low_surrogate(point):
    if is_bmp(point):
        return None
    return 0xDC00 + (TEN_BITS & (ord(point) - 0x10000))


def codepoints_from_for(file, code):
    """
    Extract the characters of a given category from the source tsv.
    :param file: The tab separated values file-like-object with category information, closed on finish
    :param code: The character code to look for
    :return: Generator of characters with the given category
    """
    for line in file:
        info = line.split('\t')
        if info[1] == code:
            yield chr(int(info[0], base=16))
    file.close()


def codepoints_to_tuples(points):
    """
    Collapse individual characters into range tuples.
    Will never construct a range across changing first byte of surrogate pairs.
    :param points: the characters to collapse
    :return: Generator of (chr, chr) ranges from the param
    """
    pair = [points[0], points[0]]
    for point in points[1:]:
        if ord(pair[1]) + 1 == ord(point) and \
                (is_bmp(point) or pair[1].encode('utf-16')[0] == point.encode('utf-16')[0]):
            pair[1] = point
        else:
            yield tuple(pair)
            pair = [point, point]
    yield tuple(pair)


def token_for(pair):
    if is_bmp(pair[0]):
        if pair[0] == pair[1]:
            return "{}".format(ANTLR_UNICODE_TOKEN_FORMAT) \
                .format(ord(pair[0]))
        else:
            return "{0}..{0}".format(ANTLR_UNICODE_TOKEN_FORMAT) \
                .format(ord(pair[0]),
                        ord(pair[1]))
    else:
        if pair[0] == pair[1]:
            return "{}{}".format(ANTLR_UNICODE_TOKEN_FORMAT[:-1], ANTLR_UNICODE_TOKEN_FORMAT[1:]) \
                .format(high_surrogate(pair[0]),
                        low_surrogate(pair[0]))
        else:
            return "{0} {0}..{0}".format(ANTLR_UNICODE_TOKEN_FORMAT) \
                .format(high_surrogate(pair[0]),
                        low_surrogate(pair[0]),
                        low_surrogate(pair[1]))


def extract(code):
    codepoints = list(codepoints_from_for(open('Categories.txt', 'r'), code))

    if len(codepoints) == 0:
        return "// No characters with category code {}".format(code)

    codepoint_ranges = codepoints_to_tuples(codepoints)

    lexer_tokens = list(map(token_for, codepoint_ranges))

    output = "fragment {} : {}".format(code, lexer_tokens.pop(0))
    line_length = len(output)

    for token in lexer_tokens:
        if line_length + 3 + len(token) > 120:
            output += "\n           "
            line_length = 11
        output += " | {}".format(token)
        line_length += 3 + len(token)

    if line_length + 2 > 120:
        output += "\n           "
    output += " ;"

    return output


def main():
    code = argv[1]
    extract(code)


if __name__ == '__main__':
    main()

gen_fragments.py

import sys

from scripts.extract_fragment import extract


def main(name):
    if name:
        print('// Generated by {}'.format(__file__[__file__.rfind('scripts'):]))
        print()
        print('lexer grammar {};'.format(name))
        print()

    print('// [C] Other')
    print('fragment C  : Cc | Cf | /* Cn | Co | Cs */ ;')
    print()

    print('// [Cc] Other, Control')
    print(extract('Cc'))
    print()

    print('// [Cf] Other, Format')
    print(extract('Cf'))
    print()

    print('// [Cn] Other, Not Assigned')
    print(extract('Cn'))
    print()

    print('// [Co] Other, Private Use')
    print(extract('Co'))
    print()

    print('// [Cs] Other, Surrogate')
    print(extract('Cs'))
    print()

    print('// [L] Letter')
    print('fragment L  : /* LC | */ Ll | Lm | Lo | Lt | Lu ;')
    print()

    print('// [LC] Letter, Cased')
    print(extract('LC'))
    print()

    print('// [Ll] Letter, Lowercase')
    print(extract('Ll'))
    print()

    print('// [Lm] Letter, Modifier')
    print(extract('Lm'))
    print()

    print('// [Lo] Letter, Other')
    print(extract('Lo'))
    print()

    print('// [Lt] Letter, Titlecase')
    print(extract('Lt'))
    print()

    print('// [Lu] Letter, Uppercase')
    print(extract('Lu'))
    print()

    print('// [M] Mark')
    print('fragment M  : Mc | Me | Mn ;')
    print()

    print('// [Mc] Mark, Spacing Combining')
    print(extract('Mc'))
    print()

    print('// [Me] Mark, Enclosing')
    print(extract('Me'))
    print()

    print('// [Mn] Mark, Nonspacing')
    print(extract('Mn'))
    print()

    print('// [N] Number')
    print('fragment N  : Nd | Nl | No ;')
    print()

    print('// [Nd] Number, Decimal Digit')
    print(extract('Nd'))
    print()

    print('// [Nl] Number, Letter')
    print(extract('Nl'))
    print()

    print('// [No] Number, Other')
    print(extract('No'))
    print()

    print('// [P] Punctuation')
    print('fragment P  : Pc | Pd | Pe | Pf | Pi | Po | Ps ;')
    print()

    print('// [Pc] Punctuation, Connector')
    print(extract('Pc'))
    print()

    print('// [Pd] Punctuation, Dash')
    print(extract('Pd'))
    print()

    print('// [Pe] Punctuation, Close')
    print(extract('Pe'))
    print()

    print('// [Pf] Punctuation, Final quote')
    print(extract('Pf'))
    print()

    print('// [Pi] Punctuation, Initial quote')
    print(extract('Pi'))
    print()

    print('// [Po] Punctuation, Other')
    print(extract('Po'))
    print()

    print('// [Ps] Punctuation, Open')
    print(extract('Ps'))
    print()

    print('// [S] Symbol')
    print('fragment S  : Sc | Sk | Sm | So ;')
    print()

    print('// [Sc] Symbol, Currency')
    print(extract('Sc'))
    print()

    print('// [Sk] Symbol, Modifier')
    print(extract('Sk'))
    print()

    print('// [Sm] Symbol, Math')
    print(extract('Sm'))
    print()

    print('// [So] Symbol, Other')
    print(extract('So'))
    print()

    print('// [Z] Separator')
    print('fragment Z  : Zl | Zp | Zs ;')
    print()

    print('// [Zl] Separator, Line')
    print(extract('Zl'))
    print()

    print('// [Zp] Separator, Paragraph')
    print(extract('Zp'))
    print()

    print('// [Zs] Separator, Space')
    print(extract('Zs'))


if __name__ == '__main__':
    if sys.argv[1]:
        sys.stdout = open(sys.argv[1], 'w')
        main(sys.argv[1][sys.argv[1].rfind('\\') + 1:sys.argv[1].find('.')])
    else:
        main(None)

The output can be found here. The full project is available on GitHub, if you're interested. I'm specifically looking to reduce repetition in gen_fragments.py, but all potential improvements are welcome.

\$\endgroup\$

2 Answers 2

2
\$\begingroup\$
print('fragment N  : Nd | Nl | No ;')

It probably doesn't matter, but there's an extraneous whitespace there, before the colon. And the whitespace before the semicolon isn't needed either, so you could output like this:

print('fragment N : Nd | Nl | No;')

Now, your script is generating fragment rules for a lexer grammar: I would expect to see a function whose role it is to output the fragment part.

I don't do any , so consider this pseudo-code:

def printFragment(token, rule, comment)
    print('// ' + comment)
    print('fragment ' + token + ' : ' + rule + ';')
    print()

So instead of this:

print('// [C] Other')
print('fragment C  : Cc | Cf | /* Cn | Co | Cs */ ;')
print()

You could have that:

printFragment('C', 'Cc | Cf | /* Cn | Co | Cs */', '[C] Other')

Similarly, there could be a function dedicated to printing the extract output:

printExtracted('Cc', '[Cc] Other, Control')
printExtracted('Cf', '[Cf] Other, Format')
printExtracted('Cn', '[Cn] Other, Not Assigned')
printExtracted('Co', '[Co] Other, Private Use')
printExtracted('Cs', '[Cs] Other, Surrogate')

Looking at what extract does, it seems it could reuse that printFragment function... the idea is to try to find a way to write a function that prints a single lexer token's definition, in a standard way.

Consider putting the comments on the same line - this is a legal fragment definition:

fragment C : Cc | Cf /*| Cn | Co | Cs */;           // [C] Other

Note I moved the /* marker up one character here, so as to avoid that empty alternative fragment C : Cc | Cf | ??? - not sure if that makes ANTLR warn or complain in any way, but it's always best to avoid grammar problems that can be avoided ;-)

If you can make all the comments aligned, the resulting lexer grammar will look pretty neat, with one fragment per line.

\$\endgroup\$
2
\$\begingroup\$

Even though it is already quite good that you abstract most of the functionality using magic constants into their own function. However, it would help a lot if you gave all those magic constants a name and defined them up top. This way the code becomes easier to read and should the spec ever change, there is only one place where you have to change them.

This regards 0x10000, 0xD800 and 0x10000.

Another thing: You overwrite the already defined sys.stdout. This is probably not a very good idea. In addition, you open the file for writing, but never close it. Use with..as to ensure the file is properly closed (even when aborting the program with CTRL-C):

with open(sys.argv[1], 'w') as file_out:
    start = sys.argv[1].rfind('\\') + 1
    end = sys.argv[1].find('.')
    main(sys.argv[1][start:end])

I also moved that in-line calculation of the start and end indices out into a variable, because otherwise it is really hard to understand.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.