Frustrated by the lack of syntax highlighting on Stack Exchange sites for code snippets written in x86 assembly language, I decided to do something about it. The network uses Google Code Prettify as its syntax-highlighting engine, so I wrote a new module that adds support for this language to Prettify.

The Prettify project's (limited) documentation says this is as simple as adding a new file following the lang-*.js pattern, so that's is what I did, following the model of the other language files included with the library. Here is the final result:

lang-x86.js

/**
*
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
*
* Unless required by applicable law or agreed to in writing, software
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
*/

/**
* @fileoverview
* Registers a language handler for x86 assembly language,
* supporting both Intel/MASM and AT&T/GAS syntax.
*
* Note that some of the more esoteric syntax and features of these assemblers
* is not supported. The highlighting focuses on constructs that would be found
* in typical code samples. All constructs are matched without respect to case.
*
* This file is an amalgamation of patterns taken from the other lang-*.js files
* included with this library, as well as novel patterns written by the author.
*
* The authoritative documentation for the registers and instruction mnemonics is
* the "Intel IA-32 Architecture Software Developer's Manual", available online
* at: https://software.intel.com/en-us/articles/intel-sdm.
* Documentation for Microsoft's assembler (MASM) is available online here:
*   https://msdn.microsoft.com/en-us/library/afzk3475.aspx.
* Documentation for the GNU assembler 'as' (GAS) is available online here:
*   https://sourceware.org/binutils/docs/as/.
*
* To use, include prettify.js and this file in your HTML page.
* Then, put your code in an HTML tag like:
*      <pre class="prettyprint lang-x86">...</pre>
*
* @author cody@<myfullname>.com
*/

PR['registerLangHandler'](
PR['createSimpleLexer']([], [
// Comments begin with either ';' (Intel/MASM) or '#' (AT&T/GAS),
// and extend to the end of that line.
[PR['PR_COMMENT'],      /^[;#].*/],

// Instruction mnemonics are like "names" for attributes.
// Unfortunately, regular expressions don't help us much here: we have to exhaustively enumerate every possibility.
// However, we can use a regex to match AT&T/GAS's suffixes, albeit at the risk of potentially matching instructions
// that are not valid (and this is especially true for those with alternate encodings in Intel vs. AT&T syntax).
// (Base)
// (Base, x86-64)
[PR['PR_ATTRIB_NAME'],  /^\s(?:cdqe|cqo|cmpsq|cmpxchg16|iretq|jrcxz|lodsq|movsxd|popfq|pushfq|rdtscp|scasq|stosq|swapgs)[bwlq]?\b/i],
// (Jumps/Branches)
[PR['PR_ATTRIB_NAME'],  /^\sj(?:mp|cxz|ecxz|o|no|s|ns|e|z|ne|nz|b|nae|c|nb|ae|nc|be|na|a|nbe|l|nge|ge|nl|le|ng|g|nle|p|pe|po|np)\b/i],
// (Other conditional instructions)
[PR['PR_ATTRIB_NAME'],  /^\s(?:set|cmov)(?:o|no|s|ns|e|z|ne|nz|b|nae|c|nb|ae|nc|be|na|a|nbe|l|nge|ge|nl|le|ng|g|nle|p|pe|po|np)[bwlq]?\b/i],
[PR['PR_ATTRIB_NAME'],  /^\sfcmov(?:b|e|be|u|nb|ne|nbe|nu)\b/i],
// (x87 FPU)
// (MMX)
// (SSE)
// (SSE2)
// (SSE3)
// (SSSE3)
// (SSE4.1)
// (SSE4.2)
[PR['PR_ATTRIB_NAME'],  /^\s(?:crc32|pcmpestri|pcmpestrm|pcmpistri|pcmpistrm|pcmpgtq|popcnt)[bwlq]?\b/i],
// (BMI1)
[PR['PR_ATTRIB_NAME'],  /^\s(?:andn|bextr|blsi|blsmsk|blsr|lzcnt|tzcnt)[bwlq]?\b/i],
// (BMI2)
[PR['PR_ATTRIB_NAME'],  /^\s(?:bzhi|mulx|pdep|pext|rorx|sarx|shrx|shlx)[bwlq]?\b/i],
// (AVX)
// (FMA)

// Registers are like the corresponding "values" for attributes.
// REVIEW: Would it be better (simpler to match and thus more performant)
//         for this to simply be an exhaustive list of candidates?
[PR['PR_ATTRIB_VALUE'], /^%?\b(?:(?:dil|sil|bpl)|(?:[re]?(?:ax|bx|cx|dx|di|si|sp|bp|ip))|(?:(?:a|b|c|d|sp|bp)[lh]+)|(?:[cdefgs]+s)|(?:r([8-9]|1[0-5])[blwd]?)|(?:[xy]?mm[0-7]))\b/i],
// The x87 FPU registers are matched using a separate expression because of the optional parentheses.
// For some reason, "/^%?\b(?:st$$?[0-7]?$$?)/i" doesn't highlight the trailing parenthesis.
[PR['PR_ATTRIB_VALUE'], /^%?\b(?:st?[0-7]?)\W/i], // Keywords/directives relating to types are highlighted accordingly. [PR['PR_TYPE'], /^(?:(?:\bshort)|(?:\b(?:(?:s|d|sd|q|f|mm|xmm|ymm)?word|[st]?byte)(?: ptr)?)|(?:\breal(?:4|8|10)\b)|(?:\bd[bwdqft]\b)|\s\.(?:byte|short|hword|word|int|long|quad|octa|sleb128|uleb128|float|single|double|ascii|asciz|string|string8|string16))\b/i], // All other keywords/directives/pseudo-ops are highlighted as "keywords". // First, match GAS-style with the initial '.', and then match MASM. [PR['PR_KEYWORD'], /^\.(?:type|title|text|tag|subsection|struct|space|skip|size|section|rept|reloc|org|noaltmacro|model|macro|local|loc_mark_labels|loc|ln|list|line|internal|intel_syntax|include|ident|gnu_attribute|globl|global|func|fill|file|extern|exitm|eqv|equiv|equ|endm|endfunc|endef|dim|desc|def|data|altmacro|cfi_window_save|cfi_val_encoded_addr|cfi_undefined|cfi_startproc|cfi_signal_frame|cfi_sections|cfi_same_value|cfi_return_column|cfi_restore|cfi_remember_state|cfi_rel_offset|cfi_register|cfi_personality|cfi_offset|cfi_lsda|cfi_escape|cfi_endproc|cfi_def_cfa_register|cfi_def_cfa_offset|cfi_def_cfa|cfi_adjust_cfa_offset|balignw|balignl|align)\s/i], [PR['PR_KEYWORD'], /^(?:xdata|tiny|times|struct|stdcall|small|sizeof|size|segment|seg|section|rept|repeat|proto|proc|pdata|org|option|offset|npad|near|medium|macro|length|large|istruc|invoke|includelib|include|import|huge|group|global|flat|far|extrn|externdef|extern|export|exitm|even|equ|eq|endstruc|ends|endp|endm|end|dup|dosseg|const|compact|comdat|assumes|assume|alignb|align|alias|addr)|(?:[\._]+(?:text|data|code|bss))\s/i], // Any text that is followed by a colon is considered to be a label. // This is a little bit over-eager, since labels can only be a single token (no spaces), // but that's actually OK because it will match GAS-style leading function signatures. // Note that we must explicitly ignore comments on a line following a label, especially // those with embedded colons, to prevent them being treated as part of the label. // Colons inside of string & character literals are also ignored. // The beginning of line anchor ('^') does not seem to do what it is expected to do, // so '\b' is used as a workaround, but this means we need to explicitly include any // symbols that might be found preceding a label, like '.' and ''. [PR['PR_DECLARATION'], /^[._]*\b[^'":;#\r\n]*:/], // String & character literals. [PR['PR_STRING'], /^"[^"\\]*(?:\\.[^"\\]*)*"/], [PR['PR_STRING'], /^'[^'\\]*(?:\\.[^'\\]*)*'/], // Decimal and hexadecimal integer literals. // (Rarely-used octal and binary literals are not yet supported.) // The current implementation highlights the prefixes and suffixes, mostly for convenience // since I can't get positive lookaheads to work, but also because it makes sense to me to // group them semantically with the number when reading code. However, this is up for // debate and may need to be reconsidered/ [PR['PR_LITERAL'], /^(?=$?)0x[\da-f]+/i], // "0x" prefix [PR['PR_LITERAL'], /^\d[\da-f]*(?=h)/i], // MASM-style "h" suffix (must start with a decimal digit) // In Intel and certain places in AT&T syntax, decimal literals occur with no identifying prefix // or suffix (although MASM allows an optional 'd' suffix). These are tricky because they must // be matched in just about any context, except when they are embedded within another identifier // (like those commonly generated by compilers). This is handled by explicitly detecting such // identifers and marking them as "plain text" to exclude them from future matches, and then // detecting and highlighting the remaining integer literals. [PR['PR_PLAIN'], /^(?:[a-z_\.]+\d+)|(?:\d+[a-z_\.]+)/i], // Also, although we normally don't want to highlight any preceding symbols (like arithmetic // operators) as being part of numeric literals, we *do* need to highlight the leading // negative sign for negative integer literals. // BUG: This matches the negative sign in expressions like "[eax-4]", where the operator // actually denotes subtraction. A simple workaround is to add a space: "[eax - 4]". [PR['PR_LITERAL'], /^(?=$?)\-?\d+(?=d?)(?![^\s\](),\+\-*])/], // Significant punctuation: grouping, comma, and arithmetic operators. [PR['PR_PUNCTUATION'], /^[\(,+\-*?]/],
]),
['x86']);


This works perfectly when I test it locally, but before I submit a pull request, I wanted to get my work reviewed. The code consists primarily of regular expressions inside a JavaScript wrapper—two technologies in which I am only barely competent! Therefore, I am looking for:

• general advice on improving the code,
• specific ways to optimize the regular expressions for maximum performance without sacrificing functionality,
• anywhere that I did something incorrectly or non-idiomatically.

However, most of the JavaScript that comprises the file's general structure is not up for review, as these conventions were established by the Prettify project and are not things that I can change. (See the existing language modules here.)

The biggest thing that I'm unhappy with is having to provide a nearly-exhaustive list of all the instruction opcodes/mnemonics. This severely bloats the code, which is a concern since it is downloaded with each page view. It also increases the maintenance burden as new instructions are added to future processor generations, which happens much faster than keywords are added to other languages. But I don't see a better approach. Although I've taken a couple of shortcuts at the expense of perfect matching accuracy in order to use regular expressions to their fullest potential, there are a lot of instructions that need to be supported and their syntax doesn't follow a regular pattern. If these aren't recognized and highlighted, they will be difficult to distinguish from user-defined identifiers, defeating much of the point.

What is especially tricky about this compared to other "languages" is, in order for it to be most useful, two different types of syntax must be supported: Intel/MASM and AT&T/GAS. Simple yet representative examples of both are shown below (the code is nonsense, with intentionally inconsistent formatting):

CONST SEGMENT
CONST ENDS

_TEXT SEGMENT
?TestCode PROC
xor    eax, eax
2:   push   eax
8:   push   0x6e696228
d:   mov    ebx,esp        ; save current stack pointer
or     al,-1
13:  int    21h            ; call interrupt
ret    0
?TestCode ENDP
_TEXT ENDS

.global _start

.text
_start:
movl  $4, %eax # put some stuff in registers xorl$1,%bx
movl  $msg,%ecx movl MsgLength, %edx int$0x80               # call interrupt

.data
msg:
.ascii  "Hello, world!\n"
MsgLength:      .long 14


Though too long to reproduce in full in this question, more complete samples that utilize a variety of instructions and syntactical constructs are included in this live demo of the code for testing purposes. These code samples are cleaned-up output from major compilers, so if the syntax highlighting works here, it should work perfectly on the relatively simple code samples that might be displayed on a web page.

• This is gorgeous! – Der Kommissar Jan 17 '17 at 6:48
• You should not get to worried about RegExp performance. The parsing of any code via regexp will take much less time than updating the DOM with the new styles (That is the real bottleneck). I would also suggest that you not write the lang-???.js files directly but create it from a higher level data source (something a little more human readable). This will make it easier to stay up to date, allow you apply optimisations (for regexp) that are hard for us humans to spot, and save you a ton of hard fiddly work. – Blindman67 Jan 17 '17 at 23:35
• Oh, that's a good point, @blindman. I didn't think about the bottleneck being DOM manipulation. Still, I imagine that, like any language, it's possible for someone who is a complete novice to write something so pathologically sub-optimal that it might matter, which is why I asked. As for creating from a higher-level data source, that's intriguing, but I wouldn't know where to begin. I also am hesitant to try and convince the Google Prettify maintainers to accept a sub-project that generates this one language module, so I'd have to maintain that as a separate project myself. Is that worth it? – Cody Gray Jan 18 '17 at 7:20
• What I think would make this more readable would be to break up the regexes onto multiple lines and/or to introduce variables that store the long lists of identifiers (and then concatenate those when building the regex. Either of those would go a long way towards increasing readability and maintainability, without vastly increasing the cost or complexity, but I couldn't find a way to do this. – Cody Gray Jan 18 '17 at 7:21
• Having had another look and a few experiments suggests that the order of the different types is important. When items have been matched they will not be matched again even if they match other regexps . Thus comments first ensures all comments are safe from further matches. Then in order of easy to match to hardest Keywords, types, literals, punctuation, and so on. In the end you are left with the mnemonics and registers which you can class on position more so than textual content. – Blindman67 Jan 18 '17 at 11:49

What is especially tricky about this compared to other "languages" is, in order for it to be most useful, two different types of syntax must be supported: Intel/MASM and AT&T/GAS

Why does one configuration have to handle two different syntax styles? On the bases of KISS, of doing one thing well, and of potential ambiguities I would think it makes more sense to register two handlers (perhaps x86-masm and x86-gas), even if they share regexes for opcodes.

specific ways to optimize the regular expressions for maximum performance

That depends on the regex engines, which might do more or fewer optimisations themselves, but basically you want to reduce backtracking. The obvious way you can reduce backtracking for a set of literal strings such as the opcodes is to turn them into a tree. E.g. instead of

[PR['PR_ATTRIB_NAME'],  /^\s(?:crc32|pcmpestri|pcmpestrm|pcmpistri|pcmpistrm|pcmpgtq|popcnt)[bwlq]?\b/i],


you could write

[PR['PR_ATTRIB_NAME'],  /^\s(?:crc32|pcmp(estr[im]|istr[im]|gtq)|popcnt)[bwlq]?\b/i],


That isn't quite a tree, and could be taken further, but backtracking over the p shared by pcmp(...) and popcnt isn't nearly as bad as testing pcmpestr twice (once for pcmpestri and once for pcmpestrm) if someone decides to use pcmpestrz as a name. There is an argument for combining the pcmp?str? as

[PR['PR_ATTRIB_NAME'],  /^\s(?:crc32|pcmp([ei]str[im]|gtq)|popcnt)[bwlq]?\b/i],


One thing which puzzles me about that regex is the (?: I'm sure there's a reason for wanting not to capture the keyword, but it's odd. However, looking at one of the examples you list (specifically lang-ml.js) I note that they're using lookaround for the whole keyword. In the example I picked above (more or less at random), the [bwlq]? is outside the lookaround. I find that suspicious, and I suggest that you double-check the lookarounds.
• (3) I used the non-capturing syntax because I read a couple of places online that it was more optimal than the capturing syntax. All I really needed was a grouping effect. I thought a lookaround was either the lookahead or lookbehind syntax (the latter, I believe, not supported by JS regexes), and although I do use lookaheads in a few places, the regexes that have [bwlq]?are not one of those places. So I'm not really sure what you mean to say here, or how to double-check it. As I understand it, (?: is not a lookaround. – Cody Gray Jan 17 '17 at 12:04