Converting Markdown to HTML using Go

Question

I recently ported a blog of mine from Python to Go (to improve speed and performance) and while all is great so far, I'd like some help optimising the Markdown function to improve the general performance, maintenance and readability of the function.

I have this function because I write my blog articles in Markdown (.md) and then use Python Go to convert the raw Markdown to HTML for output as this saves me from having to write ridiculous amounts of HTML. (which can be tedious to say the least)

The Markdown function takes one argument (raw) which is a string that contains the raw Markdown (obtained using ioutil.ReadFile).

It then splits the Markdown by \n (removing the empty lines) and converts:

• Bold and italic text (***,**,*)
• Strikethrough text (~~blah blah blah~~)
• Underscored text (__blah blah blah__)
• Links ([https://example.com](Example Link))
• Blockquotes (> sample quote by an important person)
• Inline code (`abcccc`)
• Headings (h1-h6)

While some of the supported features aren't exactly standard, this function works and outputs the expected result without any errors but being a new Go programmer and this being my first "real" Go project I'd like to know whether or not my code could be optimised for better performance, maintainability and readability.

Here a few questions I have regarding optimisation:

• Would it make a difference to performance if I reduced the amount of imports?
• Would it improve readability if I put the regexp.MustCompile functions into variables above the Markdown function?
• Would it improve performance if I used individual regexes to convert Markdown headings instead of using for i := 6; i >= 1; i-- {...}?
• If not, is there a way to convert i (an integer) to a string without using strconv.Itoa(i) (to help reduce the amount of imports)?

Here is my code:

package parse

import (
  "regexp"
  "strings"
  "strconv"
)

func Markdown(raw string) string {
  // ignore empty lines with "string.Split(...)"
  lines := strings.FieldsFunc(raw, func(c rune) bool {
    return c == '\n'
  })
  for i, line := range lines {
    // wrap bold and italic text in "<b>" and "<i>" elements
    line = regexp.MustCompile(`\*\*\*(.*?)\*\*\*`).ReplaceAllString(line, `<b><i>$1</i></b>`)
    line = regexp.MustCompile(`\*\*(.*?)\*\*`).ReplaceAllString(line, `<b>$1</b>`)
    line = regexp.MustCompile(`\*(.*?)\*`).ReplaceAllString(line, `<i>$1</i>`)
    // wrap strikethrough text in "<s>" tags
    line = regexp.MustCompile(`\~\~(.*?)\~\~`).ReplaceAllString(line, `<s>$1</s>`)
    // wrap underscored text in "<u>" tags
    line = regexp.MustCompile(`__(.*?)__`).ReplaceAllString(line, `<u>$1</u>`)
    // convert links to anchor tags
    line = regexp.MustCompile(`\[(.*?)\]\((.*?)\)[^\)]`).ReplaceAllString(line, `<a href="$2">$1</a>`)
    // escape and wrap blockquotes in "<blockquote>" tags
    line = regexp.MustCompile(`^\>(\s|)`).ReplaceAllString(line, `&gt;`)
    line = regexp.MustCompile(`\&gt\;(.*?)$`).ReplaceAllString(line, `<blockquote>$1</blockquote>`)
    // wrap the content of backticks inside of "<code>" tags
    line = regexp.MustCompile("`(.*?)`").ReplaceAllString(line, `<code>$1</code>`)
    // convert headings
    for i := 6; i >= 1; i-- {
      size, md_header := strconv.Itoa(i), strings.Repeat("#", i)
      line = regexp.MustCompile(`^` + md_header + `(\s|)(.*?)$`).ReplaceAllString(line, `<h` + size + `>$2</h` + size + `>`)
    }
    // update the line
    lines[i] = line
  }
  // return the joined lines
  return strings.Join(lines, "\n")
}

Answer 1

Performance

Regex

regex.MustCompile() is very expensive! Do not use this method inside a loop !

instead, define your regex as global variables only once:

var (
    boldItalicReg = regexp.MustCompile(`\*\*\*(.*?)\*\*\*`)
    boldReg       = regexp.MustCompile(`\*\*(.*?)\*\*`)
    ...
)

Headers

If a line is a header, it will start by a #. We can check for this before calling ReplaceAllString() 6 times ! All we need to do is to trim the line, and then check if it starts with #:

line = strings.TrimSpace(line)
if strings.HasPrefix(line, "#") {
    // convert headings
    ...
}

We could go further and unrolling the loop to avoid unecessary allocations:

count := strings.Count(line, "#")
switch count {
case 1:
    line = h1Reg.ReplaceAllString(line, `<h1>$2</h1>`)
case 2: 
    ...
}

Use a scanner

The idiomatic way to read a file line by line in go is to use a scanner. It takes an io.Reader as parameters, so you can directly pass your mardown file instead of converting it into a string first:

func NewMarkdown(input io.Reader) string {


    scanner := bufio.NewScanner(input)
    for scanner.Scan() {

        line := scanner.Text()
        ...
    }
}

Use []byte instead of string

In go, a string is a read-only slice of bytes. Working with strings is usually more expensive than working with slice of bytes, so use []byte instead of strings when you can:

line := scanner.Bytes()
line = boldItalicReg.ReplaceAll(line, []byte(`<b><i>$1</i></b>`))

Write result to a bytes.Buffer

Instead of string.Join(), we can use a buffer to write each line in order to further reduce the number of allocations:

buf := bytes.NewBuffer(nil)
scanner := bufio.NewScanner(input)
for scanner.Scan() {

    line := scanner.Bytes()
    ...
    buf.Write(line)
    buf.WriteByte('\n')
}

return buf.String()

final code:

package parse

import (
    "bufio"
    "bytes"
    "io"
    "regexp"
)

var (
    boldItalicReg = regexp.MustCompile(`\*\*\*(.*?)\*\*\*`)
    boldReg       = regexp.MustCompile(`\*\*(.*?)\*\*`)
    italicReg     = regexp.MustCompile(`\*(.*?)\*`)
    strikeReg     = regexp.MustCompile(`\~\~(.*?)\~\~`)
    underscoreReg = regexp.MustCompile(`__(.*?)__`)
    anchorReg     = regexp.MustCompile(`\[(.*?)\]\((.*?)\)[^\)]`)
    escapeReg     = regexp.MustCompile(`^\>(\s|)`)
    blockquoteReg = regexp.MustCompile(`\&gt\;(.*?)$`)
    backtipReg    = regexp.MustCompile("`(.*?)`")

    h1Reg = regexp.MustCompile(`^#(\s|)(.*?)$`)
    h2Reg = regexp.MustCompile(`^##(\s|)(.*?)$`)
    h3Reg = regexp.MustCompile(`^###(\s|)(.*?)$`)
    h4Reg = regexp.MustCompile(`^####(\s|)(.*?)$`)
    h5Reg = regexp.MustCompile(`^#####(\s|)(.*?)$`)
    h6Reg = regexp.MustCompile(`^######(\s|)(.*?)$`)
)

func NewMarkdown(input io.Reader) string {

    buf := bytes.NewBuffer(nil)

    scanner := bufio.NewScanner(input)
    for scanner.Scan() {

        line := bytes.TrimSpace(scanner.Bytes())
        if len(line) == 0 {
            buf.WriteByte('\n')
            continue
        }

        // wrap bold and italic text in "<b>" and "<i>" elements
        line = boldItalicReg.ReplaceAll(line, []byte(`<b><i>$1</i></b>`))
        line = boldReg.ReplaceAll(line, []byte(`<b>$1</b>`))
        line = italicReg.ReplaceAll(line, []byte(`<i>$1</i>`))
        // wrap strikethrough text in "<s>" tags
        line = strikeReg.ReplaceAll(line, []byte(`<s>$1</s>`))
        // wrap underscored text in "<u>" tags
        line = underscoreReg.ReplaceAll(line, []byte(`<u>$1</u>`))
        // convert links to anchor tags
        line = anchorReg.ReplaceAll(line, []byte(`<a href="$2">$1</a>`))
        // escape and wrap blockquotes in "<blockquote>" tags
        line = escapeReg.ReplaceAll(line, []byte(`&gt;`))
        line = blockquoteReg.ReplaceAll(line, []byte(`<blockquote>$1</blockquote>`))
        // wrap the content of backticks inside of "<code>" tags
        line = backtipReg.ReplaceAll(line, []byte(`<code>$1</code>`))
        // convert headings
        if line[0] == '#' {

            count := bytes.Count(line, []byte(`#`))
            switch count {
            case 1:
                line = h1Reg.ReplaceAll(line, []byte(`<h1>$2</h1>`))
            case 2:
                line = h2Reg.ReplaceAll(line, []byte(`<h2>$2</h2>`))
            case 3:
                line = h3Reg.ReplaceAll(line, []byte(`<h3>$2</h3>`))
            case 4:
                line = h4Reg.ReplaceAll(line, []byte(`<h4>$2</h4>`))
            case 5:
                line = h5Reg.ReplaceAll(line, []byte(`<h5>$2</h5>`))
            case 6:
                line = h6Reg.ReplaceAll(line, []byte(`<h6>$2</h6>`))
            }
        }
        buf.Write(line)
        buf.WriteByte('\n')
    }
    return buf.String()
}

Benchmarks

I used the folowing code for benchmarks, on a 20kB md file:

func BenchmarkMarkdown(b *testing.B) {

    md, err := ioutil.ReadFile("README.md")
    if err != nil {
        b.Fail()
    }
    raw := string(md)
    b.ResetTimer()

    for n := 0; n < b.N; n++ {
        _ = Markdown(raw)
    }
}

func BenchmarkMarkdownNew(b *testing.B) {

    for n := 0; n < b.N; n++ {
        file, err := os.Open("README.md")
        if err != nil {
            b.Fail()
        }
        _ = NewMarkdown(file)
        file.Close()
    }
}

Results:

> go test -bench=. -benchmem

goos: linux
goarch: amd64
BenchmarkMarkdown-4               10     104990431 ns/op    364617427 B/op    493813 allocs/op
BenchmarkMarkdownNew-4          1000       1464745 ns/op      379376 B/op      11085 allocs/op

benchstat diff:

name        old time/op    new time/op    delta
Markdown-4     105ms ± 0%       1ms ± 0%   ~     (p=1.000 n=1+1)

name        old alloc/op   new alloc/op   delta
Markdown-4     365MB ± 0%       0MB ± 0%   ~     (p=1.000 n=1+1)

name        old allocs/op  new allocs/op  delta
Markdown-4      494k ± 0%       11k ± 0%   ~     (p=1.000 n=1+1)