12
\$\begingroup\$

Since I am not a fan of having using large web-frameworks and libraries just to setup a simple blog, I decided to write my own static blog site generator in pure shell script. The script uses pandoc, to convert markdown files to html files.

To test the script, check out the μblog repository. It contains documentation + some example blog files. The output should look like this:

I would like to know, what I can improve. Especially the part about sorting the posts, and extracting information seems still a bit to convoluted for my liking.

mublog.sh

#!/bin/bash

declare -A post_info
declare -a posts

dst_root_dir="dst"
dst_posts_dir="${dst_root_dir}/posts"
dst_css_dir="${dst_root_dir}/css"
dst_assets_dir="${dst_root_dir}/assets"
src_root_dir="src"
src_posts_dir="${src_root_dir}/posts"
src_css_dir="${src_root_dir}/css"
src_assets_dir="${src_root_dir}/assets"
post_ignore_delim="_"
footer_copyright="Copyright © 2023 John Doe :)"
author_mail="[email protected]"

# Description:
#    Removes old build artefacts, and generates the build directories
#    The /dst directory is the root directory of the blog
#    The /dst/posts directory contains all the blog post files
#    The /dst/assets directory stores images, videos etc of posts
#    The /dst/css directory contains the style sheets of the blog
initialize_directories() {
    
    rm -rf "$dst_root_dir"

    # Create output directories
    if mkdir -p "$dst_root_dir" &&
        mkdir -p "$dst_posts_dir" &&
        mkdir -p "$dst_css_dir" &&
        mkdir -p "$dst_assets_dir" &&
        cp "$src_css_dir"/*.css "$dst_css_dir" &&
        cp -r "$src_assets_dir/." "$dst_assets_dir/"; then
        echo "Build directories initialized."
    else
        echo "Failed to create build directories. Aborting."
        exit 1
    fi  
}

# Description:
#     Verifies presence and validity of header fields line by line.
#     If a field is not present, or its value is not valid, the variables 
#     will be set to empty. Leading and trailing whitespaces will be stripped, 
#     if present, except or the markers, where only trailing whitespace is stripped.
# Parameters:
#     $1: The path to the src post file to validate
function validate_header() {
  echo "Validating post $1 ..."
  # Line 1: Check for --- start-marker
  marker1=$(sed -n '1p' "$1" | sed 's/^---[[:space:]]*$/---/; t; s/.*//')
  # Line 2: Check for title-field
  title=$(sed -n '2p' "$1" | sed -n 's/^title:\s*\(.*\)\s*$/\1/p')
  # Line 3: Check for description-field
  description=$(sed -n '3p' "$1" | sed -n 's/^description:\s*\(.*\)\s*$/\1/p')
  # Line 4: Check for date-field with valid date in YYYY-MM-DD format
  date=$(sed -n '4p' "$1" | sed -n 's/^date:\s*\(.*\)\s*$/\1/p')
  regex='^[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1])$'
  date=$(echo "$date" | grep -P "$regex" | awk '{print $1}')
  # Line 5: Check for tags-field
  tags=$(sed -n '5p' "$1" | sed -n 's/^tags:\s*\(.*\)\s*$/\1/p')
  # Line 6: Check for --- end-marker
  marker2=$(sed -n '6p' "$1" | sed 's/^---[[:space:]]*$/---/; t; s/.*//')
  
  # Check if the header is invalid (aka, non-empty fields)
  if [ -z "$marker1" ]; then
      echo "Invalid Header: Starting markers missing or incorrect" && exit 1
  elif [ -z "$title" ]; then
      echo "Invalid Header: Title field missing or incorrect" && exit 1
  elif [ -z "$description" ]; then
    echo "Invalid Header: Description field missing or incorrect" && exit 1
  elif [ -z "$date" ]; then
      echo "Invalid Header: Date field missing, incorrect or in wrong format." && exit 1
  elif [ -z "$tags" ]; then
      echo "Invalid Header: Tags field missing or incorrect" && exit 1
  elif [ -z "$marker2" ]; then
      echo "Invalid Header: Ending marker missing or incorrect" && exit 1
  fi
}

# Description:
#     Converts the markdown post or page into html format using pandoc.
#     During this process, the header is prepended and the footer appended to the post.
# Parameters:
#     $1: The source path to the markdown post/page file
#     $2: The destination path where the converted html file will be saved.
build_pages() {

    local header="
<html>
<meta charset="utf-8">
<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">
<link rel=\"stylesheet\" href=\"/css/normalize.css\" type=\"text/css\" media=\"all\">
<link rel=\"stylesheet\" href=\"/css/style.css\" type=\"text/css\" media=\"all\">
<nav>
<a href=\"/index.html\">home</a>
<a href=\"/articles.html\">articles</a>
<a href=\"mailto:$author_mail\">mail</a>
<a href=\"/about.html\">about</a>
</nav>
<hr>"
    local footer="
</main>
<footer>
<hr>
<p>
$footer_copyright
<br/>
</p>
</footer>
</body>
</html>"

    pandoc "$1" -f markdown -t html | { echo -e "$header"; cat; echo -e "$footer"; } > "$2"
}

# Description:
#     Iterate through all source post files, and extract values stored in their headers
#     such as date, title, but also stores source path and destination path.
# Parameters:
#     $1: The path to the source directory of the posts
process_files() {
    local src_posts_dir="$1"

    # Find all .md posts in the post directory and extract info from the headers
    while IFS= read -r -d '' src_post_path; do
        if validate_header "$src_post_path"; then
            local date=$(grep -oP "(?<=date: ).*" "$src_post_path")
            local title=$(grep -oP "(?<=title: ).*" "$src_post_path")

            base_name=$(basename "$src_post_path")
            local dst_post_path="${dst_posts_dir}/${base_name%.md}.html"

            posts+=("$date|$title|$src_post_path|$dst_post_path")
        fi
    done < <(find "$src_posts_dir" -name "*.md" -print0)
}

# Description:
#     Sorts posts in reverse chronological order, based on the extracted date
sort_posts() {
    IFS=$'\n' sorted_posts=($(sort -r <<<"${posts[*]}"))
    unset IFS
}

initialize_directories
build_pages "$src_root_dir/about.md" "$dst_root_dir/about.html"
build_pages "$src_root_dir/index.md" "$dst_root_dir/index.html"
build_pages "$src_root_dir/articles.md" "$dst_root_dir/articles.html"
process_files "$src_posts_dir"
sort_posts

posts_processed=0
posts_skipped=0

article_list="<ul class=\"articles\">"
for post_info in "${sorted_posts[@]}"; do
    date=$(cut -d '|' -f 1 <<<"$post_info")
    title=$(cut -d '|' -f 2 <<<"$post_info")
    src=$(cut -d '|' -f 3 <<<"$post_info")
    dst=$(cut -d '|' -f 4 <<<"$post_info")
    dst_link=${dst#*/} 
    echo "Processing post: $src"
    echo "  title: $title"
    echo "  date: $date"
    echo "  output: $dst"
    echo "  dst_link: $dst_link"
    
    # Check if the file should be ignored (if it starts with the ignore delimter)
    filename=$(basename "$src")
    if [[ $filename == $post_ignore_delim* ]]; then
        posts_skipped=$(($posts_skipped+1))
        continue
    else
        # Build article list
        article_item="<li><b style=\"color: #14263b;\">"[${date}]"</b> <a href="\"/${dst_link}\"">${title}</a></li>"
        article_list=$article_list$article_item

        # Build post file
        build_pages "$src" "$dst"
        posts_processed=$(($posts_processed+1))
    fi
done

article_list=$article_list"</ul>"

echo "Generating article listing ..."

# Replace article tags in the article.html file with the generated article list
sed -i -e '/<article>/ {
    N
    s|<article>\(.*\)</article>|<article>\1\n'"$(sed 's/[&/\]/\\&/g' <<< "$article_list")"'\n</article>|
}' "$dst_root_dir/articles.html"

echo "Finished! (built: $posts_processed, skipped: $posts_skipped)"
\$\endgroup\$
5
  • 2
    \$\begingroup\$ I like the idea. In fact I have been working on something similar once in a while, also using bash. Instead of creating the whole web page, I think that using templates and substituting the content into it is simpler. I number the posts: post1.html, post2.html, etc. and also plunk the title into an index.html page and create a comment1.html etc. page that processes comments with a perl script. The places to insert title and content are marked with html comments: <!--Title--> I don't use pandoc. I plan to generate html text to plug in. I have a long way to go. I commend your effort. \$\endgroup\$
    – Wastrel
    Jun 24 at 15:32
  • 3
    \$\begingroup\$ Not enough for an answer, but please consider using long, descriptive error message. Your current error message of "Failed to create build directories." tells me nothing what I'd need to debug the issue. Neither which directory failed, nor why. Worse, it is misleading as also the population of the directories could have failed (meaning copying a file failed, not the creation of a directory). I know it is a pain to setup detailed error messages, but if the error occurs, the users does not need to go hunting for the information of what went wrong. \$\endgroup\$
    – Bobby
    Jun 24 at 20:23
  • 1
    \$\begingroup\$ In a similar matter, consider your log output. Make sure that the "noise" from the build does not drown out error or other important messages. An nice negative example of that is Maven, it throws so much log output at you, information you don't need to know (I don't care how many files you've compiled), that it can be hard to spot errors or warnings that you need to care about. \$\endgroup\$
    – Bobby
    Jun 24 at 20:25
  • \$\begingroup\$ @Wastrel Thanks for the nice words. Yea, I like keeping things simple. Creating a beautiful and functional blog really does not require as many things as someone might think. \$\endgroup\$
    – 766F6964
    Jun 25 at 0:25
  • \$\begingroup\$ @Bobby Thank you for the feedback! I agree, error messages are important, and the current ones certainly can be improved. I will do this soon! \$\endgroup\$
    – 766F6964
    Jun 25 at 0:26

3 Answers 3

17
\$\begingroup\$
#! /bin/bash

That's where it's located on your system. But for some folks it might be in /usr/bin or /usr/local/bin.

When publishing a portable script, consider using this shebang:

#! /usr/bin/env bash

The env utility is not exactly what you'd call feature rich. It has been there since the dawn of time, or at least the 70's. It is soooo boring that even during the great Unix Wars no one thought to mess with it or move it.

The big thing that it brings to this party is it obeys ${PATH}, so it will find the interpreter in whichever of the popular locations it happened to land on this particular host.


BTW, I really like your CSS reset. It's reminiscent of the one offered by Y!UI. Putting all browsers on equal footing is really important for being able to test once and have confidence the test result translates to other browsers.


I personally find it's a bit of a challenge to author good Bourne code, since there are so many places that values can get expanded and re-globbed, erasing the difference between $@ and $*.

Before submitting code for review, it's pretty important to use ruff *.py, cc -Wall -Wextra -Wpendantic *.c, or whatever the relevant linter would be. Here, either you didn't or you chose to ignore the advice without writing any # comment about that decision. I will just reproduce the output below without comment.

$ shellcheck mublog.sh

In mublog.sh line 92:
<meta charset="utf-8">
               ^---^ SC2140 (warning): Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?


In mublog.sh line 129:
            local date=$(grep -oP "(?<=date: ).*" "$src_post_path")
                  ^--^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In mublog.sh line 130:
            local title=$(grep -oP "(?<=title: ).*" "$src_post_path")
                  ^---^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In mublog.sh line 143:
    IFS=$'\n' sorted_posts=($(sort -r <<<"${posts[*]}"))
                            ^-------------------------^ SC2207 (warning): Prefer mapfile or read -a to split command output (or quote to avoid splitting).


In mublog.sh line 173:
        posts_skipped=$(($posts_skipped+1))
                         ^------------^ SC2004 (style): $/${} is unnecessary on arithmetic variables.


In mublog.sh line 182:
        posts_processed=$(($posts_processed+1))
                           ^--------------^ SC2004 (style): $/${} is unnecessary on arithmetic variables.

For more information:
  https://www.shellcheck.net/wiki/SC2140 -- Word is of the form "A"B"C" (B in...
  https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to ...
  https://www.shellcheck.net/wiki/SC2207 -- Prefer mapfile or read -a to spli...

footer_copyright="Copyright &copy; 2023 John Doe :)"

No.

Please elide the smiley. You're trying to offer legal advice to your users. Offer good advice.

The &copy; glyph isn't exactly bad. But I recommend you elide it. The USPTO describes certain notice benefits and offers some advice:

A copyright notice consists of three elements:

• The copyright symbol © ..., the word “Copyright,” or the abbreviation “Copr.”;

• The year of first publication of the work ...; and

• The name of the copyright owner....

I don't recommend using that ancient "Copr." abbreviation.

And I don't recommend saying saying the same same thing twice twice.

Use the C-in-a-circle glyph, or the word. Not both.

Let's switch gears slightly.

Your users probably want folks to make copies of their work. Consider encouraging them to license their copyrighted work, perhaps using Creative Commons' CC attribution no-derivatives. Adding a line of boilerplate is all it would take.


author_mail="[email protected]"

Best practice here is to use a domain of example.com.

I see this greeting ATM when I start sending mail to that address:

220 mail.com (mxgmxus007) Nemesis ESMTP Service ready

It's a sad fact of life that if [email protected] appears in a document, MTAs will wind up processing a non-zero number of messages for that address.


I like initialize_directories(). Consider putting set -e, for early bailout, at the very top of the script -- it will still be compatible with the && clauses. And if you do that, go all the way: set -e -o pipefail

Probably worth adding set -u, too, since we don't expect any unset variables.

These are just belt-and-suspender approaches, something we routinely put on top of presumably correct code that strictly shouldn't need it.


function validate_header() {

Ok, you lost me a little bit, there.

It's unclear which porting setup you are targeting.

The function keyword is perfectly nice. Recommend you uniformly use it everywhere, or else use it nowhere.

The various checks seem nice enough. My offhand impression is that, as a user, a blog author, they are maybe not very diagnostic errors. That is, upon viewing an error, I might understand what went wrong and what I should fix. I'm just not yet confident of that. Sorry, don't have any specific advice here, it's just a matter of testing the system with users, looking over their shoulder, and seeing what speedbumps they actually stumble over in production. That takes time and bugreports to work out.


build_pages() {
    local header= ...

The proliferation of \ backwhacked " quotes seems tedious. Consider using the syntax of a << HERE document to make writing and reading HTML quotes a little more pleasant.

    pandoc ... | { echo -e "$header"; cat; echo -e "$footer"; } > "$2"

Kudos, that is really nicely phrased. It calls the reader's attention to the "hard" part, pandoc with its args, and it does a great job of showing off the parallel structure of the {header, text, footer} sandwich. I like it.


process_files() {
    ...
    while IFS= read ...

It's worth adding a comment that IFS= lets us parse NUL delimited records.

            posts+=("$date|$title|$src_post_path|$dst_post_path")

Up in validate_header() we never checked whether the title contains prohibited characters such as | pipe. That's kind of a problem, here.


sort_posts() {
    IFS=$'\n' sorted_posts=($(sort -r <<<"${posts[*]}"))
    unset IFS

I find the IFS thing non-obvious, and I feel it warrants a # comment.

Ok, now I understand it, having looked at this pair of identical examples, with BASH_VERSION='3.2.57(1)-release' on MacOS:

$  echo one $'\n' two
one 
 two
$
$  echo -e one '\n' two
one 
 two

The unset is weird, avoid doing that.


Kudos on this mainline code:

initialize_directories
build_pages ...
build_pages ...
build_pages ...
process_files ...
sort_posts

Nicely organized. Including how we validate at just the right point in the pipeline. Maybe put these few lines in a function called main() ?

process_files "$src_posts_dir"

Let me lean on that line, which relies on a find.

My concern is that after some dozens of blog articles have been published, we wind up

  1. taking a "long" time running pandoc on unchanged inputs, and
  2. we essentially touch *.html, changing webserver HTTP timestamp headers.

Consider using a technique like make so you only update HTML files that need it.


posts_processed=0
...
for post_info in ...

This stuff wants to be in a named function. Just sayin'.

Also, we see a bunch of cut invocations that operate on field 1 .. 4. Consider extracting that into a tiny helper function.

Similarly you really need a replace_article_tags() helper. And it would be lovely if it had an automated unit test, so I could better see just what it does.


Based on what I read, it is unclear to me what platform(s) you intend for this to be portable to. CentOS? Cygwin? MacOS? FreeBSD? Certainly debian linux.

The version of bash plus coreutils really matters. Consider bailing out with a fatal diagnostic if the target host's versions are out-of-whack with the one or more systems that you have tested on. Whatever automated tests you write, definitely publish them so as to invite contributors to run them as part of submitting PRs.


This codebase achieves its design goals.

I would be willing to delegate or accept maintenance tasks on it.

\$\endgroup\$
3
  • \$\begingroup\$ Thank you for the feedback, it helped a lot. I cleaned up the code base, and made sure error messages are more descriptive. I also introduced color coded prefixes, to easier distinguish verbose logging from error messages. \$\endgroup\$
    – 766F6964
    Jun 25 at 12:48
  • \$\begingroup\$ For more details/variations on set -e -o pipefail -u advice, google "bash strict mode", github.com/olivergondza/bash-strict-mode is a good starting point. \$\endgroup\$ Jun 26 at 7:29
  • \$\begingroup\$ #!/bin/bash is extremely common. Maybe even more common than env being /usr/bin/env? \$\endgroup\$
    – user253751
    Jun 26 at 23:02
11
\$\begingroup\$

Consider writing this in a programming language

I think that generating a static blog content is complex enough that you'd be much better off writing it in a programming language. Like Python.

Check requirements early

The script expects some non-standard programs to be installed, such as pandoc. It would be good to raise an error early if it doesn't exist, rather than failing in the middle of the script.

Avoid repeated disk I/O

Working with reasonable small files it's probably not an issue in this script, but I cringe when I see repeated re-reading of the same file to parse selected lines like this:

  marker1=$(sed -n '1p' "$1" | ...)
  title=$(sed -n '2p' "$1" | ...)
  description=$(sed -n '3p' "$1" | ...)

I would do more like this:

{
  read -r marker1
  read -r title
  read -r description
  ...
} < <(sed -n '1,6p' "$1")

This way you can collect the top N lines by reading only once.

Fail faster

validate_header parses all the header fields, and then checks them one by one and exits on the first failure. There's no need to parse the 2nd header field if something's wrong with the first.

I suggest to move the validation check of each header line right after it's parsed.

Don't repeat yourself

Instead of writing repetitive lines like this:

  if [ -z "$marker1" ]; then
      echo "Invalid Header: Starting markers missing or incorrect" && exit 1
  elif [ -z "$title" ]; then
      echo "Invalid Header: Title field missing or incorrect" && exit 1

I recommend to create a helper function, for example:

validate_header() {
  local value label
  value=$1
  label=$2

  if [ -z "$value" ]; then
    echo "Invalid Header: $label missing or incorrect" >&2
    exit 1
  fi
}

validate_header "$marker1" "Starting markers"
validate_header "$title" "Title field"
...

Do not unset IFS

I'm not sure why this function has unset IFS at the end:

sort_posts() {
    IFS=$'\n' sorted_posts=($(sort -r <<<"${posts[*]}"))
    unset IFS
}

The command IFS=... some_command sets IFS only for some_command, the value of IFS in the enclosing shell is unchanged.

As such, unset IFS has nothing to do with the rest of the function. This will erase any existing value of IFS, and I don't see a reason for this potential side effect.

Prefer native Bash functions over external processes

Executing commands has an overhead, that becomes easily visible when called in a loop.

Consider this part:

date=$(cut -d '|' -f 1 <<<"$post_info")
title=$(cut -d '|' -f 2 <<<"$post_info")
src=$(cut -d '|' -f 3 <<<"$post_info")
dst=$(cut -d '|' -f 4 <<<"$post_info")

Instead of executing the cut command 4 times, you can use the read builtin:

IFS='|' read date title src dst rest <<< "$post_info"

In other places too, consider replacing commands with native Bash techniques.

When something is difficult to replace exactly with native Bash, for example sed -n 's/^title:\s*\(.*\)\s*$/\1/p' looks not trivial, consider if a simpler logic might actually be good enough for the purpose.

For example, given $title with value title: foo , this will extract foo:

read header title <<< "$title"

My point is that this is not exactly equivalent as the sed command, but maybe it's good enough. And faster.

\$\endgroup\$
1
  • \$\begingroup\$ Thanks for the feedback. Definitely some good advice. I didn't even think of a check to ensure pandoc is installed - added that. Also, I like the fail-fast approach. \$\endgroup\$
    – 766F6964
    Jun 25 at 12:47
2
\$\begingroup\$
  • The html doesn't validate - a few tags are mismatched.
  • The title metadata is not included in the output, even though it's parsed.
  • pandoc is a rather heavy dependency

The code really looks like it wants to be Python. So the best improvement would probably be to actually write it in Python. Incidentally, it's the same length as the bash script. And it doesn't need pandoc - just the markdown2.py file from the markdown2 module.

The Python port lives in https://github.com/kubao/mublog-py

#! /usr/bin/env python

from glob import glob
import markdown2
import os
import re
import shutil

dst_root_dir = "dst"
dst_posts_dir = f"{dst_root_dir}/posts"
dst_css_dir = f"{dst_root_dir}/css"
dst_assets_dir = f"{dst_root_dir}/assets"
src_root_dir = "src"
src_posts_dir = f"{src_root_dir}/posts"
src_css_dir = f"{src_root_dir}/css"
src_assets_dir = f"{src_root_dir}/assets"
post_ignore_delim = "_"
footer_copyright = "Copyright &copy; 2023 John Doe :)"
author_mail = "[email protected]"


def dedent(text: str) -> str:
    """Dedents the indentation common to all lines in the text"""
    common_indent = re.match(r"^(\s*)[^s]", text)[1]
    unindent = f"^{common_indent}"
    return re.sub(unindent, "", text, flags=re.MULTILINE)


def initialize_directories():
    """Removes old build artefacts, and generates the build directories

    The /dst directory is the root directory of the blog
    The /dst/posts directory contains all the blog post files
    The /dst/assets directory stores images, videos etc of posts
    The /dst/css directory contains the style sheets of the blog
    """
    shutil.rmtree(dst_root_dir, ignore_errors=True)

    # Create output directories
    for d in (dst_root_dir, dst_posts_dir, dst_css_dir, dst_assets_dir):
        print(d)
        os.makedirs(d, exist_ok=True)

    for f in glob(f"{src_css_dir}/*.css"):
        shutil.copy(f, dst_css_dir)
    shutil.copytree(src_assets_dir, dst_assets_dir, dirs_exist_ok=True)

    print("Build directories initialized.")


def parse_header(src_post) -> dict[str, str]:
    """Parses the header and returns a dictionary of fields from it.
    """

    print(f"Parsing post {src_post} ...")

    with open(src_post, encoding="utf-8") as f:
        lines = f.readlines()

    patterns = (
        # Line 1: Check for --- start-marker
        (r"^---\s*$",
         "Starting markers missing or incorrect"),
        # Line 2: Check for title-field
        (r"^title:\s*(?P<title>.*)\s*$",
         "Title field missing or incorrect"),
        # Line 3: Check for description-field
        (r"^description:\s*(?P<description>.*)\s*$",
         "Description field missing or incorrect"),
        # Line 4: Check for date-field with valid date in YYYY-MM-DD format
        (r"^date:\s*(?P<date>[0-9]{4}-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1]))\s*$",
         "Date field missing, incorrect or in wrong format."),
        # Line 5: Check for tags-field
        (r"^tags:\s*(?P<tags>.*)\s*$",
         "Tags field missing or incorrect"),
        # Line 6: Check for --- end-marker
        (r"^---\s*$",
         "Ending marker missing or incorrect")
    )
    results: dict[str, str] = {}
    for line, pat_msg in zip(lines, patterns):
        pattern, msg = pat_msg
        match = re.match(pattern, line)
        if match:
            results |= match.groupdict()
        else:
            print(f"{src_post}: Invalid header \"{line}\".\n{msg}")
            return {}
    return results


def build_page(src_md: str, dst_html: str, root: str):
    """Converts the markdown post or page into html format using pandoc.

    During this process, the header is prepended and the footer appended to the post.
    Arguments:

      src_md: The source path to the markdown post/page file
      dst_html: The destination file where the converted html file will be saved.
    """

    md = markdown2.Markdown(extras={"metadata": None})
    with open(src_md, encoding='utf-8') as f:
        html = md.convert(f.read())

    title = f"<title>{md.metadata['title']}</title>\n" if "title" in md.metadata else ""

    with open(dst_html, "w", encoding='utf-8') as f:
        f.write(dedent(f"""\
            <!DOCTYPE html>
            <html>
            <head>
            <meta charset="utf-8">
            <meta name="viewport" content="width=device-width, initial-scale=1">
            <link rel="stylesheet" href="{root}/css/normalize.css" type="text/css" media="all">
            <link rel="stylesheet" href="{root}/css/style.css" type="text/css" media="all">
            {title}</head>
            <body>
            <nav>
            <a href="{root}/index.html">home</a>
            <a href="{root}/articles.html">articles</a>
            <a href="mailto:{author_mail}">mail</a>
            <a href="{root}/about.html">about</a>
            </nav>
            <main>
            <hr>"""))
        f.write(html)
        f.write(dedent(f"""\
            </main>
            <footer>
            <hr>
            <p>
            {footer_copyright}
            <br>
            </p>
            </footer>
            </body>
            </html>"""))


def process_files(src_posts_dir) -> list[dict[str, str]]:
    """
    Iterate through all source post files, and extract values stored in their headers
    such as date, title, but also stores source path and destination path.
    Return the list of dictionaries containing the data of the posts.
    """

    # Find all .md posts in the post directory and extract info from the headers
    posts = []
    for src_post_path in glob(f"{src_posts_dir}/*.md"):
        post = parse_header(src_post_path)
        if post:
            post["basename"] = os.path.basename(src_post_path)
            root_name, _ext = os.path.splitext(post["basename"])
            dst_post_path = f"{dst_posts_dir}/{root_name}.html"
            post["src"] = src_post_path
            post["dst"] = dst_post_path
            posts.append(post)
    return posts


def sort_posts(posts: list[dict[str, str]]) -> list[dict[str, str]]:
    """Sorts posts in place in reverse chronological order, based on the extracted date"""
    def key(post): return post["date"]
    posts.sort(key=key)
    return posts


if __name__ == "__main__":
    initialize_directories()
    build_page(f"{src_root_dir}/about.md", f"{dst_root_dir}/about.html", ".")
    build_page(f"{src_root_dir}/index.md", f"{dst_root_dir}/index.html", ".")
    build_page(f"{src_root_dir}/articles.md",
               f"{dst_root_dir}/articles.html", ".")
    posts = process_files(src_posts_dir)
    posts = sort_posts(posts)

    posts_processed = 0
    posts_skipped = 0

    article_list = '<ul class="articles">\n'

    for post in posts:
        dst_link = post["dst"][len("dst/"):]
        print(f'Processing post: {post["src"]}')
        print(f'    title:  {post["title"]}')
        print(f'    date:   {post["date"]}')
        print(f'    output: {post["dst"]}')

        if post["basename"].startswith(post_ignore_delim):
            posts_skipped += 1
        else:
            article_list += f'<li><b style="color: #14263b;">{post["date"]}</b> <a href="{dst_link}">{post["title"]}</a></li>\n'
            build_page(post["src"], post["dst"], "..")
            posts_processed += 1

    article_list += '</ul>'

    print("Generating article listing ...")

    # Replace article tags in the article.html file with the generated article list
    with open(f"{dst_root_dir}/articles.html", encoding="utf-8") as f:
        articles_html = f.read()

    articles_html = articles_html.replace(
        "<article>", f"<article>\n{article_list}")

    with open(f"{dst_root_dir}/articles.html", "w", encoding="utf-8") as f:
        f.write(articles_html)

    print(f"Finished! (built: {posts_processed}, skipped: {posts_skipped})")

This is not the best python out there, but it should show how relatively little it takes to go from bash to python. If you really wanted to have it all in bash, it is possible to port markdown2.py to bash, although I'll not be the one to do it.

The errors are raised as Python exceptions and cause an informative display, so that takes care of most error handling - although catching a few exceptions would be nice.

Thing about bash: most systems where you'd want to run this thing also have python either installed or a click away. And it's a bit easier to get python going on windows than bash. So - at least with python, it's quite portable.

\$\endgroup\$
3
  • \$\begingroup\$ Thanks for the feedback, maybe I should actually rewrite it in python to save myself some headache.. :D \$\endgroup\$
    – 766F6964
    Jun 26 at 12:29
  • \$\begingroup\$ @766F6964 Already done in the fork :) It’s a cool project, I really like the simplicity! \$\endgroup\$ Jun 26 at 14:09
  • \$\begingroup\$ @766F6964 Although you're probably going to do an even better job. This was a quick hack mostly. \$\endgroup\$ Jun 26 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.