1
\$\begingroup\$

The main concern is the hard requirement for python3.12 since I couldn't find a clean replacement for pathlib.Path.relative_to(root, walk_up=True) in earlier pythons. I need it in order to generate relative URLs like ../../../.thumbs/some/dir/image.jpg

Also not sure if the multithreading is sane for parallel thumbnail generation as concurrency has never been my forte.

Would also be good to know if there is anything buggy, specifically ugly or inefficient that can be improved.

This is a stripped down version of the gist here — the only difference is in the amount of the js/css code which I cut down here in order to make it shorter. The gist version adds JS keyboard navigation and opens the assets in an overlay div rather than a new tab.

#!/usr/bin/env python3.12
# ^^^^^^^^^^^^^^^^^^^^^^^

#  DEPENDENCIES:
# imagemagick:
#     ubunbtu/debian: sudo apt install imagemagick
#     macos: brew install imagemagick


# !!!MUST USE python3.12 of later!!!

# install python 3.12 on ubuntu:
#     sudo add-apt-repository ppa:deadsnakes/ppa
#     sudo apt install python3.12
#
# install python 3.12 on macos:
#     brew install [email protected]

import os.path
import subprocess
from datetime import datetime
from pathlib import Path
from sys import argv
from urllib.parse import quote
from concurrent.futures import ThreadPoolExecutor

THUMBS_DIR = '~/.thumbs'

OUTPUT_FILE_NAME = "index.html"

IMAGE_SVG = ".svg"
IMAGE_HEIC = ".heic"
IMAGE_EXTENSIONS = (".jpg", ".jpeg", ".png", ".bmp", ".webp", ".gif", IMAGE_HEIC)
VIDEO_EXTENSIONS = (".mp4", ".mkv", ".webm", ".3gp", ".mov", ".mkv", ".ogv", ".mpg", ".mpeg")
AUDIO_EXTENSIONS = (".mp3", ".ogg", ".wav")

ASSET_EXTENSIONS = IMAGE_EXTENSIONS + VIDEO_EXTENSIONS + AUDIO_EXTENSIONS

class Asset:
    def __init__(self, type, path, created_date, html, thumbnail_path=None):
        self.type = type
        self.path = path
        self.created_date = created_date
        self.html = html
        self.thumbnail = thumbnail_path


def get_creation_date(file_stat):
    try:
        return file_stat.st_birthtime
    except AttributeError:
        # We are probably on Linux. No way to get the creation date, only the last modification date.
        return file_stat.st_mtime


UNITS_MAPPING = [
    (1024 ** 5, 'P'),
    (1024 ** 4, 'T'),
    (1024 ** 3, 'G'),
    (1024 ** 2, 'M'),
    (1024 ** 1, 'K'),
    (1024 ** 0, (' byte', ' bytes')),
]


def pretty_size(bytes, units=UNITS_MAPPING):
    """Human-readable file sizes.

    ripped from https://pypi.python.org/pypi/hurry.filesize/
    """
    for factor, suffix in units:
        if bytes >= factor:
            break
    amount = int(bytes / factor)

    if isinstance(suffix, tuple):
        singular, multiple = suffix
        if amount == 1:
            suffix = singular
        else:
            suffix = multiple
    return str(amount) + suffix


def collect_media_assets(root: Path, thumbsdir: Path):
    homedir = os.path.expanduser('~')
    assets: list[Asset] = []

    for asset_path in root.rglob("*"):
        if asset_path.is_file() and asset_path.suffix.lower() in ASSET_EXTENSIONS:
            if asset_path.name.startswith('._'):  # skip macos junk
                continue
            size_pretty = pretty_size(asset_path.stat().st_size)
            created_date = datetime.fromtimestamp(get_creation_date(asset_path.stat()))
            created_date_formatted = created_date.strftime("%Y-%m-%d %H:%M:%S")
            relative_path = str(asset_path.relative_to(root))
            escaped_path = quote(relative_path)

            if asset_path.suffix.lower() == IMAGE_SVG:
                assets.append(Asset('svg',
                                    relative_path,
                                    created_date, f"""<a class="item" href="{escaped_path}" target="_blank" title="{relative_path} size: {size_pretty}; created: {created_date_formatted}">
                    <img src="{relative_path}" loading="lazy">
                </a>"""))
            elif asset_path.suffix.lower() in IMAGE_EXTENSIONS:
                segment = str(root.absolute()).replace(homedir, '').lstrip('/')
                thumbnail_path: Path = thumbsdir / segment / relative_path

                if asset_path.suffix.lower() == IMAGE_HEIC:
                    thumbnail_path = thumbnail_path.with_suffix(".jpg")

                thumbnail_path_relative = thumbnail_path.relative_to(root, walk_up=True)  ## IMPORTANT: REQUIRED python 3.12 minimum!
                thumbnail_path_relative_escaped = quote(str(thumbnail_path_relative))

                assets.append(Asset('image',
                                    relative_path,
                                    created_date, f"""<a class="item" href="{escaped_path}" target="_blank" title="{relative_path} size: {size_pretty}; created: {created_date_formatted}">
                    <img src="{thumbnail_path_relative_escaped}" loading="lazy">
                </a>""", thumbnail_path))

            elif asset_path.suffix.lower() in VIDEO_EXTENSIONS:
                assets.append(Asset('video', relative_path, created_date, f"""<a class="item" href="{escaped_path}" target="_blank" title="{relative_path} size: {size_pretty}; created: {created_date_formatted}">
                    <video preload="none" controls><source src="{escaped_path}"></video>{asset_path.name}
                </a>"""))

            elif asset_path.suffix.lower() in AUDIO_EXTENSIONS:
                assets.append(Asset('audio', relative_path, created_date, f"""<a class="item" href="{escaped_path}" target="_blank" title="{relative_path} size: {size_pretty} KB; created: {created_date_formatted}">
                    <audio controls src="{escaped_path}"></audio>{asset_path.name}
                </a>"""))
            print(f'{asset_path}')
    return assets


def generate_gallery_html(html_data, output_file_path: Path):
    with output_file_path.open(mode="w", encoding="utf-8") as fout:
        fout.write('''<html>
    <head>
        <style>
        * {
            font-family: system-ui, sans-serif;
        }
        body>div {
            display: grid; 
            grid-template-columns: repeat(auto-fill, minmax(321px, 1fr)); 
            gap: 5px;
        }
        .item {
            display: flex;
            flex-direction: column;
            align-items: center;
            justify-content: center;
        }
        img, video {max-width: 321px;
            height: auto;
            display: table-cell;
        }
        a {
            text-decoration: none;
            font-size: 14px;
        }
        a:hover {
            color: #0095e4;
        }

        a:visited {
            color: #800080;
        }

        a:visited:hover {
            color: #b900b9;
        }
        </style>
    </head>
    <body>
        <div>'''
                   + html_data
                   + '''</div>
    </body>
</html>''')

    print(f"Gallery HTML file generated: {output_file_path.absolute()}")


def generate_thumbnails(assets: list[Asset], basedir: Path):
    with ThreadPoolExecutor(max_workers=42) as executor:
        asset: Asset
        for asset in assets:
            if asset.type == 'image':
                executor.submit(make_thumbnail, asset, basedir)


def make_thumbnail(asset: Asset, basedir: Path):
    thumbnail_dir: Path = asset.thumbnail.parent
    thumbnail_dir.mkdir(parents=True, exist_ok=True)

    if asset.thumbnail.exists():
        print(f'⚡️\tSKIP existing thumbnail: {asset.thumbnail.absolute()}')
        return
    #### TODO: FIXME: validate path to convert: can be e.g. /usr/bin/convert, /usr/local/bin/convert...
    args = ["/opt/homebrew/bin/convert",
        str(basedir / asset.path),
        "-strip",
        "-quality", "60",
        "-thumbnail", "300x"]

    if Path(asset.path).suffix == '.heic':
        args += ['-format', 'jpg']

    args.append(str(asset.thumbnail.absolute()))

    proc = subprocess.run(args,
                          stdout=subprocess.PIPE,
                          stderr=subprocess.PIPE)

    result = proc.stdout.decode()
    print('✅', asset.path, str(asset.thumbnail.absolute()), result)
    err = proc.stderr.decode()

    if err:
        print("😱", err)


if __name__ == "__main__":
    thumbnails_dir: Path = Path(os.path.expanduser(THUMBS_DIR))
    thumbnails_dir.mkdir(exist_ok=True)

    root_dir = len(argv) > 1 and argv[1] or '.'  # Use 1st arg or current directory as root
    base_dir: Path = Path(root_dir).absolute()
    asset_list: list[Asset] = collect_media_assets(base_dir, thumbnails_dir)

    if not asset_list:
        print("No media files found.")
    else:
        asset_list.sort(key=lambda a: a.created_date, reverse=True)
        html_content: str = '\n'.join(asset.html for asset in asset_list)

        generate_thumbnails(asset_list, base_dir)
        generate_gallery_html(html_content, base_dir / OUTPUT_FILE_NAME)

        if asset_list and len(asset_list):
            print(f'''
Total assets:    {len(asset_list)}
Images:          {len([a for a in asset_list if a.type == 'image'])}
Audios:          {len([a for a in asset_list if a.type == 'audio'])}
Videos:          {len([a for a in asset_list if a.type == 'video'])}
''')
\$\endgroup\$

2 Answers 2

2
\$\begingroup\$

It's clear the author took great care when writing this code, DRYing things up as the opportunity arose.

modern interpreter

#!/usr/bin/env python3.12
# ^^^^^^^^^^^^^^^^^^^^^^^
#  ...
# !!!MUST USE python3.12 [or] later!!!

Gosh, that sure is a lot of shouting.

main concern is the hard requirement for python3.12

I'm afraid I don't share your concern. It's not an onerous requirement, and it will only get easier over time.

You helpfully include install instructions covering two popular platforms; I thank you for that.

The python3.12 shebang works today, but will need updating soon. I use apt and brew all the time, but I'm afraid I seldom use them to affect interpreter details. I would expect a conda environment.yml or pip requirements.txt to choose the interpreter being used in a particular venv.

Writing readme's and comments is nice, don't stop doing it. However the machine pays attention to details much better than humans do. Strive to produce an informative diagnostic. What you really care about is something like this:

assert sys.version_info[:2] >= (3. 12)  # we need Path.relative_to()

Or perhaps raise with a helpful message for downrev interpreter.

With that in hand, a simple python (or python3) shebang should suffice, working fine next year plus the year after that.

source control

... version of the gist

You are clearly familiar with creating multiple git repos. Git is a much better way of tracking version details than gist, both for the code's author and its consumers. 'Nuff said.

modern libraries

import os.path
...
from pathlib import Path

We have established that pathlib will definitely be available. Consider changing your os.path.expanduser() calls to be Path( ... ).expanduser(), just for consistency.

Habits like that will tend to improve the design of your Public APIs, with str parameters being replaced by Path parameters.

main guard

if __name__ == "__main__":
    thumbnails_dir: Path = Path( ...
    base_dir: Path = Path( ...

I really like the guard, thank you. It's good that a unittest can safely import this without side effects.

Sorry, but I'm not quite getting the whole javaesque Bond james = Bond() thing. It's no mystery to mypy, nor to a human reading this code, what type those directory variables will be.

This code stanza isn't exactly too long. But it's starting to get that way, which suggests banishing it to within a def main(): function.

cracking argv

    root_dir = len(argv) > 1 and argv[1] or '.'  # Use 1st arg or current directory as root

This is mostly clear. But not as clear as could be. You're relying on the (unambiguous!) precedence of and vs or. Consider helping out humans by adding redundant ( ) parens, so there's no need for that comment.

Better, you might import typer and punt the whole thing to a library, getting --usage help for free. Kernighan & Richie would always crack argv by hand, but you don't need to.

It's unclear how base_dir and root_dir are distinct concepts. Prefer to .absolute() the input before assigning it to a name.

annotations

I don't understand this line:

    asset_list: list[Asset] = collect_media_assets(base_dir, thumbnails_dir)

Oohhh, wait, now I get it. Look at the signature of the callee.

def collect_media_assets(root: Path, thumbsdir: Path):

It doesn't describe the return type. (Based on an annotated assignment + return it's easy to work out. But still.)

Describe the return type, please.

In general, if you feel the need to annotate some variable assignment, take a step back and wonder if mypy is really asking you to annotate a function signature. Fixing up the signature would be far more valuable, at this and at other call sites.

Granted, there are some situations that call for annotation in the middle of a chunk of code, usually when creating an empty container. While dict() is beautiful, it is alas also very flexible. So we might need to distinguish
d: dict[str, int] = {} from
d: dict[int, str] = {}, and so on.

attrgetter

Clearly this works:

        asset_list.sort(key=lambda a: a.created_date, reverse=True)

Prefer attrgetter, so we have key=attrgetter('created_date').

truthy

I don't understand this.

        if asset_list and len(asset_list):

Your annotation clearly explained that None shall not be coming back from collect_media_assets. To say nothing of the fact that we just called .sort() on it, a function which None lacks. No need for comparing length to zero again, as the first conjunct already did that.

filter

Images:          {len([a for a in asset_list if a.type == 'image'])}

We construct a list, then throw it away. But len() doesn't need a container -- len(generator) works just fine. These would work well as filter() expressions.

aliasing builtins

class Asset:
    def __init__(self, type, path, ... ):
        self.type = type

This didn't cause any confusion with the type() function. But still, consider using the conventional approach, which would name it type_.

Similarly for bytes_.

It looks like the Asset class could be a dataclass.

tricky assumptions

I get what you're trying to do here with the segment assignment.

            elif asset_path.suffix.lower() in IMAGE_EXTENSIONS:
                segment = str(root.absolute()).replace(homedir, '').lstrip('/')
                thumbnail_path: Path = thumbsdir / segment / relative_path

But it depends on some assumptions about public variable relationships which are valid for default values, but not in general. We could assert thumbnail_path.exists(), but without confidence that it shall always be True.

Consider having more strictly defined relationships among your variables. This might come in the form of insisting on certain names under $HOME. Many packages choose to claim a chunk of FS namespace, insisting that files reside under a package-owned root.

separated

        fout.write('''<html> ...
</html>''')

Uggh, I hate that.

Pascal has ; semicolon separated statements, unlike PL/I's semicolon terminated statements. C and others follow one or the other convention.

I assert that a text file (such as *.html) consists of newline terminated lines. Not newline separated lines as we see here. Last character before EOF should be \n newline.

redundant annotation

def generate_thumbnails(assets: list[Asset], basedir: Path):
    ...
        asset: Asset
        for asset in assets:

What a lovely signature! Consider ending it with ... ) -> None:

I can't imagine why man or machine would need the asset hint, given that first parameter offered a very clear introduction.

execlvp

    #### TODO: FIXME: validate path to convert: can be e.g. /usr/bin/convert, /usr/local/bin/convert...
    args = ["/opt/homebrew/bin/convert",

We don't like , shell=True ? OK. Perhaps there are filename quoting concerns.

Write a def which_convert(): helper that asks the shell to consult $PATH and report what $ which convert says. Decorate it with @lru_cache.

check child exit value

Rather than proc = subprocess.run( ... ), you might prefer .check_call( ... ). The status integer is different from determining whether stderr is of greater than zero length.


There's more than one "static gallery generator" available. It would be helpful for a comment paragraph (or URL) to offer an elevator pitch. Why is this generator a better choice? For which use case is it better? What deficiencies of competing libraries does it remedy?


This codebase appears to achieve its design goals.

I would be willing to delegate or accept maintenance tasks on it.

\$\endgroup\$
3
  • \$\begingroup\$ Thanks for breaking it down! input is much appreciated; learned something new \$\endgroup\$
    – ccpizza
    Feb 13 at 10:24
  • \$\begingroup\$ regarding github gists — they are legitimate git repos with versions and everything (probably no way to have meanginful commit messages though); the repo clone URL is hidden under the Embed dropdown. \$\endgroup\$
    – ccpizza
    Feb 13 at 10:28
  • \$\begingroup\$ regardingconvert — just discovered that there is also shutil.which() which is just the thing I needed and which can be run once in __main__ (to avoid using shell=True given that thousands of processes will be spawned) \$\endgroup\$
    – ccpizza
    Feb 13 at 14:53
2
\$\begingroup\$

Templating

I think it would make sense to use Jinja as a templating engine here. Thus you can make edits to your template without touching the Python code (and breaking things). Also, this would make it easier and more scalable to switch between different templates. BTW the CSS should preferably reside in an external file as well.

The learning curve isn't steep and this could be a good investment since Jinja is used with Flask, and is also used by Ansible. Note that Jinja also takes care of escaping values. This is a bonus.

Some minimalistic code to show how you would use it in Python:

from jinja2 import Environment, FileSystemLoader
environment = Environment(loader=FileSystemLoader("/home/user/test/"))
template = environment.get_template("library.html")

html_data = "Hello world"

content = template.render(html_data=html_data)
print(content)

Then in the HTML template you'll need a few {% for item in whatever %} loops as shown here: Template Designer Documentation. And then you can also drop the assets.append statements in your code. Delegate the presentation to the template, just collect the data.


Human readable values

It seems to me that the pretty_size function could be shortened. It comes across as slightly pedantic and could be more to the point. The UNITS_MAPPING list is redundant. I found a lot of implementations like this one or this one but there is more.


Class definition

To push the logic further I think class Asset should be reduced to the lowest common denominator eg:

def __init__(self, type, path, created_date):
    self.type = type
    self.path = path
    self.created_date = created_date

And subclasses should be derived off the top-level Asset class. For instance, pictures, videos and sound files can be classes of their own.

Key points:

  • the thumbnail attribute is not relevant for sound files
  • on the other hand, pictures and video files can have attributes such as resolution that do not apply to other file types
  • The html attribute is unneeded if you use a templating engine
  • many more useful attributes can be added at class level, such as file size
  • thus, there is room for increased differentiation depending on the file type

Some useful reading and inspiration: Multiple Inheritance in Python

\$\endgroup\$
1
  • \$\begingroup\$ thanks, yes, totally agree and, in fact, I had the same thoughts, but the intent was to keep it raw, as dumb as possible, and easily hackable, even if it implied spaghettiness; extracting stuff and cleaning up was outside the scope, given that every dev has their own preferred ways to tidy up (including architecting advanced strategies for avoiding it) \$\endgroup\$
    – ccpizza
    Feb 14 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.