Explanation
The script is designed to be run from the command-line, and takes any number of positional arguments as paths of "input" files (it also takes some optional arguments to customize its behavior, such as changing the character sequence that marks the start of a single-line comment). If one of the paths is - (or no positional arguments are given at all), that file will be read from STDIN. The files can end in any extension and should contain source code. It'll then go through these files and strip all single-line comments in them.
As for language, any language whose syntax rules for string literals and single-line comments are similiar to C should work (I've only tested it on Python source files so far).
Lines that only contain a single-line comment would be removed entirely, while lines that contain code and a single-line comment at the end would be modified to remove the comment at the end (and the whitespace before the start of the comment).
Dependencies
The regex module from PyPI, everything else is from the stdlib.
Why I Wrote it
Occasionally, after finishing work on a source file, I would want to remove all comments in it for various reasons. Maybe I decided the comments I wrote weren't very helpful and want to start over, or maybe the script never has to be read or modified again anyway. Going through the file manually to remove all single-line comments is very tedious, so I decided to write this script to automate the process and save some time.
Why I would like a review
I usually use object-oriented-programming for structuring and writing my full-sized projects, and in this script I decided to use functional programming instead, which I'm not as familiar with. So I would like to know how my code looks.
Also, I'm mostly a self-taught programmer (I'm still a student by the way, and haven't started university yet), so I have no idea how my code quality is. Hopefully it's not horrible.
I followed PEP8 as closely as I could (as I do for all my projects, single-file scripts or full-sized projects), and also kept the column width to less than or equal to 80 characters as PEP8 recommends. The code is also fully type-annotated and passes Pyright's type-check.
Source File
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
This is a simple command-line utility that takes an arbitrary number of
paths to source code files as positional arguments, and will strip all
single-line comments in those files. The character sequence that signifies a
comment can be configured via an optional argument, so it can be used for most
programming languages.
"""
import typing as tp
import platform
import pathlib
import fileinput
import contextlib
import argparse
import regex
import os
import sys
if tp.TYPE_CHECKING:
import types
BACKUP_EXT = ".backup"
stmt_parser: regex.Pattern[str]
args: argparse.Namespace
ExcType = tp.TypeVar("ExcType", bound=Exception)
# Matches a single valid C-like single or double quoted string literal,
# such as "Dick's sister said: \"But what about Timmy?\"" or "backslash: \\"
# This is useful on its own.
# The group <begin_quote> stores the type of quotes that the string uses if it
# matches (either ' or ").
str_lit = (
r"^(?P<str_lit>(?P<begin_quote>(?P<single_quote>')|\")"
r"(?:(?(single_quote)\"|')|\\[\"']|\\[^\r\n ]|[^\"'\\\r\n])*?"
r"(?P=begin_quote))$"
)
# Matches any sequence of characters that are allowed outside a string literal.
# Not very useful on its own as it matches indentation as well.
# Replacement field is for comment char sequence (like '#').
non_str_lit = r"(?P<non_str_lit>(?:(?!{})[^\"'\r\n])*?)"
# Matches a line of valid C-like code into the groups <indentation>, <stmt>,
# <whitespace>, <comment>, and <line_ending>. Lines with unclosed string
# literals won't match.
# Groups <comment> and <line_ending> can be missing if the code doesn't have it.
# Other groups can be empty.
stmt = (
r"^(?P<indentation>(?P<sp_or_tab>[ \t])*+)"
r"(?P<stmt>{non_str_lit}(?:{str_lit}(?&non_str_lit))*?)"
r"(?P<whitespace>(?&sp_or_tab)*+)"
r"(?P<comment>(?:{comment_seq_escaped})[^\r\n]*+)?"
r"(?P<line_ending>\r\n|\r|\n)?$"
)
# Matches a line that is empty except for the line ending. This is used to skip
# empty source lines to save time, as 'stmt' matches empty lines.
empty = regex.compile(r"^(\r|\n|\r\n)?$")
def setup_regex_patterns(comment_seq: str) -> None:
"""Takes the character sequence that denotes the start of a single-line
comment, and sets up regex patterns in-place."""
global non_str_lit, stmt, stmt_parser
comment = regex.escape(comment_seq)
non_str_lit = non_str_lit.format(comment)
stmt = stmt.format(non_str_lit=non_str_lit, str_lit=str_lit[1:-1],
comment_seq_escaped=comment)
stmt_parser = regex.compile(stmt)
def setup_cmdline_interface() -> argparse.ArgumentParser:
"""Sets up an 'ArgumentParser' instance and returns it."""
def directory(arg: str) -> pathlib.Path:
"""Converts string argument to a 'pathlib.Path' object if and only if
it is a valid existing directory on the filesystem."""
# Expand "~" and convert to absolute path.
try:
# '.resolve()' can raise OSError or RuntimeError if path is invalid.
path = pathlib.Path(arg).expanduser().resolve()
except (OSError, RuntimeError):
raise ValueError from None
try:
# '.is_dir()' can *also* raise an error on insufficient perms.
is_a_dir = path.is_dir()
except OSError:
raise ValueError from None
if is_a_dir:
return path
else:
raise ValueError
parser = argparse.ArgumentParser(
description="A simple command-line utility for stripping out "
"all single-line comments from one or more source code "
"files. Input will be read from STDIN for one file if "
"that file's name is \"-\", or if no files are given. For "
"files read from STDIN, the stripped output would be sent "
"to STDOUT. For the rest, the output would be written to "
"new files whose names are the original filenames plus a "
"suffix (writing to the original files can be enabled)\n\n"
"All files are read and written with UTF-8 encoding.")
parser.add_argument(
"files", nargs="*", type=os.path.expanduser,
help="One or more source files whose comments will be stripped.")
parser.add_argument(
"-d", "--directory", type=directory, default=".",
help="The directory where stripped files are outputted. Defaults to "
"the current working directory.")
parser.add_argument(
"-c", "--comment", default="#",
help="The character sequence that the source file(s) uses to signify "
"single-line comments. Defaults to \"#\", which is what Python "
"uses.")
parser.add_argument(
"-s", "--suffix", default="_stripped",
help="The suffix to add to the original filename(s) for output files. "
"Defaults to \"_stripped\".")
group = parser.add_mutually_exclusive_group()
group.add_argument(
"-o", "--overwrite", action="store_true",
help="If the output filename of a file clashes with an existing file, "
"output would be skipped for that file to prevent data loss. "
"Specify this flag to override this behavior.")
group.add_argument(
"-i", "--in-place", action="store_true",
help="This would strip the files in-place, instead of writing the "
"stripped versions to separate files. The contents of the original"
" files would be temporarily backed up, and would be restored "
"if an IO error is encountered before the stripping is completed.")
return parser
class HandleStdoutClose:
def __init__(self) -> None:
"""Context manager that allows the script to exit without errors if its
STDOUT output is cut off by an external program (such as a pager). The
code should not attempt to output anything after exiting this context.
"""
self.win32 = (platform.system() == "Windows")
def _check_exception(self, exc: Exception) -> bool:
"""Determines if the exception was caused by a broken pipe,
in a platform-dependent way."""
if self.win32:
# This is the exception Python raises on SIGPIPE on Windows.
# OSError: [Errno 22] Invalid argument
return isinstance(exc, OSError) and exc.errno == 22
else:
# This is the exception Python raises on SIGPIPE on Linux.
return isinstance(exc, BrokenPipeError)
def _try_flush(self) -> bool:
"""Flushes the standard output and returns whether SIGPIPE was
encountered."""
try:
sys.stdout.flush()
except (OSError, BrokenPipeError) as err:
return self._check_exception(err)
else:
return False
def __enter__(self) -> None:
return
@tp.overload
def __exit__(self, exc_type: None, exc_value: None,
traceback: None) -> bool:
...
@tp.overload
def __exit__(self, exc_type: tp.Type[ExcType], exc_value: ExcType,
traceback: "types.TracebackType") -> bool:
...
def __exit__(self, exc_type, exc_value, traceback):
if exc_type is None:
return False
elif not self._check_exception(exc_value) and not self._try_flush():
# Only 'BrokenPipeError' should be suppressed by context manager.
return False
devnull = os.open(os.devnull, os.O_WRONLY)
os.dup2(devnull, sys.stdout.fileno())
return True # Suppress the exception.
@tp.overload
def process(line: tp.Union[str, bytes], out: tp.BinaryIO,
is_stdin: tp.Literal[False]) -> None:
...
@tp.overload
def process(line: tp.Union[str, bytes], out: tp.TextIO,
is_stdin: tp.Literal[True]) -> None:
...
def process(line, out, is_stdin):
"""Process a single line and writes modified version to output file."""
def sub_func(match: regex.Match[str]) -> str:
if match.group("stmt"):
# If statement is non-empty, strip comment and keep line-ending.
return "".join(match.group("indentation", "stmt", "line_ending"))
else:
# Line is comment-only, remove entire line.
return ""
# When reading from STDIN, line *should* be bytes because we opened
# 'fileinput.input' with mode 'rb', but line *can* be 'r' if script is being
# run in a pseudo-terminal like IDLE.
if isinstance(line, bytes):
line = tp.cast(str, line.decode("utf-8"))
else:
line = tp.cast(str, line)
if empty.fullmatch(line): # line is empty except for line ending.
# Empty lines do not have to be changed.
filtered_line = line
else:
# If <line_ending> is 'None', it would be replaced with an empty string.
filtered_line = stmt_parser.sub(
sub_func, line, count=1
)
# Should output as 'str' if reading from STDIN.
if not is_stdin:
filtered_line = filtered_line.encode("utf-8")
out.write(filtered_line)
def switch_file(prev_file: tp.Optional[str], filename: str,
close_only: bool = False
) -> tp.Optional[tp.Union[tp.TextIO, tp.BinaryIO]]:
"""Initializes the output file for each input. Should be called everytime a
new file has started being read, including STDIN."""
if not close_only and (fileinput.isstdin() or args.in_place):
# If file is being read from STDIN or in-place filtering, return STDOUT.
return sys.stdout
if prev_file is not None:
# The previous input file has been successfully processed,
# delete the backup file.
with contextlib.suppress(OSError):
# May raise an exception on Windows if another process opened the
# backup file for some reason.
os.remove(prev_file + BACKUP_EXT)
if close_only:
return
# Output filepath generation procedure:
# args.directory → Path("path/to/some/directory")
# filename → "path/to/some/file.txt"
# new_filename = "path/to/some/file" + "_stripped" + ".txt"
# out_path = Path(Path("path/to/some/directory"),
# "path/to/some/file_stripped.txt")
base, ext = os.path.splitext(filename)
new_filename = "".join((base, args.suffix, ext))
out_path = pathlib.Path(args.directory, new_filename)
if args.overwrite:
return out_path.open(mode="wb")
else:
# If user doesn't want silent over-write, then be more careful.
try:
file_obj = out_path.open(mode="xb")
except (OSError, FileExistsError):
# File already exists, abort instead of overwriting.
return
else:
return file_obj
def reader(files: fileinput.FileInput) -> None:
"""Main logic for reading through files and creating the filtered version.
Returns a bool indicating whether to skip to the next input file."""
def read_line(line: tp.Union[str, bytes]) -> bool:
nonlocal out_file, current_file
if tp.TYPE_CHECKING:
out_file = tp.cast(tp.Union[tp.TextIO, tp.BinaryIO], out_file)
if fileinput.isfirstline():
if out_file is not None and current_file != "<stdin>":
# If previous file was read from STDIN, don't close it.
out_file.close()
# We just read the first line of a new file. Prepare out file.
out_file = switch_file(
current_file,
current_file := ("<stdin>" if fileinput.isstdin()
else fileinput.filename()))
if out_file is None:
return True
if fileinput.isstdin():
process(line, tp.cast(tp.TextIO, out_file), True)
else:
process(line, tp.cast(tp.BinaryIO, out_file), False)
return False
def restore_backup() -> None:
nonlocal current_file
if tp.TYPE_CHECKING:
current_file = tp.cast(str, current_file)
with contextlib.suppress(OSError):
# Attempt to restore original file.
os.replace(current_file + BACKUP_EXT, current_file)
out_file: tp.Optional[tp.Union[tp.TextIO, tp.BinaryIO]] = None
current_file: tp.Optional[str] = None
# Input would be str if read from stdin.
line: tp.Union[str, bytes]
# 'file_input.__next__' can possibly error at the start of every iteration.
# 'read_line' can also possibly error every iteration.
while True:
try:
# Read a new line from the files.
for line in files:
try:
abort = read_line(line)
except (OSError, UnicodeDecodeError):
restore_backup()
else:
if abort:
fileinput.nextfile()
break
except OSError:
# Problematic file, skip to next one.
fileinput.nextfile()
if current_file is not None:
# delete backup of last file processed
switch_file(current_file, "", close_only=True)
def iter_files() -> None:
# Add comment character sequence to regex.
setup_regex_patterns(args.comment)
# 'args.files' is of type 'tp.List[str]', can possibly be empty.
with fileinput.input(
files=args.files, inplace=args.in_place,
backup=BACKUP_EXT if args.in_place else "",
mode="rb") as f:
reader(f)
def main() -> None:
global args
args = setup_cmdline_interface().parse_args()
with HandleStdoutClose():
iter_files()
if __name__ == "__main__":
main()
regexmodule from PyPI, not theremodule from the standard library (which doesn't support subroutine calls in regular expressions). So you have to runpip3 install regexin a terminal (orpipif you're on Windows instead of Linux), instead of changing the script to usereinstead ofregex. \$\endgroup\$