Today I wanted to find a program that I wrote a while ago. I knew that it contained a certain regex, but I couldn't for the life of me remember the file name I saved it under. I knew I could use Windows search, but it takes more time then it would for me to write a Python program to do the same.
The main two things I use are os.walk
and re
, the former to traverse the entire directory tree, where the latter is to match the data. I also use codecs
to allow me to read files with special characters. And finally I use argparse
to get the input from the end user.
Some files still error when using codecs
such as pngs or other raw data files, so I skip these.
I kept the arguments simple, you pass a regex and a path. You can also pass any of the regex flags. So the following will search for 'metaclass' in any case, in the files below 'D:\data
'.
python search.py "metaclass" "D:\data" -i
The code is fairly small and mostly just adds information to the parser. It also runs in both Python2 and Python3.
import re
import codecs
import argparse
import operator
from os import walk
from os.path import join
# Add reduce to global scope for Python3
try:
from functools import reduce
except ImportError:
pass
# Descriptions are the same as Python's re descriptions
# https://docs.python.org/2.7/library/re.html#module-contents
# https://docs.python.org/3.5/library/re.html#module-contents
parser = argparse.ArgumentParser(description='Search file contense.')
parser.add_argument('regex', help='regex to search for')
parser.add_argument('path', help='path to root of recursive search')
parser.add_argument('-a', '--ascii', action="store_true",
help='(Python3 only) Make `\w`, `\W`, `\b`, `\B`, `\d`, '
'`\D`, `\s` and `\S` perform ASCII-only matching '
'instead of full Unicode matching. This is only '
'meaningful for Unicode patterns, and is ignored for '
'byte patterns.')
parser.add_argument('-i', '--ignorecase', action="store_true",
help='Perform case-insensitive matching; expressions like '
'`[A-Z]` will match lowercase letters, too. This is '
'not affected by the current locale.')
parser.add_argument('-l', '--locale', action="store_true",
help='Make `\w`, `\W`, `\b`, `\B`, `\s` and `\S` '
'dependent on the current locale.')
parser.add_argument('-m', '--multiline', action="store_true",
help='When specified, the pattern character \'^\' matches '
'at the beginning of the string and at the beginning '
'of each line (immediately following each newline); '
'and the pattern character \'$\' matches at the end '
'of the string and at the end of each line '
'(immediately preceding each newline). By default, '
'\'^\' matches only at the beginning of the string, '
'and \'$\' only at the end of the string and '
'immediately before the newline (if any) at the end '
'of the string.')
parser.add_argument('-s', '--dotall', action="store_true",
help='Make the \'.\' special character match any '
'character at all, including a newline; without this '
'flag, \'.\' will match anything except a newline.')
parser.add_argument('-u', '--unicode', action="store_true",
help='Make `\w`, `\W`, `\b`, `\B`, `\d`, `\D`, `\s` and '
'`\S` dependent on the Unicode character properties '
'database.')
parser.add_argument('-x', '--verbose', action="store_true",
help='This flag allows you to write regular expressions '
'that look nicer and are more readable by allowing '
'you to visually separate logical sections of the '
'pattern and add comments. Whitespace within the '
'pattern is ignored, except when in a character '
'class or when preceded by an unescaped backslash. '
'When a line contains a # that is not in a character '
'class and is not preceded by an unescaped '
'backslash, all characters from the leftmost such # '
'through the end of the line are ignored.')
parser.add_argument('--debug', action="store_true",
help='Display debug information about compiled expression.')
def get_args():
args = vars(parser.parse_args())
path = args.pop('path')
regex = args.pop('regex')
regex_flags = [getattr(re, k.upper()) for k, v in args.items() if v]
regex = re.compile(regex, reduce(operator.or_, regex_flags, 0))
return path, regex
def get_files(path, regex):
search = regex.search
for root, dirs, files in walk(path):
for name in files:
with codecs.open(join(root, name), 'r', 'utf-8') as f:
try:
data = f.read()
except (UnicodeError, ValueError):
continue
match = search(data)
if match is not None:
yield join(root, name), match
if __name__ == '__main__':
for path, match in get_files(*get_args()):
print('match: {!r} ({})'.format(path, match.group()))
My main concerns are:
- How/where I write the parser.
- How much my functions do. Do they violate SRP?
- Is there a more memory efficient way to
search
? Reading the entire file seems bad.