# Track changes inside a directory

I have built a Python 2.7 script to track all file and subdir changes inside a nominated directory. It is used with directories that have multiple levels of subdirectories, hundreds of thousands of files, and hundreds of GB of file data. The filenames can have Unicode characters (encoded in UTF-8). By "changes" I mean additions/deletions of files and subdirs, or changes in filesizes (i.e., we are not concerned with the content of the files). The tracking is not continuous, we are just comparing to the last time we checked (typically checking twice a day). The script works fine to the best of my knowledge.

I would gladly receive feedback on any aspect of the script including best coding practices and design pattern usage, handling of unexpected cases, and performance.

I include the whole script here. It's 310 lines long and I am wondering whether this might be too long as a question body, but I could not find size guidelines on the site. I opted to include everything instead of code snippets since this seems to be the recommended practice here. I also recognise that the width of my lines does not offer the best viewing opportunity inside the code box (which seems to fit 93 char lines). I normally use 120-char vertical rulers in my code, and sometimes I allow lines to go past them. I am not sure if I should modify my code to offer a better viewing chance here. Let me know if reading the code here is too annoying, and I'll wrap-line it.

You can find the code, with more backstory, details, and other pieces of code that help run the tool as a background agent here: https://github.com/boulis/Track-Dir-Changes

import json, subprocess
from argparse import ArgumentParser
from os import walk
from os.path import join, getsize
from datetime import datetime

parser = ArgumentParser(description="Tracks any changes in a specified directory. Additions, deletions,\n \
changes of files and subdirs are tracked and recorded in a log file.\n \
If user-defined thresholds are exceeded an alert is also created.\n")
parser.add_argument("-d","--dir", dest="dir", required=True, help="The directory monitored (required parameter)")
parser.add_argument("-s","--sizeabs", dest="size_abs", default=30, type=float, help="Number of MB of change in size to trigger an alert (default=30)")
parser.add_argument("-r","--sizerel", dest="size_rel", default=0.05, type=float, help="Fraction of change in size to trigger an alert (default=0.05)")
parser.add_argument("-q","--numrel", dest="num_rel", default=0.05, type=float, help="Fraction of files+dirs that need to be added/deleted to trigger an alert (default=0.05)")
parser.add_argument("-l","--logdir", dest="log_dir", default="logs/", help="The directory where log files and status information is kept (default logs/)")
parser.add_argument("--schedule", dest="daySchedule", default="",  help="Defines a series of times to run the script. The argument is a continuous string with no spaces, and times are comma-separated given in this form:  HH:MM,HH:MM,.. (NOT currently implemented)")
parser.add_argument("--persistentAlert", action='store_true',  help="If this flag is set then the alert to the user is a foreground window that requires pressing OK to dismiss")

args = parser.parse_args()

'''
In order to find all the changes done to a folder we have to keep a detailed snapshop of the folder structure.
This means we need to know all directories and files the main directory contains (along with individual item sizes)
as well as the same information for all subdirs (recursively). Note that we only rely on file size to determine that
a file has changed or not, not on a hash generated by the file. This is acceptable because the main application of
this script is to track size changes. The script is designed to be run periodically as an agent/daemon, and thus
should function after shut downs and restarts. Hence we use two files to store needed information.
One is a .json file recording the last state snapshop:  <log_dir>/Track<dir path>.json
The other is a log of the changes we find over time:    <log_dir>/Track<dir path>changes.log
'''

class Tracker:

def __init__(self, root, log_dir=''):

self.root = root # the directory that we will monitor and track changes
self.log_dir = log_dir # the directory where the logs and json files will be kept

self.previous_state = self.readPrevState()  # a structure to hold all dir and file info of our previous snapshot
self.current_state = {}  # a structure to hold all dir and file info of the current state

self.added_dirs = []    # a list of directories added along with file and subdir info
self.deleted_dirs = []  # a list of directories deleted along with the total size and number of files in them
self.added_files = {}   # a dictionary of files added. filename is the key, size is the value
self.deleted_files = {} # a dictionary of files deleted. filename is the key, size is the value
self.changed_files = {} # a dictionary of files changed. filename is the key, a tuple (old_size, new_size) is the value

self.added_total_size = 0   # The total size in bytes of all files added
self.deleted_total_size = 0 # The total size in bytes of all files deleted
self.changed_total_size = 0 # The total size in bytes of all files changes
self.deleted_total_num = 0 # The total number of all files deleted
self.changed_total_num = 0 # The total number of all files changed

self.current_total_size = 0 # The total size of all files inside the tracked dir
self.current_total_file_num = 0 # The total number of all files inside the tracked dir
self.current_total_dir_num = 0 # The total number of all subdirs inside the tracked dir

'''
Read the previous state of the root directory from a special file.
If the file does not exist or is corrupted, return an empty dict
'''

prev_state_filename = 'track{}.json'.format(self.root.replace('/','_'))
try:
with open(join(self.log_dir, prev_state_filename)) as state_file:
return json.load(state_file) # note: strings are returned as unicode strings

except (IOError, ValueError):
return {}

'''
Write the current state of the root directory to the special file. Overwrite file.
Note that all strings will be written as unicode. Non-ascii characters will be written as codepoints in ascii
(e.g., the letter alpha will be written as u'\u03b8'). We could write them as non-ascii utf-8 characters by
using the parameter ensure_ascii=False, but when reading this file the json.load() method will return unicode
strings anyway. Moreover, it is best practice to work with unicode strings, instead of a particular encoding.
'''
def writeCurrentState(self):
prev_state_filename = 'track{}.json'.format(self.root.replace('/','_'))
with open(join(self.log_dir, prev_state_filename), 'w') as state_file:
json.dump(self.current_state, state_file, encoding='utf-8', separators=(',', ':'))

'''
A function to return the total size of files and number of files of a deleted directory.
This implies that we are searching inside previous_state. The pathname should be a unicode string.
'''
def getSizeAndNum(self, pathname):

# Check if pathname is unicode. Left unchecked, the rest of the function's code will silently fail
# to find a regular string  with non-ascii chars inside the previous_state dict.
if type(pathname) is not unicode:
raise ValueError('pathname needs to be a unicode string, you have passed:', type(pathname))

if pathname not in self.previous_state: return (0, 0)

dirs, files_with_sizes = self.previous_state[pathname]
total_size = sum(files_with_sizes.values())
total_num = len(files_with_sizes)
# Recursively visit all subdirs
for d in dirs:
subdir_size, subdir_num = self.getSizeAndNum(join(pathname, d))
total_size += subdir_size
total_num += subdir_num

return (total_size, total_num)

'''
A function to find all additions, deletions, and changes in all files and subdirs
'''
def findChanges(self):

# reset the current state
self.current_state = {}

for path, curr_dirs, curr_files in walk(self.root):

# convert the path to Unicode, keep the original path to be used in finding filesizes
path_unicode = path.decode('utf-8')

# convert dir names in Unicode
curr_dirs = [d.decode('utf-8') for d in curr_dirs]

# Find the sizes of all the files in this directory, and create a dictionary.
# Use a dictionary comprehension. Store filenames in Unicode
curr_files_and_sizes = {fname.decode('utf-8'): getsize(join(path, fname)) for fname in curr_files}

# update the total counts
self.current_total_size += sum(curr_files_and_sizes.values())
self.current_total_file_num += len(curr_files_and_sizes)
self.current_total_dir_num += 1

# update the current state
self.current_state[path_unicode] = [curr_dirs, curr_files_and_sizes]

# check is this path existed in the previous state
if path_unicode in self.previous_state:
prev_dirs, prev_files_and_sizes = self.previous_state[path_unicode]

# check if sub dirs are the same
if prev_dirs != curr_dirs:
# we only want to get the deleted dirs here, since the
# added ones will appear in the path as we walk the tree
deleted_dirs_list = list(set(prev_dirs)-set(curr_dirs))
for dname in deleted_dirs_list:
# use a recursive function to get the total size and file number in the deleted dir
dir_size , dir_file_num = self.getSizeAndNum(join(path_unicode, dname))
self.deleted_dirs.append([join(path_unicode, dname), dir_size, dir_file_num])

# check if files are the same
if prev_files_and_sizes != curr_files_and_sizes:
# find the differences
deleted_files_set = set(prev_files_and_sizes) - set(curr_files_and_sizes)
for f in deleted_files_set:
self.deleted_files[join(path_unicode,f)] = prev_files_and_sizes[f]

for f in common_files_set:
if prev_files_and_sizes[f] != curr_files_and_sizes[f]:
self.changed_files[join(path_unicode,f)] = (prev_files_and_sizes[f], curr_files_and_sizes[f])

else:

# Finally calculate various aggregates:

# sum up the deleted files sizes, and count the files
self.deleted_total_size = sum(self.deleted_files.values())
self.deleted_total_num = len(self.deleted_files)
# add the total sizes and numbers of deleted directories
self.deleted_total_size += sum([size for p, size, num in self.deleted_dirs])
self.deleted_total_num  += sum([num for p, size, num in self.deleted_dirs])

# sum up the added files sizes, and count the files

# for changed files sum up the differences between old and new sizes for every changed file
self.changed_total_size = sum(map(lambda (old_size, new_size):abs(old_size-new_size), self.changed_files.values()))
self.changed_total_num = len(self.changed_files)

'''
Write all changes to the log file. Record the timestamp, summary of changes, and detailed list of changes.
If no changes happened, only the timestamp is recorded. If no previous snapshot is present we record
summary information of the new directory tracked.
'''
def writeChanges(self):

change_log_filename = 'track{}changes.log'.format(self.root.replace('/','_') )
with open(join(self.log_dir, change_log_filename), 'ab') as log_file:
# write a timestamp
log_file.write('----------------  {}  ----------------\n'.format(datetime.now().strftime('%Y-%m-%d %H:%M:%S')))
# check to see if we had no previous state, in which case we do not want to record all additions
if self.previous_state == {}:
log_file.write('New directory tracked.\n(Or .json file holding the previous state was deleted or corrupted)\n')
log_file.write('{} dirs and {} files occupying {}\n'.format(self.current_total_dir_num,
self.current_total_file_num,
else:
# provide a summary of the changes
report_for_added = ''; report_for_deleted =''; report_for_changed = ''
if self.deleted_total_size > 0:
report_for_deleted = '- Deleted {} files totalling {} -'.format(self.deleted_total_num, self.humanReadableSize(self.deleted_total_size))
if self.changed_total_size > 0:
report_for_changed = '- {} files changed by {} -'.format(self.changed_total_num, self.humanReadableSize(self.changed_total_size))

log_file.write(report_for_added + report_for_deleted + report_for_changed + '\n')

# provide a detailed list of all changes
for d in self.deleted_dirs:
path, size, files_num = d
log_file.write('Deleted dir: {} contained {} in {} files\n'.format(path.encode('utf-8'), self.humanReadableSize(size), files_num))

path, dirs, files_with_sizes = d
size = sum(files_with_sizes.values());
files_num = len(files_with_sizes)

for f, size in self.deleted_files.iteritems():
log_file.write('Deleted file: {} was {} bytes\n'.format(f.encode('utf-8'), size))

log_file.write('Added file: {} is {} bytes\n'.format(f.encode('utf-8'), size))

for f, (old_size, new_size) in self.changed_files.iteritems():
log_file.write('Changed file: {} from {} to {} bytes\n'.format(f.encode('utf-8'), old_size, new_size))

'''
A function to decide whether to alert the user and what kind of alert to present (persistent, or notification)
Note that size_abs is expressed in bytes, not MB
'''

if self.previous_state == {}: return

size = self.added_total_size + self.deleted_total_size + self.changed_total_size
num  = self.added_total_num  + self.deleted_total_num  + self.changed_total_num

if (size > size_abs or
size > size_rel * self.current_total_size or
num  > num_abs or
num  > num_rel * self.current_total_file_num):

alt_applescript = 'display dialog "Added/Deleted/Changed: {}, {} files" with title "Boulis Directory Tracker" with icon caution buttons {{"OK"}}'.format(self.humanReadableSize(size), num)

# check whether we need a persistent window or just a notification, and run the appropriate applescript
subprocess.call("osascript -e '{}'".format(alt_applescript), shell=True)
else:
subprocess.call("osascript -e '{}'".format(applescript), shell=True)

'''
A function to print file sizes in a more human readable form (using KB, MB, GB)
It also uses valiable decimal precision for different sizes. 0 decimals for bytes and KB, 2 for MB, 3 for GB
'''

for unit, decimals_printed in zip(['bytes','KB','MB', 'GB'], [0, 0, 2, 3]):
if abs(num) < 1024.0:
if decimals_printed == 0:
return '{} {}'.format(int(round(num)), unit)
else:
# create the format string to fit the desired decimal precision
# use {{ and }} to escape the special characters { and }. for decimals= 3 this will return '{:.3f} {}'
format_string = '{{:.{}f}} {{}}'.format(decimals_printed)
return format_string.format(num, unit)
num /= 1024.0
# if the num is bigger than 1024 after all divisions, just use the larger unit
return '{.3f} GB'.format(num)

def singleRun(root, log_dir, size_abs, size_rel, num_abs, num_rel, persistentAlert):

t = Tracker(root, log_dir)
t.findChanges()
t.writeChanges()
t.writeCurrentState()

def main_loop():
singleRun(args.dir, args.log_dir, args.size_abs *1024*1024, args.size_rel, args.num_abs, args.num_rel, args.persistentAlert)
# There are provisions to include multiple scheduled runs in the future, hence the parameter --schedule.
# Not needed for the Mac OSX environment

if __name__ == '__main__':
main_loop()


You've done a great job documenting your code.

Here are a few code style and code organization things I would work on:

• read arguments either inside the main_loop() function, or inside the if __name__ == '__main__': block of the code. This way, if your script would be imported, arguments would not be parsed. Also, consider having a separate parse_args() function that would be responsible for parsing arguments. This may lead to better modularity and testability
• docstrings should go right after the class, method or function definitions; should be enclosed into triple double-quotes, start with a capital letter and end with a dot (PEP8 reference)
• naming; follow the lower_case_with_underscores naming recommendations
• the humanReadableSize() method should not be under the Tracker class - it feels like this is more of a helper/utility type of a function - consider extracting it into a separate "libs"/"utils" module
• you may remove redundant parenthesis replacing return (total_size, total_num) with return total_size, total_num and return (0, 0) with return 0, 0
• the unicode type check can be done with isinstance():
if not isinstance(pathname, unicode):

• the emptiness check can be simplified to if not self.previous_state:
• @Thanassis ah, I guess my editor expanded them into multiple lines. But, you still have a single trailing ; here: size = sum(files_with_sizes.values()); :) And, yes, you should avoid putting multiple statements on a single line like this. You can also do: report_for_added = report_for_deleted = report_for_changed = ''. The 80 chars limit is too old and harsh of a requirement. I usually configure my linter and editor for 100 or 120. Thanks. – alecxe Apr 16 '17 at 3:03