I have some code that searches for duplicate files and then removes those files from a csv. To make the set of files I am searching over much smaller, I first group the files by size; if a file has a size that it is unique, it is safe to assume that it is unique and does not need to be included in the set of files I am checking for duplicates. Additionally, I must only consider files that are listed in the csv. Occasionally the csv has filenames of files that are not in the directory, which leaves me with several choices.
- I can initially check my list of files for those which do not exist and remove them. This adds a loop over all the files and seems unnecessary/inefficient; it is very rare that a file does not exist.
- Use the following code with exception handling when the file is not found and delete file from my list at this point. However this introduces a side effect to the grouping function. Clearly this function does more than its name suggests.
Code:
def group_by_size(self, files_to_group):
grouped_by_size = dict()
for fileid in files_to_group:
full_path = os.path.join(self.folder, fileid)
try:
size = os.path.getsize(full_path)
except WindowsError:
print WindowsError
print "The file wasn't found. Deleting row from PremsPhotos."
del self.files[fileid]
del self.spreadsheet_reader.main_row_dict[fileid]
if grouped_by_size.get(size):
grouped_by_size[size].append(fileid)
else:
grouped_by_size[size]=[]
grouped_by_size[size].append(fileid)
return grouped_by_size
Alternatively, eliminate duplicates beforehand:
def remove_nonexistant_photos(self):
for photo in self.files:
os.path.exists(os.path.join(self.folder, photo))
The context of this code is in a script that will likely be run a lot, but will not likely be modified once it works. Looping over all the files in the directory takes about half a second when there are 15k files in the directory. With both of these in mind, I suppose the decision is not of huge importance, but I have been doing a lot of reading about best practices, and want to improve my coding intuition/style as I go forward.