I have some code that searches for duplicate files and then removes those files from a csv. To make the set of files I am searching over much smaller, I first group the files by size; if a file has a size that it is unique, it is safe to assume that it is unique and does not need to be included in the set of files I am checking for duplicates. Additionally, I must only consider files that are listed in the csv. Occasionally the csv has filenames of files that are not in the directory, which leaves me with several choices.
- I can initially check my list of files for those which do not exist and remove them. This adds a loop over all the files and seems unnecessary/inefficient; it is very rare that a file does not exist.
- Use the following code with exception handling when the file is not found and delete file from my list at this point. However this introduces a side effect to the grouping function. Clearly this function does more than its name suggests.
def group_by_size(self, files_to_group): grouped_by_size = dict() for fileid in files_to_group: full_path = os.path.join(self.folder, fileid) try: size = os.path.getsize(full_path) except WindowsError: print WindowsError print "The file wasn't found. Deleting row from PremsPhotos." del self.files[fileid] del self.spreadsheet_reader.main_row_dict[fileid] if grouped_by_size.get(size): grouped_by_size[size].append(fileid) else: grouped_by_size[size]= grouped_by_size[size].append(fileid) return grouped_by_size
Alternatively, eliminate duplicates beforehand:
def remove_nonexistant_photos(self): for photo in self.files: os.path.exists(os.path.join(self.folder, photo))
The context of this code is in a script that will likely be run a lot, but will not likely be modified once it works. Looping over all the files in the directory takes about half a second when there are 15k files in the directory. With both of these in mind, I suppose the decision is not of huge importance, but I have been doing a lot of reading about best practices, and want to improve my coding intuition/style as I go forward.