To formalize my comments above as an answer (even though it will be an incomplete one as to possible optimizations):

1. You can check other "significant" parts of the files before hashing the whole thing. For example, as you've done, not only the first 10 characters but then the final 10 (even a middle 10 afterwards in case they also share identical closing headers).

2. To do the above, and as a bonus to optimize the I/O aspect, you can avoid re-opening each file, which is quite slow. Instead, keep them open and read as necessary, or even read a large chunk and then close, but don't reopen.

3. For style commentary, since you requested it in version 1 of this thread:

   • Add a little whitespace between functions and for loop blocks, and generally between any self-contained block of 3-4 lines or more.
   • You presumably named the variable files because you'd already used filenames, but files actually refers to groups of filenames. What's more, the variable name doesn't indicate what's special about those "files". One suggestion might be collision_set.

4. Speaking of sets, for a minor improvement, you could store the values in sets instead of appending to lists.

To formalize my comments above as an answer (even though it will be an incomplete one as to possible optimizations):

1. You can check other "significant" parts of the files before hashing the whole thing. For example, as you've done, not only the first 10 characters but then the final 10 (even a middle 10 afterwards in case they also share identical closing headers).

2. To do the above, and as a bonus to optimize the I/O aspect, you can avoid re-opening each file, which is quite slow. Instead, keep them open and read as necessary, or even read a large chunk and then close, but don't reopen.

3. For style commentary, since you requested it in version 1 of this thread:

   • Add a little whitespace between functions and for loop blocks, and generally between any self-contained block of 3-4 lines or more.
   • You presumably named the variable files because you'd already used filenames, but files actually refers to groups of filenames. What's more, the variable name doesn't indicate what's special about those "files". One suggestion might be collision_set.

4. Speaking of sets, for a minor improvement, you could store the values in sets instead of appending to lists.

To formalize my comments above as an answer (even though it will be an incomplete one as to possible optimizations):

1. You can check other "significant" parts of the files before hashing the whole thing. For example, as you've done, not only the first 10 characters but then the final 10 (even a middle 10 afterwards in case they also share identical closing headers).

2. To do the above, and as a bonus to optimize the I/O aspect, you can avoid re-opening each file, which is quite slow. Instead, keep them open and read as necessary, or even read a bunch and discard, but don't reopen.

3. For style commentary, since you requested it in version 1 of this thread:

   • Add a little whitespace between functions and for loop blocks, and generally between any self-contained block of 3-4 lines or more.
   • You presumably named the variable files because you'd already used filenames, but files actually refers to groups of filenames. What's more, the variable name doesn't indicate what's special about those "files". One suggestion might be collision_set.

4. Speaking of sets, for a minor improvement, you could store the values in sets instead of appending to lists.

To formalize my comments above as an answer (even though it will be an incomplete one as to possible optimizations):

1. You can check other "significant" parts of the files before hashing the whole thing. For example, as you've done, not only the first 10 characters but then the final 10 (even a middle 10 afterwards in case they also share identical closing headers).

2. To do the above, and as a bonus to optimize the I/O aspect, you can avoid re-opening each file, which is quite slow. Instead, keep them open and read as necessary, or even read a large chunk and then close, but don't reopen.

3. For style commentary, since you requested it in version 1 of this thread:

   • Add a little whitespace between functions and for loop blocks, and generally between any self-contained block of 3-4 lines or more.
   • You presumably named the variable files because you'd already used filenames, but files actually refers to groups of filenames. What's more, the variable name doesn't indicate what's special about those "files". One suggestion might be collision_set.

4. Speaking of sets, for a minor improvement, you could store the values in sets instead of appending to lists.