Find all files that match a filename pattern in a folder tree

I am an old python newbie here, migrating from fortran77 ;) Still unsure of the python terminology at the moment :(

The python3 script below fetches a large list of matching files in a folder tree, in the format:

[ fully_qualified_folder_name, [list of matching files in this folder] ]


Being only a python newbie, there HAS to be a better way ?

I hope to use the code to generate a nightly json file underpinning a chromecasting web server on a Raspberry Pi 4, based on Google's example chrome (web page) "sender". The web page will eventually use javascript on the client-side for the user to choose a file from a list and "cast" it from the Pi4's apache2 web server over to the chromecast device. So, I thought, why not give try Python a try ?

Suggestions would be appreciated.

import os
import fnmatch
def yield_files_with_extensions(folder_path, file_match):
for root, dirs, files in os.walk(folder_path):
for file in files:
if fnmatch.fnmatch(file.lower(),file_match.lower()):
#yield os.path.join(root, file)
yield file
break  # without this line it traverses the subfolders too
def yield_files_in_subfolders(folder_path, file_match):
for root, dirs, files in os.walk(folder_path):
for d in dirs:
subfolder = os.path.join(root, d)
mp4_files = [subfolder, [f for f in yield_files_with_extensions(subfolder, file_match)]]
yield mp4_files
#
the_path = r'/mnt/mp4library/mp4library'
# retieve the topmost folder's files
mp4_files1 = [[ the_path, [f for f in yield_files_with_extensions(the_path, "*.mp4")] ]]
mp4_files2 = [f for f in yield_files_in_subfolders(the_path, "*.mp4")]
mp4_files2 = mp4_files1 + mp4_files2
crecords=0
cfiles=0
for a in mp4_files2:
crecords=crecords+1
print ("-----record "+str(crecords)+"---" + a[0] + " ... files=" + str(len(a[1])))
#print (a[0])
#print (a[1])
c=0
for b in a[1]:
c=c+1
print ("mp4 File "+str(c)+"---" + b)
cfiles=cfiles+c
print ('Count of files: ' + str(cfiles))

• Please can you explain what this code does. Currently your description just says "doesn't work" multiple times over - which makes this question off-topic as the code doesn't work the way you intend it to. If it's true that it doesn't work, then we can't help you until you get it working. Dec 8, 2019 at 15:42
• OK. Sorry. Perhaps in the other thread, however here the code does work. It walks a nominated folder root and fetches a large list of matching filenames in the folder/subfolder tree, in the format: [ fully_qualified_folder_name, [list of matching files in this folder] ] To clarify: Is there a better way to achioeve it ? Is there a way for os.walk to return stuff in alphabetical order ? Given these are .mp4 files, is there any way to determine the file size,duration,resolution of each file so I can add it to the list ? Dec 8, 2019 at 16:00
• Given that you've asked in the question, and the comments, ways to change your code to perform actions it currently doesn't seem to do I'm voting to close this as off-topic. We aren't a code writing service, you need to figure out how to sort and extract MP4 data by yourself or through a different service. Dec 8, 2019 at 16:06
• OK. I did ask if there was a better way though (a review question). Dec 8, 2019 at 16:10
• Yes, and you've asked off-topic things too. Maybe you can edit your post to make it only ask on-topic things? Dec 8, 2019 at 16:13

As you tagged this Python 3.x, I'd suggest using pathlib.Path.rglob() and collections.defaultdict. Also, check out f-strings.

import collections
import pathlib

mp4s = collections.defaultdict(list)

root = pathlib.Path(r'/mnt/mp4library/mp4library')

file_count = 0

for filepath in root.rglob('*.mp4'):
#filepath is relative to root. Uncomment the next line for absolute paths
#filepath = filepath.resolve()

mp4s[filepath.parent].append(filepath.name)

file_count += 1

for record_number, (folder, filenames) in enumerate(sorted(mp4s.items())):

print(f"-----record {record_number}---{folder} ... files={len(filenames)}")

for c,filename in enumerate(filenames):
print(f"mp4 File {c}---filename")

print('Count of files: {filecount}')

• Thanks ! I'll look into these. Dec 9, 2019 at 3:36
• This answer doesn't really review the code but just presents an alternative solution. Please, see: Why are alternative solutions not welcome? and What IS a Code Review?. Dec 9, 2019 at 13:43
• thank you, i reviewed it and discovered case sensitivity was an issue for me, so had to use '*.[mM][pP]4' Dec 10, 2019 at 2:28
• A dictionary is the more appropriate data structure to use here, since your code is essentially building a mapping from directory paths to lists of matched filenames. That is, instead of building a List[List[Union[str, List[str]]]], build a Dict[str, List[str]].

• A single call to os.walk is sufficient to perform the job that yield_files_in_subfolders and yield_files_with_extensions are currently doing together. For each 3-tuple (root, dirs, files), root is the containing directory and files is a list of non-directory files that reside directly under root.

• Do note that if we want each root directory (as mentioned above) to be an absolute path, we need to pass in an absolute path to os.walk. Calling os.path.abspath on the input directory path ensures this.

• To make the script easier to use and test, I'd recommend reading in the target (top-level) directory and the filename extension as command-line arguments. We can do this with argparse or sys.argv.

Here is the script with the above suggestions implemented:

#!/usr/bin/env python3

import os
import fnmatch
import argparse
from collections import defaultdict

def find_matching_files(directory, file_pattern):
# directory could be a relative path, so transform it into an absolute path
directory = os.path.abspath(directory)
directory_to_matched_files = defaultdict(list)

for root, _, files in os.walk(directory):
for file in files:
if fnmatch.fnmatch(file.lower(), file_pattern):
directory_to_matched_files[root].append(file)

return directory_to_matched_files

if __name__ == '__main__':
parser = argparse.ArgumentParser()
args = parser.parse_args()

files = find_matching_files(args.target_directory,
f'*.{args.filename_extension.lower()}')

# print report of directories & files
# [...]


Sample invocation:

\$ ./script.py /mnt/mp4library/mp4library -x mp4

• Thank you ! A nice example, as an os.walk alternative implementation to rglob. I see argparse would also be useful in my use case. Dec 9, 2019 at 9:47
• @RolandIllig You're right, just fixed it. Thanks! Dec 9, 2019 at 10:49