I'm working with directory of raw data .csv files, stored daily in a directory with the same name except date stamp in the name, for example ABCD_20190221_version2.csv.
The below function works fine but given that directory 10K files and growing, it's slow. Any idea of how to speed it up?

import re
from pathlib import Path
from datetime import datetime

def recent_files(directory, pattern, days_old_cutoff):
    """  helper1: globs the dir for a list of most recent file paths
         subject to days_old_cutoff     """
    res = {}
    for f in Path(directory).iterdir():
        short_file_name, time = re.sub(pattern,'', f.name), f.stat().st_mtime
        if short_file_name not in res: 
           res[short_file_name] = f, time
            _,t_ = res[short_file_name]
            if time > t_:
                res[short_file_name] = f, time

    file_list = [f for f, _ in res.values()
                if abs(datetime.today() - 
                       < days_old_cutoff]
    return file_list  

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.