I have Python code to copy the Files locally on the Server from one Directory to another using rsync and subprocess module, this is just in continuation of the post from this post where I tried to get various insight but opted to use below code finally.

Please advise if this further can be improved or optimized in pythonic way and help to provide your esteemed reviews please.

import os
import glob
import datetime
import subprocess

def Copy_Logs():
   # Variable Declaration to get the month and Curr_date_month
    Info_month = datetime.datetime.now().strftime("%B")
    Curr_date_month = datetime.datetime.now().strftime("%b_%d_%y") 
    Sourcedir = "/data1/logs"
    Destdir = "/data2/logs/"
    ###### End of your variable section #######################
    # The result of the below glob _is_ a full path
    for filename in glob.glob("{2}/{0}/{1}/*.txt".format(Info_month, Curr_date_month, Sourcedir)):
        if os.path.getsize(filename) > 0:
            if not os.path.exists(Destdir + os.path.basename(filename)):
                subprocess.call(['rsync', '-avz', '--min-size=1', filename, Destdir ])

if __name__ == '__main__':

1 Answer 1


You're calling now() twice - what will happen if the month changes between those two calls?

Your code skips all files that exist, even though your linked question indicates that you need to update changed files.

Your code checks that file is non-empty, but rsync is already doing this for you with --min-size=1.

More generally, before you code up file-sync logic, it's best to read the rsync man page and see if your problem is already solved. Syncing files is a task full of corner cases and gotchas; life is too short for you to find them all. Just let rsync take care of it, whenever possible.

The code below will update changed files while skipping unchanged files (as in your problem description). If you want to copy only new, non-existing files (as in your posted code), add --ignore-existing to the rsync options.

import datetime
import subprocess

def Copy_Logs():
    Sourcedir = datetime.datetime.now().strftime("/data1/logs/%B/%b_%d_%y/")
    Destdir = "/data2/logs/"
    subprocess.call(['rsync', '-avz', '--min-size=1', '--include=*.txt', '--exclude=*', Sourcedir, Destdir ])
  • 3
    \$\begingroup\$ This is a good answer! I'd recommend using PEP8 names though (copy_logs, source_dir, and dest_dir). It may also be prudent to note that you may be able to achieve this with a bash oneliner like rsync -avz --min-size=1 --include=*.txt --exclude=* /data/logs/$(date "+%B/%b_%d_%y/") /data2/logs/. Don't pull out Python when you don't need to! \$\endgroup\$ Jan 24, 2019 at 1:05
  • 2
    \$\begingroup\$ If this isn't part of some larger Python project, you're absolutely right and there's no reason to use more than rsync and a shell. I will point out that your one-liner should quote the wildcards (yes, they'll almost never get interpolated by the shell as-is, but will it ever be baffling on the day that you have a file named --exclude=!) and doesn't need to quote the date format: rsync -avz --min-size=1 --include="*.txt" --exclude="*" /data1/logs/$(date +%B/%b_%d_%y/) /data2/logs/ \$\endgroup\$ Jan 24, 2019 at 1:27
  • \$\begingroup\$ @OhMyGoodness, thanks for giving the nice and precise details, However, current code I have do not skips all the files it only skips the empty files as I test the code by running it on my data but its current not to use own method while rsync is taking care that already. \$\endgroup\$
    – krock1516
    Jan 24, 2019 at 3:47
  • \$\begingroup\$ @BaileyParker, many thanks for a suggesting a nice one liner i'll keep that handy though. \$\endgroup\$
    – krock1516
    Jan 24, 2019 at 3:48
  • \$\begingroup\$ @krock1516 I don't understand how this doesn't skip existing files. What does if not os.path.exists(Destdir + os.path.basename(filename)): do? It tests that the file does not exist... \$\endgroup\$ Jan 24, 2019 at 4:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.