# Copying files locally with rsync and subprocess

I have Python code to copy the Files locally on the Server from one Directory to another using rsync and subprocess module, this is just in continuation of the post from this post where I tried to get various insight but opted to use below code finally.

#!/bin/python3
import os
import glob
import datetime
import subprocess

def Copy_Logs():
# Variable Declaration to get the month and Curr_date_month
Info_month = datetime.datetime.now().strftime("%B")
Curr_date_month = datetime.datetime.now().strftime("%b_%d_%y")
Sourcedir = "/data1/logs"
Destdir = "/data2/logs/"
###### End of your variable section #######################
# The result of the below glob _is_ a full path
for filename in glob.glob("{2}/{0}/{1}/*.txt".format(Info_month, Curr_date_month, Sourcedir)):
if os.path.getsize(filename) > 0:
if not os.path.exists(Destdir + os.path.basename(filename)):
subprocess.call(['rsync', '-avz', '--min-size=1', filename, Destdir ])

if __name__ == '__main__':
Copy_Logs()


You're calling now() twice - what will happen if the month changes between those two calls?

Your code skips all files that exist, even though your linked question indicates that you need to update changed files.

Your code checks that file is non-empty, but rsync is already doing this for you with --min-size=1.

More generally, before you code up file-sync logic, it's best to read the rsync man page and see if your problem is already solved. Syncing files is a task full of corner cases and gotchas; life is too short for you to find them all. Just let rsync take care of it, whenever possible.

The code below will update changed files while skipping unchanged files (as in your problem description). If you want to copy only new, non-existing files (as in your posted code), add --ignore-existing to the rsync options.

import datetime
import subprocess

def Copy_Logs():
Sourcedir = datetime.datetime.now().strftime("/data1/logs/%B/%b_%d_%y/")
Destdir = "/data2/logs/"
subprocess.call(['rsync', '-avz', '--min-size=1', '--include=*.txt', '--exclude=*', Sourcedir, Destdir ])

• This is a good answer! I'd recommend using PEP8 names though (copy_logs, source_dir, and dest_dir). It may also be prudent to note that you may be able to achieve this with a bash oneliner like rsync -avz --min-size=1 --include=*.txt --exclude=* /data/logs/$(date "+%B/%b_%d_%y/") /data2/logs/. Don't pull out Python when you don't need to! – Bailey Parker Jan 24 '19 at 1:05 • If this isn't part of some larger Python project, you're absolutely right and there's no reason to use more than rsync and a shell. I will point out that your one-liner should quote the wildcards (yes, they'll almost never get interpolated by the shell as-is, but will it ever be baffling on the day that you have a file named --exclude=!) and doesn't need to quote the date format: rsync -avz --min-size=1 --include="*.txt" --exclude="*" /data1/logs/$(date +%B/%b_%d_%y/) /data2/logs/ – Oh My Goodness Jan 24 '19 at 1:27
• @OhMyGoodness, thanks for giving the nice and precise details, However, current code I have do not skips all the files it only skips the empty files as I test the code by running it on my data but its current not to use own method while rsync is taking care that already. – krock1516 Jan 24 '19 at 3:47
• @BaileyParker, many thanks for a suggesting a nice one liner i'll keep that handy though. – krock1516 Jan 24 '19 at 3:48
• @krock1516 I don't understand how this doesn't skip existing files. What does if not os.path.exists(Destdir + os.path.basename(filename)): do? It tests that the file does not exist... – Oh My Goodness Jan 24 '19 at 4:01