I'm trying to extract all files (XMLs) from an ftp site with python and the ftplib module, but it's incredibly slow so I suspect I'm doing something wrong.
The files are in subfolders, and all at the same "depth" eg: 'startingdir/folder1/place1/file.xml', 'startingdir/folder1/place1/file2.xml', 'startingdir/folder1/place2/file.xml', 'startingdir/folder2/place1/file.xml'
I want to extract everything regardless of location.
As a side note, if this can be put into a function which iterates the process until it finds .xml files in the directory and then extracts those, that would be incredibly helpful (rather than knowing the "depth" and putting the right number of nested for loops).
Code as it stands:
from ftplib import FTP import os destination = r'C:\Destination' ftp = FTP('ftp.com') ftp.login('user','pw')
Once I'm in, I get a list of all the folders in my starting directory:
ftp.cwd('startingdir') templist =  ftp.retrlines('LIST',templist.append) subdirlist = [i.split(None,8)[-1].lstrip() for i in templist]
Then, I iterate this process a few times:
for i in subdirlist: ftp.cwd('startingdir/'+i) templist =  ftp.retrlines('LIST',templist.append) subdirlist2 = [i.split(None,8)[-1].lstrip() for i in templist] for j in subdirlist2: ftp.cwd('startingdir/'+ i + '/' + j) templist =  ftp.retrlines('LIST',templist.append) subdirlist3 = [i.split(None,8)[-1].lstrip() for i in templist]
Finally, I extract all files from the right "depth" folder:
for k in subdirlist3: local_filename = os.path.join(destination,k) lf = open(local_filename, 'wb') ftp.retbinary('RETR '+ k, lf.write, 8*1024) lf.close
The code works, but it's incredibly slow (without counting precisely, it looks like 3-4 files/second). Is there anything obvious I'm doing wrong? Having 4-5 nested for loops doesn't look good, but I don't know how else to achieve this.
Thank you in advance for the suggestions!