1
\$\begingroup\$

I'm trying to extract all files (XMLs) from an ftp site with python and the ftplib module, but it's incredibly slow so I suspect I'm doing something wrong.

The files are in subfolders, and all at the same "depth" eg: 'startingdir/folder1/place1/file.xml', 'startingdir/folder1/place1/file2.xml', 'startingdir/folder1/place2/file.xml', 'startingdir/folder2/place1/file.xml'

I want to extract everything regardless of location.

As a side note, if this can be put into a function which iterates the process until it finds .xml files in the directory and then extracts those, that would be incredibly helpful (rather than knowing the "depth" and putting the right number of nested for loops).

Code as it stands:

from ftplib import FTP
import os
destination = r'C:\Destination'

ftp = FTP('ftp.com')
ftp.login('user','pw')

Once I'm in, I get a list of all the folders in my starting directory:

ftp.cwd('startingdir')
templist = []
ftp.retrlines('LIST',templist.append)
subdirlist = [i.split(None,8)[-1].lstrip() for i in templist]

Then, I iterate this process a few times:

for i in subdirlist:
    ftp.cwd('startingdir/'+i)
    templist = []
    ftp.retrlines('LIST',templist.append)
    subdirlist2 = [i.split(None,8)[-1].lstrip() for i in templist]

    for j in subdirlist2:
        ftp.cwd('startingdir/'+ i + '/' + j)
        templist = []
        ftp.retrlines('LIST',templist.append)
        subdirlist3 = [i.split(None,8)[-1].lstrip() for i in templist]

Finally, I extract all files from the right "depth" folder:

        for k in subdirlist3:
            local_filename = os.path.join(destination,k)
            lf = open(local_filename, 'wb')
            ftp.retbinary('RETR '+ k, lf.write, 8*1024)
            lf.close

The code works, but it's incredibly slow (without counting precisely, it looks like 3-4 files/second). Is there anything obvious I'm doing wrong? Having 4-5 nested for loops doesn't look good, but I don't know how else to achieve this.

Thank you in advance for the suggestions!

\$\endgroup\$
  • \$\begingroup\$ Could you post your entire code instead of several stripped down snippets. To make life easier for a reviewer \$\endgroup\$ – Ludisposed Jan 11 at 12:45
  • \$\begingroup\$ Sorry if it wasn't clear - this is the entire code, just annotated to be clearer. There are two extra for loops that I excluded because they are exactly identical (this relates to the "depth" of the objects), so technically the final box would start with for m in subdirlist5:. That's all. \$\endgroup\$ – Jim Eisenberg Jan 11 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.