The purpose of the following function is to find all non-empty directories, and the files in those non-empty directories. It recursively checks each directory on an SFTP server to see if it has any files, and if it does, adds it to a default dict using the path as the key. The function uses paramiko.SFTPClient
and stat
. I am specifically concerned about the performance; it is rather slow.
Prereqsuite information
sftp.listdir_attr
returns a list ofSFTPAttribute
s which represent either files, directories, symlinks, etc., and contain ast_mode
, which is used to determine if it is a directory or file. This can throw an IOException for example if you don't have permissions to inspect the path.stat.S_ISDIR
will inspect the mode to determine if its a directory
The function in question:
def recursive_ftp(sftp, path='.', files=None):
if files is None:
files = defaultdict(list)
# loop over list of SFTPAttributes (files with modes)
for attr in sftp.listdir_attr(path):
if stat.S_ISDIR(attr.st_mode):
# If the file is a directory, recurse it
recursive_ftp(sftp, os.path.join(path,attr.filename), files)
else:
# if the file is a file, add it to our dict
files[path].append(attr.filename)
return files
Use:
import paramiko
import stat
transport = paramiko.Transport((host, port))
transport.connect(username=username, password=password)
sftp = paramiko.SFTPClient.from_transport(transport)
files = recursive_ftp(sftp)
If we have an SFTP server that looks like this:
/foo
----a.csv
----b.csv
/bar
----c.csv
/baz
The function will return a dictionary like so:
{
'./foo': ['a.csv', 'b.csv'],
'./bar': ['c.csv']
}