8
\$\begingroup\$

I created this function to recursively copy an entire directory from an FTP server. It works just fine except that it is about 4 times slower than using FileZilla to do the same operation. It takes approximately 55 seconds to download the directory in FileZilla but it takes 229 seconds with this function. What can I do to make it download/run faster?

Private Sub CopyEntireDirectory(ByVal directory As String)
    Dim localPath = localDirectory & formatPath(directory)
    'creates directory in destination path
    IO.Directory.CreateDirectory(localPath)

    'Gets the directory details so I can separate folders from files
    Dim fileList As ArrayList = Ftp.ListDirectoryDetails(directory, "")

    For Each item In fileList
        'checks if it's a folder or file: d=folder
        If (item.ToString().StartsWith("d")) Then
            'gets the directory from the details
            Dim subDirectory As String = item.ToString().Substring(item.ToString().LastIndexOf(" ") + 1)
            CopyEntireDirectory(directory & "/" & subDirectory)
        Else
            Dim remoteFilePath As String = directory & "/" & item.ToString().Substring(item.ToString().LastIndexOf(" ") + 1)
            Dim destinationPath = localPath & "\" & item.ToString().Substring(item.ToString().LastIndexOf(" ") + 1)
            'downloads file to destination directory
            Ftp.DownLoadFile(remoteFilePath, destinationPath)
        End If
    Next
End Sub

This function below is what is actually taking up 97% of the time.

Public Sub DownLoadFile(ByVal fromFilename As String, ByVal toFilename As String)
    Dim files As ArrayList = Me.ListDirectory(fromFilename, "")
    Dim request As FtpWebRequest = Me.CreateRequestObject(fromFilename)
    request.Method = WebRequestMethods.Ftp.DownloadFile

    Dim response As FtpWebResponse = CType(request.GetResponse(), FtpWebResponse)
    If response.StatusCode <> FtpStatusCode.OpeningData AndAlso response.StatusCode <> FtpStatusCode.DataAlreadyOpen Then
        Throw New ApplicationException(Me.BuildCustomFtpErrorMessage(request, response))
    End If

    Dim fromFilenameStream As Stream = response.GetResponseStream()
    Dim toFilenameStream As FileStream = File.Create(toFilename)

    Dim buffer(BLOCK_SIZE) As Byte
    Dim bytesRead As Integer = fromFilenameStream.Read(buffer, 0, buffer.Length)
    Do While bytesRead > 0
        toFilenameStream.Write(buffer, 0, bytesRead)
        Array.Clear(buffer, 0, buffer.Length)
        bytesRead = fromFilenameStream.Read(buffer, 0, buffer.Length)
    Loop

    response.Close()
    fromFilenameStream.Close()
    toFilenameStream.Close()
End Sub
\$\endgroup\$
  • 3
    \$\begingroup\$ FileZilla is probably using threads to download multiple files at the same time. You may want to consider first creating the directory structure and a list of files with full paths to download, then using the Task Parallel Library to split that list in to multiple download jobs. Specifically, Parallel.ForEach msdn.microsoft.com/en-us/library/dd460720(v=vs.110).aspx \$\endgroup\$ – Bradley Uffner Dec 2 '15 at 20:28
  • \$\begingroup\$ It appears that FileZilla sets up a queue and downloads the files one after another. \$\endgroup\$ – Programmer Dec 2 '15 at 20:30
  • \$\begingroup\$ I wouldn't know how this compares speedwise, but I've previously found this useful: stackoverflow.com/a/7535879/1316573 \$\endgroup\$ – Daniel Dec 4 '15 at 14:59
  • \$\begingroup\$ @BradleyUffner I am using Parallel.ForEach but there is a hangup when it uses Me.ListDirectory(fromFilename, ""). Somewhere it doesn't allow Parallel Tasks and is blocking the operation. \$\endgroup\$ – Programmer Dec 14 '15 at 17:14
-3
\$\begingroup\$

Check your buffer/BLOCKSIZE. If it's too small, it may be making many more reads and writes, and each has to wait for the network to respond.

I'm not given any information as to what block size you're using, so I can't help anymore.

\$\endgroup\$
  • 1
    \$\begingroup\$ Please expand on what is harmful about having a too-small buffer. Is there a method to determine the optimal buffer size? \$\endgroup\$ – user34073 Jan 16 '16 at 2:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.