1
\$\begingroup\$

After searching and reading some threads (1, 2, 3, 4 and 5) on how to split a big file into smaller ones and then merge them back into the original one I made the following VB.NET code on an empty project with module Module1 and reference to System.Windows.Forms:

' <Summary>
'
'   3 files: 1; 2; 3
'   1 has data
'   2 and 3 are empty
'
'   How to split file 1 into files 2 and 3:
'     remove 2º comment (leave 1º comment)
'     run program (select file 1 in the 1º file system prompt and file 2 in the 2º file system prompt)
'     add 2º comment back and remove 1º comment
'     run program (select file 1 in the 1º file system prompt and file 3 in the 2º file system prompt)
'
'   How to join/merge files 2 and 3 into file 1:
'     leave both 1º and 2º comments commented
'     run program (select file 3 in the 1º file system prompt and file 2 in the 2º file system prompt)
'     file 2 will be the merge of files 2 and 3 (same as file 1)
'
' </Summary>

Imports System.IO
Module Module1
    Sub Main()
        With New Windows.Forms.OpenFileDialog()
            .ShowDialog()
            Dim inputStream As FileStream = New FileStream(.FileName(), FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite)
            .ShowDialog()
            Dim outputStream As FileStream = New FileStream(.FileName(), FileMode.Append, FileAccess.Write, FileShare.ReadWrite)
            'inputStream.Position = inputStream.Length \ 2 - inputStream.Length \ 2 Mod 10000
            Dim buffer As Byte() = New Byte(9999) {}
            Dim bytesRead As Integer
            bytesRead = inputStream.Read(buffer, 0, buffer.Length)
            While bytesRead > 0 'And inputStream.Position <= inputStream.Length \ 2
                outputStream.Write(buffer, 0, bytesRead)
                bytesRead = inputStream.Read(buffer, 0, buffer.Length)
            End While
            outputStream.Flush()
            inputStream.Close()
            outputStream.Close()
        End With
    End Sub
End Module

I'm looking to know if there's any different way to do this with a noticeable performance improvement. The size of the original file is almost 30 GB, which will be split into 2 files, sent online together with the program and then merged back together.

Being of that size it takes some time so performance is the focus here, there's no checks of the user's input, no exception/error handling, no pretty code with forms, options and methods/functions. It's just some quick and dirty code.

\$\endgroup\$
1
\$\begingroup\$

I would suggest, to get performance and still keep the original intact, that .net libraries are probably not the best choice.

One suggestion that would work, if you can allow smaller chunks(8GB), is 7-Zip. It's free, open source and includes a file explorer. It will split a 32GB file into 5 volumes in about 20 min. Anyone with the same software installed can combine the volumes back again. It even includes the option to calculate a checksum for the file to help confirm it's still the same file.

\$\endgroup\$
  • \$\begingroup\$ I didn't knew 7-Zip could do that! Always thought of 7-Zip and WinRAR as just file compression and decompression programs. Thanks a lot. \$\endgroup\$ – user7393973 Jul 25 '18 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.