I have users uploading files sometimes as large as 100+ GB to a local web server. The upload process works well and chunks come in at 50MB. The problem seems to be after the file is uploaded, when the web server assembles the files and the server (24GB RAM), despite not showing any graphical signs of memory pressure, gets very sluggish. I want to make sure my code isn't causing any unnecessary slow-downs.
Please suggest any more efficient way to do it. If it is already ok, then I'll know to look at other aspects of the process.
# open the temp file to write into with open(temp_filename, 'wb') as temp_file: # loop over the chunks for i in range(total_chunks): with open(os.path.join(get_chunk_filename(chunk_identifier, i + 1)), 'rb') as chunk_file: # write the chunk to the temp file temp_file.write(chunk_file.read())