I needed to make a base64 file encoder where you can control the read buffer size. This is what I came up with and it's quite fast. It might be able to be simpler but still maintain its performance characteristics. Any suggestions?
def chunked_base64_encode(input, input_size, output, read_size=1024): """ Read a file in configurable sized chunks and write to it base64 encoded to an output file. This is an optimization over ``base64.encode`` which only reads 57 bytes at a time from the input file. Normally this is OK if the file in question is opened with ``open`` because Python will actually read the data into a larger buffer and only feed out 57 bytes at a time. But if the input file is something like a file stream that's read over the network, only 57 bytes will be read at a time. This is very slow if the file stream is not buffered some other way. This is the case for MongoDB GridFS. The GridOut file returned by GridFS is not a normal file on disk. Instead it's a file read in 256 KB chunks from MongoDB. If you read from it 57 bytes at a time, GridFS will read 256 KB then make lots of copies of that chunk to return only 57 bytes at a time. By reading in chunks equal to the GridFS chunk size, performance is 300 times better. Performance comparison: File size 10 MB Save to MongoDB took 0.271495819092 seconds Fast Base 64 encode (chunk size 261120) took 0.250380992889 seconds Base 64 encode (chunk size 57) took 62.9280769825 seconds File size 100 MB Save to MongoDB took 0.994009971619 seconds Fast Base 64 encode (chunk size 261120) took 2.78231501579 seconds Base 64 encode (chunk size 57) took 645.734956026 seconds For regular files on disk, there is no noticeable performance gain for this function over ``base64.encode`` because of Python's built in buffering for disk files. Args: input (file): File like object (implements ``read()``). input_size (int): Size of file in bytes output (file): File like object (implements ``write()``). read_size (int): How many bytes to read from ``input`` at a time """ # 57 bytes of input will be 76 bytes of base64 chunk_size = base64.MAXBINSIZE base64_line_size = base64.MAXLINESIZE # Read size needs to be in increments of chunk size for base64 # output to be RFC 3548 compliant. read_size = read_size - (read_size % chunk_size) num_reads = int(ceil(input_size / float(read_size))) # RFC 3548 says lines should be 76 chars base64_lines_per_read = read_size / chunk_size input.seek(0) for r in xrange(num_reads): is_last_read = r == num_reads - 1 s = input.read(read_size) if not s: # If this were to happen, then ``input_size`` is wrong or # the file is corrupt. raise ValueError( u'Expected to need to read %d times but got no data back on read %d' % ( num_reads, r + 1)) data = b2a_base64(s) if is_last_read: # The last chunk will be smaller than the others so the # line count needs to be calculated. b2a_base64 adds a line # break so we don't count that char base64_lines_per_read = int(ceil((len(data) - 1) / float(base64_line_size))) # Split the data chunks into base64_lines_per_read number of # lines, each 76 chars long. for l in xrange(base64_lines_per_read): is_last_line = l == base64_lines_per_read - 1 pos = l * base64_line_size line = data[pos:pos + base64_line_size] output.write(line) if not (is_last_line and is_last_read): # The very last line will already have a \n because of # b2a_base64. The other lines will not so we add it output.write('\n')