5
\$\begingroup\$

I needed to make a base64 file encoder where you can control the read buffer size. This is what I came up with and it's quite fast. It might be able to be simpler but still maintain its performance characteristics. Any suggestions?

def chunked_base64_encode(input, input_size, output, read_size=1024):
    """
    Read a file in configurable sized chunks and write to it base64
    encoded to an output file.

    This is an optimization over ``base64.encode`` which only reads 57
    bytes at a time from the input file. Normally this is OK if the
    file in question is opened with ``open`` because Python will
    actually read the data into a larger buffer and only feed out
    57 bytes at a time. But if the input file is something like a
    file stream that's read over the network, only 57 bytes will be
    read at a time. This is very slow if the file stream is not
    buffered some other way.

    This is the case for MongoDB GridFS. The GridOut file returned by
    GridFS is not a normal file on disk. Instead it's a file read in
    256 KB chunks from MongoDB. If you read from it 57 bytes at a time,
    GridFS will read 256 KB then make lots of copies of that chunk
    to return only 57 bytes at a time. By reading in chunks equal
    to the GridFS chunk size, performance is 300 times better.

    Performance comparison:

        File size 10 MB
        Save to MongoDB took 0.271495819092 seconds
        Fast Base 64 encode (chunk size 261120) took 0.250380992889 seconds
        Base 64 encode (chunk size 57) took 62.9280769825 seconds

        File size 100 MB
        Save to MongoDB took 0.994009971619 seconds
        Fast Base 64 encode (chunk size 261120) took 2.78231501579 seconds
        Base 64 encode (chunk size 57) took 645.734956026 seconds

    For regular files on disk, there is no noticeable performance gain
    for this function over ``base64.encode`` because of Python's built
    in buffering for disk files.

    Args:
        input (file): File like object (implements ``read()``).
        input_size (int): Size of file in bytes
        output (file): File like object (implements ``write()``).
        read_size (int): How many bytes to read from ``input`` at
            a time
    """
    # 57 bytes of input will be 76 bytes of base64
    chunk_size = base64.MAXBINSIZE
    base64_line_size = base64.MAXLINESIZE
    # Read size needs to be in increments of chunk size for base64
    # output to be RFC 3548 compliant.
    read_size = read_size - (read_size % chunk_size)
    num_reads = int(ceil(input_size / float(read_size)))
    # RFC 3548 says lines should be 76 chars
    base64_lines_per_read = read_size / chunk_size

    input.seek(0)
    for r in xrange(num_reads):
        is_last_read = r == num_reads - 1
        s = input.read(read_size)
        if not s:
            # If this were to happen, then ``input_size`` is wrong or
            # the file is corrupt.
            raise ValueError(
                u'Expected to need to read %d times but got no data back on read %d' % (
                    num_reads, r + 1))

        data = b2a_base64(s)

        if is_last_read:
            # The last chunk will be smaller than the others so the
            # line count needs to be calculated. b2a_base64 adds a line
            # break so we don't count that char
            base64_lines_per_read = int(ceil((len(data) - 1) / float(base64_line_size)))

        # Split the data chunks into base64_lines_per_read number of
        # lines, each 76 chars long.
        for l in xrange(base64_lines_per_read):
            is_last_line = l == base64_lines_per_read - 1
            pos = l * base64_line_size
            line = data[pos:pos + base64_line_size]
            output.write(line)

            if not (is_last_line and is_last_read):
                # The very last line will already have a \n because of
                # b2a_base64. The other lines will not so we add it
                output.write('\n')
\$\endgroup\$
  • \$\begingroup\$ Please do not update the code in your question to incorporate feedback from answers, doing so goes against the Question + Answer style of Code Review. This is not a forum where you should keep the most updated version in your question. Please see what you may and may not do after receiving answers. \$\endgroup\$ – Heslacher Sep 28 '18 at 4:35
2
\$\begingroup\$

I ended up using bytearray as an input and output buffer. If found that if the output was something that doesn't buffer output (like a socket), then writing 77 bytes at a time would be very slow. Also my original code rounded the read size to be advantageous for base64, but not advantageous for MongoDB. It's better for the read size to match the MongoDB chunk size exactly. So the input is read into a bytearray with the exact size passed in, but then read in smaller base64 size chunks.

def chunked_encode(
        input, output, read_size=DEFAULT_READ_SIZE, write_size=(base64.MAXLINESIZE + 1) * 64):
    """
    Read a file in configurable sized chunks and write to it base64
    encoded to an output file.

    Args:
        input (file): File like object (implements ``read()``).
        output (file): File like object (implements ``write()``).
        read_size (int): How many bytes to read from ``input`` at
            a time. More efficient if in increments of 57.
        write_size (int): How many bytes to write at a time. More efficient
            if in increments of 77.
    """
    # 57 bytes of input will be 76 bytes of base64
    chunk_size = base64.MAXBINSIZE
    base64_line_size = base64.MAXLINESIZE
    # Read size needs to be in increments of chunk size for base64
    # output to be RFC 3548 compliant.
    buffer_read_size = max(chunk_size, read_size - (read_size % chunk_size))

    input.seek(0)

    read_buffer = bytearray()
    write_buffer = bytearray()

    while True:
        # Read from file and store in buffer until we have enough data
        # to meet buffer_read_size
        while input and len(read_buffer) < buffer_read_size:
            s = input.read(read_size)
            if s:
                read_buffer.extend(s)
            else:
                # Nothing left to read
                input = None

        if not len(read_buffer):
            # Nothing in buffer to read, finished
            break

        # Base 64 encode up to buffer_read_size and remove the trailing
        # line break.
        data = memoryview(b2a_base64(read_buffer[:buffer_read_size]))[:-1]
        # Put any unread data back into the buffer
        read_buffer = read_buffer[buffer_read_size:]

        # Read the data in chunks of base64_line_size and append a
        # linebreak
        for pos in xrange(0, len(data), base64_line_size):
            write_buffer.extend(data[pos:pos + base64_line_size])
            write_buffer.extend('\n')

            if len(write_buffer) >= write_size:
                # Flush write buffer
                output.write(write_buffer)
                del write_buffer[:]

    if len(write_buffer):
        output.write(write_buffer)
        del write_buffer[:]

For 10 iterations of a 10 MB file (complete test), this version is up to 5 times faster than standard base64 with large buffer sizes (>969) when reading a file without buffering (like a socket). For small buffer sizes (~100) it is about the same or worse than standard base64.

--- bufsize 4096
standard_base64_encode 5.70770692825 seconds for 10 iterations
original_chunked_encode 2.07641100883 seconds for 10 iterations
latest_chunked_encode  1.44510507584 seconds for 10 iterations
--- bufsize 2048
standard_base64_encode 5.71355605125 seconds for 10 iterations
original_chunked_encode 2.17808198929 seconds for 10 iterations
latest_chunked_encode  1.5746011734 seconds for 10 iterations
--- bufsize 1024
standard_base64_encode 5.7339630127 seconds for 10 iterations
original_chunked_encode 2.35343503952 seconds for 10 iterations
latest_chunked_encode  1.83091807365 seconds for 10 iterations
--- bufsize  969
standard_base64_encode 5.87562203407 seconds for 10 iterations
original_chunked_encode 2.3832950592 seconds for 10 iterations
latest_chunked_encode  1.81391692162 seconds for 10 iterations
--- bufsize  100
standard_base64_encode 5.84305310249 seconds for 10 iterations
original_chunked_encode 6.96859192848 seconds for 10 iterations
latest_chunked_encode  6.85651683807 seconds for 10 iterations
--- bufsize   57
standard_base64_encode 5.72181987762 seconds for 10 iterations
original_chunked_encode 6.98394799232 seconds for 10 iterations
latest_chunked_encode  8.28728795052 seconds for 10 iterations
\$\endgroup\$
  • \$\begingroup\$ How much faster is it? \$\endgroup\$ – Oscar Smith Apr 26 at 20:45
  • \$\begingroup\$ I updated with timing results. \$\endgroup\$ – six8 Apr 29 at 15:49
  • \$\begingroup\$ Thanks, have an upvote. \$\endgroup\$ – Oscar Smith Apr 29 at 17:40
0
\$\begingroup\$

The first thing I notice is that you are using Python2. This is almost certainly wrong. Python3 is faster for most applications, and Python2 is going EOL in 15 months.

Other than that, my main comments would be that this would probably benefit from async as this is an IO heavy function, so you could be doing the computation while waiting for a different IO task to finish.

\$\endgroup\$
  • \$\begingroup\$ This is a legacy application. Upgrading Python is not always a practical option. \$\endgroup\$ – six8 Sep 26 '18 at 2:56
  • \$\begingroup\$ I figured it was worth a try. I know the pain of legacy environments (centos servers that only supported python 2.5). In that case my async advice won't work either. You're best bet for speeding this up then is probably docs.python.org/2/library/base64.html \$\endgroup\$ – Oscar Smith Sep 26 '18 at 15:56
  • \$\begingroup\$ In this case, base64.encode is the bottleneck because of its 57 byte read size. \$\endgroup\$ – six8 Sep 26 '18 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.