As you might know, Python's tarfile does not have the ability to append to the files that compressed with e.g. gz or bz2. I tried to implement this functionality which works fine but operates slowly. The following function accepts a string or bytes object and appends it as a file to an existing tarfile. I'm not sure if this code has the best performance, and it may have issues. I also tried writing files to memory instead of to a temporary directory, but this didn't impact performance.

import os
import tarfile
import tempfile
import time
from pathlib import Path

def append_tar_file(buffer, file_name, output_path, replace=True):
    append a buffer to an existing tar file
    # extract files
    # check for existing file and overwrite if need to
    # compress files
    if not os.path.isfile(output_path):
    buffer = buffer.encode("utf-8") if isinstance(buffer, str) else buffer

    with tempfile.TemporaryDirectory() as tempdir:
        tempdirp = Path(tempdir)
        with tarfile.open(output_path, "r:bz2") as tar:
            except Exception as err:   #tar file is empty
        buffer_path = os.path.join(tempdir, os.path.basename(file_name))
        if replace or (buffer_path not in list(os.path.abspath(f) for f in tempdirp.iterdir())):
            with open(buffer_path, "wb") as f:

        with tarfile.open(output_path, "w:bz2") as tar:
            for file in tempdirp.iterdir():
                    tar.add(file, arcname=os.path.basename(os.path.normpath(file)))
                except Exception as err:

if __name__ == "__main__":
    path = "./test.tar.gz"
    buffer = "Test String"
    filename = "somefile"
    for i in range(1, 100):
        append_tar_file(buffer, filename+str(i), path)

2 Answers 2


Indeed, the tarfile package doesn't support appending to a compressed tar. But I think you can do better than your current attempt. Instead of extracting the content to disk, you could keep it in memory, write and append to a new compressed file, and finally rename the compressed file. That way you will write a single temporary file, instead of many. The reduced disk I/O should improve the performance.

def append_tar_file(buf, file_name, output_path, replace=True):
    append a buf to an existing tar file if not already there, or if replace=True
    if not os.path.isfile(output_path):

    with tempfile.TemporaryDirectory() as tempdir:
        tmp_path = os.path.join(tempdir, 'tmp.tar.bz2')

        with tarfile.open(output_path, "r:bz2") as tar:
            if not replace:
                if file_name in (member.name for member in tar):

            if isinstance(buf, str):
                buf = buf.encode("utf-8")

            fileobj = BytesIO(buf)
            tarinfo = tarfile.TarInfo(file_name)
            tarinfo.size = len(fileobj.getvalue())

            with tarfile.open(tmp_path, "w:bz2") as tmp:
                for member in tar:
                    if member.name != file_name:
                        tmp.addfile(member, tar.extractfile(member.name))
                tmp.addfile(tarinfo, fileobj)

        os.rename(tmp_path, output_path)
  • \$\begingroup\$ Can multiprocessing used while working with memory? I mean when adding files back to output tar file use it. When the compressed file getting larger, it would take more time to append. \$\endgroup\$ Jul 30, 2018 at 7:59

Instead of hardcoding the compression scheme (and possibly compressing a .tar.gz file using BZIP2 as you do), you should try to infer that information. Note that tarfile let you open a compressed file without knowing the compression scheme using tarfile.open(filename, 'r:*') but there is no equivalent for writing the archive.

Since the compression scheme understood by the tarfile module are usual extensions of the files, inspecting the suffixes of output_path should be enough:

def get_compression(filename):
    suffixes = Path(filename).suffixes
    tar, *compression = (s.lstrip('.') for s in suffixes)

    if tar == 'tgz':
        if compression:
            raise RuntimeError('Too much suffixes, cannot infer compression scheme from {}'.format(''.join(suffixes)))
        return 'gz'

    if tar != 'tar':
        raise RuntimeError('Not a tar archive')

    if not compression:
        return ''

        compression, = compression
    except ValueError:
        raise RuntimeError('Too much compression scheme: {}'.format(', '.join(compression))) from None
        return compression

Now you can use compression = get_compression(output_path) and then open the tar file using tarfile.open(<name>, 'r:{}'.format(compression)) and open the file for writing using tarfile.open(<name>, 'w:{}'.format(compression)).

Note that I used RuntimeError here but you should probably come up with something less generic.

  • \$\begingroup\$ Thanks. The function return gz for the files that created with bz2 in tarfile. Thus it should return bz2 instead of gz. \$\endgroup\$ Jul 30, 2018 at 6:41
  • \$\begingroup\$ If your file is named something.tar.gz, it is not coherent to use anything else than GZIP compression to create the file. This is the assumption this function is doing, hence the choice to only check the filename and not trying to do anything fancy with the file content. If you store BZIP2 compressed data in a .tar.gz this is your mistake to start with. \$\endgroup\$ Jul 30, 2018 at 7:15
  • \$\begingroup\$ So you mean I should use tar.bz2 right? \$\endgroup\$ Jul 30, 2018 at 7:17
  • 2
    \$\begingroup\$ If you want to store BZIP2 compressed data, definitely. \$\endgroup\$ Jul 30, 2018 at 7:23
  • 1
    \$\begingroup\$ On the other hand, if you want something fancier based on the file content rather than its name, you can check how tarfile handle the r:* mode. Basically, it tries to open the file using, in turn gzip.GzipFile, bz2.BZ2File, lzma.LZMAFile and if all failed fallback to trying without compression. \$\endgroup\$ Jul 30, 2018 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.