6
\$\begingroup\$

Is there a way in Python 2.7 to improve reading/writing speed (or memory consumption of the file) compared to this version?

import gzip
import cPickle
import io

# save zipped and pickled file
def save_zipped_pickle(obj, filename):
    # disable garbage collector (hack for faster reading/writing)
    gc.disable()
    with gzip.open(filename, 'wb') as f:
        cPickle.dump(obj, io.BufferedWriter(f), -1)
        # enable garbage collector again
        gc.enable()

# load zipped and pickled file
def load_zipped_pickle(filename):
    # disable garbage collector (hack for faster reading/writing)
    gc.disable()
    with gzip.open(filename, 'rb') as f:
        loaded_object = cPickle.load(io.BufferedReader(f))
        # enable garbage collector again
        gc.enable()
        return loaded_object
\$\endgroup\$
3
\$\begingroup\$

This looks really efficient and potentially the best you can do, but here are some thoughts:

FYI, the gc.disable/gc.enable() pair is planned to be implemented as a context manager in Python 3.7. You can borrow a Python sample from the issue tracker:

@contextmanager
def gc_disabled():
    if gc.isenabled():
        gc.disable()
        yield
        gc.enable()
    else:
        yield
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.