2
\$\begingroup\$

I am requesting a JSON file that's gzipped. First, I download file:

import urllib.request
testfile = urllib.request.URLopener()
testfile.retrieve("https://xxxxx.auth0.com/1537150574", "file.gz")

After that, I will read the file.gz and get the data.

with gzip.GzipFile("file.gz", 'r') as fin:   
    json_bytes = fin.read()                      
json_str = json_bytes.decode('utf-8')            
data = json.loads(json_str)                      
print(data)

Actually, This above code can work well for me. But I would like to find another way (faster and brief code). Could I have a suggestion?

\$\endgroup\$
  • \$\begingroup\$ You probably can't speed this code up since you can't stream from a .gz file, so you need to load the whole thing into memory anyway, and the code is perfectly readable. If you want you could one line the read part of your code, but why? \$\endgroup\$ – Turksarama Sep 17 '18 at 3:19
  • \$\begingroup\$ Hi @Turksarama, In this case, I want to read all the data in the file.gz file. Could you help me fix it? The data in file like this: {"email":"phamvanhan681111@gmail.com","provider":"Username-Password-Authentication"} {"email":"123456@gmail.com","provider":"Username-Password-Authentication"} \$\endgroup\$ – Han Van Pham Sep 17 '18 at 5:26
1
\$\begingroup\$

Your bottleneck is probably that you write the file to disk first and then read it again (I/O). If the file does not exceed your machines random access memory, decompressing the file on the fly in memory might be a faster option:

from gzip import decompress
from json import loads

from requests import get

def get_gzipped_json(url):
    return loads(decompress(get(url).content))

if __name__ == '__main__':
    print(get_gzipped_json("https://xxxxx.auth0.com/1537150574"))

Also note, that I put the running code into an if __name__ == '__main__': guard.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.