python: is my program optimal

Question

I wrote code in python that works slow. Because I am new to python, I am not sure that I am doing everything right. My question is what I can do optimally? About the problem: I have 25 *.json files, each is about 80 MB. Each file just contain json strings. I need make some histogram based on data.

In this part I want create list of all dictionaries ( one dictionary represent json object):

d = [] # filename is list of name of files
for x in filename:
d.extend(map(json.loads, open(x)))

then I want to create list u :

u = []
for x in d:
  s = x['key_1'] # s is sting which I use to get useful value
  t1 = 60*int(s[11:13]) + int(s[14:16])# t1 is useful value
  u.append(t1)

Now I am creating histogram:

plt.hist(u, bins = (max(u) - min(u)))
plt.show()

Any thought and suggestions are appreciated. Thank you!

Answer 1

You might be able to save some run time by using a couple of generator expressions and a list comprehension. For example:

def read_json_file(name):
    with open(name, "r") as f:
        return json.load(f)

def compute(s):
    return 60 * int(s[11:13]) + int(s[14:16])

d = (read_json_file(n) for n in filename)
u = list(compute(x['key_1']) for x in d)

plt.hist(u, bins = (max(u) - min(u)))
plt.show()

This should save on memory, since anything that isn't needed is discarded.

Edit: it's difficult to discern from the information available, but I think the OP's json files contain multiple json objects, so calling json.load(f) won't work. If that is the case, then this code should fix the problem

def read_json_file(name):
    "Return an iterable of objects loaded from the json file 'name'"
    with open(name, "r") as f:
        for s in f:
            yield json.loads(s)

def compute(s):
    return 60 * int(s[11:13]) + int(s[14:16])

# d is a generator yielding an iterable at each iteration
d = (read_json_file(n) for n in filename)

# j is the flattened version of d
j = (obj for iterable in d for obj in iterable)

u = list(compute(x['key_1']) for x in j)

plt.hist(u, bins = (max(u) - min(u)))
plt.show()

Answer 2

Python uses a surprisingly large amount of memory when reading files, often 3-4 times the actual file size. You never close each file after you open it, so all of that memory is still in use later in the program.

Try changing the flow of your program to

1. Open a file
2. Compute a histogram for that file
3. Close the file
4. Merge it with a "global" histogram
5. Repeat until there are no files left.

Something like

u = []
for f in filenames:
    with open(f) as file:
        # process individual file contents
        contents = file.read()
        data = json.loads(contents)
        for obj in data:
            s = obj['key_1']
            t1 = 60 * int(s[11:13]) + int(s[14:16])
            u.append(t1)

# make the global histogram
plt.hist(u, bins = (max(u) - min(u)))
plt.show()

with open as automatically closes files when you're done, and handles cases where the file can't be read or there are other errors.

Answer 3

I'd use this, as it avoids loading and keeping all of json data in memory:

u = []
for name in filename:
  d = json.load(open(name,"r"))
  for x in d:
    s = x['key_1']
    t1 = 60*int(s[11:13]) + int(s[14:16])
    u.append(t1)
  d = None

plt.hist(u, bins = (max(u) - min(u)))
plt.show()