6
\$\begingroup\$

I have two lists of dictionaries where both have the same "id" key but otherwise different data.

list1 = [{"id":1, "spam": 3, "eggs": 5}, {"id": 2, "spam": 5, "eggs": 7}]
list2 = [{"id":1, "knights": 5, "coconuts": 2}, {"id": 2, "knights": 3, "cocounts": 8}]

I know I can do a nested loop something like this (I know I can use dict.update as well, but just writing it out as an exampel):

for id in list1:
    for i in list2:
        if id['id'] == i['id]:
           id['knights'] = i['knights']
           id['coconuts'] = i['coconuts']

but can anyone tell me the more pythonic or quicker way to do this? matching two lists with millions of rows in each does not practically work this way.

Expected result is:

[{"id": 1, "spam": 3, "eggs": 5, "knights": 5, "coconuts": 2}, {"id": 2, "spam": 5, "eggs": 7, "knights": 3, "coconuts": 8}]
\$\endgroup\$
2
  • \$\begingroup\$ Where's id['knights'] and id['coconuts'] in list1 ?? \$\endgroup\$
    – Ahmed
    Commented Nov 17, 2020 at 9:43
  • 2
    \$\begingroup\$ You've already received an excellent answer, but for future reference: you should include all of your code to have sufficient context for meaningful reviews. The code you've shown is both a snippet and pretty clearly hypothetical. \$\endgroup\$
    – Reinderien
    Commented Nov 17, 2020 at 15:34

1 Answer 1

7
\$\begingroup\$

To improve time complexity you must sometimes trade-off space complexity. It's known as space-time tradeoff

You can transform list1 into a dictionary where key is id and value is the dictionary itself.

list1 = {d['id']: d for d in list1}
# print(list1)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}}

Now, iterate through list2 and update list1 based on id

for d in list2:
    list1[d['id']].update(d)

#print(list1)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}}

Take dict.values() to get desired output

print(list(list1.values()))
# [{'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}]

Dealing with Edge cases

The above approach fails when list2 has an id which doesn't exist in list1. If that's the case then use this and you want to add that new id to output too. Using dict.setdefault

# for example list2 is as below
# [{'id': 1, 'knights': 5, 'coconuts': 2},
#  {'id': 2, 'knights': 3, 'coconuts': 8},
#  {'id': 3, 'knight': 4, 'coconuts': 5}] -> new `id` that's not in `list1`

# Here we leverage on `dict.setdefault`

for d in list2:
    list1.setdefault(d['id'], dict()).update(d)

print(list1)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8},
#  3: {'id': 3, 'knight': 4, 'coconuts': 5}}

# To get list of dicts use `list(list1.values())

In case you dont want to update with new id, we can resolve this using if block here.

for d in list2:
    if d['id'] in list1:
        list1[d['id']].update(d)
# {1: {'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  2: {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}}
# No `id` 3 present in output

Using libraries that support vectorization?

If you are working on millions of rows you can leverage on pandas which has many vectorized operations. In your case we can use DataFrame.merge

import pandas as pd

df1 = pd.DataFrame(list1)
df2 = pd.DataFrame(list2)

# Now, you have to merge them on `id`

out = df1.merge(df2) 

#    id  spam  eggs  knights  coconuts
# 0   1     3     5        5         2
# 1   2     5     7        3         8

To get output as a list of dict you df.to_dict with orient param set to records

out.to_dict(orient='records')

# [{'id': 1, 'spam': 3, 'eggs': 5, 'knights': 5, 'coconuts': 2},
#  {'id': 2, 'spam': 5, 'eggs': 7, 'knights': 3, 'coconuts': 8}]

If you want include new ids then set how param of df.merge to 'outer' by default it's 'inner'

df1.merge(df2, how='outer')

#    id  spam  eggs  knights  coconuts
# 0   1   3.0   5.0        5         2
# 1   2   5.0   7.0        3         8
# 2   3   NaN   NaN        4         5

Code Review

  • Naming variables

    list1 = [...]
    list2 = [...]
    

    A new reader might not understand what's list1 and list2. Name them so they make sense to everyone.

    current_users_details = [...]
    new_details = [...]
    

    This is just an example, name them accordingly. What I do is ask myself what is it storing in the data-structure?... family member details? then variable name is family_members_details, user details then user_details, pi value? then PI_VALUE... etc.

  • Violation of E231

    list1 = [{"id":1, "spam": 3, "eggs": 5}, {"id": 2, "spam": 5, "eggs": 7}]
    list2 = [{"id":1, "knights": 5, "coconuts": 2}, {"id": 2, "knights": 3, "cocounts": 8}]
    

    The above lines use inconsistent use of spaces after :, PEP-8 suggest add a space after :.

  • Violation of E501

    list2 = [{"id":1, "knights": 5, "coconuts": 2}, {"id": 2, "knights": 3, "cocounts": 8}]
    

    Lines longer than 79 characters, PEP-8 standard suggests to keep characters in a line <= 79

  • You have missing ' here if id['id'] == i['id]:

  • Reusability

    Make a function which takes both the lists and updates them.

    def update_details(current_details, to_update_with):
        """
        updates user details
        """
    
        details = {d['id']: d for d in current_details}
        for user in to_update_with:
            details[user['id']].update(user)
        return details
    

    Write a much better docstring :P


I suggest using black python code formatter.

ch3ster@ch3ster:~$ black --line-length 79 your_file.py

Formatted code

Formatted your code using black.

list1 = [{"id": 1, "spam": 3, "eggs": 5}, {"id": 2, "spam": 5, "eggs": 7}]
list2 = [
    {"id": 1, "knights": 5, "coconuts": 2},
    {"id": 2, "knights": 3, "cocounts": 8},
]

for d in list2:
    list1[d["id"]].update(d)

print(list(list1.values())) # Equivalent to print([*list1.values()])
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.