Iterating and updating 35000 dictionaries

Question

I'd like the function below reviewed. I'm especially interested in improvement ideas to process the API responses faster.

products_list is the API response in the form of list of dictionaries which returns all products for one supermarket only (out of many).

def encode_products_response_list(products_list, rows_list):
    """
    Convert unicode values to utf-8
    Convert all other values (e.g. Int, Float) to str
    Return list of unique rows

    :param products_list:  list of dictionaries
    :param rows_list:          empty list
    """
    for item in products_list:
        # extract name from list
        item['product_name'] = item['product_name'][0]

        for key, value in item.iteritems():
            if isinstance(value, unicode):
                item[key] = value.encode('utf-8')
            else:
                item[key] = str(value)

        if item not in rows_list:
            rows_list.append(item)

    return rows_list

if __name__ == '__main__':
    # all work in same product
    rows_list = []
    # the api products_list response is a list of dicts like the one below
    products_list = [
        {
            u'product_name': [u'Super Bleach 5'], 
            u'product_description': 'Cleans like nothing you have ever seen', 
            u'cost': 2.55, 
        },
        {
            u'product_name': [u'Magic Breakfast'], 
            u'product_description': 'Start your day with proper breakfast!', 
            u'cost': 5, 
        }
    ]
    products_list = products_list * 354342

    encode_products_response_list(products_list, rows_list)

The metrics below is just regarding one of many supermarkets (using line_profiler):

Timer unit: 1e-06 s

Total time: 108.972 s
File: example_1.py
Function: encode_products_response_list at line 2

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
 1                                           @profile
 2                                           def encode_products_response_list(products_list, rows_list):
 3    354343       362103      1.0      0.3      for row in response:
 4    354342       539821      1.5      0.5          item['product_name'] = item['product_name'][0]
 5                                           
 6   2480394      1392104      0.6      1.3          for key, value in item.iteritems():
 7   2126052      1460806      0.7      1.3              if isinstance(value, unicode):
 8    354342       661158      1.9      0.6                  item[key] = value.encode('utf-8')
 9                                                       else:
10   1771710      1728154      1.0      1.6                  item[key] = str(value)
11                                           
12   354342    102436117    289.1     94.0          if item not in rows_list:
13     13634        28912      2.1      0.0              rows_list.append(item)
14                                           
15         1            1      1.0      0.0      return rows_list

Answer 1

Not surprisingly, almost all time is spent on the repeated if item not in rows_list, as that takes linear time every time. Instead of doing the duplicate-check with a list, you'd better do it with a set, where that check averages constant time. That would require hashable elements, though, which your items, being dicts, aren't. But you can probably extract something hashable correctly identifying each item, no? Maybe even just item['product_name']? If not, then the tuple (item['product_name'], item['product_description'], item['cost']).

Answer 2

This has no bearing on performance, but there is no reason for you to pass rows_list to your function. If you instantiate an empty list in your function, then you're already returning it at the end, so you can just get the value of it like this:

rows_list = encode_products_response_list(products_list)

Answer 3

I don't see any reason why you couldn't process the dictionary in parallel.

Perhaps try looking into multiprocessing, specifically pool.map().