I'd like the function below reviewed. I'm especially interested in improvement ideas to process the API responses faster.products_list is the API response in the form of list of dictionaries which returns all products for one supermarket only (out of many).def encode_products_response_list(products_list, rows_list):        Convert unicode values to utf-8    Convert all other values (e.g. Int, Float) to str    Return list of unique rows    :param products_list:  list of dictionaries    :param rows_list:          empty list        for item in products_list:        # extract name from list        item['product_name'] = item['product_name'][0]        for key, value in item.iteritems():            if isinstance(value, unicode):                item[key] = value.encode('utf-8')            else:                item[key] = str(value)        if item not in rows_list:            rows_list.append(item)    return rows_listif __name__ == '__main__':    # all work in same product    rows_list = []    # the api products_list response is a list of dicts like the one below    products_list = [        {            u'product_name': [u'Super Bleach 5'],             u'product_description': 'Cleans like nothing you have ever seen',             u'cost': 2.55,         },        {            u'product_name': [u'Magic Breakfast'],             u'product_description': 'Start your day with proper breakfast!',             u'cost': 5,         }    ]    products_list = products_list * 354342    encode_products_response_list(products_list, rows_list)The metrics below is just regarding one of many supermarkets (using line_profiler):Timer unit: 1e-06 sTotal time: 108.972 sFile: example_1.pyFunction: encode_products_response_list at line 2Line #      Hits         Time  Per Hit   % Time  Line Contents============================================================== 1                                           @profile 2                                           def encode_products_response_list(products_list, rows_list): 3    354343       362103      1.0      0.3      for row in response: 4    354342       539821      1.5      0.5          item['product_name'] = item['product_name'][0] 5                                            6   2480394      1392104      0.6      1.3          for key, value in item.iteritems(): 7   2126052      1460806      0.7      1.3              if isinstance(value, unicode): 8    354342       661158      1.9      0.6                  item[key] = value.encode('utf-8') 9                                                       else:10   1771710      1728154      1.0      1.6                  item[key] = str(value)11                                           12   354342    102436117    289.1     94.0          if item not in rows_list:13     13634        28912      2.1      0.0              rows_list.append(item)14                                           15         1            1      1.0      0.0      return rows_list
Iterating and updating 35000 dictionaries
python;performance
Not surprisingly, almost all time is spent on the repeated if item not in rows_list, as that takes linear time every time. Instead of doing the duplicate-check with a list, you'd better do it with a set, where that check averages constant time. That would require hashable elements, though, which your items, being dicts, aren't. But you can probably extract something hashable correctly identifying each item, no? Maybe even just item['product_name']? If not, then the tuple (item['product_name'], item['product_description'], item['cost']).