A large .csv file I was given has a large table of flight data. A function I wrote to help parse it iterates over the column of Flight IDs, and then returns a dictionary containing the index and value of every unique Flight ID in order of first appearance.Dictionary = { Index: FID, ... }This comes as a quick adjustment to an older function that didn't require having to worry about FID repeats in the column (a few hundred thousand rows later...).Example:20110117559515, ... 20110117559515, ... 20110117559515, ...                     20110117559572, ...   20110117559572, ...   20110117559572, ...                               20110117559574, ...                               20110117559587, ...                             20110117559588, ...and so on for 5.3 million some rows.Right now, I have it iterating over and comparing each value in order. If a unique value appears, it only stores the first occurrence in the dictionary. I changed it to now also check if that value has already occurred before, and if so, to skip it.def DiscoverEarliestIndex(self, number):    result = {}    columnvalues = self.column(number)    column_enum = {}    for a, b in enumerate(columnvalues):        column_enum[a] = b        i = 0    while i < (len(columnvalues) - 1):        next = column_enum[i+1]        if columnvalues[i] == next:            i += 1        else:            if next in result.values():                i += 1                continue            else:                result[i+1]= next                i += 1    else:        return resultIt's very inefficient, and slows down as the dictionary grows. The column has 5.2 million rows, so it's obviously not a good idea to handle this much with Python, but I'm stuck with it for now.Is there a more efficient way to write this function?
Retrieving the first occurrence of every unique value from a CSV column
python;performance;parsing;csv;iteration
So here is what I came up with, now that I had access to a computer:raw_data.csv:20110117559515,1,10,faa20110117559515,2,20,bar20110117559515,3,30,baz20110117559572,4,40,fii20110117559572,5,50,bir20110117559572,6,60,biz20110117559574,7,70,foo20110117559587,8,80,bor20110117559588,9,90,bozcode:import csvfrom collections import defaultdictfrom pprint import pprintimport timeitdef method1():    rows = list(csv.reader(open('raw_data.csv', 'r'), delimiter=','))    cols = zip(*rows)    unik = set(cols[0])    indexed = defaultdict(list)    for x in unik:        i = cols[0].index(x)        indexed[i] = rows[i]    return indexeddef method2():    rows = list(csv.reader(open('raw_data.csv', 'r'), delimiter=','))    cols = zip(*rows)    unik = set(cols[0])    indexed = defaultdict(list)    for x in unik:        i = next(index for index,fid in enumerate(cols[0]) if fid == x)        indexed[i] = rows[i]    return indexeddef method3():    rows = list(csv.reader(open('raw_data.csv', 'r'), delimiter=','))    cols = zip(*rows)    indexes = [cols[0].index(x) for x in set(cols[0])]    for index in indexes:        yield (index,rows[index])if __name__ == '__main__':    results = method1()        print 'indexed:'    pprint(dict(results))    print '-' * 80    results = method2()        print 'indexed:'    pprint(dict(results))    print '-' * 80    results = dict(method3())    print 'indexed:'    pprint(results)    #--- Timeit ---    print 'method1:', timeit.timeit('dict(method1())', setup=from __main__ import method1, number=10000)    print 'method2:', timeit.timeit('dict(method2())', setup=from __main__ import method2, number=10000)    print 'method3:', timeit.timeit('dict(method3())', setup=from __main__ import method3, number=10000)Output:indexed:{0: ['20110117559515', '1', '10', 'faa'], 3: ['20110117559572', '4', '40', 'fii'], 6: ['20110117559574', '7', '70', 'foo'], 7: ['20110117559587', '8', '80', 'bor'], 8: ['20110117559588', '9', '90', 'boz']}--------------------------------------------------------------------------------indexed:{0: ['20110117559515', '1', '10', 'faa'], 3: ['20110117559572', '4', '40', 'fii'], 6: ['20110117559574', '7', '70', 'foo'], 7: ['20110117559587', '8', '80', 'bor'], 8: ['20110117559588', '9', '90', 'boz']}--------------------------------------------------------------------------------indexed:{0: ['20110117559515', '1', '10', 'faa'], 3: ['20110117559572', '4', '40', 'fii'], 6: ['20110117559574', '7', '70', 'foo'], 7: ['20110117559587', '8', '80', 'bor'], 8: ['20110117559588', '9', '90', 'boz']}method1: 0.283623933792method2: 0.37960600853method3: 0.293814182281