I'm much more fluent in JS, but I needed to sort a lot of dates, ages, genders, etc. from a tab-delimited text file so I wrote this. Could I get some tips on how to make this more efficient and more Pythonic? The more Python I write the more I like it, but I definitely need some help.One thing I noticed is that when I use a ~600 MB file Python uses up to 25% of my RAM. That seems like a bit much. Am I leaking somewhere? I couldn't make heads or tails of Guppy, which printed something like this:Partition of a set of 8273952 objects. Total size = 1747556688 bytes. Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)     0 414691   5 1389344200  80 1389344200  80 dict (no owner)     1 7427576  90 338925432  19 1728269632  99 str     2 414362   5  9944688   1 1738214320  99 float     3    216   0  6847872   0 1745062192 100 list     4   7040   0   580152   0 1745642344 100 tuple     5     95   0   288488   0 1745930832 100 dict of module     6   1917   0   245376   0 1746176208 100 types.CodeType     7    235   0   243592   0 1746419800 100 dict of type     8   1840   0   220800   0 1746640600 100 function     9    235   0   209104   0 1746849704 100 type<112 more rows. Type e.g. '_.more' to view.>Which I'm assuming means that the dict is using 80% of my memory, and my variables are using 19%? The documentation is, uh, not incredibly user-friendly.#!/usr/bin/env pythonfrom __future__ import divisionimport csvimport datetimeimport subprocessimport gc'''#from guppy import hpy  # This is used only if you want to see where memory is allocated#h = hpy()              # I woudn't uncomment unless you want to see your memory double'''                     # Or if you want to see memory usagevrdb = 'active.txt'# Write headings to three output fileswith open('legdata.txt', 'wb+') as myfile:    myfile.write('LegDist,AvgAge,NumMales,PerMales,Q1,Q2,Q3,Q4,Q5,Q6,NumFemales,PerFemales,Q1,Q2,Q3,Q4,Q5,Q6' + '\r\n')with open('citydata.txt', 'wb+') as myfile:    myfile.write('City,AvgAge,NumMales,PerMales,NumFemales,Q1,Q2,Q3,Q4,Q5,Q6,PerFemales,Q1,Q2,Q3,Q4,Q5,Q6' + '\r\n')with open('precinctdata.txt', 'wb+') as myfile:    myfile.write('Precinct,AvgAge,NumMales,PerMales,NumFemales,Q1,Q2,Q3,Q4,Q5,Q6,PerFemales,Q1,Q2,Q3,Q4,Q5,Q6' + '\r\n')with open('congressdata.txt', 'wb+') as myfile:    myfile.write('CongressionalDistrict,AvgAge,NumMales,PerMales,NumFemales,Q1,Q2,Q3,Q4,Q5,Q6,PerFemales,Q1,Q2,Q3,Q4,Q5,Q6' + '\r\n')def getCities():    reader = csv.DictReader(open(vrdb, 'rb'), delimiter='\t')    cities = []    # Appends all the data from each person inside a specific city    for row in reader:        if row['RegCity']:            cities.append(row.get('RegCity'))    return citiesdef getPrecincts():    reader = csv.DictReader(open(vrdb, 'rb'), delimiter='\t')    precincts = []    # In Washington counties can use their own precinct codes which are the county code (e.g. King is KI) + precinct code + precinct part    # To keep each unique we concat each part with a '+' -- this keeps the values separate but still unique    for row in reader:        if row['PrecinctCode']:            precincts.append(str(row.get('CountyCode')) + '+'  + str(row.get('PrecinctCode')) + '+' + str(row.get('PrecinctPart')))    with open('precincts.txt', 'ab+') as myfile:        myfile.write(str(precincts))def sortLists(function, output_file, shell_script):    numcities = function    numcities = str(numcities)    with open(output_file, 'ab+') as myfile:        myfile.write(numcities)    subprocess.call([shell_script])def getInformation(location, identifier, output_file):    # Opens VRDB and parses it line by line    reader = csv.DictReader(open(vrdb, 'rb'), delimiter= '\t')    master_list = []    ages = []    nummale = 0    numfemale = 0    # Here we have the different quantiles    # m denotes male, f denotes female    fq1 = 0 # 18 - 25    fq2 = 0 # 26 - 35    fq3 = 0 # 36 - 45    fq4 = 0 # 46 - 55    fq5 = 0 # 56 - 65    fq6 = 0 # 66 +    mq1 = 0 # 18 - 25    mq2 = 0 # 26 - 35    mq3 = 0 # 36 - 45    mq4 = 0 # 46 - 55    mq5 = 0 # 56 - 65    mq6 = 0 # 66 +    if useprecincts != True:        for row in reader:            if row[location] == str(identifier):                    master_list.append(row)    if useprecincts:        for row in reader:            if str(row['CountyCode']) + '+' + str(row['PrecinctCode']) + '+' + str(row['PrecinctPart']) == identifier:                master_list.append(row)    # For each person in our specific LD/city/precinct, get their information and append it to the correct lists    # 31556952 is used because it's the representation of 365.2425 days in seconds.     # 365.2425 is used because there's only 97 leap years every 400 years, not 100 (Gregorian calendar)    # We also get the number of each gender    for x,value in enumerate(master_list):        dates = datetime.datetime.now() - datetime.datetime.strptime(value['Birthdate'], '%m/%d/%Y')        age = datetime.timedelta.total_seconds(dates) / 31556952        ages.append(age)        if value['Gender'] == M:            nummale += 1            if 18 <= age < 26:                mq1 += 1            if 26 <= age < 35:                mq2 += 1            if 36 <= age < 45:                mq3 += 1            if 46 <= age < 55:                mq4 += 1            if 56 <= age < 65:                mq5 += 1            if 66 <= age:                mq6 += 1        if value['Gender'] == F:            numfemale += 1            if 18 <= age < 26:                fq1 += 1            if 26 <= age < 35:                fq2 += 1            if 36 <= age < 45:                fq3 += 1            if 46 <= age < 55:                fq4 += 1            if 56 <= age < 65:                fq5 += 1            if 66 <= age:                fq6 += 1    # This defines the ages. Takes the sum of all the ages and divides it by the total values    # It also takes the total number of each gender and divides that by the total    # This gives us a percentage with 3 decimal places    total_values = len(ages)    average_age = sum(ages) / total_values    perfemale = '{percent:.3%}'.format(percent=numfemale/total_values)    permale = '{percent:.3%}'.format(percent=nummale/total_values)    results = str(identifier) + ',' + str(average_age) + ',' + str(nummale) + ',' + str(permale) + ',' + str(mq1)+ ',' + str(mq2) + ',' + str(mq3) + ',' + str(mq4) + ',' + str(mq5) + ',' + str(mq6) + ',' + str(numfemale) + ',' +  str(perfemale) + ',' + str(fq1)+ ',' + str(fq2) + ',' + str(fq3) + ',' + str(fq4) + ',' + str(fq5) + ',' + str(fq6)    # Prints output to console so we can see our script is working    # `with... as myfile` appends each line to with \r\n so we can work with both unix and windows    print results    # h.heap() used to see where memory is allocated    #print h.heap()    with open(output_file, 'ab+') as myfile:        myfile.write(results + '\r\n')    gc.collect()    # We define set useprecincts to false because we turn it to true later so we can use a modified for-loop to handle the precinctsuseprecincts = False# Gets Congressional District datafor e in xrange(1,11):    getInformation('CongressionalDistrict', e, 'congressdata.txt')# Gets Legislative District datafor r in xrange(1, 50):    getInformation('LegislativeDistrict', r, 'legdata.txt')# Gets city datagetCities()sortLists(getCities(), 'citylist.txt', './shellsubprocess.sh')listofcities = [line.rstrip() for line in open('citylist.txt')]for i in listofcities:    getInformation('RegCity', i, 'citydata.txt')# Gets precinct datagetPrecincts()sortLists(getPrecincts(), 'precinctlist.txt', './subshellprecincts.sh')listofprecincts = [line.rstrip() for line in open('precincts.txt')]useprecincts = True# I'm passing 'a' because getInformation takes 3 args when I only need to pass two# I suppose I *could* implement *args, but this works for such a simple scraper scriptfor c in listofprecincts:    getInformation('a', c, 'precinctdata.txt')Since I call two shell processes I'll include them here. (They're the same except with different file names). Yes, I know they could be better but they take seconds to run, are only run once a piece, and work. But if you want to you can critique them. They're incredibly simple though. You can ignore this if you want.#!/bin/bashtouch precincts.bak;cat precincts.txt | sed 's/, /\n/g' > precincts.bak;cat precincts.bak | sed s/'//g > precincts.txt;cat precincts.txt | tr -d '][' > precincts.bak;sort precincts.bak | awk '!seen[$0]++' > precincts.txt;exit
CSV demographics analyzer seems to waste memory/move slowly
python;performance;memory management;csv
There are quite a few things you can do to make your code more Pythonic:First off, look at PEP8. Its the official Python style guide. It has a LOT of good Python coding practices.Use list comprehensions whenever is feasible. Take your getCities() function. This can be written in two lines:def get_cities():    reader = csv.DictReader(open(vrdb, 'rb'), delimiter='\t')    return [row.get('RegCity') for row in reader if row['RegCity']]Use elif instead of chaining multiple if statements. Currently in your getInformation function (and many other places), you have a long list of if statements. Each statement is getting processed even though each of the conditions are mutually exclusive (if age is between 18 and 26 then age cannot be between 26 and 35).Using string formatting instead of string concatenation. Though the performance benefits are debatable, string formatting is more Pythonic:# Instead of this...precincts.append(str(row.get('CountyCode')) + '+'  + str(row.get('PrecinctCode')) + '+' + str(row.get('PrecinctPart')))# Do this.precincts.append('{}+{}+{}'.format(row.get('CountyCode'),                                   row.get('PrecinctCode'),                                   row.get('PrecinctPart'))Use underscores_in_names instead of camelCase or noseparateratall. The former bad formatting can be tolerated as it just goes against convention but is still readable. However the latter, which you use (numcities, nummale, useprecinct, etc.), is very hard to read.As a follow-up to the previous point, make your names descriptive. As a reader, I have no idea what vrdb is supposed to do. I see that it holds a string, but what does that string supposed to do? What does it represent? It is better to be too verbose than too terse.This same point goes for function names. Make sure the name clearly represents the actions taken in the function. Your getPrecincts function is misleading. You grab all the precincts (expected) then you save them to a file (unexpected). Be as clear as possible.There are other improvements I can suggest. However, implementing these will probably take some time. So I'll leave the rest to other reviewers.