<jupyter_start><jupyter_text>Assignment 9 Example*Last updated: 11/6/2019, <NAME>*This notebook gives an example of what a completed Week 9 assignment might look like. It builds a forecast of the 2020 Democratic Presidential Primary based on the polls collected by 538 (using the dataset as of 11/6/2019).  Setting up the environment  Setting the working directory<jupyter_code>import os
os.chdir('C:\\working\\')<jupyter_output><empty_output><jupyter_text>Importing numpy and pandas for analysis and setting display options for pandas<jupyter_code>import numpy as np
import pandas as pd
pd.options.display.max_colwidth = 1000
pd.options.display.max_columns = 1000
pd.options.display.max_rows = 1000<jupyter_output><empty_output><jupyter_text>Loading, transforming, and exploring data  Loading just the columns we actually care about<jupyter_code>columns_to_keep = [
    'poll_id',
    'state',
    'pollster',
    'fte_grade',
    'sample_size',
    'population',
    'methodology',
    'start_date',
    'end_date',
    'tracking',
    'answer',
    'pct']
raw = pd.read_csv('president_primary_polls.csv', usecols=columns_to_keep, parse_dates=['start_date','end_date'])<jupyter_output><empty_output><jupyter_text>Filling in some missing data<jupyter_code>for i in ['fte_grade','methodology']:
    raw[i] = raw[i].fillna('Missing')<jupyter_output><empty_output><jupyter_text>Filtering to just the candidates we care aboutI'm limiting to the 9 candidates still in the race who have qualified for inclusion in the November 20th debate as of 11/5. (The only noteworthy exclusions based on this are Gabbard and Castro, but both are excluded because they don't show enough polling strength to meet the threshold, so that means they wouldn't affect a forecast much anyway.)<jupyter_code>cands = [
    'Biden',
    'Harris',
    'Warren',
    'Sanders',
    'Booker',
    'Klobuchar',
    'Buttigieg',
    'Yang',
    'Steyer'
    ]
raw = raw[raw.answer.map(lambda x: x in cands)]<jupyter_output><empty_output><jupyter_text>Limiting to states with at least three unique polls<jupyter_code>raw['state'] = raw.state.fillna('National')
polls_by_state = raw.drop_duplicates(['state','poll_id']).groupby('state').poll_id.count()
states = polls_by_state[polls_by_state.map(lambda x: x >= 3)].index
raw = raw[raw.state.map(lambda x: x in states)]
raw.sample(5).T<jupyter_output><empty_output><jupyter_text>Looping over categorical variables to see value counts<jupyter_code>for i in raw.dtypes[raw.dtypes == 'object'].index:
    print('\n{}:'.format(i))
    print(raw[i].value_counts()[:25])<jupyter_output>state:
National          3377
New Hampshire      301
Iowa               266
South Carolina     251
California         214
Texas              152
Florida             96
Nevada              87
Wisconsin           77
Pennsylvania        55
Michigan            50
Massachusetts       40
North Carolina      40
Ohio                35
Arizona             34
Virginia            32
Georgia             26
Name: state, dtype: int64

pollster:
Harris Insights & Analytics                     1435
Morning Consult                                  432
Change Research                                  416
YouGov                                           403
Emerson College                                  252
Ipsos                                            163
Quinnipiac University                            162
Monmouth University                              141
Fox News/Beacon Research/Shaw & Co. Research      98
SSRS                                              97
Zogby Interactive/JZ Analytics    [...]<jupyter_text>Creating dataframe of polls for merging later (this is to allow for cleaning up polls which publish multiple sets of results) Looking at poll_ids with multiple sets of results<jupyter_code>polls = raw[['poll_id', 'state', 'pollster', 'fte_grade', 'sample_size', 'population', 'methodology',
                'start_date', 'end_date', 'tracking']].drop_duplicates()
result_count = polls.poll_id.value_counts()
multi_results = result_count[result_count > 1].index
polls[polls.poll_id.map(lambda x: x in multi_results)].sort_values(by='poll_id')[:20]<jupyter_output><empty_output><jupyter_text>Limiting to the set of results with the smallest sample size for each poll, which is the tightest screen (likely voters before registered voters, registered voters before all adults, etc.)<jupyter_code>polls = polls.sort_values(by='sample_size',ascending=True).drop_duplicates('poll_id')<jupyter_output><empty_output><jupyter_text>Calculating age of polls and duration in the field<jupyter_code>polls['age'] = (pd.Timestamp('now') - polls.end_date).dt.days
polls['duration'] = (polls.end_date - polls.start_date).dt.days + 1
polls.head()<jupyter_output><empty_output><jupyter_text>Joining in results that match to those included poll_ids (with all the other fields matching)<jupyter_code>results = polls.merge(raw, how='inner', on=['poll_id', 'state', 'pollster', 'fte_grade', 'sample_size', 'population', 
                                            'methodology', 'start_date', 'end_date', 'tracking'])
results.head()<jupyter_output><empty_output><jupyter_text>Averaging across a handful of remaining duplicate entriesThere aren't many of these, so I'm assuming they're additional variants not captured in the standard columns, and rather than deal with them individually I'm just averaging the results across them.<jupyter_code>results = results.groupby(['poll_id', 'state', 'pollster', 'fte_grade', 'sample_size', 'population', 
                            'methodology', 'start_date', 'end_date', 'tracking', 'answer']
                            ).pct.mean().reset_index()
results.head()<jupyter_output><empty_output><jupyter_text>Pivoting long to wide (one column per candidate), filling in with zeros, and calculating undecideds<jupyter_code>results_wide = results.pivot(index='poll_id', columns='answer', values='pct').fillna(0)
results_wide['TOTAL'] = results_wide.sum(axis=1)
results_wide['UNDECIDED'] = 100 - results_wide['TOTAL']
results_wide.head(10)<jupyter_output><empty_output><jupyter_text>Making the final dataframe<jupyter_code>data = polls.merge(results_wide, on='poll_id').set_index('poll_id').sort_index()
data.head(20)<jupyter_output><empty_output><jupyter_text>Setting up the forecast functionThis function combines poll results using the following fixed rules and options:* Fixed rules:  * Each poll result is given a weight based on a variety of different characteristics (see "options" below), and the weighted average of these is used as the forecast. (This means that we're estimating where the race stands now, not where it's going to be in the future - there's no imputation of a future state, e.g. by forecasting a trend line.)  * Each poll characteristic is given a weight value, and these are all multiplied together to calculate the overall weight per poll.  * "Undecided" is treated as a distinct response choice, * Options:   * Which state the calculation is for ("National" is treated as a separate state here)  * Which candidates are included  * Max number of poll results included per pollster (keeps only most recent)  * Max age of poll results included  * A set of pre-defined weights based on values of:    * fte_grade (pollster rating by 538)    * population    * methodology    * tracking  * A set of weighting functions to apply to values of:    * sample_size    * duration    * age    Some things *not* included in here that could be: * Adjusting to down-weight additional polls from the same pollster * Giving lower weights to older results from the same pollster (other than because of increased age) * Removing results from tracking polls with overlapping timeframes (but 538 has done some of this already, and the weight given to the "tracking" flag can partially adjust for this)  Setting pre-defined weights for values of each characteristic (as dicts)Adding a "Missing" option for each (which will get filled in during processing) except for Tracking, where missing values are treated as False<jupyter_code>w_fte_grade = {
    'A+' : 4.3,
    'A' : 4.0,
    'A-' : 3.7,
    'A/B' : 3.5,
    'B+' : 3.3,
    'B' : 3.0,
    'B-' : 2.7,
    'B/C' : 2.5,
    'C+' : 2.3,
    'C' : 2.0,
    'C/D' : 1.5,
    'D-' : 0.7,
    'Missing' : 1.0 
}
w_population = {
    'lv' : 2.0,
    'rv' : 1.5,
    'v' : 1.0,
    'a' : 1.0,
    'Missing' : 1.0
}
w_methodology = {
    'Live Phone' : 3.0,
    'Live Phone/Text' : 3.0,
    'Live Phone/Online' : 3.0,
    'Online' : 2.0,
    'IVR/Text': 2.0,
    'Online/IVR' : 2.0,
    'IVR/Online' : 2.0,
    'Automated Phone': 1.0,
    'Missing' : 1.0
}
w_tracking = {
    True: 0.5,
    False: 1.0,
}<jupyter_output><empty_output><jupyter_text>Setting weighting functions for sample size, poll age, and duration<jupyter_code>def f_sample_size(x):
    z = np.amin([x,2000]) #topcoding at 2000 to avoid over-weighting giant online panels
    return (z**0.5)/(1000**0.5)

#showing how it works in practice
for i in [0,100,500,1000,2000,10000]:
    print('{}: {}'.format(i,f_sample_size(i)))
def f_age(x):
    return 3 / (x+9)**0.5

#showing how it works in practice
for i in [0,3,7,30,90,365]:
    print('{}: {}'.format(i,f_age(i)))
def f_duration(x):
    if (x > 0) & (x < 4):
        return x**0.5
    elif x >= 4:
        return 2.0

#showing how it works in practice
for i in [0,1,2,3,4,5]:
    print('{}: {}'.format(i,f_duration(i)))<jupyter_output>0: None
1: 1.0
2: 1.4142135623730951
3: 1.7320508075688772
4: 2.0
5: 2.0<jupyter_text>Creating the forecast function<jupyter_code>def forecast(x, state, candidates, max_results, max_age,
                p_fte_grade, p_population, p_methodology, p_tracking,
                p_sample_size, p_age, p_duration):
    
    #limiting to the state being forecast and the max number of polls per pollster and max poll age
    x = x[x.state==state].copy()
    x['recency'] = x.sort_values(by='age', ascending=True).groupby('pollster').cumcount()
    x = x[x.recency < max_results]
    x = x[x.age <= max_age]
    
    #applying weight mappings
    x['w1'] = x.fte_grade.map(p_fte_grade)
    x['w2'] = x.population.map(p_population)
    x['w3'] = x.methodology.map(p_methodology)
    x['w4'] = x.tracking.map(p_tracking)
    x['w5'] = x.sample_size.map(p_sample_size)
    x['w6'] = x.age.map(p_age)
    x['w7'] = x.duration.map(p_duration)
    
    #calculating overall weights and adjusting to mean=1
    x['w'] = x.w1 * x.w2 * x.w3 * x.w4 * x.w5 * x.w6 * x.w7
    x['w'] = x.w / x.w.mean()
    
    #calculating forecasts
    forecast = dict()
    for i in (candidates + ['UNDECIDED']):
        forecast[i] = np.round((x[i]*x.w).sum() / x.w.sum(),1)
    
    #showing metadata
    print('\n\nFORECAST: {}'.format(state))
    print('\nNumber of polls included: {}'.format(len(x)))
    top5 = x.sort_values(by='w', ascending=False).head(5)[
            ['pollster', 'fte_grade', 'sample_size', 'population', 'methodology', 'end_date',
             'tracking', 'age', 'duration', 'w']]
    print('\nTop 5 highest-weighted polls:')
    print(top5)
        
    #showing forecast
    print('\nForecast:')
    print(pd.Series(forecast).sort_values(ascending=False))<jupyter_output><empty_output><jupyter_text>Generating forecasts  National forecast<jupyter_code>forecast(data, 'National', candidates=cands, max_results=3, max_age = 90,       
            p_fte_grade=w_fte_grade, p_population=w_population, p_methodology=w_methodology, p_tracking=w_tracking,
            p_sample_size=f_sample_size, p_age=f_age, p_duration=f_duration)<jupyter_output>FORECAST: National

Number of polls included: 55

Top 5 highest-weighted polls:
                                             pollster fte_grade  sample_size  \
poll_id                                                                        
62557                                 Morning Consult       B/C        16071   
62536    Fox News/Beacon Research/Shaw & Co. Research        A-          471   
62537                    NBC News/Wall Street Journal       A/B          414   
62563                             Monmouth University        A+          345   
62503                                 Morning Consult       B/C        15431   

        population methodology   end_date  tracking  age  duration         w  
poll_id                                                                       
62557           lv      Online 2019-11-03     False    3         7  2.544276  
62536           lv  Live Phone 2019-10-30     False    7         4  2.373795  
62537           lv  Live Phone 2019-10-30[...]<jupyter_text>State-by-state forecasts<jupyter_code>for i in sorted(data.state.unique()):
    if i != 'National':
        forecast(data, state=i, candidates=cands, max_results=3, max_age = 365,       
                    p_fte_grade=w_fte_grade, p_population=w_population, p_methodology=w_methodology, p_tracking=w_tracking,
                    p_sample_size=f_sample_size, p_age=f_age, p_duration=f_duration)<jupyter_output>FORECAST: Arizona

Number of polls included: 4

Top 5 highest-weighted polls:
                                pollster fte_grade  sample_size population  \
poll_id                                                                      
62508                    Emerson College        A-          339         lv   
59598    Bendixen & Amandi International       B/C          250         rv   
59778                    Change Research         C          396         lv   
58455     Zogby Interactive/JZ Analytics         C          197         lv   

        methodology   end_date  tracking  age  duration         w  
poll_id                                                            
62508    IVR/Online 2019-10-28     False    9         4  2.307815  
59598    Live Phone 2019-09-12     False   55         4  0.798929  
59778        Online 2019-09-28     False   39         2  0.583818  
58455        Online 2019-05-29     False  161         7  0.309438  

Forecast:
Biden        26.8
Warren       2[...]
{'fraction_non_alphanumeric': 0.10441365519211541, 'fraction_numerical': 0.04027996000571347, 'mean_word_length': 2.7113702623906706, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '264917', 'license': 'Unlicense', 'n_tokens_mistral': 4782, 'n_tokens_neox': 4289, 'n_words': 1242, 'path': '.ipynb_checkpoints/Assignment 9 Example-checkpoint.ipynb', 'repo_name': 'therriault/dsforcampaigns_fall2019_public', 'size': '14013'}