<jupyter_start><jupyter_text>Before filling values, I wanna check and see min and max in comparison to see if there are bad outliers for means<jupyter_code>for column in  df.columns[3:]:
    var = df[column]
    small = var.min()
    big = var.max()
    deviate = var.std()
    
    print(column)
    print(f'Min is:  {small}')
    print(f'Max is:  {big}')
    print(f'STD is:  {deviate}')
   

#look at histograms to see general curve and attempt to detect mass outliers
df.hist(figsize = (15,15))
plt.show()<jupyter_output><empty_output><jupyter_text>I don't see too many outliers so I'm gonna fill by mean<jupyter_code>fill_df = df.copy()
need_to_fill = ['Life expectancy','Adult Mortality', 'Alcohol', 'Hepatitis B', 'BMI', 'Polio', 'Total expenditure', 
               'Diphtheria', 'GDP', 'Population', 'thinness  1-19 years', 'thinness 5-9 years', 'Income composition of resources', 
               'Schooling']

for col in need_to_fill:
    fill_df.loc[:,col] = fill_df.loc[:,col].fillna(fill_df.loc[:,col].mean())
#make sure this worked
fill_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Now on to interpolate and comparisons among the two to see which one is better<jupyter_code>inter_df = df.copy()
for col in need_to_fill:
    inter_df.loc[:,col] = inter_df.loc[:, col].interpolate()

#let's do the mean by year because I can't figure out how to sort by country. I'm not smart

pop_df = df.copy()
years = pop_df["Year"].unique()

for col in need_to_fill:
    for year in years:
        pop_df.loc[pop_df["Year"] == year, col] = pop_df.loc[pop_df["Year"] == year, col].fillna(
            pop_df[pop_df["Year"] == year][col].mean())
pop_df.isna().sum()
for col in need_to_fill:
    print('Stats for: {}'.format(col))
    print(pd.concat([df[col], fill_df[col], inter_df[col], pop_df[col]], axis = 1).describe())<jupyter_output>Stats for: Life expectancy
       Life expectancy  Life expectancy  Life expectancy  Life expectancy
count      2928.000000      2938.000000      2938.000000      2938.000000
mean         69.224932        69.224932        69.214874        69.231777
std           9.523867         9.507640         9.510819         9.508362
min          36.300000        36.300000        36.300000        36.300000
25%          63.100000        63.200000        63.100000        63.200000
50%          72.100000        72.000000        72.000000        72.000000
75%          75.700000        75.600000        75.600000        75.600000
max          89.000000        89.000000        89.000000        89.000000
Stats for: Adult Mortality
       Adult Mortality  Adult Mortality  Adult Mortality  Adult Mortality
count      2928.000000      2938.000000      2938.000000      2938.000000
mean        164.796448       164.796448       164.744554       164.741882
std         124.292079       124.080302       124.226596  [...]<jupyter_text>Looks like doing general mean is better than specialized mean and interpolation. If I could figure out how to fill based on country, I'd want to try that for next pass through.<jupyter_code>def find_z_score(sample):
    for column in  sample.columns:
        var = stats.zscore(sample[column])
        small = var.min()
        big = var.max()
        print(column)
        print(f'Min is:  {small}')
        print(f'Max is:  {big}')
    
find_z_score(fill_df.select_dtypes(include=['float64', 'int64']))<jupyter_output>Year
Min is:  -1.6298784908947028
Max is:  1.6217623010351567
Life expectancy
Min is:  -3.463586777506765
Max is:  2.0802674929010387
Adult Mortality
Min is:  -1.3203089339714467
Max is:  4.499494007167724
infant deaths
Min is:  -0.2570169256653636
Max is:  15.009325999864597
Alcohol
Min is:  -1.1729584018986523
Max is:  3.388259111251576
percentage expenditure
Min is:  -0.37143289259454365
Max is:  9.429403166006338
Hepatitis B
Min is:  -3.539849644426931
Max is:  0.7996958107436184
Measles
Min is:  -0.21103572121748299
Max is:  18.295467854833035
BMI
Min is:  -1.8731536327605787
Max is:  2.4582439871615565
under-five deaths
Min is:  -0.2620383964766972
Max is:  15.322224258249507
Polio
Min is:  -3.4071275910584027
Max is:  0.70454398678701
Total expenditure
Min is:  -2.3202091677166066
Max is:  4.859360377924712
Diphtheria
Min is:  -3.3983717740773227
Max is:  0.7055289161190191
HIV/AIDS
Min is:  -0.3234448000817086
Max is:  9.62353033585425
GDP
Min is:  -0.5696022191662836
Max is:  [...]<jupyter_text>Is population really that weird?<jupyter_code>population = fill_df[['Population', 'Country']].sort_values(by = 'Population')
population
sns.scatterplot(x = 'Country', y = 'Population', data = population)

population = fill_df.sort_values(by = ['Population'], ascending = False)
population<jupyter_output><empty_output><jupyter_text>I can't take off India's population. It is a population anomoly, but needs to stay since without India, the population curve is much more normal.<jupyter_code>fill_df.boxplot(figsize = (40, 40))
plt.show()<jupyter_output><empty_output><jupyter_text>clearly is India well and above  Based on the z_values, I'm gonna keep limits of 4 on each side. I'll compare log and box_cox<jupyter_code>log_pop = np.log(fill_df['Population'])
z_scores = stats.zscore(log_pop)
outlier_filter = np.where(abs(z_scores) <= 3)
filtered_log_pop = log_pop.values[outlier_filter]

# Show count of lost observations
print(f'There were {len(log_pop) - len(filtered_log_pop)} observations removed by z-score.')

# Re-check histogram and normal QQ plot
fig, axes = plt.subplots(1, 2)

axes[0].hist(filtered_log_pop)
stats.probplot(filtered_log_pop, dist='norm', plot=axes[1])

axes[0].set_title('Log transform hist')
axes[1].set_title('Log transform QQ')

fig.tight_layout()
plt.show()
boxcox_pop,_ = stats.boxcox(fill_df['Population'])
z_scores_pop = stats.zscore(boxcox_pop)
outlier_filter_pop = np.where(abs(z_scores_pop) <= 3)
filtered_boxcox_pop = boxcox_pop[outlier_filter_pop]
#fill_df.values
#boxcox_pop

#Show count of lost observations
print(f'There were {len(boxcox_pop) - len(filtered_boxcox_pop)} observations removed by z-score.')

# Re-check histogram and normal QQ plot
fig, axes = plt.subplots(1, 2)

axes[0].hist(filtered_boxcox_pop)
stats.probplot(filtered_boxcox_pop, dist='norm', plot=axes[1])

axes[0].set_title('Box cox transform hist')
axes[1].set_title('Box cox transform QQ')

fig.tight_layout()
plt.show()<jupyter_output>There were 5 observations removed by z-score.<jupyter_text>Boxcox looks better, but let's be sure<jupyter_code>log_df = fill_df.copy()
log_df = np.log(log_df.select_dtypes(include=['float64', 'int64']))
log_df.apply(find_z_score(log_df.select_dtypes(include=['float64', 'int64'])))
outlier_filter = np.where(abs(z_scores) <= 3)
filtered_log_pop = log_pop.values[outlier_filter]
def boxed_df(sample):
    box_df = {}
    for col in sample.select_dtypes(include= 'number'):
        smol_value = sample[col].min()
        fixed_value = abs(smol_value) + 1
        box_pop,_ = stats.boxcox(sample[col] + fixed_value)
        box_df[col] = box_pop
    box_df = pd.DataFrame(box_df)
    return box_df


box_df 





#fill_df.loc[30, 'Population']<jupyter_output><empty_output><jupyter_text>Let's check some histograms and coorelation heatmaps<jupyter_code>box_df.select_dtypes(include=['float64', 'int64']).hist(figsize = (15,15))

log_df.hist(figsize = (15,15))
sns.heatmap(fill_df.corr(), vmin = -1, vmax = 1)

sns.heatmap(log_df.corr(), vmin = -1, vmax = 1)

sns.heatmap(box_df.corr(), vmin = -1, vmax = 1)
log_df.info()
box_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2938 entries, 0 to 2937
Data columns (total 20 columns):
Year                               2938 non-null float64
Life expectancy                    2938 non-null float64
Adult Mortality                    2938 non-null float64
infant deaths                      2938 non-null float64
Alcohol                            2938 non-null float64
percentage expenditure             2938 non-null float64
Hepatitis B                        2938 non-null float64
Measles                            2938 non-null float64
BMI                                2938 non-null float64
under-five deaths                  2938 non-null float64
Polio                              2938 non-null float64
Total expenditure                  2938 non-null float64
Diphtheria                         2938 non-null float64
HIV/AIDS                           2938 non-null float64
GDP                                2938 non-null float64
Population                         293[...]<jupyter_text>I think boxcox dealt with outliers pretty well. I'm gonna keep pressing on with that one.<jupyter_code>probplot(box_df['GDP'], dist = 'norm', plot = plt)
plt.show()
stats.ttest_ind(box_df['Life expectancy'], box_df['GDP'])<jupyter_output><empty_output><jupyter_text>Here I re-refer to the heatmap<jupyter_code>sns.heatmap(box_df.corr(), vmin = -1, vmax = 1)
# I realized I'm trash with colors. 
#I'd want python to show me data with high coorelations then use the heatmap to double check is it positive or negative

corred = box_df.corr().abs()
s = corred.unstack()
s = s.reset_index()

s = s[s['level_0']=='Life expectancy'].sort_values(by = 0, ascending = False)#
s.head(10)<jupyter_output><empty_output><jupyter_text>So I would pick everything above except for life expectancy(redundant), and I wouldn't pick thinness 5-9 years due to redundancy with 1-19 years<jupyter_code>s.tail(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1102878916172735, 'fraction_numerical': 0.1096528365791702, 'mean_word_length': 3.4216190921853067, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '411364', 'license': 'MIT', 'n_tokens_mistral': 4025, 'n_tokens_neox': 3286, 'n_words': 838, 'path': 'Challenge.ipynb', 'repo_name': 'AsterLaoWhy/Thinkful', 'size': '9448'}