<jupyter_start><jupyter_text>Risk Adjustment and Machine Learning  Loading health data<jupyter_code># Import pandas and assign to it the pd alias
import pandas as pd

# Load csv to pd.dataframe using pd.read_csv
df_salud = pd.read_csv('../suficiencia.csv')

# Index is not appropiately set
print(df_salud.head())

# pd.read_csv inferred unconvenient data types for some columns
for columna in df_salud.columns:
    print(columna,df_salud[columna].dtype)
# TO DO: declare a dict named dtype with column names as keys and data types as values
# We need MUNI_2010, MUNI_2011, DPTO_2010 and DPTO_2011 as data type 'category'. We need SEXO_M and SEXO_F as bool as well.
dtype = {}

# TO DO: declare a integer variable with the column number to be taken as index 
index_col = 

# We reload csv file using index_col and dtype parameters  
df_salud = pd.read_csv('../suficiencia.csv',index_col= index_col,dtype=dtype)

# Index is appropriately set
print(df_salud.head())

# TO DO: check pd.read_csv has convenient data types 
# Check last code cell for help.

# TO DO: print mean value for expenditure in 2010 and 2011
# Expenditure is given by variables 'VALOR_TOT_2010' and 'VALOR_TOT_2011'<jupyter_output><empty_output><jupyter_text>Exploring health dataWe are interested in exploring risk profiles of individuals. Lets estimate expenditure and enrollee density distribution for different expenditure intervals. We will consider intervals of \$10,000 COP between \$0 and \$3,000,000 COP.<jupyter_code># We will be using plotly to graph the distributions. 
import plotly 
import plotly.graph_objs as go 
plotly.offline.init_notebook_mode(connected=True)

# Set interval and step size
tamanho = 10**6*3
step_size = 10**4
# Enrollee distribution is straightforward using plotly. 
trace2010 = go.Histogram(
                x=df_salud['VALOR_TOT_2010'],
                name='2010',
                histnorm='probability',
                xbins=dict(start=0.0,end=tamanho,size=step_size),
                legendgroup='2010'
            )

# TO DO: declare a second trace for the 2011 enrollee distribution
trace2011 = go.Histogram(
                
            )

layout = go.Layout(
            legend=dict(
                xanchor='center',
                yanchor='top',
                orientation='h',
                y=-0.25,
                x=0.5,
            ),
            yaxis=dict(
                title='Density',
                rangemode='tozero'
            ),
            xaxis=dict(
                title='Expenditure'
            ),
            title='Enrolle density'
         )

# TO DO Add both traces to a list and pass it to go.Figure data parameter
fig = go.Figure(data=, layout=layout)
plotly.offline.iplot(fig)<jupyter_output><empty_output><jupyter_text>Expenditure distribution needs extra work since we are accumulating expenditure and not enrollees. For this purpose we first sort enrollees, then we calculate accumulated expenditure up to each interval and normalize it by total expenditure and finally we differentiate the series.<jupyter_code># TO DO: import numpy with alias np

# TO DO: write function to calculate expenditure cumulative density for a given year
def calculate_expenditure_cumulative_density(year):
        
    return cumulative_density
density_2010 = np.diff(calculate_expenditure_cumulative_density('2010'))
density_2011 = np.diff(calculate_expenditure_cumulative_density('2011'))

# TO DO: declare a trace for 2010 expenditure distribution. Use color '#1f77b4' for markers.
trace_2010 = go.Scatter(
                
             )

trace_2011 = go.Scatter(
                x=list(range(0,tamanho,step_size)),
                y=density_2011,
                legendgroup='2011',
                name='2011',
                marker=dict(color='#ff7f0e'),
                type='bar'
             )

layout = go.Layout(
            legend=dict(
                xanchor='center',
                yanchor='top',
                orientation='h',
                y=-0.25,
                x=0.5,
            ),
            yaxis=dict(
                title='Density',
                rangemode='tozero'
            ),
            xaxis=dict(
                title='Expenditure'
            ),
            title='Expenditure density'
         )

# Add both traces to a list and pass it to go.Figure data parameter. Add the layout parameter as well.
fig = go.Figure(data=,layout=)
plotly.offline.iplot(fig)<jupyter_output><empty_output><jupyter_text>How about cumulative density for enrollees and expenditure? Enrollee cumulative density needs some extra work since we did not explicitly calculate enrollee density before.<jupyter_code># We will be using scipy
from scipy import stats
# TO DO: scipy.stats.percentileofscore(series,score) returns percentile value of score in series
def calculate_enrollee_cumulative_density(year):
    
    return cumulative_density
enrollee_cumulative_density_2010 = calculate_enrollee_cumulative_density('2010')
enrollee_cumulative_density_2011 = calculate_enrollee_cumulative_density('2011')
expenditure_cumulative_density_2010 = calculate_expenditure_cumulative_density('2010')
expenditure_cumulative_density_2011 = calculate_expenditure_cumulative_density('2011')

# TO DO: Create cumulative expenditure and enrollees traces and plot them. Use previous color conventions.
trace_enrollee_2010 = go.Scatter(
                        
                     )

trace_enrollee_2011 = go.Scatter(
                            
                      )

trace_expenditure_2010 = go.Scatter(
                            
                         )


trace_expenditure_2011 = go.Scatter(
                            
                         )

layout = go.Layout(
            legend=dict(
                xanchor='center',
                yanchor='top',
                orientation='h',
                y=-0.25,
                x=0.5,
            ),
            yaxis=dict(
                title='Cumulative density (%)',
                rangemode='tozero'
            ),
            xaxis=dict(
                title='Expenditure'
            ),
            title='Cumulative density of enrollees and expenditure'
         )<jupyter_output><empty_output><jupyter_text>Benchmarking the problemBefore fitting any models it is convenient to have a benchmark from a model as simple as possible. We estimate the mean absolute error (MAE) of the simple model $$ y_{it}^{pred} = \frac{1}{N}\sum_{N}{y_{it}} $$<jupyter_code>ymean = df_salud['VALOR_TOT_2011'].mean()

# TO DO : write a function that calculates beanchmark MAE
def calculate_benchmark_mae(row):
    
    return mae

print('BENCHMARK MAE',df_salud.apply(calculate_mae,axis=1).mean())<jupyter_output><empty_output><jupyter_text>MSPS risk adjustment Colombian Ministry of Health and Social Protection currently employs a linear regression of annual health expenditure on sociodemographic risk factors that include gender, age groups and location as the risk-adjustment mechanism.$$y_{it} = \beta_{0} + \sum_{K}{\beta_{j}D_{jit}} + \epsilon_{i}$$ We will start by calculating age groups from variable 'EDAD_2011'.<jupyter_code># Creating a grouping variable is straightforward with pd.cut
bins = [0,1,4,18,44,49,54,59,64,69,74,150]
labels = ['0_1','2_4','5_18','19_44','45_49','50_54','55_59','60_64','65_69','70_74','74_']
df_salud['AGE_GROUP'] = pd.cut(df_salud['EDAD_2011'],bins,labels=labels,include_lowest=True)
print(df_salud[['EDAD_2011','AGE_GROUP']])

# We also need to create dummy variables using pd.get_dummies
age_group_dummies = pd.get_dummies(df_salud['AGE_GROUP'],prefix='AGE_GROUP')
df_salud = pd.concat([df_salud,age_group_dummies],axis=1)
for column in df_salud.columns:
    print(column)<jupyter_output><empty_output><jupyter_text>We also need to group location codes into government defined categories. This requires some extra work. Make sure you have divipola.csv file in your home.<jupyter_code># Download divipola.csv from your email and mo
divipola = pd.read_csv('../divipola.csv',index_col=0)

def give_location_group(row,divipola=divipola):
    codigo_dpto = str(row['DPTO_2011']).rjust(2,'0')
    codigo_muni = str(row['MUNI_2011']).rjust(3,'0')
    codigo = codigo_dpto + codigo_muni
    try:
        grupo = divipola.loc[int(codigo)]['zona']
    # Exception management for a single observation where last digit of municipality code is not valid
    except KeyError:
        return 'C'
    return grupo

location_group_dummies = pd.get_dummies(df_salud.apply(give_location_group,axis=1),prefix='LOCATION_GROUP')
df_salud = pd.concat([df_salud,location_group_dummies],axis=1)
for column in df_salud.columns:
    print(column)<jupyter_output><empty_output><jupyter_text>Now we are ready to fit MSPS linear model.<jupyter_code># We will be using sklearn
from sklearn import linear_model
from sklearn.model_selection import cross_val_score
# Feature space
# One reference category is excluded for each dummy group
features = ['SEXO_M',
            'AGE_GROUP_2_4',
            'AGE_GROUP_5_18',
            'AGE_GROUP_19_44',
            'AGE_GROUP_45_49',
            'AGE_GROUP_50_54',
            'AGE_GROUP_55_59',
            'AGE_GROUP_60_64',
            'AGE_GROUP_65_69',
            'AGE_GROUP_70_74',
            'AGE_GROUP_74_',
            'LOCATION_GROUP_N',
            'LOCATION_GROUP_Z',]

# Target space
target = ['VALOR_TOT_2011']

# TO DO: calculate 10 cv mae for linear regression model using sklearn.model_selection.cross_val_score. Take a look at the needed parameters.
reg = linear_model.LinearRegression()
neg_mae = cross_val_score(estimator=,X=,y=,cv=,scoring=)
print('REGRESSION MAE',-1*neg_mae.mean())
reg = reg.fit(df_salud[features].values,df_salud[target].values)

# TO DO: predict over enrollees with enrollees with 2011 expenditure above $3,000,000
upper = 
y_pred_upper = [y[0] for y in reg.predict(upper[features])]
print('REGRESSION MAE UPPER',(y_pred_upper-upper['VALOR_TOT_2011']).mean())

# TO DO: predict over enrollees with enrollees with 2011 expenditure below or equal to $3,000,000
lower = 
y_pred_lower = [y[0] for y in reg.predict(lower[features])]
print('REGRESSION MAE LOWER',(y_pred_lower-lower['VALOR_TOT_2011']).mean())<jupyter_output><empty_output><jupyter_text>Risk adjustment using machine learningHow about a regression tree?<jupyter_code># We will be using sklearn
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import cross_val_score
# Feature space
# One reference category is excluded for each dummy group
features = ['SEXO_M',
            'AGE_GROUP_2_4',
            'AGE_GROUP_5_18',
            'AGE_GROUP_19_44',
            'AGE_GROUP_45_49',
            'AGE_GROUP_50_54',
            'AGE_GROUP_55_59',
            'AGE_GROUP_60_64',
            'AGE_GROUP_65_69',
            'AGE_GROUP_70_74',
            'AGE_GROUP_74_',
            'LOCATION_GROUP_N',
            'LOCATION_GROUP_Z',]

# Target space
target = ['VALOR_TOT_2011']

reg_tree = DecisionTreeRegressor(min_samples_leaf=1000)
neg_mae = cross_val_score(estimator=,X=,y=,cv=,scoring=)
print('TREE REGRESSION MAE',-1*neg_mae.mean())<jupyter_output><empty_output><jupyter_text>What does a tree look like?<jupyter_code># We will use modules from sklearn, ipython and pydotplus to visualize trees
from sklearn.externals.six import StringIO
from IPython.display import Image, display
from sklearn.tree import export_graphviz
import pydotplus
def plot_tree(tree):
    dot_data = StringIO()
    export_graphviz(
        tree,
        out_file=dot_data,
        filled=True,
        special_characters=True,
        precision=0,
        feature_names=features
    )
    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
    display(Image(graph.create_png()))
    
reg_tree = DecisionTreeRegressor(min_samples_leaf=1000)
reg_tree = reg_tree.fit(df_salud[features].values,df_salud[target].values)
plot_tree(reg_tree)

upper = df_salud[df_salud['VALOR_TOT_2011'] > (3*10**6)]
y_pred_upper = reg_tree.predict(upper[features])
print('TREE REGRESSION MAE UPPER',(y_pred_upper-upper['VALOR_TOT_2011']).mean())

lower = df_salud[df_salud['VALOR_TOT_2011'] <= (3*10**6)]
y_pred_lower = reg_tree.predict(lower[features])
print('TREE REGRESSION MAE LOWER',(y_pred_lower-lower['VALOR_TOT_2011']).mean())
# Feature space
# One reference category is excluded for each dummy group
features = ['SEXO_M',
            'AGE_GROUP_2_4',
            'AGE_GROUP_5_18',
            'AGE_GROUP_19_44',
            'AGE_GROUP_45_49',
            'AGE_GROUP_50_54',
            'AGE_GROUP_55_59',
            'AGE_GROUP_60_64',
            'AGE_GROUP_65_69',
            'AGE_GROUP_70_74',
            'AGE_GROUP_74_',
            'LOCATION_GROUP_N',
            'LOCATION_GROUP_Z',
            'DIAG_1_C_2010',
            'DIAG_1_P_2010',
            'DIAG_1_D_2010',]

# Target space
target = ['VALOR_TOT_2011']

reg_tree = DecisionTreeRegressor(min_samples_leaf=100)
neg_mae = cross_val_score(estimator=reg_tree,X=df_salud[features],y=df_salud[target],cv=10,scoring='neg_mean_absolute_error')
print('TREE REGRESSION MAE',-1*neg_mae.mean())
reg_tree = DecisionTreeRegressor(min_samples_leaf=100)
reg_tree = reg_tree.fit(df_salud[features].values,df_salud[target].values)
plot_tree(reg_tree)

upper = df_salud[df_salud['VALOR_TOT_2011'] > (3*10**6)]
y_pred_upper = reg_tree.predict(upper[features])
print('TREE REGRESSION MAE UPPER',(y_pred_upper-upper['VALOR_TOT_2011']).mean())

lower = df_salud[df_salud['VALOR_TOT_2011'] <= (3*10**6)]
y_pred_lower = reg_tree.predict(lower[features])
print('TREE REGRESSION MAE LOWER',(y_pred_lower-lower['VALOR_TOT_2011']).mean())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09844938560561732, 'fraction_numerical': 0.037741369221767115, 'mean_word_length': 3.3865896695540583, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '43659', 'license': 'MIT', 'n_tokens_mistral': 4779, 'n_tokens_neox': 4341, 'n_words': 1012, 'path': 'health-students.ipynb', 'repo_name': 'gcaballero01/mlandpp-uniandes', 'size': '13672'}