<jupyter_start><jupyter_text>What is the median loan amount?<jupyter_code>pd.concat([df14.loan_amnt, df15.loan_amnt]).median(axis = 0, skipna = True)<jupyter_output><empty_output><jupyter_text>Each loan is categorized into a single purpose. What fraction of all loans are for the most common purpose?<jupyter_code>pd.concat([df14.purpose, df15.purpose]).value_counts() / pd.concat([df14.purpose, df15.purpose]).value_counts().sum()<jupyter_output><empty_output><jupyter_text>Calculate the average interest rate across loans for each purpose. What is the ratio of minimum average rate to the maximum average rate? (The ratio should be less than 1.)<jupyter_code>df = pd.concat([df14, df15])
df['int_rate'].head()
df['int_rate'] = df['int_rate'].apply(lambda x: float(str(x).replace('%', '')) )
ratios_by_purposes = df.groupby('purpose')['int_rate'].agg('mean')
min(ratios_by_purposes) / max(ratios_by_purposes)<jupyter_output><empty_output><jupyter_text>What is the difference in the fraction of the loans with a 36-month term between 2014 and 2015?<jupyter_code>(df14['term'].value_counts() / df14.shape[0])[0] - (df15['term'].value_counts() / df15.shape[0])[0]<jupyter_output><empty_output><jupyter_text>We will consider all loans that are not in the 'Fully Paid', 'Current', 'In Grace Period' statuses to be in default. Calculate the ratio of the time spent paying the loan, defined as the difference between the last payment date and the issue date, divided by the term of loan. What is the standard deviation of this ratio for all the loans in default?<jupyter_code>df['pymnt_d'] = pd.to_datetime(df['last_pymnt_d']) - pd.to_datetime(df['issue_d']) 
df['pymnt_d'] = df['pymnt_d'].apply(lambda x: x.days)
df['term_int'] = df['term'].apply(lambda x: int(str(x).replace(' months', '')) if pd.notnull(x) else x)
df['pymnt_d_ratio'] = df['pymnt_d'] / df['term_int']
default_idx = df['loan_status'].apply(lambda x: x not in ('Fully Paid', 'Current', 'In Grace Period'))
df.loc[default_idx,:]['pymnt_d_ratio'].std()<jupyter_output><empty_output><jupyter_text>What is the Pearson correlation coefficient between the total rate of return, as figured from the total payments and the loan amount, and the interest rate? Consider only loans that have reached the end of their term.<jupyter_code>df['rate_of_return'] = df['total_pymnt'] / df['loan_amnt']
df['pymnt_m'] = pd.to_datetime(df['last_credit_pull_d']).dt.to_period('M') - pd.to_datetime(df['issue_d']).dt.to_period('M')
idx_ended = (df['term_int'] - df['pymnt_m'] < 0)
df.loc[idx_ended,:][['rate_of_return', 'int_rate']].corr(method = 'pearson')<jupyter_output><empty_output><jupyter_text>Let's find a loan purpose that shows up abnormally often in one state. Call A the probability of a loan going to a specific purpose nationwide. Call B the probability of a loan going to a specific purpose for each state. Out of all (state, purpose) pairs with at least 10 loans, what is the highest ratio of B / A (i.e. the most surprising)?<jupyter_code># nation-wide
nation_purposes = df['purpose'].value_counts() / df['purpose'].value_counts().sum()
nation_purposes = nation_purposes.to_frame().reset_index(level=0)
nation_purposes.columns = ['purpose', 'prop']
nation_purposes.head()
state_purposes = df[['addr_state', 'purpose']].groupby(['addr_state', 'purpose']).size()
state_purposes = state_purposes[state_purposes >= 10]
state_purposes_prop = state_purposes.groupby(level = 0).apply(lambda x: x / float(x.sum())).rename('prop').reset_index()
state_purposes_prop.head()
state_purposes_prop = state_purposes_prop.merge(nation_purposes, left_on = 'purpose', right_on = 'purpose', how = 'left')
state_purposes_prop.head()
max(state_purposes_prop['prop_x'] / state_purposes_prop['prop_y'])<jupyter_output><empty_output><jupyter_text>Group all loans by their sub-grade and calculate their average inerest rate and average default rate. Build a linear model to predict the (average) default rate from the (average) interest rate. Find that sub-grade with the largest absolute deviation from this model. What is the deviation of the actual default rate from the predicted value?<jupyter_code>df_int_rate = df.groupby('sub_grade')['int_rate'].agg('mean').to_frame().reset_index(level = 0)
df_int_rate.head()
### taken from the previous question
df.loc[default_idx, 'loan_status'] = 'default'
df_default = df.groupby(['sub_grade', 'loan_status'])['loan_status'].agg('count') \
    .groupby(level = 0).apply(lambda x: x / float(x.sum())).rename('default_prop').reset_index()
df_default = df_default.loc[df_default.loan_status == 'default']
df_default.head()
df_int_rate = df_int_rate.merge(df_default[['sub_grade', 'default_prop']], left_on = 'sub_grade', right_on = 'sub_grade',
                 how = 'left')
df_int_rate['int_rate'] - df_int_rate['default_prop']
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(df_int_rate['int_rate'].reshape(-1, 1), df_int_rate['default_prop'])
max(abs(lm.predict(df_int_rate['int_rate'].reshape(-1, 1)) - df_int_rate['default_prop']))<jupyter_output>//anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:1: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
  if __name__ == '__main__':
{'fraction_non_alphanumeric': 0.1425317897134181, 'fraction_numerical': 0.01062820269500854, 'mean_word_length': 6.8774289985052315, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '223692', 'license': 'MIT', 'n_tokens_mistral': 1833, 'n_tokens_neox': 1758, 'n_words': 449, 'path': 'others/Q2.ipynb', 'repo_name': 'qjin2016/project_demo1', 'size': '5269'}