<jupyter_start><jupyter_text>ALL OF THIS IS TO GET A SMALLER CSV TO WORK WITH!<jupyter_code>import pandas as pd
pd.set_option('display.max_columns', 500)
datapath = 'C:\\Users\\Nick\\LendingClub Project\\New\\Data'
df = pd.read_csv(datapath + '\\loan.csv', low_memory=False)
df.shape
df.head()<jupyter_output><empty_output><jupyter_text>ONLY WANT LOANS THAT ARE FINALIZED (NO CURRENT LOANS)<jupyter_code>df['loan_status'].value_counts()
loans_usable = ['Fully Paid', 'Charged Off', 'Default']
df1 = df[df['loan_status'].isin(loans_usable)]
df1.shape
df1['loan_status'].value_counts()<jupyter_output><empty_output><jupyter_text>MANY COLUMNS WITH ~HALF NULL VALUES, DROP TO INCREASE PERFORMANCE<jupyter_code>(df1.isnull().sum() > 500000).value_counts()<jupyter_output><empty_output><jupyter_text>CREATE NEW DF WITH COLUMNS AND WHETHER THEY ARE MOSTLY NULL TO CREATE A LIST TO DROP<jupyter_code>clean = pd.DataFrame({'Name' : (df1.isnull().sum() > 500000).index.tolist(), 'Null' : (df1.isnull().sum() > 500000).tolist()})
clean.head()<jupyter_output><empty_output><jupyter_text>NEW DF TO EXPORT WITH CLEANED COLUMNS<jupyter_code>drop_null_cols = clean[clean['Null'] == True]['Name'].tolist()
x = df1.drop(drop_null_cols, axis=1)
x.head()
# Create new CSV to work with
x.to_csv(datapath + '\\clean_loan.csv', index=False)
df.columns<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15537555228276878, 'fraction_numerical': 0.021354933726067747, 'mean_word_length': 7.656050955414012, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '179748', 'license': 'MIT', 'n_tokens_mistral': 569, 'n_tokens_neox': 525, 'n_words': 99, 'path': 'notebooks/CSV_Clean.ipynb', 'repo_name': 'Nckflannery/lending-club-predictions', 'size': '1358'}