<jupyter_start><jupyter_text>```NoteBook Focus```---1. Exploring the data to see what the data says.2. Most of the EDA was done through a "Try and test" analysis by fitting models and checking for overfitting and unusually high accuracy.  ```Imports```---<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
drugs = pd.read_csv('../Claire/data/drugs_2020_simply_imputed.csv')
drugs.head()
drugs.columns<jupyter_output><empty_output><jupyter_text>```EDA1```---1. General EDA, checking for multicolinearity, etc..<jupyter_code>sns.heatmap(drugs[['prisdum', 'statmax', 'statmin']].corr(), annot=True);
plt.figure(figsize=(8,5))
sns.heatmap(drugs.corr()[['prisdum']].sort_values(by='prisdum', ascending=False).head(10), annot=True);<jupyter_output><empty_output><jupyter_text>**The only feature that stands out as being highly correlated is 'suprdum' (Indicates whether the defendant received supervised release). This feature will be removed from all models to see how models perform. What i'm expecting is that a model missing this information will perform worst.**  EDA2---1. Trying to identify which features are making the model on unbalanced classes work well.<jupyter_code>df = drugs[['statmax', 'statmin', 'supermax', 'supermin','suprel','timservc', 'totchpts', 'totrest', 'totunit', 'typemony','prisdum']]
plt.figure(figsize=(8,5))
sns.heatmap(df.corr(), annot=True);
drugs['casetype'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14421768707482993, 'fraction_numerical': 0.010884353741496598, 'mean_word_length': 7.038251366120218, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '391950', 'license': 'MIT', 'n_tokens_mistral': 511, 'n_tokens_neox': 475, 'n_words': 134, 'path': 'code/02_EDA.ipynb', 'repo_name': '123CarlosRivera/project5', 'size': '1470'}