<jupyter_start><jupyter_text>**Table of Content:*** Summary* Introduction* Methods* Results* Conclusions* References  1. Summary In this project, we try to find the best features that best predict default customers using machine learning tools. Logistic Regression was found to achieve acceptable results on the test data provided to the trained model. The accuracy of the model on test data was about {{test_accuracy}} and the recall on test data found to be {{recall}}. The precision for the model on the test was about {{precision}} . The area under the ROC Curve for the final model is {{auc}}.Due to the risk associated with customers failing to pay, the model was designed to maximize the recall rate, identifying customers that will default to the greatest extent. This was also balanced with the overall accuracy of the training and test dataset. The model predicts the following 7 features to be the most important features to predict customers default.1. Amount of the given credit (NT dollar): it includes both the individual consumer credit and his/her family (supplementary) credit.2. EDUCATION3. MARRIAGE4. AGE5. Past monthly repayment status in September 2005 (which is the most recent month before the prediction month)6. Past monthly repayment status in August 2005 (which is the second from the most recent month before the prediction month)7. Amount of previous payment (NT dollar) in September 2005 (which is the most recent month before the prediction month)  2. Introduction Prediction of customers default behaviour is critically important in Risk Management by lenders. In particular,  there has been a significant interest in identifying features that are associated with the highest prediction power to reduce the overall lender's credit risk. In this study, we perform a data-informed analysis to build a model that can successfully capture features that predict default payment.  3. Methods  DataWe used credit default data collected from the Taiwanese market in 2005. The Data Set is available from [UCI Machine Learning Repository Irvine, CA: University of California, School of Information and Computer Science](https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients). The data that contains 23 features from 30,000 customers. was originally publicized by Chung Hua University of Taiwan and Tamkang University of Taiwan. Features include :- `LIMIT_BAL`: Amount of the given credit (NT dollar): it includes both the individual consumer credit and his/her family (supplementary) credit. - `SEX`: Gender(1 = male; 2 = female).- `EDUCATION`: Education (1 = graduate school; 2 = university; 3 = high school; 4 = others). - `MARRIAGE`: Marital status (1 = married; 2 = single; 3 = others).  - `AGE`: Age (year).  - `PAY_1`, `PAY_2`, ..., `PAY_6`: Past monthly repayment status in September 2005, August 2005, ..., April 2005 respectively. ( -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months; . . .; 8 = payment delay for eight months; 9 = payment delay for nine months and above.)  - `BILL_AMT1`, `BILL_AMT2`, ..., `BILL_AMT6`: Amount of bill statement (NT dollar) in September 2005, August 2005, ..., April 2005 respectively.  - `PAY_AMT1`, `PAY_AMT2`, ..., `PAY_AMT6`: Amount of previous payment (NT dollar) in September 2005, August 2005, ..., April 2005 respectively.    Analysis Immediately after importing the data it was split into training and test data. Only 75% of the data was used to train the models and the test data was only used to obtain the test performance of the model on unseen data.  Next, we created a list for numeric and categorical features, below is the summary statistic of the training data. It showing mean, standard deviation, min, max etc. The bill amount, payment amount and credit limit ranges are roughly similar which are around 800,000. Interestingly, the median for the bill statement amounts is around 20,000, but the median for payment amounts is 2,000. Age ranges from 21 to 75 which is reasonable<jupyter_code>summary<jupyter_output><empty_output><jupyter_text>Table 1. Summary the data used in this study To learn the association between numeric features we explored their inter-correlations which can be seen below. We can observe that some features have a stronger co-linearity such as BILL-AMT1,BILL-AMT2,.. to BILL-AMT6 ![](../results/num_corr_chart.png) Figure 1. Inter-correlation between numeric features We can also study the correlation between the features and the response variable. We can see that some of the features such as LIMIT_BALANCE and Age have a stronger correlation with the response variable than other features.  ![](../results/num_res_chart.png) [](roc.png) Figure 2. Correlation between numeric features and response Figure 2 also shows that many of the features have a heavy tail distribution.  To mitigate this issue we applied [SMOTE](https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html) (Synthetic Minority Oversampling Technique) on the response variable to create a balanced data set to fit the model. Furthermore, we implemented [`RobustScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html) to scale predictors  4. Results  We selected logistic regression model(`LogisticRegression`) and [`RFE`](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.htmlsklearn.feature_selection.RFE)(recursive feature elimination) as our model since it is more robust given that the dataset has many of the features are not normally distributed. One additional advantage of (`LogisticRegression`) that is much interpretable than more complex modelsWe started the analysis by applying a robust scalar on the training data-set. Following that we build a model with the full set of features as our base-case model. The confusion matrix, evaluation matrix and ROC results were obtained to set a bench-mark for comparison purposes. `RFE` was then used to identify the most useful predictors and consequently we dropped those columns that are deemed as less useful. Eventually, 7 features were used to train the model.The hyperparameters `C` was tunned in the range from -4 to 20 using 5-fold cross-validation and the model was then fitted with the best hyperparameter. Let us now look at the result by glancing into the confusion matrix. Best-case Model            |  Base-case Model:-------------------------:|:-------------------------:![](../results/confusion_matrix.png)  |  ![](../results_baseline/confusion_matrix.png) Figure 3. Confusion matrix of the fitted model with 7 features We can see that the best-case model which uses 7 features out-performs the base-case model in many aspects. First, it has obtained more true negatives on the test data than the base-case model. Furthermore, the best-case model has less false positives than the base-case model making it more precise as depicted in the evaluation matrix below.<jupyter_code>evaluation_matrix<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07007818052594172, 'fraction_numerical': 0.01847903340440654, 'mean_word_length': 4.601910828025478, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '456551', 'license': 'MIT', 'n_tokens_mistral': 1828, 'n_tokens_neox': 1639, 'n_words': 959, 'path': 'doc/final_report.ipynb', 'repo_name': 'elliott-ribner/DSCI_522_group-314', 'size': '7035'}