<jupyter_start><jupyter_text>Finding Patterns in Data using IBM Power and PowerAIIn this lab we will explore an open source data set, and discover how we can use the tools that are part of **PowerAI** to explore and discover patterns in the data.  For this lab, we will make use of the Lending Club data set, pandas, numpy and scikit learn libraries.  Here is a brief description about Lending Club.  [Lending Club (LC)](https://www.lendingclub.com/) is the world’s largest online marketplace connecting borrowers and investors. It is transforming the banking system to make credit more affordable and investing more rewarding. Lending Club operates at a lower cost than traditional bank lending programs and pass the savings on to borrowers in the form of lower rates and to investors in the form of solid risk-adjusted returns.**The DATA**  The original data set is downloaded from [LC](https://www.lendingclub.com/info/download-data.action) covering complete loan data for all loans issued through the 2007-2018, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information. Additional features include credit history, number of finance inquiries, address including zip codes, and state, and collections among others. It is quite rich and is an excellent example of credit risk data.  Interestingly, Goldman Sachs’ new peer-to-peer lending platform called Marcus was built almost entirely using the Lending Club data.Here is a link to some extra information regarding the fields of the data set.[Data Dictionary](https://github.com/dustinvanstee/mldl-101/blob/master/lab5-powerai-lc/LCDataDictionary.csv)**Important**In this notebook, we will play with the lending club data, conduct a set of exploratory analysis and try to apply various machine learning techniques to predict borrower’s default. We took a small sample of loans made in 2016 (130K) to help speed up the processing time for the labNote : to remove a lot of the busy verbose code, we are making using of a utility python file called lc_utils.py.  For implemenation details you can refer here [python code](https://github.com/dustinvanstee/mldl-101/blob/master/lab5-powerai-lc/lc_utils.py)  Quick word on the data science methodHere we will use these simple high level steps to work through a typical data science problem.  This workflow is meant to be a high level guide, but in practice this is a highly iterative approach ... Goals* Perform some initial analysis of the data for **Business Understanding*** **Prepare the Data** for our visualization and modeling* **Visualize** the data* Model using **Dimension Reduction** and **Classification** techniques* **Evaluate** the approach  Business/Data Understanding and Preparation  Import Libraries<jupyter_code># Code functions that are needed to run this lab
import pandas as pd
import warnings
warnings.filterwarnings('ignore')
import time
from datetime import datetime
import math

import pandas as pd
#pd.set_option('display.height', 1000)
pd.set_option('display.max_rows', 500)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)

import glob

# custom library for some helper functions 
from sklearn.base import BaseEstimator, TransformerMixin
import matplotlib
import matplotlib.pyplot as plt
import numpy as np

import myenv as myenv
from lc_utils_2020 import *

%load_ext autoreload
%autoreload 2
from lc_utils_2020 import *<jupyter_output><empty_output><jupyter_text>Load the DataHere we load data that was previously downloaded from lendingclub.com.  For speed of this lab, we are restricting the number of loans ~ 130K**Key functions*** pd.read_csv* pd.concat<jupyter_code>loan_df = load_sample_data('acc')
loan_df_orig = loan_df.copy()
loan_df.head(30)<jupyter_output>**load_sample_data** : Setting data location to /gpfs/home/s4s004/vanstee/2019-06-lendingclub-git/rawdata/
**load_sample_data** : Found 1 files.  CSV files = ['/gpfs/home/s4s004/vanstee/2019-06-lendingclub-git/rawdata/LoanStats_securev1_2016Q1.csv.gz']
**load_sample_data** : Loading /gpfs/home/s4s004/vanstee/2019-06-lendingclub-git/rawdata/LoanStats_securev1_2016Q1.csv.gz<jupyter_text>Descriptive Statistics (1D)Lets look at some 1D and 2D descriptive statistics for this datasetIn this dataset, we have all types of data.  Numerical, Categorical, Ranked data.  This small module will take you through what is typical done to quickly understand the data**Key Functions*** df[column].value_counts()* df.describe()<jupyter_code># This function provide the number of rows/cols
# Information on the types of data
# and a report of descriptive statistics

# quick_overview_1d(loan_df)
categorical_overview,numerical_overview = quick_overview_1d_v2(loan_df)
print("Categorical report")
display(categorical_overview)
print("Numerical report")
display(numerical_overview)<jupyter_output>**quick_overview_1d_v2** : There are 133889 observations in the dataset.
**quick_overview_1d_v2** : There are 151 variables in the dataset.
**quick_overview_1d_v2** : 


**quick_overview_1d_v2** : 
****************** Histogram of data types  *****************************

**quick_overview_1d_v2** : use df.dtypes ...
float64    112
object      39
Name: dtype, dtype: int64
**quick_overview_1d_v2** : 


**quick_overview_1d_v2** : 
****************** Generating Cardinality Report (all types) *****************************

**quick_overview_1d_v2** : 
****************** Generating NaNs Report *****************************

**quick_overview_1d_v2** : 
******************Generating Descriptive Statistics (numerical columns only) *****************************

Categorical report<jupyter_text>Here we can get a quick assessment of the statistics for each column.  **Quick Question** can you answer what was the average income for the 133K loan applicants ?  Descriptive Statistics (2D)Since we have over 100 numerical variables, creating a 2D correlation plot may be time consuming and difficult to interpret.  Lets look at correlations on a smaller scale for now....<jupyter_code># Grab only a subset of columns
cols = ["loan_amnt","annual_inc","dti","fico_range_high","open_acc",'funded_amnt', 'total_acc']
quick_overview_2d(loan_df, cols)<jupyter_output>**quick_overview_2d** : There are 133889 observations in the dataset.
**quick_overview_2d** : There are 151 variables in the dataset.<jupyter_text>**Quick Question** : Can you find a negatively correlated variable to annual_inc in the chart above?  Can you think of a reason for this result ?  Create Loan Default column.  This is the column we will predict laterThe **loan_status** column contains the information of whether or not the loan is in default. This column has more than just a 'default or paid' status.  Since our goal is to build a simple default classifier , we need to make a new column based off the **loan_status** column.Here we will look at all the categorical values in **loan_status**, and create a new column called **default** based off that one.<jupyter_code># function to create loan status .... 
loan_df = create_loan_default(loan_df)
loan_df.head(3) # scroll to the right, and see the new 'default' column<jupyter_output>**create_loan_default** : Dropping other values that are highly correlated with loan_status
**create_loan_default** : Dropping loan_status,total_rec_prncp,total_pymnt,total_pymnt_inv
**create_loan_default** : Unique values in default
0    117220
1     16669
Name: default, dtype: int64<jupyter_text>Data Preparation - Handle Null Values aka NaNs ...One part of the data science process thats especially time consuming is working with unclean data.  This lending club data set is a great example of that.  If you look at the dataframe shown above, you will see a number of columns with the indicator **NaN** .  This means 'not a number' and needs to be dealt with prior to any machine learning steps.  You have many options here.  Some options are listed below...* Fill with a value -> impute mean/median/min/max/other* drop rows with NaNs* drop columns with large number of NaNs * use data in other columns to deriveAll these methods are possible, but its up to the data scientist / domain expert to figure out the best approach.  There is definitely some grey area involved in whats the best approach.First, lets understand which columns have NaNs...<jupyter_code># For every column, count the number of NaNs .... 
# code hint : uses df.isna().sum()

columns_with_nans(loan_df)<jupyter_output><empty_output><jupyter_text>As you can see, we have some work to do to clean up the NaN values.  Beyond NaN values, we also have to transform columns if they aren't formatted correctly, or maybe we want to transform a column based on custom requirements.  ```Example : column=employee_length , values=[1,2,3,4,5,6,7,8,9,10+] formatted as a string          transform into           column=employee_length, [0_3yrs,4_6yrs,gt_6yrs] (categorical:strings)```          Luckily, we took care to process and clean this data below using a few functions.  In practice, **this is where data scientists spend a large portion of their time** as this requires detailed domain knowledge to clean the data.  We have made a fair number of assumptions about how to process the data which we won't go into due to time contraints for the lab.<jupyter_code># The following cleaning of the data makes use of the steps shown below.....

# loan_df1 = drop_sparse_numeric_columns(loan_df, threshold=0.03)
loan_df1 = drop_sparse_columns(loan_df,pct_missing_threshold=0.6)
loan_df2 = impute_columns(loan_df1)
loan_df3 = handle_employee_length(loan_df2)
loan_df4 = handle_revol_util(loan_df3)
loan_df = loan_df4
columns_with_nans(loan_df4)

# Final Sanity check ....
# If we did our job right, there should not be any NaN's left.  
# Use this convenience function to check

# code hint df.isna().sum()

columns_with_nans(loan_df)<jupyter_output><empty_output><jupyter_text>Data Preparation - Handle Time ObjectsSometimes for columns that contain date information, you may want to break them down into individual columns like month, day, day of week etc.  For our use case, we will create a new column called `time_history` that will indicate how long an applicant has been a borrower.  This is an example of **feature engineering**.  Essentially, using business logic to create a new column (feature) that may have predictive value.<jupyter_code>#loan_df[loan_df["earliest_cr_line"]!="unknown"]
loan_df = create_time_features(loan_df)
loan_df.head(3)<jupyter_output>**create_time_features** : Creating new column called time_history : Calculated feature showing how long applicant has been a borrower..<jupyter_text>Convert Categorical Data to One hot encode If you look above at the data frame, we are almost ready to start building models.  However, there is one important step to complete.  Notice we have some columns that are still built out of string data ```example column=home_ownership values=[RENT, MORTGAGE, OWN]```Machine learning algorithms only process numerical data, so we need to transform these **categorical columns** into **indicator columns**From the example above, the transform would yield 3 new columns```example column=RENT values=[0,1]        column=MORTGAGE values=[0,1]        column=OWN values=[0,1]```Conveniently pandas has a nice function called **get_dummies** that we will use for this purpose<jupyter_code># Transform categorical data into binary indicator columns
# code hint, uses pd.get_dummies

loan_df = one_hot_encode_keep_cols(loan_df)
loan_df.head() # once complete, see how many new columns you have!<jupyter_output>**one_hot_encode_keep_cols** : Dropping these columns since they are greater than cardinality limit of 50
**one_hot_encode_keep_cols** : ['id', 'int_rate', 'emp_title', 'url', 'zip_code', 'addr_state']
**one_hot_encode_keep_cols** : Keeping these cols
**one_hot_encode_keep_cols** : ['term', 'grade', 'sub_grade', 'home_ownership', 'verification_status', 'pymnt_plan', 'purpose', 'title', 'initial_list_status', 'last_pymnt_d', 'next_pymnt_d', 'last_credit_pull_d', 'application_type', 'hardship_flag', 'disbursement_method', 'debt_settlement_flag']<jupyter_text>Final Result after data preparation ....Ok, so you made it here, lets take a look at the final results of your data preparation work.  It may be helpful to  **qualitatively compare** your original data frame to this one and see how different they look..  Execute the cells below to get a sense of what the tranformations accomplished.<jupyter_code>loan_df_orig.head(3)
loan_df.head(3)<jupyter_output><empty_output><jupyter_text>Data VisualizationAs you saw, when you 'describe' a data frame, you get a table statistics showing you the mean,min,max and other statistics about each column.  This is good, but sometimes its also good to look at the histograms of the data as well.  Lets Visualize some of the distributions from our dataset<jupyter_code># Here we plot distribution charts for all the numerical columns in our dataframe
plot_histograms(loan_df_orig)<jupyter_output>skipping column id of type  object
skipping column term of type  object
skipping column int_rate of type  object
skipping column grade of type  object
skipping column sub_grade of type  object
skipping column emp_title of type  object
skipping column emp_length of type  object
skipping column home_ownership of type  object
skipping column verification_status of type  object
skipping column issue_d of type  object
skipping column loan_status of type  object
skipping column pymnt_plan of type  object
skipping column url of type  object
skipping column desc of type  object
skipping column purpose of type  object
skipping column title of type  object
skipping column zip_code of type  object
skipping column addr_state of type  object
skipping column earliest_cr_line of type  object
skipping column revol_util of type  object
skipping column initial_list_status of type  object
skipping column last_pymnt_d of type  object
skipping column next_pymnt_d of type  object
skipping column last_credit[...]<jupyter_text>Data Visualization Examples The Growth of Lending Club<jupyter_code># Build a statistics data frame based on issue date
# aggregate on loan amount
vis_df = loan_df_orig.copy().sample(5000)<jupyter_output><empty_output><jupyter_text>Outcome Variable: Loan StatusOn the left is the breakdown of all loan status classifications.  On the right is our simple default classification based on our data prep<jupyter_code># Example of a groupby and aggregate 
df=vis_df[['loan_status', 'loan_amnt', 'funded_amnt']].groupby(['loan_status']).agg(['sum', 'count'])
df.columns = ['_'.join(col).strip() for col in df.columns.values]
df=df.reset_index()
df
import seaborn as sns
plt.figure(figsize=(20,5))
sns.set(style="whitegrid")

#tips = sns.load_dataset("tips")
#ax = sns.barplot(x="loan_status", y="loan_amnt_count", data=df)
#ax = sns.barplot(x="loan_status", y="loan_amnt", data=vis_df)
ax = sns.countplot(x="loan_status", hue="home_ownership", data=vis_df)<jupyter_output><empty_output><jupyter_text>Borrowing by State Most of the money in terms of absolute borrowing is borrowed by people from California. For average loan amount per state, Alaska ranks on top.<jupyter_code># Create an aggregated dataframe ...
df=vis_df[['addr_state', 'loan_amnt', ]].groupby(['addr_state']).agg(['mean', 'sum'])
df.columns = ['_'.join(col).strip() for col in df.columns.values]
df=df.reset_index()
df

#Plotly Choropleth Example 
import plotly.graph_objects as go

fig = go.Figure(data=go.Choropleth(
    locations=df['addr_state'], # Spatial coordinates
    z = df['loan_amnt_mean'].astype(float), # Data to be color-coded
    locationmode = 'USA-states', # set of locations match entries in `locations`
    colorscale = 'Reds',
    colorbar_title = "USD",
))

fig.update_layout(
    title_text = '2016 Lending Club Avg Loans by State',
    geo_scope='usa', # limite map scope to USA
)

fig.show()<jupyter_output><empty_output><jupyter_text>Loan Purpose WordcloudLets try to get a sense of why people are borrowing ...<jupyter_code>text = " ".join(str(purpose).lower() for purpose in loan_df_orig['title'])

from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
wordcloud = WordCloud(background_color="white").generate(text)
# Display the generated image:
plt.figure(figsize=(20,5))
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")
plt.show()<jupyter_output><empty_output><jupyter_text>As you can see, this could go on forever, but hopefully you get a sense of the power of data visualization  Modelling Phase   Train / Test set creationOne of the key points in any machine learning workflow is the **partitioning** of the data set into **train** and **test** sets.  The key idea here is that a model is built using the training data, and evaluated using the test data.  There are more nuances to how you partition data into train/test sets, but for purposes of this lab we will omit these finer points.<jupyter_code>%load_ext autoreload
%autoreload 2
from lc_utils_2020 import *
# Instantiate lendingclub_ml object that will hold our test, and contain methods used for testing.
# Implementation done like this to ease the burden on users for keeping track of train/test sets for different
# models we are going to build.

my_analysis = lendingclub_ml(loan_df)
# Create a train / test split of your data set.  Paramter is test set size percentage 
# Returns data in the form of dataframes

my_analysis.create_train_test(test_size=0.33)<jupyter_output>**create_train_test** : Dropping 2 timestamp columns issue_d and earliest_cr_line
**create_train_test** : Training set size: (89704, 225)
**create_train_test** : Testing set size: (44183, 225)
**create_train_test** : Train set loan_default:loan_paid ratio : 11277/78427
**create_train_test** : Test  set loan_default:loan_paid ratio : 5392/38791<jupyter_text>Dimension ReductionFor this modeling exercise we will perform a couple of tasks, **dimension reduction** and **classification** as shown in the following diagram.**Dimension Reduction** is useful in scenarios when you have a large number of columns and you would like to reduce that down to a compressed representation .  In this lab we will try 2 methods of dimension reduction.  It will be your choice to decide which method you want to use for the classification part of the lab ! (you could even decice to bypass this if you want ...)  Dimension Reduction - PCAPrincipal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables.A simple way to think about PCA is that it helps compress the data in a lossy representation of the original dataset.This will also be used to help us visualize the data as you will see below<jupyter_code># Dimension Reduction using PCA
# PCA model saved in data structure ..
my_analysis.build_pca_model(n_components=30)<jupyter_output>**build_pca_model** : Explained Variance : [0.05273159 0.03915729 0.03200515 0.02697703 0.02205358 0.01884257
 0.01650913 0.01549409 0.01424003 0.0126312  0.01219791 0.01116194
 0.01107317 0.01067109 0.00986656 0.00976973 0.00951809 0.00938076
 0.00929004 0.00911505 0.00905296 0.00899589 0.00892721 0.00878349
 0.00876695 0.00870437 0.00866278 0.00846316 0.00834286 0.00792983]<jupyter_text>In the chart above, you can see that we get ok results from PCA.  Using the first 20 principal components, we can account for ~50% of the variance described in the dataset.  **Feel free to change the number of principal components above to see if adding more helps with explained variance.**  Now update our test dataframe with new columns that are predicted by our PCA.  Here we will now take the models that we built and pass our test data set through the models.   By doing this, we will have reduced the number features in our data set by a significant amount (~177 colums => ~5-20 columns!)  .  In this step we will add new columns to our test/train data frames for  our PCA model .  Don't worry about the details of this step, its just required for some followon visualization, and training steps ahead.<jupyter_code>my_analysis.add_pca_columns_to_df()
#[x for x in my_analysis.train_df.columns if 'PC' in str(x)]
#for c in my_analysis.train_df.columns: 
#    print(c)
#display(my_analysis.X_train_scaled.head(10))
#my_analysis.train_df.head(10)<jupyter_output><empty_output><jupyter_text>Cool Visualizations using our dimension reduction columnsNext we will plot a few scatterplot grids based on our pricipal component and autoencoder representations of the dataWe will color each data point using this key```Green -> Fully paid or current loanRed   -> Loan in default```<jupyter_code># This will take a minute or so ...
my_analysis.visualize_dimred_results(mode='pca')
# Example : Filter based on Principal Components ..
a = my_analysis.test_df
f_df = a[(a['PC0'] < 1) & (a['PC1'] <0)]
display(f_df.describe())
display(loan_df.describe())<jupyter_output><empty_output><jupyter_text>If you can discern a pattern between the red / green dots, its likely we can use a classifier to automatically seperate them! We'll see that in a few more sections  Lending Club Default Prediction using Random ForestHere we will build a classifier to predict if loan will fail or not.  We will us a ** Random Forest ** tree algorithm  .  You will have 2 options for data sources, * the raw data* PCA dimension reduction featuresTo evaluate our model, we will use a simple contingency table (showing true/false positve/negative).  However, this is a fairly simplistic method.  Better method that data scientists use are F1 score, and PR/ROC curves but thats beyond the scope of this lab.Step 1 here is to set our baseline result.  In this example, we are dealing with a **skewed** dataset.  This means, on average, most people will not default, and they pay their loan off.  If you built a classifier that just predicted no default, you would be right most of the time.  Lets see the stats from our dataset below....<jupyter_code># Set our baseline
my_analysis.train_df['default'].describe()<jupyter_output><empty_output><jupyter_text>As you can see, **only ~12.8% of the applicants default**.  Any classifier we build must be better than this, or we aren't doing a very good job ;)<jupyter_code># modes
# pca           : principal components only
# raw           : all the data non reduced
# raw_no_grades : all the data non reduced except the grade info provided by lending club

mode = 'raw' # ae , raw, raw_no_grades
x_cols =[]
if(mode == 'pca') :
            x_cols = [x for x in my_analysis.train_df.columns if 'PC' in x]
elif(mode == 'raw') :
            x_cols = [x for x in my_analysis.train_df.columns if 'PC' not in x]
            x_cols.remove('default')
elif(mode == 'raw_no_grades') :
            x_cols = [x for x in my_analysis.train_df.columns if 'PC' not in x]
            import re
            x_cols = [x for x in x_cols if not re.match('^[ABCDEFG]',x)]
            x_cols.remove('default')

#print(x_cols)<jupyter_output>['index', 'loan_amnt', 'funded_amnt', 'funded_amnt_inv', 'installment', 'annual_inc', 'dti', 'delinq_2yrs', 'fico_range_low', 'fico_range_high', 'inq_last_6mths', 'mths_since_last_delinq', 'open_acc', 'pub_rec', 'revol_bal', 'total_acc', 'out_prncp', 'out_prncp_inv', 'total_rec_int', 'total_rec_late_fee', 'recoveries', 'collection_recovery_fee', 'last_pymnt_amnt', 'last_fico_range_high', 'last_fico_range_low', 'collections_12_mths_ex_med', 'policy_code', 'acc_now_delinq', 'tot_coll_amt', 'tot_cur_bal', 'open_acc_6m', 'open_act_il', 'open_il_12m', 'open_il_24m', 'mths_since_rcnt_il', 'total_bal_il', 'il_util', 'open_rv_12m', 'open_rv_24m', 'max_bal_bc', 'all_util', 'total_rev_hi_lim', 'inq_fi', 'total_cu_tl', 'inq_last_12m', 'acc_open_past_24mths', 'avg_cur_bal', 'bc_open_to_buy', 'bc_util', 'chargeoff_within_12_mths', 'delinq_amnt', 'mo_sin_old_il_acct', 'mo_sin_old_rev_tl_op', 'mo_sin_rcnt_rev_tl_op', 'mo_sin_rcnt_tl', 'mort_acc', 'mths_since_recent_bc', 'mths_since_recent_inq', 'num_[...]<jupyter_text>Random Forest Example<jupyter_code># Build a dataframe with selected columns ...
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X = my_analysis.train_df[x_cols]
y = my_analysis.Y_train
## Simple Random Forest Model
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix
clf = RandomForestClassifier(max_depth=5,n_estimators=300, random_state=0)

clf.fit(X,y)<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>Y_test_predict = np.where(clf.predict(my_analysis.test_df[x_cols]) > 0.5, 1, 0 )

cnf_matrix =confusion_matrix(my_analysis.Y_test, Y_test_predict)
class_names =  ['Default','Paid']
plot_confusion_matrix(cnf_matrix, class_names)<jupyter_output><empty_output><jupyter_text>Explainability : RF Feature Importance<jupyter_code># Random Forest Explainability
#print(len(clf.feature_importances_))
#print(len(X.columns))
rf_feature_importance = dict(zip(X.columns,clf.feature_importances_))


sort_rf_feature_importance = sorted(rf_feature_importance.items(), key=lambda x: x[1], reverse=True)

for i in sort_rf_feature_importance:
    print(i[0], i[1])<jupyter_output>last_fico_range_low 0.1603399211710119
last_fico_range_high 0.1437350367254732
last_pymnt_amnt 0.08288560298999591
collection_recovery_fee 0.07642528001790322
recoveries 0.07445315257563169
out_prncp 0.026135654433055922
out_prncp_inv 0.019293547525115175
Dec-2017 0.014802720999050768
N 0.010929536871565388
total_rec_late_fee 0.005433524758877392
Y 0.004172425677844359
A 0.003462821758422586
Feb-2017 0.0032224099655267227
Nov-2017 0.0030975816699359204
 60 months 0.0021099510827911703
n 0.0019065298416625898
fico_range_low 0.0018081116618675448
E 0.001773954561134124
y 0.0017484176680806034
fico_range_high 0.0014896647763741945
F 0.001445109650358621
dti 0.0009857639969921245
 36 months 0.0009732758785704406
B 0.0008689732988178535
Jun-2017 0.0008583607006260929
total_rec_int 0.0008098484162356504
avg_cur_bal 0.0007455971204206448
mort_acc 0.0007446871990640158
D 0.0007334860931984668
bc_open_to_buy 0.0006692012793428436
funded_amnt_inv 0.0004980867610295717
A1 0.0004004139895255794
to[...]
{'fraction_non_alphanumeric': 0.08559972765442372, 'fraction_numerical': 0.04690396035858834, 'mean_word_length': 5.619429143715573, 'pattern_counts': {'":': 0, '<': 88, '<?xml version=': 0, '>': 91, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '134421', 'license': 'Apache-2.0', 'n_tokens_mistral': 8926, 'n_tokens_neox': 7867, 'n_words': 3016, 'path': 'UTSA/Lab1-LendingClub-AiImmersion-Solution.ipynb', 'repo_name': 'dustinvanstee/aicoc-ai-immersion', 'size': '26437'}