<jupyter_start><jupyter_text>AbstractionIt's important for finance companies to know risk of giving credit to their clients. Credit scores are essential for companies to decide on whether giving credit to someone or not. In this project, we experiment on features of clients to decide on which ones are more important. SVM, logistic regression and desicion trees are utilized and compared. An interface that calculates credit score for given features is provided.  Introduction&nbsp;&nbsp; Banks play a crucial role in market economies. They decide who can get finance and on what terms and can make or break investment decisions. For markets and society to function, individuals and companies need access to credit. &nbsp;&nbsp; Credit scoring algorithms, which make a guess at the probability of default, are the method banks use to determine whether or not a loan should be granted. This competition requires participants to improve on the state of the art in credit scoring, by predicting the probability that somebody will experience financial distress in the next two years. Data Dictionary| Variable Name                        | Description                                                                                                                                              | Type       ||--------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|------------|| SeriousDlqin2yrs                     | Person experienced 90 days past due delinquency or worse                                                                                                 | Y/N        || RevolvingUtilizationOfUnsecuredLines | Total balance on credit cards and personal lines of credit except real estate and no installment debt like car loans divided by the sum of credit limits | percentage || age                                  | Age of borrower in years                                                                                                                                 | integer    || NumberOfTime30-59DaysPastDueNotWorse | Number of times borrower has been 30-59 days past due but no worse in the last 2 years.                                                                  | integer    || DebtRatio                            | Monthly debt payments, alimony,living costs divided by monthy gross income                                                                               | percentage || MonthlyIncome                        | Monthly income                                                                                                                                           | real       || NumberOfOpenCreditLinesAndLoans      | Number of Open loans (installment like car loan or mortgage) and Lines of credit (e.g. credit cards)                                                     | integer    || NumberOfTimes90DaysLate              | Number of times borrower has been 90 days or more past due.                                                                                              | integer    || NumberRealEstateLoansOrLines         | Number of mortgage and real estate loans including home equity lines of credit                                                                           | integer    || NumberOfTime60-89DaysPastDueNotWorse | Number of times borrower has been 60-89 days past due but no worse in the last 2 years.                                                                  | integer    || NumberOfDependents                   | Number of dependents in family excluding themselves (spouse, children etc.)                                                                              | integer    |  Related Work A two-stage hybrid credit scoring model using artificial neural networks and multivariate adaptive regression splineshttp://www.sciencedirect.com/science/article/pii/S0957417404001782Benchmarking state-of-the-art classification algorithms for credit scoringhttps://link.springer.com/article/10.1057/palgrave.jors.2601545Using neural network ensembles for bankruptcy prediction and credit scoringhttp://www.sciencedirect.com/science/article/pii/S0957417407001558A comparative assessment of ensemble learning for credit scoringhttp://www.sciencedirect.com/science/article/pii/S095741741000552XComprehensible credit scoring models using rule extraction from support vector machineshttp://www.sciencedirect.com/science/article/pii/S0377221706011878Neural network credit scoring modelshttp://www.sciencedirect.com/science/article/pii/S0305054899001495Statistical Classification Methods in Consumer Credit Scoring: a Reviewhttp://onlinelibrary.wiley.com/doi/10.1111/j.1467-985X.1997.00078.x/full<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.feature_extraction import DictVectorizer
from pandas import DataFrame
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Train Data Loaded<jupyter_code>###
### Load Data Set
###
data = pd.read_csv('cs-training.csv').drop('Unnamed: 0', axis = 1)
data
# '-' in column name creates a problem when accessing
Cols = []
for i in range(len(data.columns)):
    Cols.append(data.columns[i].replace('-', ''))
data.columns = Cols
#
data = data.apply(lambda x: x.fillna(np.nanmedian(x),axis=0))
# Drop rows with missing column data
#data = data.dropna()

###
### Convert Data Into List Of Dict Records
###
data = data.to_dict(orient='records')

###
### Seperate Target and Outcome Features
###

vec = DictVectorizer()

df_data = vec.fit_transform(data).toarray()
feature_names = vec.get_feature_names()
df_data = DataFrame(
    df_data,
    columns=feature_names)<jupyter_output><empty_output><jupyter_text>Missing Values There are some values in the dataset that are missing or having really awkward magnitudes. Therefore, firstly, we should take care of them.<jupyter_code>#col_mean = np.nanmean(X,axis=0)

#Find indicies that you need to replace
#inds = np.where(np.isnan(X))

#Place column means in the indices. Align the arrays using take
#X[inds]=np.take(col_mean,inds[1])<jupyter_output><empty_output><jupyter_text>VisualizationIn this part we will show you the visualization of the data with respect to some aspects of them.<jupyter_code>def plot_freq(l):
    ncount = l

    ax2=ax.twinx()

    ax2.yaxis.tick_left()
    ax.yaxis.tick_right()

    ax.yaxis.set_label_position('right')
    ax2.yaxis.set_label_position('left')

    ax2.set_ylabel('Frequency [%]')

    for p in ax.patches:
        x=p.get_bbox().get_points()[:,0]
        y=p.get_bbox().get_points()[1,1]
        ax.annotate('{:.1f}%'.format(100.*y/ncount), (x.mean(), y), 
                ha='center', va='bottom')

    ax2.set_ylim(0,100)
    ax2.grid(None)    
ax = sns.countplot(x = df_data.SeriousDlqin2yrs , palette="Set2")
sns.set(font_scale=1.5)
ax.set_ylim(top = len(data))
ax.set_xlabel('Class')
ax.set_ylabel('Sample Size')
plt.title('Class Frequencies')

plot_freq(l = len(df_data.SeriousDlqin2yrs))

plt.show()<jupyter_output><empty_output><jupyter_text>Linear Discriminant Analysis<jupyter_code>from sklearn import  linear_model
#clf = linear_model.LogisticRegression(C=1e5)
#clf.fit(X_Train,Y_Train)
#clf.predict_proba(X[0])

#clf.coef_
#ind = np.where(Y_test == 1)
#ss = clf.predict_log_proba(X_test[ind]) 
#t = ss.argmax(axis = 1)
#a = np.intersect1d(ind,np.where(t == 1))
#100*len(a)/len(t)
#print(clf.predict_proba(X_test[37441].reshape(1,-1)))
#y_test[37441]<jupyter_output><empty_output><jupyter_text>Outlier detection We are going to examine some features to eliminate outliers. For instance, in *RevolvingUtilizationOfUnsecuredLines*, there are some values that are close to 50000. We should eliminate such values to prevent our error rate being very high.  RevolvingUtilizationOfUnsecuredLines Variance<jupyter_code>#plt.plot(data2.RevolvingUtilizationOfUnsecuredLines)
plt.figure(figsize=(20,15))
ax = plt.subplot(211)
#ax.set_ylim(0,20)
plt.plot(df_data.RevolvingUtilizationOfUnsecuredLines, 'bo',df_data.RevolvingUtilizationOfUnsecuredLines, 'k')
print('Median: %.7f \nMean: %.7f' %(np.median(df_data.RevolvingUtilizationOfUnsecuredLines),np.mean(df_data.RevolvingUtilizationOfUnsecuredLines)))
ruoelLt2=len(df_data[df_data.RevolvingUtilizationOfUnsecuredLines < 2])
ruoelACt=len(df_data.RevolvingUtilizationOfUnsecuredLines)
print('Values less than 2 : %d in %d. Ratio: %.5f%%' %(ruoelLt2,ruoelACt,100*ruoelLt2/ruoelACt))
#sns.kdeplot(data2.RevolvingUtilizationOfUnsecuredLines, shade=True, color="r")
#data2.age.plot.box()
#data2.RevolvingUtilizationOfUnsecuredLines.plot.box()<jupyter_output>Median: 0.1541807 
Mean: 6.0484381
Values less than 2 : 149629 in 150000. Ratio: 99.75267%<jupyter_text>Therefore we need to clean outliers by considering the ratio we obtained above.<jupyter_code>ind = np.where(df_data.RevolvingUtilizationOfUnsecuredLines>2)
df_data.RevolvingUtilizationOfUnsecuredLines[ind[0]] = 2.<jupyter_output><empty_output><jupyter_text>Age Variance<jupyter_code>from collections import Counter
plt.figure(1)
df_data.age.plot.box()
Counter(df_data.age)

plt.figure(2)
sns.set_color_codes()
sns.distplot(df_data.age, color="y")
plt.show()
ind = np.where(df_data.age<21)
df_data.age[ind[0]] = 21.
ind = np.where(df_data.age>94)
df_data.age[ind[0]] = 94.<jupyter_output><empty_output><jupyter_text>NumberOfTime30-59DaysPastDueNotWorse Varience<jupyter_code>Counter(df_data.NumberOfTime3059DaysPastDueNotWorse)
### Set outlier values to median , that is 0.
ind = np.where(df_data.NumberOfTime3059DaysPastDueNotWorse>95)
df_data.NumberOfTime3059DaysPastDueNotWorse[ind[0]] = 0.
def mad_based_outlier(points, thresh=3.5):
    if len(points.shape) == 1:
        points = points[:,None]
    median = np.median(points, axis=0)
    diff = np.sum((points - median)**2, axis=-1)
    diff = np.sqrt(diff)
    med_abs_deviation = np.median(diff)

    modified_z_score = 0.6745 * diff / med_abs_deviation

    return modified_z_score > thresh<jupyter_output><empty_output><jupyter_text>DebtRatio Variance<jupyter_code>plt.figure(figsize=(20,15))
ax = plt.subplot(211)
#ax.set_ylim(0,20)
plt.plot(df_data.DebtRatio, 'bo',df_data.DebtRatio, 'k')
print('Median: %.7f \nMean: %.7f' %(np.median(df_data.DebtRatio),np.mean(df_data.DebtRatio)))
#ruoelLt2=len(df_data[df_data.RevolvingUtilizationOfUnsecuredLines < 2])
#ruoelACt=len(df_data.RevolvingUtilizationOfUnsecuredLines)
#print('Values less than 2 : %d in %d. Ratio: %.5f%%' %(ruoelLt2,ruoelACt,100*ruoelLt2/ruoelACt))
ax = sns.countplot(mad_based_outlier(df_data.DebtRatio))
plot_freq(l = len(df_data.DebtRatio))
minUpperBound = min([val for (val, out) in zip(df_data.DebtRatio, mad_based_outlier(df_data.DebtRatio)) if out == True])
### Set outlier values to upperbound, that is minUpperBound.
ind = np.where(df_data.DebtRatio>minUpperBound)
df_data.DebtRatio[ind[0]] = minUpperBound
plt.figure(figsize=(20,15))
ax = plt.subplot(211)
plt.plot(df_data.DebtRatio, 'o')

df_data.DebtRatio.describe()<jupyter_output><empty_output><jupyter_text>MonthlyIncome Variance<jupyter_code>plt.figure(figsize=(20,15))
ax = plt.subplot(211)
#ax.set_ylim(0,20)
plt.plot(df_data.MonthlyIncome, 'bo',df_data.MonthlyIncome, 'k')
print('Median: %.7f \nMean: %.7f' %(np.median(df_data.MonthlyIncome),np.mean(df_data.MonthlyIncome)))
maxUpperBound = min([val for (val, out) in zip(df_data.MonthlyIncome, mad_based_outlier(df_data.MonthlyIncome)) if out == True])
ind = np.where(df_data.MonthlyIncome>maxUpperBound)
df_data.MonthlyIncome[ind[0]] = maxUpperBound
ind = np.where(df_data.MonthlyIncome<1500)
df_data.MonthlyIncome[ind[0]] = 1500
df_data.MonthlyIncome.describe()<jupyter_output><empty_output><jupyter_text>NumberOfTimes90DaysLate Variance<jupyter_code>Counter(df_data.NumberOfTimes90DaysLate)
### Set outlier values to median , that is 0.
ind = np.where(df_data.NumberOfTimes90DaysLate>95)
df_data.NumberOfTimes90DaysLate[ind[0]] = 0<jupyter_output><empty_output><jupyter_text>NumberRealEstateLoansOrLines Variance<jupyter_code>Counter(df_data.NumberRealEstateLoansOrLines)
### Set outlier values to 16.
ind = np.where(df_data.NumberRealEstateLoansOrLines>16)
df_data.NumberRealEstateLoansOrLines[ind[0]] = 16<jupyter_output><empty_output><jupyter_text>NumberOfTime60-89DaysPastDueNotWorse Variance<jupyter_code>Counter(df_data.NumberOfTime6089DaysPastDueNotWorse)
### Set outlier values to 0.
ind = np.where(df_data.NumberOfTime6089DaysPastDueNotWorse>11)
df_data.NumberOfTime6089DaysPastDueNotWorse[ind[0]] = 0<jupyter_output><empty_output><jupyter_text>NumberOfDependents Variance<jupyter_code>Counter(df_data.NumberOfDependents)
ind = np.where(df_data.NumberOfDependents >10)
df_data.NumberOfDependents[ind[0]] = 10<jupyter_output><empty_output><jupyter_text>Train-Test Split<jupyter_code>###
### Generate Training and Testing Set 
###
    
outcome_feature = df_data['SeriousDlqin2yrs']
target_features = df_data.drop('SeriousDlqin2yrs', axis=1)

from sklearn.model_selection import train_test_split
"""
    X_1: independent (target) variables for first data set
    Y_1: dependent (outcome) variable for first data set
    X_2: independent (target) variables for the second data set
    Y_2: dependent (outcome) variable for the second data set
"""
X_train, X_test, Y_train, Y_test = train_test_split(target_features, outcome_feature, test_size=0.5, random_state=0)
from sklearn.metrics import confusion_matrix<jupyter_output><empty_output><jupyter_text>Desicion Tree<jupyter_code>from sklearn import tree
clf2 = tree.DecisionTreeClassifier(class_weight='balanced',min_impurity_split=1e-05,max_depth=6)
clf2.fit(X_train,Y_train)
Y_pred = clf2.predict(X_test)
pd.crosstab(Y_test, Y_pred, rownames=['True'], colnames=['Predicted'], margins=True)<jupyter_output><empty_output><jupyter_text>GaussianNB<jupyter_code>from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB(priors=[0.07,0.93])
gnb.fit(X_train,Y_train)
Y_pred = gnb.predict(X_test)
pd.crosstab(Y_test, Y_pred, rownames=['True'], colnames=['Predicted'], margins=True)<jupyter_output><empty_output><jupyter_text>SVM<jupyter_code>from sklearn import svm
clf3 = svm.SVC(C=2,cache_size=7000,tol=1,class_weight={0:.1,1:.9} )
clf3.fit(X_train,Y_train)
Y_pred = clf3.predict(X_test)
pd.crosstab(Y_test, Y_pred, rownames=['True'], colnames=['Predicted'], margins=True)<jupyter_output><empty_output><jupyter_text>MLP<jupyter_code>from sklearn.neural_network import MLPClassifier
clf4 = MLPClassifier(activation="identity", tol= 0.1)
clf4.fit(X_train, Y_train) 
from sklearn.metrics import confusion_matrix
Y_pred = clf4.predict(X_test)
pd.crosstab(Y_test, Y_pred, rownames=['True'], colnames=['Predicted'], margins=True)<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>from sklearn import  linear_model
clf = linear_model.LogisticRegression(C=1e5,class_weight= {0:.1, 1:.9} )
clf.fit(X_train,Y_train)
Y_pred = clf.predict(X_test)
netMat = (Y_pred == Y_test)
clf.coef_
ind = np.where(Y_test == 1)
Counter(Y_pred[ind])
pd.crosstab(Y_test, Y_pred, rownames=['True'], colnames=['Predicted'], margins=True)<jupyter_output><empty_output><jupyter_text>A Small Application<jupyter_code>%matplotlib notebook
import pandas as pd
import matplotlib.pyplot as plt
from ipywidgets import *
from IPython.display import display
import ipywidgets as widgets
plt.style.use('ggplot')


form_item_layout = widgets.Layout(
    display='flex',
    flex_flow='row',
    justify_content='space-between'
)


# displaying the text widget
text = widgets.Text(
    placeholder='Enes',
    disabled=False
)
#display(text)
# add button that updates the graph based on the checkboxes
button = widgets.Button(description="Check credibility")
#display(button)
resultLabel = widgets.Label(
    value="",
    visible = False,
    disabled = True
)
#display(resultLabel)
revolve = widgets.FloatSlider(
    value=df_data.RevolvingUtilizationOfUnsecuredLines.median(),
    min=df_data.RevolvingUtilizationOfUnsecuredLines.min(),
    max=df_data.RevolvingUtilizationOfUnsecuredLines.max(),
    step=0.01,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    readout_format='.4f',
    slider_color='black'
)
age = widgets.IntSlider(
    value=df_data.age.median(),
    min=df_data.age.min(),
    max=df_data.age.max(),
    step=1,
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)
income = widgets.FloatSlider(
    value=df_data.MonthlyIncome.median(),
    min=df_data.MonthlyIncome.min(),
    max=df_data.MonthlyIncome.max(),
    step=0.5,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    readout_format='.1f',
    slider_color='black'
)
debtRatio = widgets.FloatSlider(
    value=df_data.DebtRatio.median(),
    min=df_data.DebtRatio.min(),
    max=df_data.DebtRatio.max(),
    step=0.01,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    readout_format='.4f',
    slider_color='black'
)
NumberOfTime3059DaysPastDueNotWorse = widgets.IntSlider(
    value=df_data.NumberOfTime3059DaysPastDueNotWorse.median(),
    min=df_data.NumberOfTime3059DaysPastDueNotWorse.min(),
    max=df_data.NumberOfTime3059DaysPastDueNotWorse.max(),
    step=1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)
NumberOfTimes90DaysLate =  widgets.IntSlider(
    value=df_data.NumberOfTimes90DaysLate.median(),
    min=df_data.NumberOfTimes90DaysLate.min(),
    max=df_data.NumberOfTimes90DaysLate.max(),
    step=1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)
NumberRealEstateLoansOrLines =  widgets.IntSlider(
    value=df_data.NumberRealEstateLoansOrLines.median(),
    min=df_data.NumberRealEstateLoansOrLines.min(),
    max=df_data.NumberRealEstateLoansOrLines.max(),
    step=1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)
NumberOfTime6089DaysPastDueNotWorse =  widgets.IntSlider(
    value=df_data.NumberOfTime6089DaysPastDueNotWorse.median(),
    min=df_data.NumberOfTime6089DaysPastDueNotWorse.min(),
    max=df_data.NumberOfTime6089DaysPastDueNotWorse.max(),
    step=1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)
NumberOfOpenCreditLinesAndLoans = widgets.IntSlider(
    value=df_data.NumberOfOpenCreditLinesAndLoans.median(),
    min=df_data.NumberOfOpenCreditLinesAndLoans.min(),
    max=df_data.NumberOfOpenCreditLinesAndLoans.max(),
    step=1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)

NumberOfDependents =  widgets.IntSlider(
    value=df_data.NumberOfDependents.median(),
    min=df_data.NumberOfDependents.min(),
    max=df_data.NumberOfDependents.max(),
    step=1,
    disabled=False,
    continuous_update=True,
    orientation='horizontal',
    readout=True,
    slider_color='black'
)
algos = widgets.Dropdown(options=['Decision Tree', 'SVM', 'Logistic Regression','GaussianNB','MLP'])
#display(revolve)

form_items = [
   Box([Label(value='Please, enter the name:'), text], layout=form_item_layout),
   Box([Label(value='Revolving Util. of Unsecured Lines:'), revolve], layout=form_item_layout),
   Box([Label(value='Age:'), age], layout=form_item_layout),
   Box([Label(value='Monthly Income:'), income], layout=form_item_layout),
   Box([Label(value='Dept Ratio:'), debtRatio], layout=form_item_layout),
   Box([Label(value='NumberOfOpenCreditLinesAndLoans:'), NumberOfOpenCreditLinesAndLoans], layout=form_item_layout),
   Box([Label(value='NumberOfTime3059DaysPastDueNotWorse:'), NumberOfTime3059DaysPastDueNotWorse], layout=form_item_layout),
   Box([Label(value='NumberOfTimes90DaysLate:'), NumberOfTimes90DaysLate], layout=form_item_layout),
   Box([Label(value='NumberRealEstateLoansOrLines:'), NumberRealEstateLoansOrLines], layout=form_item_layout),
   Box([Label(value='NumberOfTime6089DaysPastDueNotWorse:'), NumberOfTime6089DaysPastDueNotWorse], layout=form_item_layout),
   Box([Label(value='NumberOfDependents:'), NumberOfDependents], layout=form_item_layout),
   Box([Label(value='Algorithm:'),algos], layout=form_item_layout),
   button,
   Box([Label(value='Result:'), resultLabel], layout=form_item_layout),
]

form = Box(form_items, layout=Layout(
    display='flex',
    flex_flow='column',
    border='dashed 2px',
    align_items='stretch',
    width= '70%'
))

display(form)

# function to deal with the checkbox update button       
def on_button_clicked(b):
    name = text.value
    roul = revolve.value
    ageV = age.value
    monI = income.value
    dratio = debtRatio.value
    noOpCL = NumberOfOpenCreditLinesAndLoans.value
    noT3059 = NumberOfTime3059DaysPastDueNotWorse.value
    noT90 = NumberOfTimes90DaysLate.value
    nreL = NumberRealEstateLoansOrLines.value
    noT6089 = NumberOfTime6089DaysPastDueNotWorse.value
    noD = NumberOfDependents.value
    algo = algos.value
    
    testARR = [[dratio,monI,noD,noOpCL,noT3059,noT6089,noT90,nreL,roul,ageV]]
    yGuess  = []
    
    if   algo == 'Decision Tree':
        yGuess = clf2.predict(testARR)
    elif algo == 'SVM':
        yGuess = clf3.predict(testARR)
    elif algo == 'Logistic Regression':
        yGuess = clf.predict(testARR)
    elif algo == 'GaussianNB':
        yGuess = gnb.predict(testARR)
    else: 
        yGuess = clf4.predict(testARR)
    
    print(name,roul,ageV,monI,dratio,noOpCL,noT3059,noT90,nreL,noT6089,noD,algo,yGuess[0])
    
    if(resultLabel.visible == False and yGuess[0] == 0):
        resultLabel.value = text.value + ' can be provided with the loan.'    
    else:
        resultLabel.value = text.value + ' should not be provided with the loan.'
        
button.on_click(on_button_clicked)
plt.show()

ind = df_data.SeriousDlqin2yrs[df_data.SeriousDlqin2yrs == 1]
ind = ind[1:100]
ind2 = df_data.SeriousDlqin2yrs[df_data.SeriousDlqin2yrs == 0]
ind2 = ind2[1:100]
%matplotlib inline
plt.figure(figsize=(15,15))
ax = plt.subplot(211)
#ax.set_ylim(0,20)
plt.plot(df_data.MonthlyIncome[ind.index],df_data.age[ind.index], 'ko',df_data.MonthlyIncome[ind2.index],df_data.age[ind2.index], 'go')

from sklearn.metrics import roc_curve, auc

fpr, tpr, threshold = roc_curve(Y_test, Y_pred)
roc_auc = auc(fpr, tpr)

%matplotlib inline

plt.figure(figsize=(15,15))
plt.title('Receiver Operating Characteristic')
plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)
plt.legend(loc = 'lower right')
plt.plot([0, 1], [0, 1],'r--')
plt.xlim([0, 1])
plt.ylim([0, 1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12019713027170832, 'fraction_numerical': 0.02686554145405382, 'mean_word_length': 4.166741775574583, 'pattern_counts': {'":': 0, '<': 92, '<?xml version=': 0, '>': 98, 'https://': 1, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 45, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '604017', 'license': 'Apache-2.0', 'n_tokens_mistral': 8155, 'n_tokens_neox': 7515, 'n_words': 1428, 'path': 'Credit Risk Analysis.ipynb', 'repo_name': 'DiogoRibeiro7/credit-default-machine-learning', 'size': '22929'}