<jupyter_start><jupyter_text>Data increase with Enormous rate<jupyter_code>from IPython.display import display, Image
display(Image(filename='Data growth.PNG'))<jupyter_output><empty_output><jupyter_text>What is Data Mining? Data mining is a rapidly growing field that is concerned with developing techniques to assist managers and decision makers to makeintelligent use of these repositories. The goal is to discover meaningful new correlations, patterns and trends by sifting through large amounts of data stored in repositories, using techniques developed in pattern recognition, machine learning, artificial intelligence, statistics and mathematics<jupyter_code>from IPython.display import display, Image
display(Image(filename='Data mining process.png'))<jupyter_output><empty_output><jupyter_text>Sample data you can play with Lending Club: These files contain complete loan data for all loans issued through the 2007-2015, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information. The file containing loan data through the "present" contains complete loan data for all loans issued through the previous completed calendar quarter. Additional features include credit scores, number of finance inquiries, address including zip codes, and state, and collections among others. The file is a matrix of about 890 thousand observations and 75 variables. A data dictionary is provided in a separate file.  https://storage.googleapis.com/kaggle-data-sets/34/334209/bundle/archive.zip?GoogleAccessId=<EMAIL>&Expires=1582857554&Signature=CTb6MbE9v3ss2O9D8y0ZW%2FNSG40Og52KFjVzC3wYumBYZCtjOQC1RdEZhCDDR%2FpjSCVcQf8tFSUgqk%2FQnugwao4ClY6%2BoWSwflkslT%2BOMqVkCH8eEE%2BgsYPc8rEoYwqRQvO9Y2NezaRwr3iFYbkRpKoNsQlbnBMLPbNwXcOujuajDQLg64QslEH%2BZB%2FjZdFI3R87PQ7ORoa%2FEOEg0CDc4zt%2BOYJfzJSLCuGGNPZW%2F6yBTzlZG%2FW2FdY7P2tylnC7rmCaiXREDZ5fiHHRy8LQ34iM7irztZiWu0aBPLKd%2BouSYa3WnAgfJ4oO8TTTDrkufViLkjScr2l%2BG12s8anAUQ%3D%3D&response-content-disposition=attachment%3B+filename%3Dlending-club-loan-data.zip Movie Review Data:This page is a distribution site for movie-review data for use in sentiment-analysis experiments. Available are collections of movie-review documents labeled with respect to their overall sentiment polarity (positive or negative) or subjective rating (e.g., "two and a half stars") and sentences labeled with respect to their subjectivity status (subjective or objective) or polarity. These data sets were introduced in the following papers: http://www.cs.cornell.edu/people/pabo/movie-review-data/ What Kinds of Patterns Can Be Mined Frequent Patterns, Associations, and Correlations 1. Frequent patterns2. Association analysis Classification and Regression for Predictive Analysis 1. Classification2. Regression3. Cluster analysis4. Outlier Analysis Are All Patterns Interesting? Cosition vs correlation Most time, world can be explained by common sense Google Cloud Guidance Upload data into Cloud Storage https://cloud.google.com/storage/docs/quickstart-console?hl=zh-tw Link the Cloud storage data with Big query https://cloud.google.com/bigquery/external-data-cloud-storage Set up Jupyter notebook, and link it with Big query data warehouse https://jingsblog.com/2018/11/27/connect-your-jupyter-notebook-to-the-bigquery/<jupyter_code>import numpy as np
import scipy as sp
import pandas as pd
import matplotlib as mpl
import seaborn as sns
import matplotlib.pyplot as plt

pd.options.display.max_rows = 999
pd.set_option('display.max_columns', None)  
pd.set_option('display.expand_frame_repr', False)
pd.set_option('max_colwidth', -1)
import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/home/tianboli1987/.config/gcloud/application_default_credentials.json'
from google.cloud import bigquery
bigquery_client = bigquery.Client(project='double-airport-256012')
query1 = """select * from load.loan3 
"""
query_job = bigquery_client.query(query1)
df = query_job.to_dataframe()
df.head()
df.describe()<jupyter_output><empty_output><jupyter_text>Data preprocessing Data quality is defined in terms of accuracy, completeness, consistency, timeliness,believability, and interpretabilty. These qualities are assessed based on the intendeduse of the data. 1. Data Cleaning2. Data Integration3. Data Transformation and Data Discretization<jupyter_code>df["target"] = np.where(df["acc_now_delinq"]== 0, 0, 1)
df_1= df["uuid"].groupby(df['target']).count()
df_1
list_dfn = list(df.columns)
df1 = df.drop(['uuid', 'emp_title','title','zip_code',
               'addr_state','id', 'member_id', 'url', 
               'policy_code','issue_d', 'earliest_cr_line', 
               'last_pymnt_d', 'next_pymnt_d', 'last_credit_pull_d', 
               'sec_app_earliest_cr_line', 'hardship_start_date', 
               'hardship_end_date', 'payment_plan_start_date', 
               'debt_settlement_flag_date', 'settlement_date','desc'], axis=1)
df3 = pd.merge(df1.select_dtypes(include=['object','category']), df1[['target']], left_index=True, right_index=True) 
df4 = df1.select_dtypes(include='number')<jupyter_output><empty_output><jupyter_text>Association Rules Weight of Evidence and Information Value The weight of evidence tells the predictive power of an independent variable in relation to the dependent variable. It's good to understand the concept of WOE in terms of events and non-events. It is calculated by taking the natural logarithm (log to base e) of division of % of non-events and % of events.<jupyter_code>from IPython.display import display, Image
display(Image(filename='WOE.png'))
import scorecardpy as sc
bins = sc.woebin(df1, y="target")
# uuid, emp_title, title, zip_code, addr_state
list_dfn = list(df1.columns)
full_woe = pd.concat([bins[list_dfn] for list_dfn in bins.keys()]).reset_index(drop=True)
full_woe.head(10)
full_iv = full_woe[["variable","total_iv"]].drop_duplicates().sort_values(by=['total_iv'],ascending=False).reset_index(drop=True)
full_iv_selected = full_iv[full_iv["total_iv"] > 0.5]
list_sel = list(full_iv_selected["variable"])
import matplotlib.pyplot as plt
for i in list_sel:
    print(i)
    ax = sns.barplot(x="bin", y="woe", data=full_woe[full_woe["variable"]== i])
    plt.show(block=False)
full_iv_2 = full_iv[full_iv["total_iv"] > 0]
list_sel2 = list(full_iv_2["variable"])
import seaborn as sns
import matplotlib.pyplot as plt
corr = df1[list_sel].corr()
fig, ax = plt.subplots(figsize=(10,10)) 
sns.heatmap(corr, 
        xticklabels=corr.columns,
        yticklabels=corr.columns,annot=True, linewidths=.5, ax=ax)
df2 = sc.woebin_ply(df1, bins)
df2.head()
list_cat = list(set(list_sel2) & set(list(df3.columns)))
for i in list_cat:
    a = bins[i][['variable','breaks']]
    a['breaks'] = a["breaks"].replace("%", "",regex=True) 
    f =pd.concat([a[['breaks']], a["breaks"].str.split(',', expand=True)], axis=1)
    e1 = f.set_index('breaks').T
    mydict = {}
    for a, b in e1.iteritems():
        c = b.to_numpy().tolist()
        for j in range(len(c)):
            if c[j] is not None:
                currentid = c[j]
                currentvalue = a
                mydict.setdefault(currentid, [])
                mydict[currentid].append(currentvalue)
    df3[str(i)+'_bin'] = df3[i].map(mydict)
    df3['bin_'+str(i)] = df3[str(i)+'_bin'].astype(str).str.replace('\[|\]|\'', '')
df3.head()
list_cat = list(set(list_sel2) & set(list(df3.columns)))
for i in list_cat:
    figsize=(20, 16)
    g = sns.FacetGrid(df3, col="target", height=10, aspect=1)
    g = g.map(sns.countplot, "bin_"+str(i),color="r",order=np.sort(df3['bin_'+str(i)].astype(str).unique()))
list_num = list(set(list_sel2) & set(list(df4.columns)))
for i in list_num:
    a = bins[i][['breaks']]
    b = a[(a['breaks'] != 'inf') & (a['breaks'] != '-inf')&(a['breaks'] != 'missing')]["breaks"]
    c = b.values.astype(float).tolist()
    if len(c) != 0:
        innercutpoints = [-np.inf] + c + [np.inf]  # add -Inf, +Inf to cutpoints
        cutpoints = list(dict.fromkeys(list(dict.fromkeys(innercutpoints))))
        df4['bin_'+str(i)] = pd.cut(df4[i], bins = cutpoints, duplicates='drop')
df4.head()
g = sns.PairGrid(df4, y_vars=["loan_amnt"], x_vars=["num_tl_op_past_12m"], size=10)
g.map(plt.scatter)<jupyter_output>/opt/conda/lib/python3.7/site-packages/seaborn/axisgrid.py:1264: UserWarning: The `size` parameter has been renamed to `height`; please update your code.
  warnings.warn(UserWarning(msg))
{'fraction_non_alphanumeric': 0.12065269175797999, 'fraction_numerical': 0.023701762744163888, 'mean_word_length': 5.544816835541699, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '54733', 'license': 'MIT', 'n_tokens_mistral': 2976, 'n_tokens_neox': 2823, 'n_words': 696, 'path': 'py/MLF 10 - PHBS test_prep.ipynb', 'repo_name': 'Delia810/MLF', 'size': '8435'}