<jupyter_start><jupyter_text>All State Claims Severity - <NAME>  Importing Data, Packages and Visualizing missing values<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
data = pd.read_csv('C:/Users/ganes/Data Science/Datasets/digiledge/train.csv')
test = pd.read_csv('C:/Users/ganes/Data Science/Datasets/digiledge/test.csv')
data.head()
data.columns
for column in data:
    print(column)
data.shape
sns.heatmap(data[].isnull())
sns.heatmap(data.loc[:,'cat1':'cat9'].isnull())
sns.heatmap(data.loc[:,'cat10':'cat50'].isnull())
sns.heatmap(data.loc[:,'cat51':'loss'].isnull())<jupyter_output><empty_output><jupyter_text>Removal of Highly Correlated Columns Here columns 'cont1' and 'cont6' are highly correlated to columns 'cont9' and 'cont10'<jupyter_code>plt.figure(figsize=(12,10))
sns.heatmap(data.corr(), annot=True)
print(data.loc[:,'cont1':'cont10'].corr())<jupyter_output>cont1     cont2     cont3     cont4     cont5     cont6     cont7  \
cont1   1.000000 -0.085180 -0.445431  0.367549 -0.025230  0.758315  0.367384   
cont2  -0.085180  1.000000  0.455861  0.038693  0.191427  0.015864  0.048187   
cont3  -0.445431  0.455861  1.000000 -0.341633  0.089417 -0.349278  0.097516   
cont4   0.367549  0.038693 -0.341633  1.000000  0.163748  0.220932 -0.115064   
cont5  -0.025230  0.191427  0.089417  0.163748  1.000000 -0.149810 -0.249344   
cont6   0.758315  0.015864 -0.349278  0.220932 -0.149810  1.000000  0.658918   
cont7   0.367384  0.048187  0.097516 -0.115064 -0.249344  0.658918  1.000000   
cont8   0.361163  0.137468 -0.185432  0.528740  0.009015  0.437437  0.142042   
cont9   0.929912 -0.032729 -0.417054  0.328961 -0.088202  0.797544  0.384343   
cont10  0.808551  0.063526 -0.325562  0.283294 -0.064967  0.883351  0.492621   

           cont8     cont9    cont10  
cont1   0.361163  0.929912  0.808551  
cont2   0.137468 -0.032729  0.063526  
co[...]<jupyter_text>So here we drop the two columns<jupyter_code>data.drop(['cont9','cont10'], inplace = True, axis=1)
data.shape<jupyter_output><empty_output><jupyter_text>Visualization the distribution of all continuous values<jupyter_code>sns.distplot(data['cont1'])
sns.distplot(data['cont2'])
sns.distplot(data['cont3'])
sns.distplot(data['cont4'])
sns.distplot(data['cont5'])
sns.distplot(data['cont6'])
sns.distplot(data['cont7'])
sns.distplot(data['cont8'])
sns.distplot(data['cont11'])
sns.distplot(data['cont12'])
sns.distplot(data['cont13'])
sns.distplot(data['cont14'])<jupyter_output><empty_output><jupyter_text>Here we note that the around 10k bucks are lost on huge claims and very few claims have more than 20k bucks<jupyter_code>sns.distplot(data['loss'])<jupyter_output><empty_output><jupyter_text>Visualizing Number of Categories in categorical features<jupyter_code>data['cat1'].value_counts().index.value_counts().sum()
no_of_categories = []

for column in data.loc[:,'cat1':'cat116'].columns:
    no_of_categories.append(data[column].value_counts().index.value_counts().sum())
plt.figure(figsize=(20,20))
plt.plot(no_of_categories, '*')
data['cat73'].value_counts().index.value_counts().sum()
data['cat116'].value_counts().index.value_counts().sum()<jupyter_output><empty_output><jupyter_text>Splitting Categorical, Nominal and target variables in separate DataFrames<jupyter_code>data_id = data['id']
data_cat = data.loc[:,'cat1':'cat116']
data_cont = data.loc[:,'cont1':'cont14']
target = data['loss']
data_cat.shape,data_cont.shape,target.shape,data_id.shape
data.shape<jupyter_output><empty_output><jupyter_text>Assigning Dummies<jupyter_code>for column in data_cat:
    data_cat[column] = pd.get_dummies(data=data_cat)
data_cat.head(10)
plt.figure(figsize=(20,20))
plt.plot(no_of_categories, '*')<jupyter_output><empty_output><jupyter_text>Exporting Data to CSV files<jupyter_code>data_cat.to_csv('categorical_train.csv', index=None)
data_cont.to_csv('continuous_train.csv', index=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14768145161290322, 'fraction_numerical': 0.1580141129032258, 'mean_word_length': 5.370786516853933, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '467626', 'license': 'MIT', 'n_tokens_mistral': 1936, 'n_tokens_neox': 1603, 'n_words': 261, 'path': 'AllState Claims Severity Project.ipynb', 'repo_name': 'ganeshramg/Regression-Projects', 'size': '3982'}