<jupyter_start><jupyter_text>[NTDS'18] Project: Spammer on social network detection [ntds'18]: https://github.com/mfendri2/NTDS_Project_Team20  Jupyter notebook for Feature exploration  **This notebook goes through different features trying to better understand them**  1. Import library<jupyter_code>import numpy as np
from numpy import linalg as LA
import pandas as pd
import matplotlib.pyplot as plt
from scipy.sparse import csr_matrix
import networkx as nx
import collections<jupyter_output><empty_output><jupyter_text>2. Import data<jupyter_code>all_users=pd.read_csv('usersdata.csv',sep='\t',names = ["User Id", "Gender", "Time", "Age_Range","Spammer_Label"])
spammer = all_users[all_users.Spammer_Label == 1]
non_spammer = all_users[all_users.Spammer_Label == 0]<jupyter_output><empty_output><jupyter_text>3. Study the feature of Users  3.1 Age Range<jupyter_code>spammer.hist(column = 'Age_Range')
plt.title("Spammer's Age Range")
non_spammer.hist(column = 'Age_Range')
plt.title("Non-Spammer's Age Range")<jupyter_output><empty_output><jupyter_text>3.2 Time Pass Validation<jupyter_code>import random
import numpy



plt.figure(figsize=(10,8))
bins = numpy.linspace(0, 1, 100)
bins = np.ones_like(non_spammer.Time) / float(non_spammer.shape[0])
bins2 = np.ones_like(spammer.Time) / float(spammer.shape[0])
p1 = plt.hist(non_spammer.Time, weights=bins, alpha =0.5, color ='skyblue',label='Non Spammer');
p2 = plt.hist(spammer.Time, weights=bins2,alpha =0.5, color ='coral',label = 'Spammer');
plt.title("Time After Validation of Users",fontsize=20);
plt.xlabel("Time (s) ",fontsize=20)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.ylabel("Percentage",fontsize=20)
plt.legend(loc='best',fontsize = 15)
plt.show()<jupyter_output><empty_output><jupyter_text>3.3 Gender<jupyter_code>Male_spammer = spammer[spammer.Gender == 'M'].groupby(['Age_Range']).agg('count')
Female_spammer = spammer[spammer.Gender =='F'].groupby(['Age_Range']).agg('count')
Male_nonspammer = non_spammer[non_spammer.Gender =='M'].groupby(['Age_Range']).agg('count')
Female_nonspammer = non_spammer[non_spammer.Gender =='F'].groupby(['Age_Range']).agg('count')
Female_spammer
Male_spammer
import numpy as np
import matplotlib.pyplot as plt

plt.figure(figsize=(10,8))
women_pop = Female_spammer.Gender
men_pop     = Male_spammer.Gender
X = [10,20,30,40,50,60,70]

p1 = plt.barh(X, women_pop,9, color = 'coral')
p2 = plt.barh(X, -men_pop,9, color = 'deepskyblue')
plt.xlabel("Population ",fontsize=20)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.ylabel("Age-Group",fontsize=20)
plt.title("Population of Spammer with Gender",fontsize=20);
plt.legend((p1[0], p2[0]), ('Women', 'Men'),loc='best',markerscale= 1,fontsize = 15)
plt.show()
import numpy as np
import matplotlib.pyplot as plt

plt.figure(figsize=(10,8))
women_pop = Female_nonspammer.Gender
men_pop     = Male_nonspammer.Gender
X = [10,20,30,40,50,60,70]

p1 = plt.barh(X, women_pop,9, color = 'salmon')
p2 = plt.barh(X, -men_pop,9, color = 'lightskyblue')
plt.xlabel("Population ",fontsize=20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12)
plt.ylabel("Age-Group",fontsize=20)
plt.title("Population of Non Spammer with Gender",fontsize=20);
plt.legend((p1[0], p2[0]), ('Women', 'Men'),loc='best',markerscale= 1,fontsize = 15)
plt.show()
male = all_users[all_users.Gender == 'M'].agg('count')
dict = {'Male': [Male_spammer.agg('sum').Gender, Male_nonspammer.agg('sum').Gender],
         'Female': [Female_spammer.agg('sum').Gender, Female_nonspammer.agg('sum').Gender],
           '10' :[Male_spammer.Gender.loc[[10]].values[0] + Female_spammer.Gender.loc[[10]].values[0] , Male_nonspammer.Gender.loc[[10]].values[0] + Female_nonspammer.Gender.loc[[10]].values[0]] ,
            '20': [Male_spammer.Gender.loc[[20]].values[0] + Female_spammer.Gender.loc[[20]].values[0] , Male_nonspammer.Gender.loc[[20]].values[0] + Female_nonspammer.Gender.loc[[20]].values[0]],
               '30':[Male_spammer.Gender.loc[[30]].values[0] + Female_spammer.Gender.loc[[30]].values[0] , Male_nonspammer.Gender.loc[[30]].values[0] + Female_nonspammer.Gender.loc[[30]].values[0]],
                   '40': [Male_spammer.Gender.loc[[40]].values[0] + Female_spammer.Gender.loc[[40]].values[0] , Male_nonspammer.Gender.loc[[40]].values[0] + Female_nonspammer.Gender.loc[[40]].values[0]],
                        '50':[Male_spammer.Gender.loc[[50]].values[0] + Female_spammer.Gender.loc[[50]].values[0] , Male_nonspammer.Gender.loc[[50]].values[0] + Female_nonspammer.Gender.loc[[50]].values[0]],
                            '60':[Male_spammer.Gender.loc[[60]].values[0] + Female_spammer.Gender.loc[[60]].values[0] , Male_nonspammer.Gender.loc[[60]].values[0] + Female_nonspammer.Gender.loc[[60]].values[0]],
                                   '70': [Male_spammer.Gender.loc[[70]].values[0] + Female_spammer.Gender.loc[[70]].values[0] , Male_nonspammer.Gender.loc[[70]].values[0] + Female_nonspammer.Gender.loc[[70]].values[0]]}
data = pd.DataFrame(data=dict)
data
data['Total'] = data[['10','20','30','40','50','60','70']].sum(axis=1)
data<jupyter_output><empty_output><jupyter_text>**From the distribution of gender and age across the population of Spammers and Non-spammers, there is an interesting remark that we used to expect there would be more Male spammers than Female ones, however, it turns out that in the spammers' society, the woman is who runs the world. That is opposite to the non-spammers' society, in which, the male is dominant compared to the female.****Hence, we think that using gender could be efficient in classfiying Spammers and Non-spammers.**  3.4 Degree of nodes **For the following part, we will use our sub-sample data set to study the degree of the nodes**<jupyter_code>sub_users = pd.read_csv('datasets_final.csv')
sub_users = sub_users.fillna(value = 0)
sub_spammer = sub_users[sub_users.label == 1]
sub_nonspammer = sub_users[sub_users.label == 0]<jupyter_output><empty_output><jupyter_text>**We decide to work with boxplot, since when we use histogram, the comparison of the distributions is not as clear as using the box plot **<jupyter_code>fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10, 8))
medianprops = dict(linestyle='-.', linewidth=2.5, color='firebrick')
sub_users[sub_users.deg_1!= 0].boxplot(by = 'label', column = ['in_deg_1','out_deg_1','deg_1'] , ax = axes,medianprops=medianprops)
plt.suptitle("")
#plt.title("Distribution of degree of nodes for relation 1")
#plt.title("Out degree of Non Spammer and Spammer for relation 1")
#From the plot, we could see that they share similar median, and quite similar variance (variance is computed from the 
# blue horizontal line segment to the black one) The points outside the black line segment (outside the box) 
#are just outlier point 
#Degree distribution for relation 2
# points out of the blue part is outlier. the small blue is the box and the green line is the median.
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10, 8))
medianprops = dict(linestyle='-.', linewidth=2.5, color='firebrick')
sub_users[sub_users.deg_2!= 0].boxplot(by = 'label', column = ['in_deg_2','out_deg_2','deg_2'] , ax = axes, medianprops=medianprops)
plt.suptitle("")
#plt.title("Out degree of Non Spammer and Spammer for relation 1")
#From the plot, we could see that they share similar median, and quite similar variance (variance is computed from the 
# blue horizontal line segment to the black one) The points outside the black line segment (outside the box) 
#are just outlier point. For non spammer, outliers spread wider than for spammer.  
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10, 8))
sub_users[sub_users.deg_4!= 0].boxplot(by = 'label', column = ['in_deg_4','out_deg_4','deg_4'] , ax = axes, patch_artist=True)
plt.suptitle("")
#plt.title("Out degree of Non Spammer and Spammer for relation 1")
#From the plot, we could see that they share similar median, and quite similar variance (variance is computed from the 
# blue horizontal line segment to the black one) The points outside the black line segment (outside the box) 
#are just outlier point 
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10, 8))
sub_users[sub_users.deg_5!= 0].boxplot(by = 'label', column = ['in_deg_5','out_deg_5','deg_5'] , ax = axes, patch_artist=True)
plt.suptitle("")
#plt.title("Out degree of Non Spammer and Spammer for relation 1")
#From the plot, we could see that they share similar median, and quite similar variance (variance is computed from the 
# blue horizontal line segment to the black one) The points outside the black line segment (outside the box) 
#are just outlier point 
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(10, 8))
sub_users[sub_users.deg_7!= 0].boxplot(by = 'label', column = ['in_deg_7','out_deg_7','deg_7'] , ax = axes, patch_artist=True)
plt.suptitle("")
#plt.title("Out degree of Non Spammer and Spammer for relation 1")
#From the plot, we could see that they share similar median, and quite similar variance (variance is computed from the 
# blue horizontal line segment to the black one) The points outside the black line segment (outside the box) 
#are just outlier point<jupyter_output><empty_output><jupyter_text>**What we could observe from the plots is that, even though the Spammers and Non - Spammers share quite similar distribution, with low median, small variance, low mean ( which represents in the tiny box plots), their ouliers are different. The outliers of Non-Spammers could reach far higher degree than the outliers of Spammers** **Folowing, we try another type of graph to see if it could better tell the different between spammers and non-spammers.**<jupyter_code>sub_spammer_non0 = sub_spammer[sub_spammer.in_deg_2!=0]
sub_nonspammer_non0 = sub_nonspammer[sub_nonspammer.in_deg_2!=0]
plt.figure(figsize=(10,8))
bins = np.ones_like(sub_nonspammer_non0.in_deg_2) / float(sub_nonspammer_non0.shape[0])
bins2 = np.ones_like(sub_spammer_non0.in_deg_2) / float(sub_spammer_non0.shape[0])
p1 = plt.hist(sub_nonspammer_non0.in_deg_2,bins=np.arange(min(sub_nonspammer_non0.in_deg_2), max(sub_nonspammer_non0.in_deg_2) + 1, 1),weights=bins, alpha =0.8, color ='skyblue',label = 'Non Spammer');
p2 = plt.hist(sub_spammer_non0.in_deg_2,bins=np.arange(min(sub_spammer_non0.in_deg_2), max(sub_spammer_non0.in_deg_2) + 1, 1),weights=bins2, alpha =0.5, color ='coral',label = 'Spammer');

plt.title("In degree of User nodes for relation 2",fontsize=20);
plt.xlabel("Degree ",fontsize=20)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.ylabel("Percentage",fontsize=20)
plt.legend(loc='best',fontsize = 15)
plt.show()
sub_spammer_non0 = sub_spammer[sub_spammer.in_deg_2!=0]
sub_nonspammer_non0 = sub_nonspammer[sub_nonspammer.in_deg_2!=0]
plt.figure(figsize=(10,8))
bins = np.ones_like(sub_nonspammer_non0.in_deg_2) / float(sub_nonspammer_non0.shape[0])
bins2 = np.ones_like(sub_spammer_non0.in_deg_2) / float(sub_spammer_non0.shape[0])
p1 = plt.hist(sub_nonspammer_non0.in_deg_2,bins=np.arange(min(sub_nonspammer_non0.in_deg_2), max(sub_nonspammer_non0.in_deg_2) + 1, 1),weights=bins, alpha =0.8, color ='skyblue',label = 'Non Spammer');
p2 = plt.hist(sub_spammer_non0.in_deg_2,bins=np.arange(min(sub_spammer_non0.in_deg_2), max(sub_spammer_non0.in_deg_2) + 1, 1),weights=bins2, alpha =0.5, color ='coral',label = 'Spammer');

plt.title("In degree of User nodes for relation 2",fontsize=20);
plt.xlabel("Degree ",fontsize=20)
plt.xticks(fontsize=14)
plt.yticks(fontsize=14)
plt.ylabel("Percentage",fontsize=20)
plt.legend(loc='best',fontsize = 15)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14566044260027664, 'fraction_numerical': 0.035529045643153526, 'mean_word_length': 6.308275426405559, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '344900', 'license': 'MIT', 'n_tokens_mistral': 4616, 'n_tokens_neox': 4276, 'n_words': 963, 'path': 'Feature_exploration.ipynb', 'repo_name': 'mfendri2/NTDS_Project_Team20', 'size': '11568'}