<jupyter_start><jupyter_text>NTDS Project:  XXx Project Title XXX[Github Link] [github link]: https://github.com/mdeff/ntds_2018  Students* Team: 50* Students: <NAME>, <NAME>, <NAME>, <NAME>* Dataset: Spammers on Social Network<jupyter_code>! pip3 install seaborn==0.9.0
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import DataProcessing as processing
%matplotlib inline
sns.__version__<jupyter_output><empty_output><jupyter_text>IntroductionSocial Networks are an essential part of the many people's daily life. With the growth and  spread of the social networks, the amount of spammers with bad intentions increase as well. In this project, we aim to find what makes a spammer and whether we can find spammers based on the features such as age, gender and time of interaction. After understanding the nature of our network data our final aim is to try to predict spammers on social networks.For this project we will use the Spammers on Social Network data which includes users and relations information in a given social network. XXXXX explain more XXX  Data ProcessingWe have processed the data differently for several different steps in our project. For the basic statistical analysis and understanding the properties of our network, we will use the below processing steps and the final created adjacency matrix.For the prediction level of our project, we decided to preprocess the data further in a way that we can feed them into neural networks. XXX further explanation from Raphael or Ilija XXXX.<jupyter_code>relations, users = processing.readData()<jupyter_output><empty_output><jupyter_text>Our data is composed from relations and users. We will explore them below separately.  Relations As we can see relations consist of 5 features: day, time_ms, src, dst and relation. Each row shows a relation between a source and destination (users in this case) and which day, how long and what type of relation it is. For our current study we will focus on relation 5.We have a total of 83176 relations in our data. Minimum interaction time is 0.129 seconds (1.290000e+02 ms) and maximum is 24 hours (8.639946e+07 ms)	 while in average interaction in these relation events are around 12.12 hours (4.364584e+07 ms).<jupyter_code>processing.displayInfo(relations, "relations")<jupyter_output><empty_output><jupyter_text>Our data is already clean, therefore before moving forward we check if there is any null value, which we don't. We also make a sanity check to see we only have relation type of 5 in our relation data. The day values are from 0 to 9.<jupyter_code>processing.printFurtherInfo(relations)<jupyter_output>Is there any NA? 
____________<jupyter_text>UsersIn the users data we have some characteristic information of the users such as gender, time, age range and whether they are manually identified as spammer or not.We have 62173 users in our data where their age ranges between 10-70.<jupyter_code>processing.displayInfo(users, "users")<jupyter_output><empty_output><jupyter_text>Create Adjacency Matrix of the NetworkAfter discovering how our data looks like, we create our adjacency matrix with the given informations. Our nodes will be specified as users and edges will be the relations. We will create both directed and undirected graph in here.<jupyter_code># create nodes from users and edges from relations
nodes = processing.createNodes(users)
edges = processing.createEdges(nodes, relations)
# created undirected adjacency matrix
adjacency = processing.createAdjacencyMatrix(nodes, edges, isDirected = False)

# created directed adjacency matrix
adjacency_directed = processing.createAdjacencyMatrix(nodes, edges, isDirected = True)<jupyter_output><empty_output><jupyter_text>Undirected Adjacency Matrix<jupyter_code># plt.spy(adjacency, markersize=1)
# plt.title('undirected adjacency matrix')<jupyter_output><empty_output><jupyter_text>Directed Adjacency Matrix<jupyter_code># plt.spy(adjacency_directed, markersize=1)
# plt.title('directed adjacency matrix')<jupyter_output><empty_output><jupyter_text>Data and Network ExplorationIn this part we will focus on understanding the data and network better by asking some specific questions and providing some statistics to better understand them.  What is the number of nodes and edges?  Our network has 62173 node and 83176 edges.<jupyter_code>num_nodes = len(nodes)
num_edges = len(edges)

print("We have ", num_nodes, " nodes and ", num_edges, " edges.")<jupyter_output>('We have ', 62173, ' nodes and ', 83176, ' edges.')<jupyter_text>What is the degree distribution of the network and maximum network degree?Our network max degree is 8216 and as we can see from our degree distribution plot, there are very few nodes that has many nodes, our network is a sparse network since most of our nodes has very low incoming and outgoing edges.<jupyter_code>in_degree, out_degree, degree = processing.calculateIn_Out_Degrees(edges, num_nodes)
max(degree)<jupyter_output><empty_output><jupyter_text>Degree Distribution Plot<jupyter_code>weights = np.ones_like(degree) / float(num_nodes)
plt.hist(np.log(degree), weights=weights, bins=100, range= (0,5));<jupyter_output><empty_output><jupyter_text>What is the average degree of the network both directed and undirected one?  The network L is 83176 and average degree is 1.33 therefore we can say that our network is pretty scarce.<jupyter_code>avg_degree = processing.calculateAvgDegree(in_degree, out_degree, num_nodes)<jupyter_output>('L is: ', 83176)
('average degree is: ', 1)<jupyter_text>Is the network connected? If not, how many connected components it has?As we can see our network is connected hence has by one big component.<jupyter_code>isConnected = processing.isConnected(adjacency)
print(isConnected)<jupyter_output>True<jupyter_text>How many of the users are manually labeled as spammers?We can see below that in our network 2448 of the users are labeled as spammers which are around 3.9374% of the total users. This means that out of 100 people around 4 of them are considered as spammers while manually labeled which is a very serious number.<jupyter_code>spammer_distribution = pd.DataFrame(nodes.groupby(["Spammer Label"])["User Id"].count())
spammer_distribution.reset_index(inplace=True)
spammer_distribution.rename(columns={"User Id":"count"}, inplace=True)
spammer_distribution
# Spammer % in total users
spammer_distribution[spammer_distribution["Spammer Label"]==1]["count"] * 100.0 / num_nodes
ax = sns.barplot(x="Spammer Label", y="count", data=spammer_distribution).set_title('Spammers Distribution')<jupyter_output><empty_output><jupyter_text>0s are non-spammers and 1s are Spammers.   Check the Attribute Distributions?  In this part, we discover more about the Age variable of our data.   What are the AGE distributions (for spammers, non-spammers and in general)?  The age range for the users are from 10-70. As we can see the 70% of the overal users are fall in age range 57.5 and below.<jupyter_code>nodes_age = pd.DataFrame(nodes.groupby(["Spammer Label", "Age Range"])["User Id"].count())
nodes_age.reset_index(inplace=True)
nodes_age.rename(columns={"User Id":"count"}, inplace=True)
nodes_age.T
nodes_age.describe()<jupyter_output><empty_output><jupyter_text>Spammers Distribution Plot by AgeIn this plot we can see that most frequent age range is from 20 to 50 in the social network data we are investigating. The non-spammers age distribution is more diversive compare to the spammers. However, we need also keep in mind that spammers are around 4% of the users, even though age distributions seems almost equally distributed in this plot, if we look them independently from non-spammers we can find a more insightful understanding about age distribution between spammers.<jupyter_code>g = sns.catplot(x="Age Range", y="count", hue="Spammer Label", data=nodes_age)<jupyter_output><empty_output><jupyter_text>Age Distribution only for SpammersIn order to see better the Spammer distributions, we draw it separetly as well. We can see that the majority of the spammers are in age range 20-40, which can tell that they are either young or adults. The numbers of spammers who are either old or teenagers are very low.<jupyter_code>nodes_age_spammers = nodes_age[nodes_age["Spammer Label"]==1]
g = sns.catplot(x="Age Range", y="count", hue="Spammer Label", data=nodes_age_spammers)<jupyter_output><empty_output><jupyter_text>What are the GENDER distributions (for spammers, non-spammers and in general)?  Another interesting idea is to look for gender of the spammers. In our data most of the users are male and both in spammers and nonspammers male gender is the dominant one. Age range would be an useful feature when we try to predict spammers.<jupyter_code>nodes_gender = pd.DataFrame(nodes.groupby(["Spammer Label", "Gender"])["User Id"].count())
nodes_gender.reset_index(inplace=True)
nodes_gender.rename(columns={"User Id":"count"}, inplace=True)
nodes_gender<jupyter_output><empty_output><jupyter_text>Spammers Distribution Plot by GenderIn this plot as well we can observe the male dominance in the social network. Also, this gap is even higher in the non-spammers. Within the spammers as well female gender is fewer.<jupyter_code>g = sns.catplot(x="Spammer Label", y="count", hue="Gender", data=nodes_gender)<jupyter_output><empty_output><jupyter_text>What % of the Spammers female?As we can see only 1.593% of the Spammers are female which would be an interesting idea to keep in mind for further network analysis and predictions. This feature could be distinctive while trying to predict Spammers.<jupyter_code>spammer_females= nodes_gender[(nodes_gender["Spammer Label"]==1) & (nodes_gender["Gender"] == "F")]["count"]
all_spammers = nodes_gender[(nodes_gender["Spammer Label"]==1)]["count"].sum()
female_spammer_percentage = (spammer_females *100.0)/ all_spammers
female_spammer_percentage<jupyter_output><empty_output><jupyter_text>What are the TIME SPENT by spammers and non-spammers?   Average time spent by Spammers are actually lower (0.264) than time spent by actual users (0.424) which are non-spammers.<jupyter_code>nodes_time = nodes.groupby(["Spammer Label"])["Time"].mean()
nodes_time<jupyter_output><empty_output><jupyter_text>As we can see from the below scatter plot, the time spent by the user is not very distinguishable for the Spammers in this case since they are spread out in the plot. We don't see any time interval where there is particularly more spammers compared with the other users.<jupyter_code>sns.scatterplot(x="User Id", y="Time" , hue="Spammer Label", data=nodes)<jupyter_output><empty_output><jupyter_text>Check Time Spent only For SpammersWhen we check for only Spammers we can see the majority of the Spammers fall below 0.2 Time interval.<jupyter_code>nodes_time_spammers =  nodes[["Spammer Label", "User Id", "Time" ]]
nodes_time_spammers = nodes_time_spammers[nodes_time_spammers["Spammer Label"]== 1]
sns.scatterplot(x="User Id", y="Time" , hue="Spammer Label", data=nodes_time_spammers)<jupyter_output><empty_output><jupyter_text>How many relations belongs to Spammers? What % of the relations they are?62055 of the edges start from a Spammer labeled source. This makes around 74.6% of the all relations.This is very interesting. In the above, analysis we actually saw that only around 4% of the users were Spammers and these 4% of the users actually are responsible for the 74.6% of the events (relation 5). Even seeing this, could lead us to think that, the dominant nodes which has more outgoing edges are potential spammers whereas more scarce nodes are non-spammers.<jupyter_code>spammer_ids = nodes[nodes["Spammer Label"]==1]["User Id"]
spammer_ids_list = spammer_ids.tolist()
# Find relations where Spammers are the Source
spammers_relations = relations[relations['src'].isin(spammer_ids_list)]
print(len(spammers_relations))
print((len(spammers_relations)*100.0) /len(relations))<jupyter_output>62055
74.6068577474<jupyter_text>What are the number of different days Spammers interact? (same for non-spammers) We also checked how many different days these spammers were doing the relation 5 in the social network. Is this activity diversed through the days or done in same day? From the below Spammers Day Count Distribution we can see that most of these relations are created in the first or second day. Therefore, this might lead us to say that Spammers do these activities within a limited time period, so spam people on the same day rather than diversed in 10 days. This actually makes sense actually, if we think that spammers generally prone to spread the same info, so technology allows them to do multiple events easier such as send a chat info to all at the same etc.<jupyter_code>spammers_count_day = pd.DataFrame(spammers_relations.groupby("src")["day"].count())
#spammers_count_day
sns.distplot(np.log(spammers_count_day))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0741915519108773, 'fraction_numerical': 0.016246325235958533, 'mean_word_length': 6.319932049830125, 'pattern_counts': {'":': 3, '<': 100, '<?xml version=': 0, '>': 100, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '335100', 'license': 'MIT', 'n_tokens_mistral': 3915, 'n_tokens_neox': 3610, 'n_words': 1527, 'path': 'gorkem_temp/Project_Spammers.ipynb', 'repo_name': 'ilijagjorgjiev/ntds2018_team50', 'size': '12954'}