<jupyter_start><jupyter_text>Social Data from the Web  Social Networking CommunitiesWe have just discussed how clustering can be an effective tool to understand political behaviour. As an unsupervised learning technique it provides a new machine reading on party affiliations. Another popular application of clustering is detecting communities in social relationships.  Next we go through an example and dataset in <NAME>’s excellent book on Machine Learning(Lantz, B. (2013). Packt Publishing Ltd.). The dataset is discussed on pp. 279. It covers the relationships in a Social Networking Service (SNS). While this is a fairly early SNS dataset, it is freely available and offers similar kind of insights you can gain from my recent examples. This section also introduces you to the intersection of digital marketing techniques and sociological studies of online networks. ![title](../img-videos/teen-social-media.jpg) Lantz explains that the dataset was compiled for sociological research on teenage identities at Notre Dame University. It represents a random sample of 30,000 US high school students who had profiles on a well-known SNS in 2006. At the time the data was collected, the SNS was a popular web destination for U.S. teenagers. Therefore, it is reasonable to assume that the profiles represent a fairly wide cross section of American adolescents in 2006. The data was sampled evenly across four high school graduation years (2006 through 2009) representing the senior, junior, second-year and freshman classes at the time of data collection. Then, the full texts of the SNS profiles were downloaded. Each teen's gender, age and number of SNS friends was recorded.  A text-mining tool was used to divide the remaining SNS page content into words. From the top 500 words appearing across all pages, 36 words were chosen to represent five categories of interests, namely extracurricular activities, fashion, religion, romance and antisocial behaviour. The 36 words include terms such as football, sexy, kissed, bible, shopping, death, drugs, etc. The final dataset indicates, for each person, how many times each word appeared in the person's SNS profile. First we load the relevant packages and the dataset.<jupyter_code>#Run the code below

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline
#Run the code below

teens = pd.read_csv("../data/snsdata.csv")<jupyter_output><empty_output><jupyter_text>Print out the first couple of rows from the teens dataset. The teens dataset is now loaded into your environment. Take a close look and make sure you understand how it is produced.We can use the info() method to output some general information about the dataframe. Part of this lesson is centered on the issue of looking into real-life data on digital society. We have mentioned earlier that a common problem is that observations/records are missing in such data, which is indicated by the NaN value in Python - as you might remember. In the info printout, you can also see that the non-null count is lower for those columns that contain NaN values. Gender is one of them.Let's watch a quick video first how to deal with dirty data in general.<jupyter_code>#Run the code below

from IPython.display import Video

Video("../img-videos/Session3.mp4")<jupyter_output><empty_output><jupyter_text>That was a lot of information. Let's go through this one step a time with the teens data.In order to see how many gender entries are missing we can use the value_counts() function. Print out the absolute non-null value counts for gender as well as the relative ones. With dropna set to False in value_counts, we can also see NaN index values. Try that ... But missing values are not our only problem. At least as common are misreported observations in real-life data. As an example, let’s look at the at the age distribution of the teens' age. You can do this in several ways but you should always print out maximum and minimum values. There are quite a few strange records here. Teens can have a minimum age of less than 4 and a maximum age of over 100! These cannot be considered teenagers. As a rule of thumb, let’s assume teenagers are between 13 and 19 years old. Let’s mark all other teens.age entries as invalid. We say that invalid entries should have a NaN value. Set this, please. The next step in our data cleaning process is to replace NaN values. Of course, we could simply remove all rows/observations, for which we have null entries. We did this for the senate example effectively. But then there was only one row that contained null values. In the SNS example, we would lose too many rows with such a brute-force approach. So, let’s try and fill the null values with estimated values.  Let’s start with the gender and replace null values by creating new columns for male and females.  To this end, we first create a new column to record all the female teenagers. Create a new column 'female' that is set to 1 if the teenager is female and 0 otherwise. Next we will create another column for the null values in gender we want to call no_gender. Set this to 1 if there is no gender recorded and otherwise to 0. This process is called dummy-coding btw. This is typical to community analysis. A dummy variable is a numerical variable used in the analysis to represent subgroups – in our case males, females and others. In research design, a dummy variable is often used to distinguish different groups to address them differently. By creating a separate column per gender entry, we can compute clusters for separate gender communities. Check out dummy-coding on the web. Can you find easier ways to do this in Pandas? After this, we have the original column, a new column called female, which contains information about whether the teen is female or not (male) and a new column with information about whether the gender value is missing. Using this column we could, for instance, check with clustering whether certain communities have a tendency not to record their gender values. How? Check out the changes with teens.head(). You have to scroll all the way to the right to find the new columns. It's very easy now to calculate the number of teenagers where we do not have gender entries for. How? Did you find that there are 2724.The age column is next. Can you find the average age and take care that null values are discounted?  What would happen if you set skipna to False?A good strategy to overwrite missing age values would be to use the average age value and assign it to all of the missing ones. This process is called mean-imputation and is employed frequently. Pandas has some real strengths here. Check out https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html.Pandas makes you life very easy with its fillna function. Run the following cell.<jupyter_code>#Run the code below

teens['age'].fillna(teens['age'].mean()).quantile([.25, .5, .75])<jupyter_output><empty_output><jupyter_text>This does look much better than the values we had beforehand. Let's further improve this with some good old-fashioned human intelligence.We feel confident that we can do better, because we know the graduation year, too. This is the year our teens are supposed to graduate. It seems a reasonable assumption that those teenagers with an earlier graduation year should be older (closer to graduation) than those for whom graduation is further away. We can easily find this out by running the mean function for each graduation year group separately. Do you know how? Our suspicion has proven right. There is a significant difference in the average ages depending on the graduation year. Let’s use this knowledge and update missing values in the age group depending on the graduation year. To this end, you actually have to do a lot of Pandas labour, which demonstrates that 80% of the work of a data scientist lies in working with data. But I am sure you know this by now.You can, e.g., proceed with the following strategy:Create first a temporary dataset with the results from the above group_by call. Then merge this new dataset with teens and replace the null values of teens.age with the ones from the temporary dataset.Feel free to name the temporary dataset as you like. Make sure to add to the teens dataset.Create the temporary dataset first. Update the teens age columns but make sure that in the end you have not added additional columns. Run the cell below to check that it all worked.<jupyter_code>#Run the code below

teens[['gradyear', 'age']].groupby(['gradyear']).mean().reset_index().head()<jupyter_output><empty_output><jupyter_text>Check out ave_age with teens.age.mean(). This is all quite advanced stuff but as long as you remember the kind of steps we have taken you should be able to impute one column's missing values by using another column as a reference. In our case, we use our knowledge that age is dependent on gradyear to find the missing values. Please, take some time to review the steps.Let’s take a look at the resulting age column and run the next cell.<jupyter_code>#Run the code below

teens.age.describe()<jupyter_output><empty_output><jupyter_text>This looks much better. We have now learned how to delete missing values completely or impute them using a background knowledge. After we have dealt with the missing records, I think we are ready to cluster again. We will use our trusted k-means without actually referring to either age nor gender. Sorry! But it was good that you learned how to deal with missing values and we will use them later. Just like in the US Senate example, we need to first understand, what we are trying to cluster. In the Senate example, we clustered voting behaviour. Now, it will be interests, which we can get from the columns 5 to 40 of the teens data frame. This time we have to select them by number as there is no clever way of selecting them by expression as for the Senate data.Please create the interests dataframe by selecting columns 5 to 40 from teens. We did not mention this before, because it was not necessary but k-means is very sensitive to input of varying size, length, etc. It was not necessary to focus on this in the previous example, because all the voting behaviour was in the range of 0 to 1. Now, we have interests of very different ranges. The interests are simply based on how many times a keyword appears in teenagers' social networking contributions.Since k-means is based on calculating distances between data points and their centroids, it will be strongly influenced by the magnitudes of the variables we cluster. Think about if for a moment! Just imagine one column having values running from 1 to 10 and another from 1 to 1000. How could we compare distances between them? We therefore need to scale the value so that they all fall into the same range. To this end, Python has the scale function in scipy.stats, which centres values around their mean. Using the apply function, we can tell Python to scale all interests values.  Do you remember apply? from scipy.stats import zscore, apply zscore and assign the results to a new dataframe interests_z. Finally, print out its first few columns.  ClusteringNow we cluster again and start by importing KMeans.<jupyter_code>#Run the code below

from sklearn.cluster import KMeans<jupyter_output><empty_output><jupyter_text>We decide 5 clusters is enough and assign k = 5.<jupyter_code>#Run the code below

k = 5<jupyter_output><empty_output><jupyter_text>Now we are ready to cluster. Create and fit teen_clusters the way you know. Let’s investigate the size of the clusters with .labels_ and np.bincount. I have noticed very different results depending on the kmeans results. I suggest to rerun kmeans a couple of times until you see a distribution that looks ok. You want to especially avoid clusters of only one 1 item.We can also look at the centroids/centres with teen_clusters.cluster_centers_. You learned earlier how to pretty-print this in a dataframe. A simple way to detect clusters is to find thr maximum values in the columns. Try it by using the idxmax() function. Hopefully, your results look similar to the table from Lantz (2013) on p. 288. ![title](../img-videos/teen-clusters.jpg)Doe the names of the clusters make sense to you? Do you remember all those teenage movies you watched?Next, let’s continue with another type of analysis. Let’s first assign each teen to a cluster, as we did before in the voting example for the senators. Please, add a column called 'cluster' to the teen data frame. Let's take a look at the teen data frame with head(). All the way to the right, you can find the cluster assignment. Let's print out the first 5 teens and only the columns 'cluster', 'gender', 'age' and 'friends'. I hope you remember how this works. How do we select the first 5 rows? How do we select the columns? Since we have learned earlier how to group by particular interests, let’s aggregate the teens' features using the clusters. Print out the average ages per cluster. Do you remember how this works? The clusters do not differ in terms of ages. There is no immediate relation between age and interest clusters. Now, let’s look at the female contribution to each cluster.  Overall, 74 per cent of the SNS's users are female. That’s why they contribute so much to each cluster. Can you see the cluster that has the most female users? Do you know why yet? You can check for the average number of friends per cluster now. Just define the target of the aggregation per cluster as friends instead of female or age in the expressions above. Here the differences are more distinct. We suspect that the number of friends played a key role in assigning the clusters. That’s the nature of a social network, I guess.  We have now completed our exercises on clustering and understanding political and social communities. For today, we have just one more important question to answer. How do we get access to the kind of data we worked on today? The teens dataset stemmed from a research project in sociology published online, while the US Senate voting behaviour was downloaded from US government websites.   Web Scraping The web has become a unique source of data for social analysis. Munzert et al. (2014) in their book on 'Automated data collection (...). A practical guide to web scraping and text mining' (<NAME>iley & Sons) emphasize in the Introduction that 'the rapid growth of the World Wide Web over the past two decades tremendously changed the way we share, collect, and publish data. Firms, public institutions, and private users provide every imaginable type of information and new channels of communication generate vast amounts of data on human behavior. What was once a fundamental problem for the social sciences — the scarcity and inaccessibility of observations — is quickly turning into an abundance of data. This turn of events does not come without problems. (…), traditional techniques for collecting and analyzing data may no longer suffice to overcome the tangled masses of data.' (p. XV).  In short, we can find lots of data on the web. A big problem with web data is, however, that it is often inconsistent and heterogeneous. To get access to it, one often has to visit multiple web sites and assemble their data together. Finally, the data is generally published without reuse in mind, which implies that the data can be of low quality. That said, the web is so vast that it still provides an often overwhelming source of exciting data. Let's take a look at how we can access web data in general by scraping web sites.<jupyter_code>#Run the code below

from IPython.display import Video

Video("img-videos/Session2.mp4")<jupyter_output><empty_output><jupyter_text>Returning to our first example of political communities, we will scrape data on the current composition of the US Senate from Wikipedia. Later we will see that this can be a complex task and requires additional libraries. But in this case, we can rely on Pandas directly with its read_html function that does all the hard work for you. Check it at https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html. All the content on the web is presented to us in a language called HyperText Markup Language (HTML; https://en.wikipedia.org/wiki/HTML). HTML is of course a way of presenting content on the web in a universal way. It also contains so-called hyperlinks that let you jump from web content to web content. If you are interested in the further details of HTML, why not take some time now to visit the excellent http://www.w3schools.com/html/, which contains a lot of practical exercises to learn everything about HTML and other web technologies.  Each document on the web is identified by a URL. We set the url to the wikipedia page of current US senators and run the below cell.<jupyter_code>#Run the code below

url = 'https://en.wikipedia.org/wiki/Current_members_of_the_United_States_Senate'<jupyter_output><empty_output><jupyter_text>With the read_html function of Pandas, we can read in the web content behind the URL. However, if you check the documentation you need to provide Pandas with further details. Unfortunately, web content in HTML format is not very structured and often simply chaotic. We would like to download only the table of the page of current US Senators and need to find a so-called 'match' for read_html to choose that table from the HTML document. Fortunately, for us there are many existing strategies to determine exactly the HTML element we would like to select. I used Google Chrome to find the correct source code and check out some specific names in the table that are not repeated in the rest of the wiki page. Find you own specific match and use it in the match parameter of read_html. Then, run read_html and assign the result to senator_wiki. If it all worked as it should run the cell below to create our senators dataframe.<jupyter_code>#Run the code below

senators = senator_wiki[0]
senators<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.039353907637655415, 'fraction_numerical': 0.005495115452930729, 'mean_word_length': 4.5522342064714945, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '654775', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4324, 'n_tokens_neox': 4114, 'n_words': 2888, 'path': 'week_3/Lecture/Lecture2-AnalysingCommunities.ipynb', 'repo_name': 'Giovanni1085/UvA_AIforSociety_2021', 'size': '18021'}