<jupyter_start><jupyter_text>**Knowledge Graph Analytics Example**Hands-On Data Analytics example for lecture 5.6 Knowledge Graph Analytics of OpenHPI Knowledge Graphs 2020 Course*Please make a copy of this notebook to try out your own adaptions via "File -> Save Copy in Drive"*<jupyter_code># To support both python 2 and python 3
from __future__ import division, print_function, unicode_literals

# Common imports
import numpy as np
import scipy as scp
import os

# To plot pretty figures
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.rcParams['axes.labelsize'] = 14
plt.rcParams['xtick.labelsize'] = 12
plt.rcParams['ytick.labelsize'] = 12<jupyter_output><empty_output><jupyter_text>First, we are going to read in the data. In the example, we are reading from a GoogleDoc spreadsheet. Therefore, we first have to **authenticate with Google Docs**...<jupyter_code>#Prepare to read Data from Google Sheets
!pip install --upgrade -q gspread
#Authenticate with Google Sheets
from google.colab import auth
auth.authenticate_user()

import gspread
from oauth2client.client import GoogleCredentials

gc = gspread.authorize(GoogleCredentials.get_application_default())<jupyter_output><empty_output><jupyter_text>We are **reading the data** into a comfortable data stracture (pandas).([Read the python documentation to learn more about the datastructure](http://pandas.pydata.org/pandas-docs/version/0.15/10min.html)). The usage here is more or less straight forward.<jupyter_code>#Read Google Spreadsheet as CSV into pandas DataFrame
import pandas as pd
physicists = pd.read_csv('https://docs.google.com/spreadsheets/d/' + '1srbAnRTUkmG1fTN_RmlLZycGnW9ZVfCnsTkc69s4nFQ' +
                   '/export?gid=270987752&format=csv'
                  )
physicists.head(10)<jupyter_output><empty_output><jupyter_text>First, let's **get some insights about our data**.<jupyter_code>physicists.describe()<jupyter_output><empty_output><jupyter_text>Let's have a look on the distribution of the numerical data via **histogram plots**.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
physicists.hist(bins=50, figsize=(20,10))<jupyter_output>/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:307: MatplotlibDeprecationWarning: 
The rowNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().rowspan.start instead.
  layout[ax.rowNum, ax.colNum] = ax.get_visible()
/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:307: MatplotlibDeprecationWarning: 
The colNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().colspan.start instead.
  layout[ax.rowNum, ax.colNum] = ax.get_visible()
/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:313: MatplotlibDeprecationWarning: 
The rowNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().rowspan.start instead.
  if not layout[ax.rowNum + 1, ax.colNum]:
/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:3[...]<jupyter_text>**Boxplots** are visualizing the numbers that we have already inspected<jupyter_code>physicists.boxplot(figsize=(20,10))
physicists.boxplot('bdate', figsize=(15,8))
physicists.boxplot(column='countries', figsize=(15,8))
physicists.boxplot('occupations', figsize=(15,8))
physicists.boxplot('employers', figsize=(15,8))
physicists.boxplot('awards', figsize=(15,8))
physicists.boxplot('members', figsize=(15,8))
physicists.boxplot('fields', figsize=(15,8))<jupyter_output><empty_output><jupyter_text>We can compare the counts for female and male physicists.<jupyter_code>physicists["gender"].value_counts()<jupyter_output><empty_output><jupyter_text>...and we can have a look at the birth year distribution (in numbers)<jupyter_code>physicists["bdate"].value_counts()<jupyter_output><empty_output><jupyter_text>With a **Scatter Plot** you can relate on measure to another, as e.g. *year of birth* with number of *working fields*.<jupyter_code>physicists.plot(kind="scatter", x="bdate", y="fields", figsize=(17,12), alpha=0.3)<jupyter_output><empty_output><jupyter_text>You might include more measures and use **colors**, **dot sizes**, and **transparency** as additional indicators...<jupyter_code>physicists.plot(kind="scatter", x="bdate", y="occupations", figsize=(17,12), alpha=0.6,
          s=physicists["fields"]*50, label="fields",
          c="awards", cmap=plt.get_cmap("jet"), colorbar=True,
          sharex=False)<jupyter_output>/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:307: MatplotlibDeprecationWarning: 
The rowNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().rowspan.start instead.
  layout[ax.rowNum, ax.colNum] = ax.get_visible()
/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:307: MatplotlibDeprecationWarning: 
The colNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().colspan.start instead.
  layout[ax.rowNum, ax.colNum] = ax.get_visible()
/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:313: MatplotlibDeprecationWarning: 
The rowNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().rowspan.start instead.
  if not layout[ax.rowNum + 1, ax.colNum]:
/usr/local/lib/python3.6/dist-packages/pandas/plotting/_matplotlib/tools.py:3[...]<jupyter_text>Now let's have a look, whether there is a corellation among the numerical data<jupyter_code>corr_matrix = physicists.corr()
corr_matrix["bdate"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>What can you read from the shown correlations?<jupyter_code>corr_matrix = physicists.corr()
corr_matrix["fields"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>What can you read from the shown correlations? Now let's see, whether there are differences between the data of male and female physicists<jupyter_code>physicists.boxplot(column='bdate', by='gender',figsize=(15,10))<jupyter_output><empty_output><jupyter_text>What can you read from the box plot? For further analysis, let's encode **gender** (given as string) by numbers<jupyter_code>physicists_gender_encoded, physicists_gender = physicists["gender"].factorize()
physicists_gender_encoded[:20]    #sample of first 20<jupyter_output><empty_output><jupyter_text>Take a look at our **gender encoding**<jupyter_code>physicists_gender     # encoding
physicists['gender_num']=physicists_gender_encoded
physicists.head(15)<jupyter_output><empty_output><jupyter_text>Now we can also include these measures (transferred into numerals) into a scatter plot<jupyter_code>physicists.plot(kind="scatter", x="gender_num", y="bdate", figsize=(17,12), alpha=0.3)
physicists.plot(kind="scatter", x="bdate", y="occupations", figsize=(17,12), alpha=0.25,
          s=physicists["countries"]*80, label="countries",
          c="gender_num", cmap=plt.get_cmap("jet"), colorbar=True,
          sharex=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11789009778267456, 'fraction_numerical': 0.021622366065280264, 'mean_word_length': 6.676532769556025, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 64, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '272646', 'license': 'Apache-2.0', 'n_tokens_mistral': 2531, 'n_tokens_neox': 2384, 'n_words': 556, 'path': 'assets/frameworks/knowledgegraphs/KnowledgeGraphs_week5/5_6_Physicist_Data_Analysis.ipynb', 'repo_name': 'governance-foundation/governance.foundation', 'size': '7261'}