<jupyter_start><jupyter_text>Prof. <NAME>, PhD* Associate Professor, Departamento de Ciencias Sociales, Pontificia Universidad Católica del Perú, [<EMAIL>](mailto:<EMAIL>)* Visiting Associate Professor, Evans School of Public Policy and Governance / Senior Data Science Fellow, eScience Institute, University of Washington, [<EMAIL>](mailto:<EMAIL>)__________  Session 1: Introduction to Python - EXERCISES  CASE 1: HDI Link from Google:<jupyter_code>link1="https://docs.google.com/spreadsheets/d/e/2PACX-1vSNMpr1Dz3tvMqEAnmGAjaQi19zcsEiLZQn6f5ZM6ap4tzNy6bQUXtctNebCL4OyEN6UODUc0U7B9pf/pub?gid=1506000439&single=true&output=csv"<jupyter_output><empty_output><jupyter_text>Collecting the data with the link from Google:<jupyter_code># need to call pandas:

import pandas as pd

hdi=pd.read_csv(link1)<jupyter_output><empty_output><jupyter_text>-  Cleaning Keep only data for data frame  * Check first rows<jupyter_code># Is the header in the right place:
hdi.head(10)<jupyter_output><empty_output><jupyter_text>Two problems:* The headers are in two different rows.* The data starts in below zero.Cleaning the top part:* Saving the headers:<jupyter_code># for the first two columns:
hdi.iloc[4,:2].tolist()
# for the rest of columns columns:
hdi.iloc[3,2:].tolist()
# saving headers:
CurrentHeaders=hdi.iloc[4,:2].tolist()+hdi.iloc[3,2:].tolist()

# you saved this:
CurrentHeaders<jupyter_output><empty_output><jupyter_text>* Setting the begining of data:We need to get rid the top rows that are not part of the data frame:<jupyter_code># PREVIES: deleting rows, NORWAY should be the first row:
hdi.drop(index=range(0,7)) #7 in range() will not be erased.<jupyter_output><empty_output><jupyter_text>Making effective the last code:<jupyter_code>hdi.drop(index=range(0,7),inplace=True)<jupyter_output><empty_output><jupyter_text>As soon as you delete rows, you should reset the indexes:<jupyter_code>hdi.reset_index(drop=True,inplace=True)<jupyter_output><empty_output><jupyter_text>You can rename the columns now:<jupyter_code># rename columns
hdi.columns=CurrentHeaders<jupyter_output><empty_output><jupyter_text>So far:<jupyter_code>hdi.head()<jupyter_output><empty_output><jupyter_text>* Check last rows<jupyter_code>hdi.tail(65) #change until you see last data row<jupyter_output><empty_output><jupyter_text>Based on what is diSplayed, you should keep data up to Tuvalu country:<jupyter_code># deleting: preview
hdi.drop(index=range(199,262)) # rows starting from 199 will be erased<jupyter_output><empty_output><jupyter_text>Making effective the last code:<jupyter_code>hdi.drop(index=range(199,262),inplace=True) # should I reset indexes?<jupyter_output><empty_output><jupyter_text>So far:<jupyter_code>hdi<jupyter_output><empty_output><jupyter_text>* Check the column names You have columns with names "missing", you want to avoid those:<jupyter_code># Get good columns

GoodHeaders=[header for header in CurrentHeaders if str(header) != 'nan']
#
GoodHeaders<jupyter_output><empty_output><jupyter_text>The column **HDI rank** appears twice, because it appears twice in the data. Also, you have another rank **GNI per capita rank minus HDI rank'**. Ranks are repetitive because we have the scores. So, you can always get rid of them in this and other similar situations:<jupyter_code>FinalHeaders=[header for header in GoodHeaders if 'rank' not in header]
# then
FinalHeaders
# remember loc works with names, not with positions:

hdi.loc[:,FinalHeaders]<jupyter_output><empty_output><jupyter_text>Saving previous result:<jupyter_code>hdi=hdi.loc[:,FinalHeaders]
# you have:
hdi.head()<jupyter_output><empty_output><jupyter_text>The remaining column names need to improve:<jupyter_code># replace with '' (empty) the "blanks":
hdi.columns.str.replace("\s","") 
# replace with '' (empty) consecutive word characters in parenthesis:
hdi.columns.str.replace("\(\w+\)","")
#or all combines
hdi.columns.str.replace("\s+|\(\w+\)","")<jupyter_output><empty_output><jupyter_text>Saving result:<jupyter_code>hdi.columns=hdi.columns.str.replace("\s+|\(\w+\)","")
hdi.head()<jupyter_output><empty_output><jupyter_text>* Check the cell values Find cells where all numeric data is missing, that is, a country with no data:<jupyter_code># check empty cells from second to last
hdi[hdi.iloc[:,1:].isnull().all(axis=1)] #ALL<jupyter_output><empty_output><jupyter_text>These are no countries. They were subtitles for groups of countries.  You need the opposite to that:<jupyter_code># the opposite
hdi[hdi.iloc[:,1:].notnull().all(axis=1)]<jupyter_output><empty_output><jupyter_text>Keepin what you need:<jupyter_code>hdi=hdi[hdi.iloc[:,1:].notnull().all(axis=1)]<jupyter_output><empty_output><jupyter_text>The last code deleted rows, then indexes need to be reset:<jupyter_code>hdi.reset_index(drop=True, inplace=True)<jupyter_output><empty_output><jupyter_text>So far:<jupyter_code>hdi<jupyter_output><empty_output><jupyter_text>* Check the cell values Above you see some characters that are not numbers, are they missing values:<jupyter_code>hdi[hdi.iloc[:,1:].isnull().any(axis=1)] #ANY<jupyter_output><empty_output><jupyter_text>There are symbols that represent missing values, but they are not recognized so. We need to check what values in those **numeric** columns are not numbers:<jupyter_code>badHDISymbols=[] # list for bad symbols

NumericColNames=hdi.iloc[:,1:].columns # save names of columns with numeric data

for columnName in NumericColNames:# visit every column name
    for cell in hdi[columnName]:# visit every cell for that column
        try:
            float(cell) # try this
        except: # if not possible:            
            if cell not in badHDISymbols:# if cell is not in the list                
                badHDISymbols.append(cell)# add it to the list

# you get:
badHDISymbols<jupyter_output><empty_output><jupyter_text>You need to replace that value in the numeric columns:<jupyter_code>import numpy as np

hdi.replace(to_replace=badHDISymbols,value=np.nan,inplace=True)
# you have:
hdi<jupyter_output><empty_output><jupyter_text>When you have numbers in your columns, you can request statistical summaries:<jupyter_code>hdi.describe(include='all')<jupyter_output><empty_output><jupyter_text>You are not getting those because they lack format.  -  Formatting Check the data types:<jupyter_code>hdi.dtypes<jupyter_output><empty_output><jupyter_text>Country can remain as an object (text), but not the rest. * **Formatting into numeric type**:<jupyter_code># as easy as:

hdi[NumericColNames]=hdi.loc[:,NumericColNames].apply(pd.to_numeric)
#recheck
hdi.dtypes
# recheck
hdi.describe(include='all')<jupyter_output><empty_output><jupyter_text>Some more information:<jupyter_code>hdi.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 195 entries, 0 to 194
Data columns (total 6 columns):
Country                         195 non-null object
Humandevelopmentindex           189 non-null float64
Lifeexpectancyatbirth           191 non-null float64
Expectedyearsofschooling        193 non-null float64
Meanyearsofschooling            189 non-null float64
Grossnationalincomepercapita    191 non-null float64
dtypes: float64(5), object(1)
memory usage: 9.3+ KB<jupyter_text>CASE 2: DEMOCRACY INDEX Link to use:<jupyter_code>link2 = "https://en.wikipedia.org/wiki/Democracy_Index" 
# import pandas as pd
# maybe you need: pip install beautifulsoup4
#collection
demodex=pd.read_html(link2,
                     header=0,
                     flavor='bs4',
                     attrs={'class': 'wikitable sortable'})[0]<jupyter_output><empty_output><jupyter_text>- Cleaning * First rows:<jupyter_code># what's on top?
# names? weird symbols? more links?
demodex.head(10)<jupyter_output><empty_output><jupyter_text>* Last rows:<jupyter_code># what's at the bottom?
# note? credits? extra info?

demodex.tail(10)<jupyter_output><empty_output><jupyter_text>From what I saw, I will get rid of the _Score_ and the _Rank_, an also of the last row (it is the repetition of the headers):<jupyter_code>#bye row 167, and two columns
demClean=demodex.drop(index=167,columns=['Rank','Score']).copy()
demClean<jupyter_output><empty_output><jupyter_text>As there are few names, we can change to smaller sizes:<jupyter_code>newNames=['pluralism','effectiveness','participation','culture','liberties']

# names from the second and before the last one '[1:-1]':
newMapper={old:new for old,new in zip(demClean.columns[1:-1],newNames)}

demClean.rename(columns=newMapper,inplace=True)
# this is what we have so far:
demClean.head()<jupyter_output><empty_output><jupyter_text>Let me check if there is any rare character in the numbers:<jupyter_code>badDemoSymbols=[] # list for bad symbols

NumericColNames=demClean.iloc[:,1:-2].columns # save names of columns with numeric data

for columnName in NumericColNames:# visit every column name
    for cell in demClean[columnName]:# visit every cell for that column
        try:
            float(cell) # try this
        except: # if not possible:            
            if cell not in badDemoSymbols:# if cell is not in the list                
                badDemoSymbols.append(cell)# add it to the list

# you get:
badDemoSymbols<jupyter_output><empty_output><jupyter_text>The last two columns are categories, we should check if the categories are consistent:<jupyter_code># get unique values in each column:
demClean.iloc[:,-2:].apply(set)
# easier to see:
demClean.iloc[:,-2:].apply(set).to_list()<jupyter_output><empty_output><jupyter_text>It looks good so far. Let's go to formatting. - Formatting<jupyter_code># checking data types:
demClean.dtypes<jupyter_output><empty_output><jupyter_text>* **Formatting into numeric**<jupyter_code>demClean[NumericColNames]=demClean[NumericColNames].apply(pd.to_numeric)<jupyter_output><empty_output><jupyter_text>* **Formatting into nominal**<jupyter_code>demClean.Continent=pd.Categorical(demClean.Continent)<jupyter_output><empty_output><jupyter_text>* **Formatting into ordinal**<jupyter_code># Check current levels:
pd.unique(demClean.Regimetype).tolist()
#rewrite the levels in order:
correctLevels=['Authoritarian', 'Hybrid regime', 'Flawed democracy','Full democracy']
#format as ordinal:
demClean.Regimetype=pd.Categorical(demClean.Regimetype,categories=correctLevels,ordered=True)<jupyter_output><empty_output><jupyter_text>The data types have changed:<jupyter_code>#then
demClean.dtypes<jupyter_output><empty_output><jupyter_text>For more detail:<jupyter_code>demClean.Regimetype.cat.ordered
# checking missing values
demClean.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 167 entries, 0 to 166
Data columns (total 8 columns):
Country          167 non-null object
pluralism        167 non-null float64
effectiveness    167 non-null float64
participation    167 non-null float64
culture          167 non-null float64
liberties        167 non-null float64
Regimetype       167 non-null category
Continent        167 non-null category
dtypes: category(2), float64(5), object(1)
memory usage: 10.0+ KB<jupyter_text>Case 3: Integrating and Saving This should be an easy step:<jupyter_code>hdi.merge(demClean)<jupyter_output><empty_output><jupyter_text>Notice you have lost countries:<jupyter_code>len(demClean),len(hdi)
hdi.merge(demClean,how='outer',indicator=True) # see last column<jupyter_output><empty_output><jupyter_text>Let me save previous result:<jupyter_code>test=hdi.merge(demClean,how='outer',indicator=True) 
# in hdi only
test.loc[test['_merge']=='left_only',"Country"]
# in demClean only
test.loc[test['_merge']=='right_only',"Country"]
# you need to change the original

#dictionary of replacements:
replacements={'South Korea[n 1]': 'Korea (Republic of)', 
              'Cape Verde':'Cabo Verde',
              'Czech Republic':'Czechia',
              'Hong Kong':'Hong Kong, China (SAR)',
              'Moldova':'Moldova (Republic of)',
              'Bolivia':'Bolivia (Plurinational State of)',
              'Tanzania':'Tanzania (United Republic of)',
              'Palestine':'Palestine, State of',
              'Ivory Coast':"Côte d'Ivoire",
              'Republic of the Congo':'Congo',
              'Venezuela':'Venezuela (Bolivarian Republic of)',
              'Vietnam':'Viet Nam',
              'Eswatini':'Eswatini (Kingdom of)',              
              'Russia':'Russian Federation',
              'Iran':'Iran (Islamic Republic of)',
              'Laos':"Lao People's Democratic Republic",
              'Democratic Republic of the Congo':'Congo (Democratic Republic of the)',
              'Syria':'Syrian Arab Republic',
              'North Korea': "Korea (Democratic People's Rep. of)" #check ""
             }

# replacing
demClean.Country.replace(replacements,inplace=True)<jupyter_output><empty_output><jupyter_text>The merge should give more rows:<jupyter_code>hdidem=hdi.merge(demClean)
# result:
hdidem<jupyter_output><empty_output><jupyter_text>________ Saving File to Disk  For future use in Python:<jupyter_code>hdidem.to_pickle("hdidemocia.pkl")
# you will need: DF=pd.read_pickle("hdidemocia.pkl")
# or:
# from urllib.request import urlopen
# DF=pd.read_pickle(urlopen("https://...../hdidemocia.pkl"),compression=None)<jupyter_output><empty_output><jupyter_text>For future  use in R:<jupyter_code>from rpy2.robjects import pandas2ri
pandas2ri.activate()

from rpy2.robjects.packages import importr

base = importr('base')
base.saveRDS(hdidem,file="hdidem.RDS")

#In R, you call it with: DF = readRDS("hdidem.RDS")
#or, if iyou read from cloud: DF = readRDS(url("https://...../hdidem.RDS")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1157091496074547, 'fraction_numerical': 0.014381099126861838, 'mean_word_length': 5.6912125675012275, 'pattern_counts': {'":': 1, '<': 202, '<?xml version=': 0, '>': 202, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '49', 'dir': 'jupyter-structured-clean-dedup', 'id': '305759', 'license': 'MIT', 'n_tokens_mistral': 4806, 'n_tokens_neox': 4569, 'n_words': 1104, 'path': '.ipynb_checkpoints/Exercises-checkpoint.ipynb', 'repo_name': 'eScienceWinterSchool/PythonSession', 'size': '13689'}