<jupyter_start><jupyter_text>Sampling Universities from the THES Rankings========================Having processed the THES files to a csv file we now want to sample a selection of the universities for our own analysis purposes. To do this we use pandas sampling tools and generate a list based on ranking groups as described below.<jupyter_code>import os.path
import pandas
input_data_folder = 'data/input/'
output_data_folder = 'data/output/'
thes_csv_file = os.path.join(output_data_folder, 'thes_2016_ranking.csv')


import pandas as pd
tdf = pd.read_csv(thes_csv_file)
regions = tdf.groupby(['rank_group','subregion'])
regions['int_rank'].count()<jupyter_output><empty_output><jupyter_text>Sampling-------We sample 15 from each rank group at random and write these out as sample files for inspection. We'll choose one that seems to have reasonable distribution and then supplement it as seems appropriate.<jupyter_code>def sample(i):
    return tdf.groupby(['rank_group']).apply(lambda x: x.sample(n=i))
for i in [0,1,2]:
    s = sample(15)
    filename = 'uni_sample_{}.csv'.format(str(i))
    s.to_csv(os.path.join(output_data_folder, filename), 
             columns = ['rank_group', 'int_rank', 'name', 'region', 'subregion', 'country'], 
             encoding='utf-8')<jupyter_output><empty_output><jupyter_text>Based on a quick look it seems like sample1 is the best of the three produced with some from both Africa and South America and a few from Southern Asia.<jupyter_code>s = pd.read_csv(os.path.join(output_data_folder, 'uni_sample_1.csv'))
s = s[s['int_rank'] <= 1000]
p1 = s.groupby(['subregion']).count().plot.pie(figsize=(6,6), y='rank_group', legend=False)
p2 = s.groupby(['country']).count().plot.pie(figsize=(6,6), y='rank_group')<jupyter_output><empty_output><jupyter_text>Supplementing the Sample List-----------------------------The current sample list does not include UCT, one of our core test list and is otherwise weak on Africa and potentially India. Going back to the full list we can select a few examples to add. We will also be only be sampling from those ranked up to 1000. A random selection of those from Africa.<jupyter_code>tdf = tdf[tdf['int_rank'] <= 1000]
africa = tdf[tdf['region']=='Africa']
africa_sample = africa.sample(5)
africa_sample['name']<jupyter_output><empty_output><jupyter_text>A random selection from southern Asia<jupyter_code>south_asia = tdf[tdf['subregion'] == 'Southern Asia']
south_asia_sample = south_asia.sample(5)
south_asia_sample['country']
south_asia_sample['name']<jupyter_output><empty_output><jupyter_text>Random selection from Latin America. Wanted to include Mexico which requires a little bit more manipulation.<jupyter_code>south_america = tdf[tdf['subregion'] == 'South America']
mexico = tdf[tdf['country'] == 'Mexico']
latin_america = pd.concat([south_america, mexico])
latin_america_sample = latin_america.sample(5)
latin_america_sample['country']
latin_america_sample['name']<jupyter_output><empty_output><jupyter_text>Concatenate and deduplicate the list and then write to the final sample list for a set of just over 100 institutions. Will still be missing some of the our core set of ten. Because the sample only wrote out a subset of columns there are some warnings thrown.<jupyter_code>sample100 = pd.concat([s, latin_america_sample, south_asia_sample, africa_sample])
sample100.drop_duplicates()
sample100.to_csv(os.path.join(output_data_folder, 'uni_sample.csv'), 
             columns = ['name', 'rank_group', 'int_rank', 'region', 'subregion', 'country'], 
             encoding='utf-8')
p1 = sample100.groupby(['subregion']).count().plot.pie(figsize=(6,6), y='rank_group', legend=False)
p2 = sample100.groupby(['country']).count().plot.pie(figsize=(6,6), y='rank_group')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13317510548523206, 'fraction_numerical': 0.015822784810126583, 'mean_word_length': 5.9724264705882355, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '210538', 'license': 'MIT', 'n_tokens_mistral': 1244, 'n_tokens_neox': 1184, 'n_words': 339, 'path': 'notebooks/2)_Sampling_the_THES_uni_list.ipynb', 'repo_name': 'ccat-lab/university-clustering', 'size': '3792'}