<jupyter_start><jupyter_text>DATA 512 Final Project Report: Analysis of World University Rankings  1. IntroductionOver the past years, university rankings have become one of essential references for applicants before submitting their applications. Knowing which countries have most globally ranked universities can give people who want to study abroad a clear direction on choosing countries. We also perform analysis to find reason why these countries have more globally ranked universities than others. Generally, ranking systems evaluate universities in many different aspects, including citations, research, influences and etc. Applicants may want to know which indicators can provide them more valuable information besides total score. While there are many different existing ranking systems, we used data from one of the most authoritative ranking systems, the Times Higher Education World University Ranking, as primary dataset on most analysis. Meanwhile, we used data from the other influential ranking system to make comparisons.  2. BackgroundMore than 4.5 million students are enrolled in university-level education outside their country of origin and this number is continually growing. Over the past years, as a new form of higher education evaluation system, university ranking has become prevalence all over the world. World university rankings are widely used to measure the competitiveness of instructing institutions and regarded as one of crucial factors to consider for applicants of overseas universities. From the perspective of schools, it is important to know which objective indicators have the largest impact on their rankings so that they can improve practices and ranks to raise their reputations. On the other hand, it is also important for the people who are seeking for universities abroad to know which countries have most high-quality universities so that they can have a clearer direction on their applications. As a human-centered based project, I start the exploration of world university rankings with several research questions and corresponding hypotheses: RQ1: What factors contribute to world university rankings a lot?The rankings are calculated based on various measures. In this research question, I want to figure out what factors really make an educational institution ranks higher than the others.  H1a: Universities with higher citation scores are more likely to rank higher.Unlike national ranking systems, global university rankings may focus more on the citation area, other factors such as teaching and learning environment have less impact on the final ranking results. H1b: Student-staff ratio affects rankings to a great extent. Higher ratio leads to lower ranking.Student-staff ratio means number of students per staff. Students may get more attention in the universities with lower ratios which leads to higher education quality. RQ2: Which countries have the most high-ranked universities on average? Why these countries have more top universities?As far as I know, lots of globally-ranked universities are in the US. But how about other countries? In this research question, I would like to find out 10 highest-ranked countries in terms of number of top globally-ranked universities and what these countries have in common. H2a: Countries have most high-ranked universities are wealthy developed countries. The governments of wealthy developed countries can invest more money in education. As a result, developed countries have more top universities than developing countries. H2b: The population of a country affects the quality of its education. Populous countries might have more globally-ranked universities than the ones with small population. Populous countries have higher demand of educational resources. The larger the number of universities in a country, the greater the chances of these universities are ranked in top global universities. RQ3: Do any biases might exist in ranking systems from various countries?Different ranking systems have different preferences of indicator selections and even their evaluation standards are quite different. Therefore, there might be some biases between these ranking systems. In this research question, I want to test bias cause by regional difference. H3a: A ranking system might include more globally-ranked universities of its producer’s original country than the system produced by another country.  3. DataIn order to answer the research questions and test hypotheses, I performed a couple of quantitative analyses and visualize data on the datasets from two of the most influential and widely observed university measures. In addition, population data is specific to some of hypotheses. Population dataset is downloaded from the Population Research Bureau website. Others are from Kaggle website and are authorized to use by their original owners. Since the population data is from a certain year and there are plenty of missing values in the data of low-ranking universities in both ranking datasets, I performed analysis on top 200 globally ranked universities in 2015. 3.1 University Ranking Data 3.1.1 Times Higher Education World University Ranking (“timesData.csv”)Times Higher Education [1] is a well-known global university ranking system which mainly focuses on a lot of non-English educational institutions and their commercialization. It collaborated with <NAME> and started to use its own methodology since 2011. The dataset, “timesData.csv”, contains 2603 rows and 14 columns (13 performance indicators and 1 column for year). The performance indicators cover 5 major fields: teaching, research, citations, income, international outlook. I used this dataset as a prime dataset to answer the majority of research questions. Columns found in the dataset are listed below: Column Name|Description|Value|:- | :------------------- | :-: world_rank|world rank for the university(contains rank ranges and equal ranks)|Numeric|university_name|name of university|String|country|country of each university|String|teaching|university score for teaching and learning environment|Numeric|international|university score for international outlook(staff,students,research)|Numeric|research|university score for research(volumn, income, reputation)|Numeric|citations|university score for citations(research, influence)|Numeric|income|university score for industry income(knowledge transfer)|Numeric|total_score|total score for university|Numeric|num_students|number of students at the university|Numeric|student_staff_ratio|number of students divided by number of staff|Numeric|international_students|percentage of students who are international|String|female_male_ratio|female student to male student ratio|String|year|year of ranking(2012 to 2016)|Numeric|  3.1.2 Center for World University Rankings (“cwurData.csv”)As one of the largest and most comprehensive academic ranking of global universities, Center for World University Rankings [2] was compiled by a Saudi Arabia-based organization since 2012. The dataset, “cwurData.csv”, contains 2200 rows and 14 columns. It is used to make comparisons with the dataset in 4.1.1 and find any existing bias among ranking systems from different countries. The system uses 6 indicators to rank. Columns found in the dataset are listed below: Column Name|Description|Value|:- | :------------------- | :-: world_rank|world rank for the university|Numeric|institution|name of university|String|country|country of each university|Stringnational_rank|rank of university within its country|Numeric|quality_of_education|rank for quality of education|Numeric|alumni_employment|rank for alumni employment|Numeric|quality_of_faculty|rank for quality of faculty|Numeric|publications|rank for publications|Numeric|influence|rank for influence|Numeric|citations|rank for citations|Numeric|broad_impact|rank for broad impact(only available for 2014 and 2015)|String|patents|rank for patents|Numeric|score|total score, used for determining world rank|Numeric|year|year of ranking(2012 to 2015)|Numeric|  3.2 Supplementary Data 3.2.1 Population Data (“Population Mid-2015.csv”)The population data is exactly the same dataset we used in the second assignment which contains the populations of a variety of counties collected in mid 2015. The data was obtained from Population Research Bureau website.[3] This dataset is used to test the second hypothesis of research question 2. Columns found in the dataset are listed below: Column Name|Description|Value|:- | :------------------- | :-: Location|name of country|String|Location Type|country|String|TimeFrame|Mid-2015|String|Data Type|Number|String|Data|population of country|String(numbers are encoded with commas)|  4. MethodsThe analysis consists of two main parts: statistical calculations and data visualizations. To answer the research questions in section 2, computational analysis such as calculate the proportions of number of top 200 globally-ranked universities per population was performed. For the research questions that can be more intuitively answered by visualizing the data, I make plots in appropriate styles for each research question such as line charts and bar charts.The code are written in Python 3 in Jupyter Notebook with using the following libraries:•	Numpy: used for quantitative analysis, especially in scientific computing•	Pandas: used for data processing and analysis•	Collections: used for data processing•	Matplotlib: used for visualizing data Importing Data<jupyter_code>import csv
from itertools import islice
import pandas as pd
import warnings
warnings.filterwarnings("ignore")
# get data from Center for World University Ranking
cwur = []
with open('cwurData.csv', newline='') as cwurData:
    readCSV = csv.reader(cwurData,delimiter=',')
    for row in readCSV:
        cwur.append(row)
# get data from Times Higher Education World University Ranking
times = []
with open('timesData.csv', newline='') as timesData:
    readCSV = csv.reader(timesData,delimiter=',')
    for row in readCSV:
        times.append(row)
# get data from population data of a variety of countries in 2015
population = []
with open('Population Mid-2015.csv') as population_file:
    reader = csv.reader(population_file)
    # note that first row is title; the second and last two rows are blank
    # skip first and last two rows in the csv file
    for row in islice(reader,2,213):
        population.append([row[0],row[4]])<jupyter_output><empty_output><jupyter_text>Extract top 200 universities in 2015 from the THE ranking dataset<jupyter_code># Convert THE ranking data to dataframe
timesDf = pd.DataFrame(times[1:],columns = times[0])
# Keep data from 2015
times2015 = timesDf[timesDf['year'] == '2015']
# Keep top 200 globally ranked universities in 2015
top200_idx_the = times2015[times2015['world_rank'] == '200'].index.values.tolist()[-1]
times2015_top200 = times2015.loc[:top200_idx_the,:]<jupyter_output><empty_output><jupyter_text>List 10 example data of top 200 globally ranked universities in 2015<jupyter_code>times2015_top200.head(10)<jupyter_output><empty_output><jupyter_text>Extract columns related to citation scores<jupyter_code>times_citation = times2015_top200[['world_rank','university_name','country','citations','total_score']]<jupyter_output><empty_output><jupyter_text>RQ1: What factors contribute to world university rankings a lot? H1a: Universities with higher citation scores are more likely to rank higher. Plot citation scores and total score of top 200 universities<jupyter_code>%matplotlib inline

import matplotlib.pyplot as plt
import matplotlib
# Plot citation score and total score for top 200 universities
fig, ax = plt.subplots(figsize=(20,10))
ax.plot(times_citation['world_rank'].astype(int),times_citation['citations'].astype(float),label="citation_score",marker='o')
ax.plot(times_citation['world_rank'].astype(int),times_citation['total_score'].astype(float),label="total_score")
plt.title('Global Ranked Top 200 Universities Citation Scores VS Total Scores')
plt.xlabel('World Rank For University')
plt.ylabel('Score')
# Add a legend
ax.legend(loc='lower left', shadow=True)<jupyter_output><empty_output><jupyter_text>Findings:The overall observation is that the higher the ranking, the greater the citation score. Schools rank high tend to have smaller variation in citation scores. On the other hand, schools rank low have bigger variations in citation scores. Some of their scores are good and in parity with high ranking schools, while other scores are really low. It is likely that some of the research labs in the low ranking schools are strong and other labs are weak. For high ranking schools, most of their research labs are strong and produce high impacting, high citation publications.  H1b: Student-staff ratio affects rankings to a great extent. Higher ratio leads to lower ranking. Extract columns related to student-staff-ratio index<jupyter_code>times_studentStaffRatio = times2015_top200[['world_rank','university_name','country','student_staff_ratio','total_score']]
import numpy as np
# Replace blanks with NaN
times_studentStaffRatio['student_staff_ratio'] = times_studentStaffRatio['student_staff_ratio'].replace(r'\s+',np.nan,regex=True).replace('',np.nan)
times_studentStaffRatio['student_staff_ratio'] = np.array(times_studentStaffRatio['student_staff_ratio'])
# Convert string to float if the value is not NaN
studentStaffRatio = []
for x in times_studentStaffRatio['student_staff_ratio']:
    if (pd.isnull(x)):
        studentStaffRatio.append(x)
        continue
    else:
        x = float(x)
        studentStaffRatio.append(x)
times_studentStaffRatio['student_staff_ratio'] = studentStaffRatio<jupyter_output><empty_output><jupyter_text>Plot student and staff ratio and total score of top 200 universities<jupyter_code>fig, ax = plt.subplots(figsize=(20,10))
ax.plot(times_studentStaffRatio['world_rank'].astype(int),times_studentStaffRatio['student_staff_ratio'],label="student_staff_ratio",marker='o')
ax.plot(times_studentStaffRatio['world_rank'].astype(int),times_studentStaffRatio['total_score'].astype(float),label="total_score")
plt.title('Global Ranked Top 200 Universities Student and Staff Ratio VS Total Scores')
plt.xlabel('World Rank For University')
plt.ylabel('Score for University or Index for Student and Staff Ratio')
# Add a legend
ax.legend(loc='upper right', shadow=True)<jupyter_output><empty_output><jupyter_text>Findings:It is observed that student staff ratio is low in high ranking universities. It is likely a way to keep their education quality high. Interesting, mid and low ranking universities can have big variance on student staff ratio. Both high and low student staff ratio can occur in mid and low ranking universities. Student staff ratio does not appear to be a linear factor that impact university ranking score.  RQ2: Which countries have the most high-ranked universities on average? Why these countries have more top universities?<jupyter_code># Extract column "country" from top 200 global ranked universities in 2015
times2015_top200_country = times2015_top200.iloc[:,2].tolist()
# Find 10 highest-ranked countries have most number of top globally-ranked universities
from collections import Counter
count_country = Counter(times2015_top200_country)
top10_countries = count_country.most_common(10)<jupyter_output><empty_output><jupyter_text>H2a: Countries have most high-ranked universities are wealthy developed countries.<jupyter_code># Plot bar chart to visualize 10 highest-ranked countries from THE rankings
plt.subplots(figsize=(20,10))
plt.bar(range(len(top10_countries)), [val[1] for val in top10_countries], align='center')
plt.xticks(range(len(top10_countries)), [val[0] for val in top10_countries])
plt.xticks(rotation=70)
for x, y in zip(range(len(top10_countries)), [val[1] for val in top10_countries]):
    plt.annotate(str(y),xy=(x, y + 1.1))
plt.title("10 highest-ranked countries in terms of number of top globally-ranked universities(THE Data)")
plt.ylabel("Number of Universities")
plt.xlabel("Country Name")
plt.show()<jupyter_output><empty_output><jupyter_text>Findings:The top 10 counties with most high-ranking universities are all developed countries. Developed countries has a better education system, which may in turn contribute to their development. U.S. and U.K. rank number one and number two in this 10 countries. The universities from U.S. and U.K. combined account for more than half of the top 200 universities. This is consistent with the status of the two countries. Both U.S. and U.k. have a lot of high-tech companies, which is likely a result of talent trained from those universities.  H2b: The population of a country affects the quality of its education. Populous countries might have more globally-ranked universities than the ones with small population.<jupyter_code># Convert data (country and population) from the population file to dataframe
population_data = pd.DataFrame(population[1:],columns=population[0])
# Renames the columns with suitable names
population_data.columns = ["Location","Population"]
top10_countries_population = []
for country in [val[0] for val in top10_countries]:
    if country == "United States of America":
        p = int(population_data[population_data["Location"] == "United States"].iloc[0][1].replace(",",""))
        top10_countries_population.append(p)
    else:
        p = int(population_data[population_data["Location"] == country].iloc[0][1].replace(",",""))
        top10_countries_population.append(p)
fig, ax1 = plt.subplots(figsize=(20,10))
ax2 = ax1.twinx()
ax1.bar(range(len(top10_countries)), [val[1] for val in top10_countries],width = -0.2,align = "edge",label = "Number of Universities")
plt.xticks(range(len(top10_countries)), [val[0] for val in top10_countries])
plt.xticks(rotation=70)
ax2.bar(range(len(top10_countries)), top10_countries_population,width = 0.2,color = "red",align = "edge",label = "Population")
ax1.set_ylabel('Number of Universities')
ax2.set_ylabel('Population')
ax1.set_xlabel('Country Name')
plt.title("10 highest-ranked countries in terms of number of top globally-ranked universities VS their population")
ax1.legend(loc='upper center')
ax2.legend(loc='upper right')<jupyter_output><empty_output><jupyter_text>Findings:U.S. has most of the population in those 10 countries as well as most of the high ranking universities. Japan has the second largest population in the 10 countries. However, the number of high ranking universities in Japan is not the second place. There can be multiple factors why Japan does not have many high ranking universities. One of the factor might the duration of Japan’s current education system is not as long as U.S. or U.K. China and India are counties with high populations, however, they are not on the list. It is likely their universities systems need more time to develop and catch up.  RQ3: Do any biases might exist in ranking systems from various countries?  H3a: A ranking system might include more globally-ranked universities of its producer’s original country than the system produced by another country.<jupyter_code># Convert CWUR ranking data to dataframe
cwurDf = pd.DataFrame(cwur[1:],columns = cwur[0])
# Keep data from 2015
cwur2015 = cwurDf[cwurDf['year'] == '2015']
# Keep top 200 globally ranked universities in 2015
top200_idx_cwur = cwur2015[cwur2015['world_rank'] == '200'].index.values.tolist()[-1]
cwur2015_top200 = cwur2015.loc[:top200_idx_cwur,:]
# Extract column "country" from top 200 global ranked universities in 2015
cwur2015_top200_country = cwur2015_top200.iloc[:,2].tolist()
# Find 10 highest-ranked countries have most number of top globally-ranked universities
from collections import Counter
count_country = Counter(cwur2015_top200_country)
top10_countries_cwur = count_country.most_common(10)
# Plot bar chart to visualize 10 highest-ranked countries from CWUR rankings
plt.subplots(figsize=(20,10))
plt.bar(range(len(top10_countries_cwur)), [val[1] for val in top10_countries_cwur], align='center')
plt.xticks(range(len(top10_countries_cwur)), [val[0] for val in top10_countries_cwur])
plt.xticks(rotation=70)
for x, y in zip(range(len(top10_countries)), [val[1] for val in top10_countries_cwur]):
    plt.annotate(str(y),xy=(x, y + 1.1))
plt.title("10 highest-ranked countries in terms of number of top globally-ranked universities(CWUR Data)")
plt.ylabel("Number of Universities")
plt.xlabel("Country Name")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.071630859375, 'fraction_numerical': 0.02353515625, 'mean_word_length': 6.052685950413223, 'pattern_counts': {'":': 1, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '324077', 'license': 'MIT', 'n_tokens_mistral': 5648, 'n_tokens_neox': 5085, 'n_words': 2329, 'path': 'data-512-final-project-report.ipynb', 'repo_name': 'yawen32/data-512-final-project', 'size': '20489'}