<jupyter_start><jupyter_text>Metrics of Institutions for Higher Education in the United States Authors: <NAME>, <NAME> Overview:We are interested in getting a closer look at the profile data of institutions for higher education in the United States. We look at the top 50 most expensive institutions and make note of some interesting outliers, asses a matrix of density data to get a feel for the US market for higher education, and visualize cost and tuition values against undergraduate populations across the entire dataset.Some high-level metrics that we take into consideration are:* Overall admissions rate* Average SAT scores of admitted students* Total number of undergraduates* Tuition for both in-state and out-of-state.<jupyter_code>#Begin by importing all the required packages you need to work with the data
import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import matplotlib.cm as cm
%matplotlib inline
import datetime as dt

#Download the data file from:
#https://ed-public-download.app.cloud.gov/downloads/Most-Recent-Cohorts-All-Data-Elements.csv
    
    #After downloading the data, assign the file path to load into pandas dataframe
url = "C:/Users/Jessiah/Desktop/DB_Data_Project/Colleges_All_Recent_Data.csv"

#import only the data needed to perform analysis
    #Data column headers pulled from data dictionary available at following url:
        #https://collegescorecard.ed.gov/assets/CollegeScorecardDataDictionary.xlsx
        
fields = ['CITY', 'INSTNM',
          'STABBR','TUITFTE','INEXPFTE','ADM_RATE',
          'SAT_AVG','UG','COSTT4_A','TUITIONFEE_OUT',
          'TUITIONFEE_IN','AVGFACSAL','PCTFLOAN',
          'FEMALE_DEBT_MDN','MALE_DEBT_MDN',
          'GRAD_DEBT_MDN']

#Create a dataframe using the read_csv function and reference the url
df = pd.read_csv(url,
                 usecols = fields,
                 index_col ='INSTNM')

#Rename all of the columns so that they are more readable and standardized
df = df.rename(columns={'CITY' : 'City', 
                        'STABBR' : 'State',
                        'ADM_RATE' : 'Admit Rate',
                        'SAT_AVG' : 'Avg. SAT',
                        'UG' : 'Undergrads', 
                        'COSTT4_A' : 'Avg. Cost',
                        'TUITIONFEE_IN' : 'IS Tuition',
                        'TUITIONFEE_OUT':'OOS Tuition',
                        'AVGFACSAL' : 'Avg. Faculty Salary',
                        'PCTFLOAN' : '% UG on Fed. Loans',
                        'FEMALE_DEBT_MDN': 'Median Female Debt',
                        'MALE_DEBT_MDN'  : 'Median Male Debt',
                        'GRAD_DEBT_MDN'  : 'Median Grad Debt'})

#Rename the index so that it makes sense
df.index.name = 'School'

#sort the table in descending order of average cost
df.sort_values(by=['Avg. Cost'], ascending = False )

#display the first 10 rows of data
df.head(50)
           

#Create a new dataframe that has sorted the table on Average Cost of attendance
df3=df.sort_values(['Avg. Cost'], ascending = False)

#Create figure that you will use to append subplots
fig=plt.figure(figsize =(15,10))

#You need multiple axes to represent multiple bar charts on the same figure.
#Create those axes here: 

#first subplot
ax1 = plt.subplot(311)

#second subplot
ax2 = plt.subplot(312)

#third subplot
ax3 = plt.subplot(313)

# Share the x axes so that all the data can be tied to the same school

ax1.get_shared_x_axes().join(ax1, ax2, ax3)

#create admit rate lines
l1 = df3['Admit Rate'].iloc[1:51].plot(ax=ax1, kind = 'bar', color = 'Blue', label = 'Admit Rate')

#create tuition cost lines
l2 = df3['TUITFTE'].iloc[1:51].plot(ax=ax2, kind = 'Bar', color = 'Green', label = 'TUITFTE')

#create Undergrads lines
l3 = df3['Undergrads'].iloc[1:51].plot(ax=ax3, kind = 'Bar', color = 'Red', label = 'Undergrads')

# hide the ticks on the x axes across admit rate and tuition cost plots
ax1.axes.get_xaxis().set_visible(False)
ax1.set_ylabel('')

ax2.axes.get_xaxis().set_visible(False)
ax2.set_ylabel('')

# show the ticks on the x axis for Undergrads plot
ax3.axes.get_xaxis().set_visible(True)
ax3.set_ylabel('')

# create a title above the group of subplots
fig.suptitle("Top 50 Schools by Cost of Attendance", x = 0.28, fontsize = 24)

#create 'proxy artists' to satisfy the legend for all subplots
red_patch =mpatches.Patch(color = 'Red', label = 'Undergrads')
blue_patch = mpatches.Patch(color = 'Blue', label = 'Admit Rate')
green_patch = mpatches.Patch(color = 'Green', label = 'Net Tuition Revenue Per Student')

#create consolidated legend for all subplots
plt.legend(handles=[blue_patch, green_patch, red_patch], bbox_to_anchor=(1.03, 3.8), loc=5, borderaxespad=2)

plt.show()<jupyter_output><empty_output><jupyter_text>An Overview of the Top 50 U.S. Institutions by CostAt first glance, we see some of the usual suspects populating this output. The University of Chicago is holding the top position as the most expensive school. Several of the Ivy-League names show up as well, including Columbia University, Dartmouth University, the University of Pennsylvania, Yale, etc. Of course, New York University is also present, holding position number 18.When reviewing this output, we are interested in counter-intuitive results. For example, it would make sense that an institution with a high admit rate would have a high undergraduate population and, potentially, a relatively low tuition cost. Elite schools tend to have a low admit rate, charge a lot for tuition, and maintain a moderate-to-large undergraduate population. Take the University of Pennsylvania, for example. The University of Pennsylvania has an undergraduate population between 10-15K, earns between $30-$40K in tuition per full time student, and admits only approximately 10% of applicants on average. This seems pretty straightforward. What about an institution that has the same revenue figures for tuition as the University of Pennsylvania, admits around 85% of applicants, and has an incredibly low population of undergraduates?  This would seem counter-intuitive, indeed. This is the case for Landmark College.Landmark College is a small private college in Vermont. Landmark was the first higher level institution that focused on providing education for students with learning disabilities such as dyslexia, ADHD, or Autism Spectrum Disorder (ASD). As a result, Landmark has a high cost per student, given that each student needs more individual attention; the school has 80 academic staff for 500 students. Another little known, anomalous school is the Webb Institute, which ranks within the top 25 most expensive institutions in the data set. Webb is an institution specifically created to serve as a pipeline for naval engineering and has an undergraduate population of fewer than 100 students, with around 10 academic staff. Despite its minuscule size, Webb has an endowment of $40 million. Like Landmark College, Webb is a costly school that caters to a very specific group of undergraduate students.<jupyter_code>#Create a figure that you will use to append subplots
fig2= plt.figure(figsize =(20,25))


# The next four charts compare Admit Rate as a base line against the other variables
ax2_1 = plt.subplot2grid((5,2),(0,0))

df3.plot(ax=ax2_1, cmap = cm.Oranges, kind = 'hexbin', x='Admit Rate', y= 'Admit Rate', gridsize = 20, mincnt = 1)

ax2_2 = plt.subplot2grid((5,2),(0,1))

df3.plot(ax=ax2_2, cmap = cm.Oranges, kind = 'hexbin', x='Admit Rate', y= 'Avg. Cost', gridsize = 20, mincnt = 1)

ax2_3 = plt.subplot2grid((5,2),(1,0))

df3.plot(ax=ax2_3, cmap = cm.Oranges, kind = 'hexbin', x='Admit Rate', y= 'Avg. SAT', gridsize = 20, mincnt = 1)

ax2_4 = plt.subplot2grid((5,2),(1,1))

df3.plot(ax=ax2_4, cmap = cm.Oranges, kind = 'hexbin', x='Admit Rate', y= 'Undergrads', gridsize = 20, mincnt = 1)


#The next three charts compare Avg. Cost as a base line against the other variables
ax2_5 = plt.subplot2grid((5,2),(2,0))

df3.plot(ax=ax2_5, cmap = cm.Purples, kind = 'hexbin', x='Avg. Cost', y= 'Avg. Cost', gridsize = 20, mincnt = 1)

ax2_6 = plt.subplot2grid((5,2),(2,1))

df3.plot(ax=ax2_6, cmap = cm.Purples, kind = 'hexbin', x='Avg. Cost', y= 'Avg. SAT', gridsize = 20, mincnt = 1)

ax2_7 = plt.subplot2grid((5,2),(3,0))

df3.plot(ax=ax2_7, cmap = cm.Purples, kind = 'hexbin', x='Avg. Cost', y= 'Undergrads', gridsize = 20, mincnt = 1)


#The next two charts compare Avg. SAT as a base line against the other variables
ax2_8 = plt.subplot2grid((5,2),(3,1))

df3.plot(ax=ax2_8, cmap=cm.Greens, kind = 'hexbin', x='Avg. SAT', y= 'Avg. SAT', gridsize = 20, mincnt = 1)


ax2_9 = plt.subplot2grid((5,2),(4,0)) 

df3.plot(ax=ax2_9, cmap=cm.Greens, kind = 'hexbin', x='Avg. SAT', y= 'Undergrads', gridsize = 20, mincnt = 1)


##The next chart breaks out the avergae number of undergraduates in U.S schools
ax2_10 = plt.subplot2grid((5,2),(4,1))

df3.plot(ax=ax2_10, cmap=cm.Blues, kind = 'hexbin', x='Undergrads', y= 'Undergrads', gridsize = 20, mincnt = 1)


# create a title above the group of subplots
fig2.suptitle("School Densities Matrix", x = 0.5, fontsize = 24)

plt.show()<jupyter_output><empty_output><jupyter_text>Based on the densities matrix above, the average institution in the United States has the following profile: **Admissions rate:** .60 - .80**Undergraduate population:**  Primarily less than 5K, likely skewed by community colleges**SAT scores:** 1,000 - 1,150**Cost:** Two primary buckets of 10,000  - 15,000  and 20,000 - 30,000 Admit rate X Avg. Cost - There appears to be three distinct, high-frequency results in this density plot, with the following characteristics: Admit Rate = .60 , Cost =  $ 22,000 Admit Rate = .75 , Cost =  $ 27,000Admit Rate = .95 , Cost =  $ 27,000 While we can’t immediately point to an explanation, it is curious that institutions with a 95 percent admission rate would charge the same amount as those with a 20 percent lower admissions rate. Admit rate X Undergraduates- The intersection with the greatest density on this plot is at approximately 100 percent admissions rate and fewer than 2,000 undergraduates. This makes sense, given that our dataset includes trade institutions (culinary, electrician training, etc.) and smaller community schools which are inclined to admit anyone who applies and pays to attend. Avg. Cost X Avg. SAT- The data set shows that institutions begin to increase in price relatively steeply as it begins to admit students that score above 1200 on the SAT. Avg. Cost X Undergraduates- The data shows that as the number of undergraduates enrolled in an institution decreases, the cost of attendance typically increases. This follows conventional wisdom for the most part, as smaller, more selective institutions typically cost more.  Avg. SAT X Undergraduates- There is nothing surprising here. The density graph resembles a nearly perfect normal distribution that is in line with our average institution profile outlined above. Avg. SAT X Admit rate - This data visualization more or less follows conventional wisdom: as the SAT scores of admitted students rises, the admission rate of the institution decreases.<jupyter_code># Create a figure for the scatter plot
plt.figure(figsize=(10,5))

#assign a subplot to the figure
ax1 = plt.subplot2grid((1,1),(0,0),colspan=2)

#Create variables for the axis and data to be used in the scatter
x1 = df3['Undergrads']
y1 = df3['Avg. Cost']
y2 = df3['IS Tuition']
y3 = df3['OOS Tuition']

# Plot scattered data in the subplot
s1 = plt.scatter(x1, y1, s=40, lw=0, color='Green', marker='o', label=r'Avg. Cost')
s2 = plt.scatter(x1, y2, s=40, lw=0, color='Red', marker='<', label=r'IS Tuit')
s3 = plt.scatter(x1, y3, s=40, lw=0, color='Purple', marker='^', label=r'OOS Tuit')

# Create a legend depicting all scatter data
lgd = plt.legend((s1, s2, s3), (r'Avg. Cost', r'IS Tuit', r'OOS Tuit'), loc='upper right')

#Create a title above the plot
plt.gcf().suptitle(r' Undergrads against Avg. Cost / IS Tuit / OOS Tuit', size=16)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09888291270169632, 'fraction_numerical': 0.02887877534133223, 'mean_word_length': 4.2072382593709605, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 15, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '661852', 'license': 'MIT', 'n_tokens_mistral': 3974, 'n_tokens_neox': 3671, 'n_words': 1495, 'path': 'UG_S17/Nair-Straw-CollegeScoreCardData.ipynb', 'repo_name': 'NYUDataBootcamp/Projects', 'size': '12098'}