<jupyter_start><jupyter_text>**Course Content**: Day 71 of 100 Days of Python Project Name: Analyzing College Majors vs Salary Things I implemented: Pandas, Beautiful Soup, Requests  Data Exploration using Pandas Analyzing College Majors vs Salary  Uploading and Reading CSV<jupyter_code>import pandas as pd
df = pd.read_csv("sample_data/salaries_by_college_major.csv")
df.head()<jupyter_output><empty_output><jupyter_text>Preliminary Data Exploration and Data Cleaning with Pandas<jupyter_code>df.shape # Number of rows and columns
df.columns # Return all columns 
df.isna() # check the whole dataframe for blank cells
df.tail() # retrieve last 5 values
df = df.dropna() # Drop all columns with NaN values
df.tail()<jupyter_output><empty_output><jupyter_text>Accessing Columns and Individual Cells in the Dataframe<jupyter_code>df.get('Starting Median Salary') # Access Columns
df.get('Starting Median Salary').max() # Find max value in the column
df.get('Starting Median Salary').idxmax() # Get index of max value in the column
df.get('Undergraduate Major').loc[43] # Get the Major using loc function
df.loc[43] # Get all columns of a single row<jupyter_output><empty_output><jupyter_text>Challenge 1: Highest mid-career salary What college major has the highest mid-career salary? How much do graduates with this major earn? (Mid-career is defined as having 10+ years of experience).<jupyter_code>df.columns
highest_mid_career_index = df.get('Mid-Career Median Salary').idxmax()
print(df.loc[highest_mid_career_index])
highest_mid_career_major = df['Undergraduate Major'][highest_mid_career_index]
highest_mid_career_salary = int(df['Mid-Career Median Salary'][highest_mid_career_index])
highest_mid_career_start = int(df['Starting Median Salary'][highest_mid_career_index])
print(f'College Major with highest mid-career salary is {highest_mid_career_major}\nwith mid-career salary of ${highest_mid_career_salary}\nand starting median salary of ${highest_mid_career_start}')<jupyter_output>College Major with highest mid-career salary is Chemical Engineering
with mid-career salary of $107000
and starting median salary of $63200<jupyter_text>Challenge 2: Lowest starting salaryWhich college major has the lowest starting salary and how much do graduates earn after university?<jupyter_code>df.columns
lowest_start_index = df.get('Starting Median Salary').idxmin()
lowest_start_index
lowest_start_major = df['Undergraduate Major'][lowest_start_index]
lowest_start_salary = int(df['Starting Median Salary'][lowest_start_index])
print(f'College Major with lowest starting salary is {lowest_start_major}\nwith starting median salary of ${lowest_start_salary}')<jupyter_output>College Major with lowest starting salary is Spanish
with starting median salary of $34000<jupyter_text>Challenge 3: Lowest mid-career salaryWhich college major has the lowest mid-career salary and how much can people expect to earn with this degree?<jupyter_code>df.columns
lowest_mid_index = df.get('Mid-Career Median Salary').idxmin()
lowest_mid_index
lowest_mid_major = df['Undergraduate Major'][lowest_mid_index]
lowest_mid_salary = int(df['Mid-Career Median Salary'][lowest_mid_index])
lowest_mid_start = int(df['Starting Median Salary'][lowest_mid_index])
print(f'College Major with lowest starting salary is {lowest_mid_major}\nwith mid-career median salary of ${lowest_mid_salary}\nand starting median salary of ${lowest_mid_start}')<jupyter_output>College Major with lowest starting salary is Education
with mid-career median salary of $52000
and starting median salary of $34900<jupyter_text>Sorting values and Adding columns<jupyter_code># Make spread column
spread_col = df['Mid-Career 90th Percentile Salary'].subtract(df['Mid-Career 10th Percentile Salary'])
df.insert(1, "Mid-Career Spread", spread_col)
df.head()
# Sort values
low_risk = df.sort_values('Mid-Career Spread', ascending = False)
low_risk.head()<jupyter_output><empty_output><jupyter_text>Challenge 1: Highest PotentialFind the top 5 degrees with the highest values in the 90th percentile.<jupyter_code>high_potential = df.sort_values('Mid-Career 90th Percentile Salary', ascending=False, ignore_index = True)
high_potential.get(['Undergraduate Major', 'Mid-Career 90th Percentile Salary']).head(5)<jupyter_output><empty_output><jupyter_text>Challenge 2: Degrees with greatest spread in salariesWhich majors have the largest difference between high and low earners after graduation.<jupyter_code>high_spread = df.sort_values('Mid-Career Spread', ascending=False, ignore_index=True)
high_spread.get(['Undergraduate Major', 'Mid-Career Spread']).head(5)<jupyter_output><empty_output><jupyter_text>Grouping and Pivoting Data<jupyter_code>df.groupby('Group').count() # Group the values together by value
df.groupby('Group').mean()<jupyter_output><empty_output><jupyter_text>Project: Update the data analysis by scraping<jupyter_code>from bs4 import BeautifulSoup # Import Beautiful Soup Module
import requests # Import Requests Module
# Create new DataFrame
new_df = pd.DataFrame(columns=['Undergraduate Major', 'Starting Median Salary', 'Mid-Career Median Salary'])
new_df
# Scrape using Beautiful Soup
major_url = 'https://www.payscale.com/college-salary-report/majors-that-pay-you-back/bachelors/page/'

for i in range(1,35):
  url = major_url+str(i)
  page_response = requests.get(url=url).text
  soup = BeautifulSoup(page_response, "html.parser")
  rows = soup.find_all(class_='data-table__row')
  for row in rows:
    major = row.select_one('.csr-col--school-name .data-table__value').get_text()
    salaries = row.select('.csr-col--right .data-table__value')
    start, mid = salaries[:2]
    add = {'Undergraduate Major':major,'Starting Median Salary':start.get_text()[1:], 'Mid-Career Median Salary':mid.get_text()[1:]}
    new_df = new_df.append(add, ignore_index=True)

new_df #still contain duplicates due to multiple test:v
new_df = new_df.drop_duplicates() # Drop all duplicates
new_df.describe()
# Convert the Salary from string into integers
new_df['Starting Median Salary'] = new_df['Starting Median Salary'].apply(lambda x: int(x.split(',')[0])*1000 + int(x.split(',')[1]))
new_df['Mid-Career Median Salary'] = new_df['Mid-Career Median Salary'].apply(lambda x: int(x.split(',')[0])*1000 + int(x.split(',')[1]))

new_df
new_df.describe()
new_df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10486537553141237, 'fraction_numerical': 0.011966619430011021, 'mean_word_length': 7.29242819843342, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '280258', 'license': 'MIT', 'n_tokens_mistral': 2188, 'n_tokens_neox': 1982, 'n_words': 549, 'path': 'Advanced/71/Day 71 - College Majors vs Salary.ipynb', 'repo_name': 'Matthew1906/100DaysOfPython', 'size': '6351'}