<jupyter_start><jupyter_text>Initial Data Exploration & Cleaning my Columns About the data* Pulled from Kaggle.com* Dataset name on Kaggle: Indeed Dataset (Data Scientist, Analyst, and Engineer)* Age of dataset: 11/02/2018* The data was originally sourced from Indeed.* 43 Columns* 5715 Rows<jupyter_code># Dependencies and Setup
# import pandas as pd
# import matplotlib.pyplot as plt

# Read Indeed Dataset and store into Pandas data frame
# indeed_df = pd.read_csv("./indeed_job_dataset.csv")
# indeed_df
# What are the column names
# indeed_df.columns<jupyter_output><empty_output><jupyter_text>Removing the following columns* Link column* Date since posted* Description<jupyter_code># Dropping unusable columns
# indeed_df = indeed_df.drop(columns = ["Unnamed: 0", "Link", "Date_Since_Posted", "Description"])
# Checking DataFrame
# indeed_df.columns<jupyter_output><empty_output><jupyter_text>My Columns for Cleaning    Originally we seperated the columns evenly (or as evenly as possible), but some columns didn't need cleaning. I started with columns: 'IL', 'WA', 'MD', 'DC', 'NC', 'Other_states', 'Consulting and Business Services', 'Internet and Software', 'Banks and Financial Services', 'Health Care', 'Insurance', and 'Other_industries'. However, all of those columns were already cleaned. Thus, I took columns 'No_of_Reviews' and 'Company_Industry' from Jennifer and Lori.     We decided it would be best if we each created a dataframe with our columns only. This is what I did before beginning the cleaninging process.<jupyter_code># Seperating my rows to clean
# suzy_col = indeed_df.loc[:, ['No_of_Reviews','Company_Industry', 'IL', 'WA', 'MD', 'DC', 'NC',
#        'Other_states', 'Consulting and Business Services',
#        'Internet and Software', 'Banks and Financial Services', 'Health Care',
#        'Insurance', 'Other_industries']]
# Checking DataFrame
# suzy_col
# What are the data types
# suzy_col.dtypes
# Filling company industry NaNs with Not Listed
# Chose to fill with string because the column was already filled with strings
# suzy_col['Company_Industry'] = suzy_col['Company_Industry'].fillna('Not Listed')
# Filling No_of_Reviews NaNs with 0
# Chose to fill with 0 because the column contains floats
# suzy_col['No_of_Reviews'] = suzy_col['No_of_Reviews'].fillna(0)
# suzy_col.to_csv('./SuzyColumns.csv')<jupyter_output><empty_output><jupyter_text>Exploring my set of columns<jupyter_code># Exploring usable data - 'No_of_Reviews'
# suzy_col.loc[suzy_col['No_of_Reviews']!= 0, :]['No_of_Reviews'].count()
# Exploring usable data - 'Company_Industry'
# suzy_col.loc[suzy_col['Company_Industry']!= 'Not Listed', :]['Company_Industry'].count()
# Exploring usable data - 'IL'
# suzy_col['IL'].sum()
# Exploring usable data - 'WA'
# suzy_col['WA'].sum()
# Exploring usable data 
# suzy_col['MD'].sum()
# Exploring usable data 
# suzy_col['DC'].sum()
# Exploring usable data 
# suzy_col['NC'].sum()
# Exploring usable data 
# suzy_col['Other_states'].sum()
# Exploring usable data 
# suzy_col['Consulting and Business Services'].sum()
# Exploring usable data 
# suzy_col['Internet and Software'].sum()
# Exploring usable data 
# suzy_col['Banks and Financial Services'].sum()
# Exploring usable data 
# suzy_col['Health Care'].sum()
# Exploring usable data 
# suzy_col['Insurance'].sum()
# Exploring usable data 
# suzy_col['Other_industries'].sum()
# Test
# suzy_col.loc[suzy_col['No_of_Reviews']!= 0, 'No_of_Reviews']
# Looking for outliars to determine if I should filter
# reviews = suzy_col.loc[suzy_col['No_of_Reviews']!= 0, 'No_of_Reviews']

# fig1, ax1 = plt.subplots()
# ax1.set_title('Number of Reviews')
# ax1.set_ylabel('Reviews')
# fig1.set_size_inches(4, 10)
# ax1.boxplot(reviews)

# # Saving figure
# # plt.savefig("./Number of Reviews.png")

# # Showing image
# plt.show()<jupyter_output><empty_output><jupyter_text>Data Exploration & Analysis on Clean DataFurther Exploring and cleaning "clean", merged dataset.<jupyter_code># Dependencies and Setup
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
# Read Clean Indeed Dataset and store into Pandas data frame
clean_indeed_df = pd.read_csv("./Indeed Data Set.csv")
clean_indeed_df
# Exploring Number of stars after filtering
test = clean_indeed_df.loc[(clean_indeed_df['No_of_Reviews'] >= 100)]
test['No_of_Stars'].value_counts()
# Checking columns
clean_indeed_df.columns
# Filtering data to keep those above 500 reviews
# Was going to use this, but decided not to.
indeed_df_500 = clean_indeed_df.loc[(clean_indeed_df['No_of_Reviews'] >= 500)]
indeed_df_500
reviews = indeed_df_500['No_of_Reviews']

fig1, ax1 = plt.subplots()
ax1.set_title('Number of Reviews above 500')
ax1.set_ylabel('Reviews')
fig1.set_size_inches(4, 10)
ax1.boxplot(reviews)

# Saving figure
plt.savefig("./Number of Reviews.png")

# Showing image
plt.show()
# What is the distributions of rating
print(indeed_df_500['No_of_Stars'].value_counts())
print(clean_indeed_df['No_of_Stars'].value_counts())

# After this decided to use clean_indeed_df instead of indeed_df_500
# Realized there are multiple rows for the same column
clean_indeed_df['Company'].value_counts()
# New Data Frame to just keep pertinent information for my analysis
new_df_undup = clean_indeed_df.loc[:, ['Queried_Salary', 'Job_Type', 'Company', 'No_of_Reviews', 
                                 'No_of_Stars', 'Location', 'Company_Revenue', 
                                 'Company_Employees', 'Company_Industry']]
new_df_undup
# Removing those records that are missing No_of_Stars
new_df_undup = new_df_undup.loc[new_df_undup['No_of_Stars'] != "Missing", :]
new_df_undup
# Removing duplicates across the columns
new_df_undup = new_df_undup.drop_duplicates()
# Checking data types
new_df_undup.dtypes
# converting No_of_Stars from object to int
# new_df_undup.astype({'No_of_Stars': 'int32'})
new_df_undup["No_of_Stars"] = new_df_undup["No_of_Stars"].astype('float')
new_df_undup
# Checking data types again
new_df_undup.dtypes<jupyter_output><empty_output><jupyter_text>Company Ratings by Job Type<jupyter_code># New Data Frame for job type
new_df_job_type = new_df_undup.loc[:, ['Job_Type', 'Company', 'No_of_Reviews', 
                                 'No_of_Stars']]

# Removing duplicates
new_df_job_type = new_df_job_type.drop_duplicates()
new_df_job_type
# Checking the number of records falling under each job type
new_df_job_type['Job_Type'].value_counts()
# Grouping by Job Type
groupby_job_type = new_df_job_type.groupby('Job_Type')

# Checking the number of reviews in each group
sum_reviews_by_job_type = groupby_job_type['No_of_Reviews'].sum()
print(sum_reviews_by_job_type)

# Creating division
print('--------------------------------')

# What is the average stars by job type
mean_star_by_job_type = groupby_job_type['No_of_Stars'].mean()
mean_star_by_job_type = mean_star_by_job_type.sort_values(ascending=True)
print(mean_star_by_job_type)
# Job Type Ratings - Barplot

# An array that contains the star rating by job type, length of array and tick mark locations
job_type = [value for value in mean_star_by_job_type]
x_axis_job = np.arange(len(mean_star_by_job_type))# Determine number of bars needed
tick_locations_job = [value for value in x_axis_job] # to tell the plot where to place tick marks

# Bar Chart
plt.bar(x_axis_job, job_type, color=['r', 'b', 'g'], alpha=1,  align="center")
plt.xticks(tick_locations_job, mean_star_by_job_type.index.values, rotation="vertical")
plt.ylim([1,5])
plt.title("Job Type vs Company Rating")
plt.ylabel("Company Star Rating")
plt.tight_layout()

# Saving figure
plt.savefig("./Job Type vs Company Ratings.png")
# Create a boxplot to compare means
job_types = ['data_analyst', 'data_engineer', 'data_scientist']
rates = []
data = []

#Setting up plot
fig1, ax1 = plt.subplots()
ax1.set_title('Rating by Job Types')
ax1.set_ylabel('Ratings')

for job in job_types:
    job_df = new_df_job_type.loc[new_df_job_type['Job_Type'] == job]
    rates.append(job_df['No_of_Stars'])
    rat = rates
    data.append(rat)
    rates = []
    
# Extracting from list fro boxplot
x1 = data[0]
x2 = data[1]
x3 = data[2]

# Convert to array source: https://www.educative.io/edpresso/how-to-convert-a-list-to-an-array-in-python
ax1.boxplot([np.array(x1[0]), np.array(x2[0]), np.array(x3[0])], labels = job_types)
plt.tight_layout()

# Saving figure
plt.savefig("./Job Type vs Company Ratings(2).png")

# Showing image
plt.show()
# Over all company rating mean
mean = new_df_job_type['No_of_Stars'].mean()
highest = new_df_job_type['No_of_Stars'].max()
lowest = new_df_job_type['No_of_Stars'].min()
print(lowest, mean , highest)<jupyter_output>1.299999952 3.784555307265497 5.0<jupyter_text>Company Rating by Company Location<jupyter_code># New Data Frame for Company Location
new_df_location = new_df_undup.loc[:, ['Company', 'No_of_Reviews', 
                                 'No_of_Stars', 'Location']]

# Removing duplicates
new_df_location = new_df_location.drop_duplicates()
new_df_location
# Grouping by Location
groupby_location = new_df_location.groupby('Location')

# Checking the number of reviews in each group
sum_reviews_by_location = groupby_location['No_of_Reviews'].sum()
sum_reviews_by_location = sum_reviews_by_location.drop('Missing')
print(sum_reviews_by_location)


# Creating division
print('----------------------------------------------------------')

# What is the average stars by job type
mean_star_by_location = groupby_location['No_of_Stars'].mean()
mean_star_by_location = mean_star_by_location.drop('Missing')
mean_star_by_location = mean_star_by_location.sort_values(ascending=True)
print(mean_star_by_location)

# Location Ratings - Barplot

# An array that contains the star rating by job type, length of array and tick mark locations
location = [value for value in mean_star_by_location]
x_axis_l = np.arange(len(mean_star_by_location))# Determine number of bars needed
tick_locations_l = [value for value in x_axis_l] # to tell the plot where to place tick marks

# Bar Chart
fig, ax = plt.subplots(figsize=(12,5))
plt.bar(x_axis_l, location, color='g', alpha=0.5, align="center")
plt.xticks(tick_locations_l, mean_star_by_location.index.values, rotation="vertical")
plt.ylim([1,5])
plt.ylim([1,5])
plt.axhline(y=mean, color = 'r')
plt.title("Job Location vs Company Rating")
plt.ylabel("Company Star Rating")
plt.tight_layout()


# Saving figure
plt.savefig("./Location vs Company Ratings.png")<jupyter_output><empty_output><jupyter_text>Company Rating by Job Salary<jupyter_code># New Data Frame for job salary
new_df_job_salary = new_df_undup.loc[:, ['Queried_Salary', 'Job_Type', 'Company', 'No_of_Reviews', 
                                 'No_of_Stars', 'Location']]

# Removing duplicates
new_df_job_salary = new_df_job_salary.drop_duplicates()
new_df_job_salary
# Grouping by Salary
groupby_salary = new_df_job_salary.groupby('Queried_Salary')

# Checking the number of reviews in each group
sum_reviews_by_salary = groupby_salary['No_of_Reviews'].sum()
sum_reviews_by_salary = sum_reviews_by_salary[['<80000', '80000-99999', '100000-119999', '120000-139999', '140000-159999', '>160000']]
print(sum_reviews_by_salary)

# Creating division
print('----------------------------------------------------------')

# What is the average stars by job type
mean_star_by_salary = groupby_salary['No_of_Stars'].mean()
mean_star_by_salary = mean_star_by_salary[['<80000', '80000-99999', '100000-119999', '120000-139999', '140000-159999', '>160000']]
mean_star_by_salary = mean_star_by_salary.sort_values(ascending=True)
print(mean_star_by_salary)
# Salary Ratings - Barplot

# An array that contains the star rating by job type, length of array and tick mark locations
salary = [value for value in mean_star_by_salary]
x_axis_sal = np.arange(len(mean_star_by_salary))# Determine number of bars needed
tick_loc_sal = [value for value in x_axis_sal] # to tell the plot where to place tick marks

# Bar Chart
fig, ax = plt.subplots(figsize=(5,5))
plt.bar(x_axis_sal, salary, color='g', alpha=0.5, align="center")
plt.xticks(tick_loc_sal, mean_star_by_salary.index.values, rotation="vertical")
plt.ylim([1,5])
plt.ylim([1,5])
plt.axhline(y=mean, color = 'r')
plt.title("Position Salary Range vs Company Rating")
plt.ylabel("Company Star Rating")
plt.tight_layout()


# Saving figure
plt.savefig("./Position Salary Range vs Company Ratings.png")<jupyter_output><empty_output><jupyter_text>Company Rating by Company Industry, Revenue, and  of Employees<jupyter_code># New Data Frame for Company Industry, Revenue, and # of Employees
new_df_indrevemp = new_df_undup.loc[:, ['Company', 'No_of_Reviews', 
                                 'No_of_Stars', 'Company_Revenue', 
                                 'Company_Employees', 'Company_Industry']]

# Removing duplicates
new_df_indrevemp = new_df_indrevemp.drop_duplicates()
new_df_indrevemp
# Grouping by Company Industry
groupby_industry = new_df_indrevemp.groupby('Company_Industry')

# Checking the number of reviews in each group
sum_reviews_by_industry = groupby_industry['No_of_Reviews'].sum()
sum_reviews_by_industry = sum_reviews_by_industry.drop("Not Listed")
print(sum_reviews_by_industry)

# Creating division
print('----------------------------------------------------------')

# What is the average stars by job type
mean_star_by_industry = groupby_industry['No_of_Stars'].mean()
mean_star_by_industry = mean_star_by_industry.drop("Not Listed")
mean_star_by_industry = mean_star_by_industry.sort_values(ascending=True)
print(mean_star_by_industry)
# Industry Ratings - Barplot

# An array that contains the star rating by job type, length of array and tick mark locations
industry = [value for value in mean_star_by_industry]
x_axis_in = np.arange(len(mean_star_by_industry))# Determine number of bars needed
tick_locations_in = [value for value in x_axis_in] # to tell the plot where to place tick marks

# Bar Chart
fig, ax = plt.subplots(figsize=(12,5))
plt.bar(x_axis_in, industry, color='g', alpha=0.5, align="center")
plt.xticks(tick_locations_in, mean_star_by_industry.index.values, rotation="vertical")
plt.ylim([1,5])
plt.ylim([1,5])
plt.axhline(y=mean, color = 'r')
plt.title("Industry Type vs Company Rating")
plt.ylabel("Company Star Rating")
plt.tight_layout()



# Saving figure
plt.savefig("./Industry vs Company Ratings.png")
# Grouping by Employees
groupby_employee = new_df_indrevemp.groupby('Company_Employees')

# Checking the number of reviews in each group
sum_reviews_by_employee = groupby_employee['No_of_Reviews'].sum()
sum_reviews_by_employee = sum_reviews_by_employee.drop('missing')
sum_reviews_by_employee = sum_reviews_by_employee[['Less than 10,000', '10,000+']]
print(sum_reviews_by_employee)

# Creating division
print('----------------------------------------------------------')

# What is the average stars by job type
mean_star_by_employee = groupby_employee['No_of_Stars'].mean()
mean_star_by_employee = mean_star_by_employee.drop('missing')
mean_star_by_employee = mean_star_by_employee[['Less than 10,000', '10,000+']]
mean_star_by_employee = mean_star_by_employee.sort_values(ascending=True)
print(mean_star_by_employee)
# Employee Ratings - Barplot

# An array that contains the star rating by job type, length of array and tick mark locations
employee = [value for value in mean_star_by_employee]
x_axis_emp = np.arange(len(mean_star_by_employee))# Determine number of bars needed
tick_loc_emp = [value for value in x_axis_emp] # to tell the plot where to place tick marks

# Bar Chart
fig, ax = plt.subplots(figsize=(5,5))
plt.bar(x_axis_emp, employee, color='g', alpha=0.5, align="center")
plt.xticks(tick_loc_emp, mean_star_by_employee.index.values, rotation="vertical")
plt.ylim([1,5])
plt.ylim([1,5])
plt.axhline(y=mean, color = 'r')
plt.title("Company Size vs Company Rating")
plt.xlabel("Number of Company Employees (Range)")
plt.ylabel("Company Star Rating")
plt.tight_layout()


# Saving figure
plt.savefig("./Number of Company Employees vs Company Ratings.png")
# Grouping by Revenue
groupby_revenue = new_df_indrevemp.groupby('Company_Revenue')

# Checking the number of reviews in each group
sum_reviews_by_revenue = groupby_revenue['No_of_Reviews'].sum()
sum_reviews_by_revenue =  sum_reviews_by_revenue.drop("Missing")
sum_reviews_by_revenue = sum_reviews_by_revenue[['Less than $1B (USD)', '$1B to $5B (USD)', '$5B to $10B (USD)', 'More than $10B (USD)']]
print(sum_reviews_by_revenue)

# Creating division
print('----------------------------------------------------------')

# What is the average stars by job type
mean_star_by_revenue = groupby_revenue['No_of_Stars'].mean()
mean_star_by_revenue = mean_star_by_revenue.drop("Missing")
mean_star_by_revenue = mean_star_by_revenue[['Less than $1B (USD)', '$1B to $5B (USD)', '$5B to $10B (USD)', 'More than $10B (USD)']]
mean_star_by_revenue = mean_star_by_revenue.sort_values(ascending=True)
print(mean_star_by_revenue)
# Revenue Ratings - Barplot

# An array that contains the star rating by job type, length of array and tick mark locations
revenue = [value for value in mean_star_by_revenue]
x_axis_rev = np.arange(len(mean_star_by_revenue))# Determine number of bars needed
tick_loc_rev = [value for value in x_axis_rev] # to tell the plot where to place tick marks

# Bar Chart
fig, ax = plt.subplots(figsize=(5,5))
plt.bar(x_axis_rev, revenue, color='g', alpha=0.5, align="center")
plt.xticks(tick_loc_rev, mean_star_by_revenue.index.values, rotation="vertical")
plt.ylim([1,5])
plt.ylim([1,5])
plt.axhline(y=mean, color = 'r')
plt.title("Company Revenue vs Company Rating")
plt.xlabel("Company Revenue")
plt.ylabel("Company Star Rating")
plt.tight_layout()


# Saving figure
plt.savefig("./Company Revenue vs Company Ratings.png")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13099724393947917, 'fraction_numerical': 0.017380055121210415, 'mean_word_length': 5.444363899963755, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '483421', 'license': 'MIT', 'n_tokens_mistral': 6450, 'n_tokens_neox': 6080, 'n_words': 1514, 'path': 'Susana Final Code.ipynb', 'repo_name': 'jennifermarie6sl/Project-1', 'size': '17779'}