<jupyter_start><jupyter_text>Data CleaningThe goal in this notebook is to remove as much of the non-tech entries as possible. The data will not be perfect but needs to be more representative of a true proportion of the tech job landscape. The approach taken here is to find the 200 (somewhat arbitrarily chosen) most common job titles and remove job titles that are not directly associated with a tech position (some sort of software or productt development involved), such as "cashier" or "truck driver". The list of jobs will then be used as a guide for a text classification model. Jobs below a certain threshold of similarity (to be decided here) will be dropped from the data.<jupyter_code>import re
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt


from sklearn.neighbors import NearestNeighbors, KNeighborsClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
jobs = pd.read_csv('indeed_jobs.csv', sep='|')
jobs.head()
# As the data stands, the more unique job titles are
# the most likely (intuitively) to not be tech related
jobs['role'].value_counts(ascending=True)[11:31]

# MEAT MANAGER! (not helpful)<jupyter_output><empty_output><jupyter_text>One of the main issues is that filtering by the most commonjob titles will not help the situation as too many titlesare unique. The graph below shows the amount of unique titles overwhelming common titles; duplicates are in the minority.<jupyter_code>plt.plot(jobs['role'].value_counts().values);
two_hundred_jobs = jobs['role'].value_counts()[:200].index
with open("jobs_200.txt", "w") as f:
    for job in two_hundred_jobs:
        f.write("{}\n".format(job))
with open("tech_jobs_edited.txt", "r") as fedit:
    valid_jobs = fedit.read()
    valid_jobs = valid_jobs.split('\n')
def format_string(s):
    ptrn = r'[^a-zA-z]'
    new = re.sub(ptrn, ' ', s)
    new = new.replace('  ', ' ')
    new = new.strip().lower()
    
    return new
valid_jobs = list(map(format_string, valid_jobs))
# vectorize job titles and fit nearest neighbors
vectorizer = TfidfVectorizer(ngram_range=(1, 2),
                             max_df=.50)
jobs_transformed = vectorizer.fit_transform(valid_jobs)
nn = NearestNeighbors(n_neighbors=1, algorithm="brute",
                      metric="cosine")
nn.fit(jobs_transformed)
test_jobs = jobs['role'].value_counts(ascending=True)[:50].index
test_jobs = test_jobs.to_list()
zeros = np.zeros(len(test_jobs))
test_data = {'job': test_jobs, 'distance': zeros}
test_df = pd.DataFrame(data=test_data)
test_df['job'] = test_df['job'].apply(format_string)
test_df.head()
test_jobs_transformed = vectorizer.transform(test_df['job'])
distances, indices = nn.kneighbors(test_jobs_transformed)
test_df['distance'] = distances
test_df['neighbor'] = list(map(lambda x: valid_jobs[x[0]], indices))
test_df[test_df['distance'] <= .20]<jupyter_output><empty_output><jupyter_text>As seen above, there are some jobs that pass through the filter with a distance threshold of 0.35, but not many (~12% of the job listings are not accurately designated as tech jobs). It's worth the time to see how this would affect the entire dataset:<jupyter_code>jobs['role_formatted'] = jobs['role'].apply(format_string)
roles_transformed = vectorizer.transform(jobs['role_formatted'])
distances, indices = nn.kneighbors(roles_transformed)
jobs['distances'] = distances
jobs.head()
# how much data is retained when the jobs are filtered via distance?
print(f"Data retained at .35: {jobs[jobs['distances'] < .35].shape[0]}")
print(f"Data retained at .30: {jobs[jobs['distances'] < .30].shape[0]}")
print(f"Data retained at .20: {jobs[jobs['distances'] < .20].shape[0]}")
jobs = jobs[jobs['distances'] < .30]
jobs['role'].value_counts()[:20]
# The more unique roles were previously more likely to
# be related to a field other than tech. Much improved!
jobs['role'].value_counts(ascending=True)[:20]<jupyter_output><empty_output><jupyter_text>The figures above are an improvement from the retention seen in past attempts (trying to align the `search_field` column with job roles) and are more likely to produce a dataset where the jobs are _actually_ related to tech. Now, we can apply a process that I attemped earlier: organizing the actual job titles into uniform categories. The following section *Individual Jobs* is from the aforementioned previous attempt. The goal here is two merge the previous process with the following one to create a totally improved, albeit thinner dataset, more representative of the population of Indeed tech jobs  Individual JobsAn explanation of what's going on here:>To answer the question of which jobs have more remote possibilities I'll take a look at the top 20 jobs. This will also help answer the question of average pay range per job. There is one issue, however: The `search_field` result column does not ensure that its value will be related to the `role` column. The job roles have a lot of variation in how they're named, and picking out groups manually would be an impractical waste of time. >The proposed method to alleviate this is to take a deterministic and a probabilistic model and use the text details of the job to classify the roles that are named with enough variation to make them appear to be different jobs. Here I will use a Multinomial Naive Bayes and a K-Nearest Neighbors to see which gets better accuracy. The winner will classify the job roles in a new column. Similarity score will also be stored. Scores below a threshold will be discarded.<jupyter_code># First, some housekeeping: The software developer role and it's
# corresponding search field have a spelling discrepancy
jobs.loc[jobs['search_field'] == 'Software Developers', 'search_field'] = "Software Developer"
# Now, the jobs where the role and search_field align exactly
# will become the training data. This could be difficult to test
# because some roles only have one example
class_data = jobs[jobs['role'] == jobs['search_field']]<jupyter_output><empty_output><jupyter_text>Because some of the roles only have one example, I'll manually choose some of the search results that align with the `search_field` job title to rename according to the target job roles. This will make it easier to create a stratified train/test split<jupyter_code>class_data['role'].value_counts()[class_data['role'].value_counts() == 1]
# create a new column that will be used as the new label
jobs['job_label'] = jobs['role']
weban_data = jobs[jobs['search_field'] == 'Web Analytics Developer'][['role', 'details']].values
weban_index = jobs[jobs['search_field'] == 'Web Analytics Developer'][['role', 'details']].index

gamedev_data = jobs[jobs['search_field'] == 'Game Developer'][['role', 'details']].values
gamedev_index = jobs[jobs['search_field'] == 'Game Developer'][['role', 'details']].index

weban = list(zip(weban_data, weban_index))
gamedev = list(zip(gamedev_data, gamedev_index))
for value, index in weban:
    print(value[1], index, value[0])
    print(">>>>>>>>>>>>>>>>>>>>>>")
jobs.loc[1186][['role', 'details']]
# create a dictionary to hold the indices of good matches for named roles
job_label_dict = {'Game Developer':[243469, 243470, 243485, 
                                    243486, 243511, 243519],
                  'Web Analytics Developer': [1175, 1187, 1186,
                                              1276, 4649, 4636]}
for job_name, indices in job_label_dict.items():
    for idx in indices:
        jobs.loc[idx, 'job_label'] = job_name<jupyter_output><empty_output><jupyter_text>Now there should be enough data to split into train and test<jupyter_code>class_data = jobs[jobs['job_label'] == jobs['search_field']][['job_label', 'details']]<jupyter_output><empty_output><jupyter_text>Creating Training DataNow the details from each job in the new training data need to be vectorized.<jupyter_code>class_data['job_label'].value_counts()
# accuracy in predictions may increase if the job_label is adds to the description
class_data['details'] = class_data['job_label'] + ' ' + class_data['details'] 
def format_text(series, inplace=False):
    """Format text in a way that optimizes vectorization"""
    if not inplace:
        series = series.copy()
    series = series.str.replace('[^a-zA-Z0-9]', ' ')
    series = series.str.replace('\s{2}', ' ')
    series = series.str.lower().str.strip()
    
    return series
documents = format_text(class_data['details'])
documents.iloc[3999]
vectorizer = TfidfVectorizer(ngram_range=(1, 2), 
                             max_df=.80, min_df=10)
le = LabelEncoder()
le.fit(class_data['job_label'])

X = vectorizer.fit_transform(documents)
y = le.transform(class_data['job_label'])
print(X.shape)
print(y.shape)
# split X into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42, stratify=y)

print(X_train.shape, X_test.shape)
print(y_train.shape, y_test.shape)
neighbors = KNeighborsClassifier(algorithm="brute", metric="cosine")
neighbors.fit(X_train, y_train)

n_bayes = MultinomialNB(alpha=0.1)
n_bayes.fit(X_train, y_train)

y_pred_neighbors = neighbors.predict(X_test)
y_pred_n_bayes = n_bayes.predict(X_test)
print(f"KNeighbors: {accuracy_score(y_test, y_pred_neighbors)}")
print(f"MultinomialNB: {accuracy_score(y_test, y_pred_n_bayes)}")
jobs['role_plus_details'] = jobs['role'] + ' ' + jobs['details']
jobs['role_plus_details'] = format_text(jobs['role_plus_details'])
jobs[['role', 'search_field', 'role_plus_details']]
desc = jobs['role_plus_details'].iloc[23094]
print(desc)
desc = vectorizer.transform([desc])
le.inverse_transform(n_bayes.predict(desc))
descriptions = jobs['role_plus_details']
descriptions = vectorizer.transform(descriptions)
descriptions = n_bayes.predict(descriptions)
jobs['predicted_role'] = le.inverse_transform(descriptions)
# top ten before label prediction
for rank, role in enumerate(jobs['role'].value_counts()[:10].index):
    print(f"{rank+1}. {role}")
# top ten after label prediction
for rank, role in enumerate(jobs['predicted_role'].value_counts()[:10].index):
    print(f"{rank+1}. {role}")
jobs[jobs['predicted_role'] == "Data Scientist"][['role', 'predicted_role','search_field']].head()
jobs[jobs['predicted_role'] == "Front-End Developer"][['role', 'predicted_role','search_field']].head()
jobs[jobs['predicted_role'] == "UI Designer"][['role', 'predicted_role','search_field']].head(10)
jobs[jobs['predicted_role'] == "Mobile App Developer"][['role', 'predicted_role','search_field']].head()<jupyter_output><empty_output><jupyter_text>As we can see from the examples above, the `search_field` column (created during scrape) did not always align with the actual role. After removing jobs that were not likely to be tech-related and categorizing them using the description snippets, we end up with data that, while not perfect, provides a far better representation of the job landscape.<jupyter_code>jobs.to_csv('jobs_cleaned.csv', sep='|', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11441627094346385, 'fraction_numerical': 0.013529253651106531, 'mean_word_length': 5.612559241706161, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '237607', 'license': 'MIT', 'n_tokens_mistral': 3587, 'n_tokens_neox': 3319, 'n_words': 1115, 'path': 'jupyter_notebooks/cleaning_jobs.ipynb', 'repo_name': 'jose-marquez89/tech-job-landscape', 'size': '11161'}