<jupyter_start><jupyter_text>Kat's Scraper Notebook Code Fellows 401d8 Python MidtermScrapes Indeed for salary information for a given keyword in a given city. Keywords are a list and multiple arguments are acceptable, but note that adding additional keywords drastically increases the time it takes for the scrape to run, since it is currently searching OR, not AND.<jupyter_code>import requests
import bs4
from bs4 import BeautifulSoup
import urllib3
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Define city and keywordsAccepts one city and multiple keywords in a list. Multi-word keywords must be separated with a plus sign.<jupyter_code>city = 'Seattle'
keywords = ['UX+designer']<jupyter_output><empty_output><jupyter_text>The scraper itself, using BeautifulSoupCreates a dataframe from the results of the scrap. Also handles cleaning up of some data in the salary field, since Indeed salary fields come in a variety of formats.<jupyter_code>url_template = 'https://www.indeed.com/jobs?q={}&l={}&fromage=any&limit=100'
max_results = 100

df = pd.DataFrame(columns=['ux'])
requests.packages.urllib3.disable_warnings()
for keyword in keywords:
    for start in range(0, max_results):
        url = url_template.format(keyword, city)
        http = urllib3.PoolManager()
        response = http.request('GET', url)
        soups = BeautifulSoup(response.data.decode('utf-8'), 'html.parser')
        for b in soups.find_all('div', attrs={'class': ' row result'}):
            try:
                salary = b.find('span', attrs={'class': 'no-wrap'}).text
            except AttributeError:
                salary = 'NA'
            df = df.append({'ux': salary}, ignore_index=True)

            df.ux.replace(regex=True,inplace=True,to_replace='\n',value='')
            df.ux.replace(regex=True,inplace=True,to_replace='$',value='')
            df.ux.replace(regex=True,inplace=True,to_replace=' a year',value='')
            df.ux.replace(regex=True,inplace=True,to_replace='(Indeed est.)',value='')<jupyter_output><empty_output><jupyter_text>CleanupThe next three cells clean up some data for us. We eliminate rows where there is no salary, remove 'a year', comma separation, and dollar signs. We also eliminate any rows that contain 'a day,' 'an hour,' or 'a month,' since we only want to work with annual salaries.<jupyter_code>df = df.query('ux != "NA"')
df = df[df.ux.str.contains('a day') == False]
df = df[df.ux.str.contains('an hour') == False]
df = df[df.ux.str.contains('a month') == False]
df.ux = df.ux.str.replace('a year', '').str.replace(',', '').str.replace('$', '')<jupyter_output><empty_output><jupyter_text>Here we just take a peek at the data to confirm the above reformatting is working correctly.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Taking lowest in the rangeSince most of the salaries are listed as a range, we assume the worst-case scenario by splitting the salary on the dash and assigning the first index as a float to a list.<jupyter_code>cleaned_salaries = []
for i in df.ux:
    a = i.split('-')
    cleaned_salaries.append(float(a[0]))<jupyter_output><empty_output><jupyter_text>Reassigning salariesThis replaces the salary column in the dataframe with the values from the list we made above.<jupyter_code>df.ux = cleaned_salaries
# df<jupyter_output><empty_output><jupyter_text>Write to CSVWe write our results to a CSV because this scrape is kind of large and it takes foreverrrrr. We want to do things with this data, but we don't want to have to run the scrapes repeatedly.<jupyter_code>df.to_csv('uxresults.csv', encoding='utf-8', index=False)<jupyter_output><empty_output><jupyter_text>Let's chart some salaries!First, read them into dataframes from the CSVs we made.<jupyter_code>c_plus = pd.read_csv('cplusresults.csv')
python = pd.read_csv('pythonresults.csv')
javascript = pd.read_csv('javascriptresults.csv')
java = pd.read_csv('javaresults.csv')
php = pd.read_csv('phpresults.csv')
csharp = pd.read_csv('csharpresults.csv')
datascience = pd.read_csv('datascienceresults.csv')
softwaredev = pd.read_csv('softwaredevresults.csv')
webdev = pd.read_csv('webdevresults.csv')
dba = pd.read_csv('DBAresults.csv')
ux = pd.read_csv('uxresults.csv')<jupyter_output><empty_output><jupyter_text>Concatenating them into relevant dataframes<jupyter_code>languages = pd.concat([c_plus, python, javascript, java, php, csharp], axis=1)
languages.head()
jobs = pd.concat([datascience, softwaredev, webdev, dba, ux])<jupyter_output><empty_output><jupyter_text>Get median values for each<jupyter_code>median_languages = languages.median()
median_jobs = jobs.median()<jupyter_output><empty_output><jupyter_text>Let's just plot the distributions of some languages<jupyter_code>python = pd.read_csv('pythonresults.csv')
plt.hist(python.python, bins=20)
plt.ylabel('Python')
javascript = pd.read_csv('javascriptresults.csv')
plt.hist(javascript.javascript, bins=20)
plt.ylabel('Javascript')
c_plus = pd.read_csv('cplusresults.csv')
plt.hist(c_plus.Cplus, bins=20)
plt.ylabel('C++')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1204117304330938, 'fraction_numerical': 0.005049524179452321, 'mean_word_length': 5.5688775510204085, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '445063', 'license': 'MIT', 'n_tokens_mistral': 1701, 'n_tokens_neox': 1645, 'n_words': 451, 'path': 'mass_scraper/MassScraper.ipynb', 'repo_name': 'Mildly-Sketchy/Mildly-Sketchy', 'size': '5149'}