<commit_before>---
title: "Web Scraping Indeed.com to Predict Salaries"
header:
  overlay_color: "#333"
date: 2017-01-03
tags: Web scraping, Indeed.com, Data science jobs
---
# Web Scraping Indeed.com to Predict Salaries

For this project, we used a set of characteristics to predict the salaries for data science jobs in the Boston area. We analyzed search results from Indeed.com, an online jobs board, and highlighted key phrases in each job posting. We also determined if each job's salary is above a threshold value. We used logistic regression to model this data. Machine learning and python experience were the strongest predictors of salary. Machine learning, python and scientist were the strongest predictors of salary. The model had poor precision --- approximately 63% precision and recall, where random guesses could have achieved 57% --- which may have been caused by variability in the source salary data.

Part I outlines the code used. [Part II](#Part-II:-Report) contains the writeup.

## Part I: Code

#### Import libraries


```python
# Webscraping
from urllib.request import urlopen
from bs4 import BeautifulSoup

#Data and analysis
import numpy as np
import pandas as pd
import statsmodels.formula.api as sm
from sklearn import preprocessing, linear_model, metrics

# Plotting
import matplotlib.pyplot as plt<commit_msg>updated project 4 post<commit_after>---
title: "Web Scraping Indeed.com to Predict Salaries"
header:
  overlay_color: "#333"
date: 2017-01-03
tags: Web scraping, Indeed.com, Data science jobs
---

For this project, we used a set of characteristics to predict the salaries for data science jobs in the Boston area. We analyzed search results from Indeed.com, an online jobs board, and highlighted key phrases in each job posting. We also determined if each job's salary is above a threshold value. We used logistic regression to model this data. Machine learning and python experience were the strongest predictors of salary. Machine learning, python and scientist were the strongest predictors of salary. The model had poor precision --- approximately 63% precision and recall, where random guesses could have achieved 57% --- which may have been caused by variability in the source salary data.

Part I outlines the code used. [Part II](#Part-II\:-Report) contains the writeup.

## Part I: Code

#### Import libraries


```python
# Webscraping
from urllib.request import urlopen
from bs4 import BeautifulSoup

#Data and analysis
import numpy as np
import pandas as pd
import statsmodels.formula.api as sm
from sklearn import preprocessing, linear_model, metrics

# Plotting
import matplotlib.pyplot as plt
{'fraction_non_alphanumeric': 0.05853474320241692, 'fraction_numerical': 0.012462235649546828, 'mean_word_length': 4.600422832980972, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2221715', 'n_tokens_mistral': 698, 'n_tokens_neox': 639, 'n_words': 372}