<jupyter_start><jupyter_text>Abstract  Problem Statement- *Customers*: League of Legends players- *Problem*: Predict and interperate champion popularity- *Solution*: Linear regression model  Work Plan- *Data*: Web scrape champion statistics by date- *Model*: Linear regression- *Measure of Success*: TBD  Obtain the Data  Map the Data Pipeline*After completing this step, be sure to edit `references/data_dictionary` to include descriptions of where you obtained your data and what information it contains.*  Build the Pipeline Tools*Make sure these steps are reproducible by code. Put some thought into the directory structures and filepaths you are using to save your data, so it's easy to load files you need.*<jupyter_code># %%writefile src/obtain.py

# def basic_lr_model(y_idx):
#     """
#     Reads cleaned data set from clean_data.csv and runs it as is
#     through an OLS linear regression model using statsmodels.api
    
#     Input
#     -----
#     y_idx: column index of the clean data set that corresponds to the predicted variable
    
#     Output
#     ------
#     model : the OLS model computed by statsmodel.api.OLS()
#     fit   : best fit of the OLS model to the data 
#     """
    
#     # Import relevant modules
#     import patsy
#     import pandas as pd
#     import statsmodels.api as sm
    
#     # Read in clean data
#     datadir = '/Users/jeremy_lehner/Documents/GitHub/metis_project2/data/processed/clean_data.csv'
#     df = pd.read_csv(datadir)
    
#     # Construct R formula style string
#     colnames = list(df.columns)
#     y_name = colnames[y_idx]
#     colnames.remove(y_name)
    
#     R_formula = y_name + ' ~ ' + ' + '.join(colnames)
    
#     # Construct target vector y and feature matrix X
#     y, X = patsy.dmatrices(R_formula, data = df, return_type = "dataframe")
    
#     # Construct OLS model
#     model = sm.OLS(y, X)

#     # Fit model to training set
#     fit = model.fit()
    
#     # Return the model and the fit
#     return model, fit

# write functions that form a data pipeline, 
# then run this cell to save them in src/obtain.py<jupyter_output>Overwriting src/obtain.py<jupyter_text>Run the Pipeline*Set this up so that you won't need to download datasets that you already have on your computer when you re-run the pipeline.*<jupyter_code># import data pipeline functions
#pipelinedir = '/Users/jeremy_lehner/Documents/GitHub/metis_project2/src'
#import sys
#sys.path.insert(0, pipelinedir)
#from obtain import basic_lr_model
#import statsmodels.api as sm

# def run_data_pipeline():
#     # call data pipeline functions in order
#     model, fit = basic_lr_model(5)
#     pass

#run_data_pipeline()<jupyter_output><empty_output><jupyter_text>Scrub the Data  Load Data  Verify Data Integrity  Engineer Features *Before moving on to exploratory analysis, write down some notes about challenges encountered while working with this data that might be helpful for anyone else (including yourself) who may work through this later on.*  Explore the Data*Before you start exploring the data, write out your thought process about what you're looking for and what you expect to find. Take a minute to confirm that your plan actually makes sense.**Calculate summary statistics and plot some charts to give you an idea what types of useful relationships might be in your dataset. Use these insights to go back and download additional data or engineer new features if necessary. Not now though... remember we're still just trying to finish the MVP!*  Inspect Raw Data  Inspect Features for Predictive Patterns*In a regression model, we are looking for clear relationships between our features and targets. Generate pairplots and look for features that have high correlations with the target variable.**In a classification model, we are looking for features that separate the population into distinct distributions. Generate pairplots that are color-coded by your categories and look for features where the categories have distinct distributions.* - *What did you learn about your data?*- *Does it look like there are clear patterns and relationships among your features that will allow you to make good predictions?*- *Which features do you think will be most helpful?*  Model the Data*Describe the algorithms that you are considering. How do they work? Why are they good choices for this data and problem space?**What nuances in the data will you have to be aware of in order to avoid introducing bias to your model? What steps will you need to take to prevent overfitting? What risks are there for data leakage?*<jupyter_code># Import relevant modules
import patsy
import pandas as pd
import statsmodels.api as sm

y_idx = 5

# Read in clean data
datadir = '/Users/jeremy_lehner/Documents/GitHub/metis_project2/data/processed/clean_data.csv'
df = pd.read_csv(datadir)

# Construct R formula style string
colnames = list(df.columns)
colnames.remove('champion')
y_name = colnames[y_idx]
colnames.remove(y_name)

R_formula = y_name + ' ~ ' + ' + '.join(colnames)

# Construct target vector y and feature matrix X
y, X = patsy.dmatrices(R_formula, data = df, return_type = "dataframe")

# Construct OLS model
model = sm.OLS(y, X)

# Fit model to training set
fit = model.fit()

# Return the model and the fit
fit.summary()
# Use statsmodels to plot the residuals vs the fitted values
import matplotlib.pyplot as plt
plt.figure(figsize=(12,8))
plt.scatter(fit.predict(), fit.resid);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07102428336680665, 'fraction_numerical': 0.0016432353478181487, 'mean_word_length': 4.039558417663294, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '446176', 'license': 'MIT', 'n_tokens_mistral': 1551, 'n_tokens_neox': 1502, 'n_words': 680, 'path': 'mvp.ipynb', 'repo_name': 'jlehnersd/metis_project2', 'size': '5477'}